Method and system for predicting protein structure using cycles through multiple iterations

Abstract

The present disclosure provides methods, systems, and apparatus, including computer programs encoded on computer storage media, for predicting protein structures including one or more chains. In one aspect, a method includes, at each subsequent iteration after the first iteration in a sequence of iterations: obtaining a network input characterizing a subsequent iteration of a protein; generating features for the subsequent iteration from (i) structure parameters generated in a previous iteration preceding the subsequent iteration in the sequence, (ii) one or more intermediate outputs generated by a protein structure prediction neural network in generating the structure parameters in a last iteration, or (iii) both; and processing the features and the network input for the subsequent iteration using the protein structure prediction neural network to generate structure parameters for the subsequent iteration defining another predicted structure of the protein.

Description

Technical Field

[0001] This manual relates to the prediction of protein structure. Background Technology

[0002] Proteins are defined by a sequence of one or more amino acids (“chains”). An amino acid is an organic compound that includes an amino group and a carboxyl group, as well as a side chain (i.e., a group of atoms) specific to that amino acid. Protein folding is the physical process by which one or more amino acid sequences fold into a three-dimensional (3-D) conformation. The structure of a protein defines the three-dimensional conformation of the atoms in its amino acid sequence after folding. When in a sequence linked by peptide bonds, an amino acid can be referred to as an amino acid residue.

[0003] Machine learning models can be used for prediction. A machine learning model takes input and generates an output, such as a predicted output, based on the received input. Some machine learning models are parametric models and generate outputs based on the received input and the model's parameter values. Some machine learning models are deep models that employ multiple layers to generate outputs for received inputs. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers, each of which applies a non-linear transformation to the received input to generate an output. Summary of the Invention

[0004] This specification describes a neural network system implemented as a computer program on one or more computers at one or more locations for predicting protein structure.

[0005] As used in this specification, the term "protein" can be understood to refer to any biomolecule specified by one or more amino acid sequences (or "chains"). For example, the term "protein" can refer to a protein domain, such as a portion of the amino acid chain of a protein, which can fold almost independently of the rest of the protein. As another example, the term "protein" can refer to a protein complex, i.e., a complex of multiple amino acid chains that fold together to form a protein structure.

[0006] Multiple sequence alignment (MSA) of an amino acid chain in a protein refers to the alignment of an amino acid chain with multiple additional amino acid chains (e.g., from other proteins, such as homologous proteins). More specifically, an MSA can define the correspondence between a position in an amino acid chain and its corresponding position in multiple additional amino acid chains. An MSA of an amino acid chain can be generated, for example, by processing a database of amino acid chains using any suitable computational sequence alignment technique (e.g., progressive alignment construction). The amino acid chains in an MSA can be understood as having evolutionary relationships; for example, each amino acid chain in the MSA may share a common ancestor. The correlations between amino acids in the amino acid chains within an MSA can encode information related to predicting the structure of the amino acid chain.

[0007] "Embedding" of entities (such as amino acid pairs) can refer to representing entities as ordered sets of numerical values, such as vectors or matrices of numerical values.

[0008] The structure of a protein can be defined by a set of structural parameters. This set of structural parameters can be represented as an ordered set of values. Several examples of possible structural parameters used to define protein structures are described in more detail below.

[0009] In one example, the structural parameters that define the structure of a protein include (i) positional parameters and (ii) rotational parameters for each amino acid in the protein.

[0010] The positional parameters of an amino acid can specify the predicted 3-D spatial position of a particular atom within the amino acid in the protein structure. For example, a specific atom could be the α-carbon atom in an amino acid, i.e., the carbon atom bonded to the amino group, carboxyl group, and side chain. As another example, a specific atom could be the β-carbon atom in an amino acid. The positional parameters of an amino acid can be represented in any suitable coordinate system, such as a 3-D [x,y,z] Cartesian coordinate system.

[0011] The rotation parameters of an amino acid specify its predicted "orientation" within a protein structure. More specifically, the rotation parameters specify a 3D spatial rotation operation, which, when applied to a coordinate system relative to the position parameters, results in the three "main chain" atoms in the amino acid occupying fixed positions relative to the rotation coordinate system. The three main chain atoms in an amino acid can refer to a series of nitrogen, α-carbon, and carbonyl carbon atoms linked together within the amino acid. The rotation parameters of an amino acid can be represented, for example, as an orthogonal 3×3 matrix with a determinant of 1.

[0012] Typically, the position and rotation parameters of an amino acid define an egocentric frame of reference for that amino acid. In this frame of reference, the side chain of each amino acid can start from a starting point, and can proceed along the first bond of the side chain (i.e., the α-carbon-β-carbon bond) in a defined direction.

[0013] In another example, structural parameters defining protein structure may include a "distance map," which characterizes the estimated distance (e.g., in angstroms) between each pair of amino acids in the protein. The distance map can characterize the estimated distances between amino acid pairs, for example, through a probability distribution of the set of possible distances between amino acid pairs.

[0014] In another example, structural parameters that define a protein's structure can define the three-dimensional (3D) spatial location of each atom in each amino acid within the protein structure.

[0015] The protein structure prediction system described herein can be used to obtain ligands, such as ligands for drugs or industrial enzymes. For example, a method for obtaining a ligand may include obtaining the target amino acid sequence, particularly the amino acid sequence of a target protein (e.g., a drug target), and processing the input based on the target amino acid sequence using the protein structure prediction system to determine the (tertiary) structure of the target protein, i.e., the predicted protein structure. The method may then include evaluating the interaction between one or more candidate ligands and the target protein structure. The method may also include selecting one or more candidate ligands as ligands based on the evaluation results of the interaction.

[0016] In some implementations, evaluating interactions may include assessing the binding of candidate ligands to target protein structures. For example, evaluating interactions may include identifying ligands that bind with sufficient affinity to achieve biological effects. In some other implementations, evaluating interactions may include assessing the association between candidate ligands and the structure of a target protein that influences the function of the target protein (e.g., an enzyme). Evaluation may include assessing the affinity between candidate ligands and target protein structures, or assessing the selectivity of the interaction. Candidate ligands (multiple) may be selected based on the ligand with the highest affinity.

[0017] Multiple candidate ligands can be derived from a candidate ligand database, and / or obtained by modifying ligands in the candidate ligand database, for example by modifying the structure or amino acid sequence of the candidate ligand, and / or obtained by stepwise or iterative assembly / optimization of the candidate ligand.

[0018] The evaluation of the interaction between candidate ligands and target protein structures can be performed using computer-aided methods, where graphical models displaying the candidate ligand and target protein structures are available for user manipulation, and / or the evaluation can be performed partially or fully automatically, for example using standard molecular (protein-ligand) docking software. In some implementations, the evaluation may include determining the interaction score of the candidate ligand, where the interaction score is a measure of the interaction between the candidate ligand and the target protein. The interaction score may depend on the strength and / or specificity of the interaction; for example, the score depends on the binding free energy. Candidate ligands can be selected based on their scores, for example, as the candidate ligand with the highest interaction score.

[0019] In some implementations, the target protein includes a receptor or enzyme, and the ligand is an agonist or antagonist of the receptor or enzyme. In some implementations, this method can be used to identify structures with cell surface markers. This can then be used to identify ligands, such as antibodies or markers, such as fluorescent markers, that bind to the cell surface markers. This can be used to identify and / or treat cancer cells.

[0020] In some implementations, the ligand is the drug, and a predicted structure for each of a plurality of target proteins is determined, and the interaction of one or more candidate ligands with the predicted structure of each target protein is evaluated. One or more candidate ligands can then be selected to obtain a ligand that interacts (functionally) with each target protein, or to obtain a ligand that interacts (functionally) with only one target protein. For example, in some implementations, it may be desirable to obtain a drug effective against multiple drug targets. Furthermore, or instead, it may be necessary to screen drugs for off-target effects. For example, in agriculture, it may be useful to determine that a drug designed for one plant species does not interact with another different plant species and / or animal species.

[0021] In some implementations, the candidate ligands may include small molecule ligands, such as organic compounds with a molecular weight <900 Daltons. In other implementations, the candidate ligands may include peptide ligands, i.e., peptide ligands defined by an amino acid sequence.

[0022] In some cases, protein structure prediction systems can be used to determine the structure of candidate polypeptide ligands, such as those for drugs or industrial enzymes. Their interaction with the target protein structure can then be assessed; the target protein structure can be determined using structure prediction neural networks or conventional physical research techniques such as X-ray crystallography and / or magnetic resonance imaging and / or cryo-electron microscopy.

[0023] On the other hand, a method is provided for obtaining peptide ligands (e.g., molecules or their sequences) using a protein structure prediction system. This method may include obtaining the amino acid sequences of one or more candidate peptide ligands. The method may further include using a protein structure prediction system to determine the (tertiary) structure of the candidate peptide ligands. The method may further include obtaining the target protein structure of the target protein on a computer and / or through physical studies, and evaluating the interaction between the structure of each of the one or more candidate peptide ligands and the target protein structure. The method may also include selecting one or more candidate peptide ligands as peptide ligands based on the evaluation results.

[0024] As previously described, evaluating interactions may include assessing the binding of candidate peptide ligands to target protein structures, for example, identifying ligands with sufficient affinity for biological effects, and / or assessing the association of candidate peptide ligands with target protein structures that influence the function of the target protein (e.g., enzymes), and / or assessing the affinity between candidate peptide ligands and target protein structures, or assessing the selectivity of the interaction. In some implementations, the peptide ligand may be an aptamer. Similarly, (multiple) peptide candidate ligands may be selected based on the ligand with the highest affinity.

[0025] As previously mentioned, the selected peptide ligand may comprise a receptor or an enzyme, and the ligand may be an agonist or antagonist of the receptor or enzyme. In some implementations, the peptide ligand may comprise an antibody, and the target protein may comprise an antibody target, such as a virus, particularly a viral capsid protein, or a protein expressed on cancer cells. In these implementations, the antibody binds to the antibody target to provide a therapeutic effect. For example, the antibody may bind to the target and act as an agonist of a specific receptor; or, the antibody may prevent the binding of another ligand to the target, thereby preventing the activation of the relevant biological pathway.

[0026] The implementation of this method may further include synthesis, i.e., the production of small molecule or peptide ligands. The ligands can be synthesized using any conventional chemical technique and / or are already available, for example, they can be derived from a compound library or synthesized using combinatorial chemistry.

[0027] This method can be further included in vitro and / or in vivo testing of the ligand's biological activity. For example, the ADME (absorption, distribution, metabolism, excretion) and / or toxicological properties of the ligand can be tested to screen out unsuitable ligands. Tests may include, for example, contacting the candidate small molecule or peptide ligand with the target protein and measuring changes in protein expression or activity.

[0028] In some implementations, candidate (peptide) ligands may include: isolated antibodies, fragments of isolated antibodies, single-domain antibodies, bispecific or multispecific antibodies, multivalent antibodies, bivariate antibodies, immunoconjugates, fibronectin molecules, attachment proteins, DARPin, antibodies, affinity molecules, anti-carlin, affinity proteins, protein epitope mimics, or combinations thereof. Candidate (peptide) ligands may include antibodies with mutated or chemically modified amino acid Fc regions, for example, those that prevent or reduce ADCC (antibody-dependent cellular cytotoxicity) activity and / or increase half-life compared to wild-type Fc regions. Candidate (peptide) ligands may include antibodies with different CDRs (Complementarity-Determining Regions).

[0029] The protein structure prediction system described herein can also be used to obtain diagnostic antibody markers for diseases. A method is also provided for determining the predicted structure of each of one or more candidate antibodies, such as those described above, using the protein structure prediction system. This method may further include obtaining the target protein structure, evaluating the interaction between the predicted structure of each of the one or more candidate antibodies and the target protein structure, and selecting one of the one or more candidate antibodies as a diagnostic antibody marker based on the evaluation results, for example, selecting one or more candidate antibodies with the highest affinity for the target protein structure. This method may include manufacturing the diagnostic antibody marker. The diagnostic antibody marker can be used to diagnose diseases by detecting whether it binds to a target protein in a sample obtained from a patient (e.g., a bodily fluid sample). As described above, the corresponding techniques can be used to obtain therapeutic antibodies (peptide ligands).

[0030] Misfolded proteins are associated with many diseases. Therefore, in another respect, a method is provided to identify the presence of protein misfolding diseases using a protein structure prediction system. This method may include obtaining the amino acid sequence of a protein and using a protein structure prediction system to determine the protein's structure. The method may also include obtaining the structure of a version of the protein obtained from a human or animal body, for example, by conventional (physical) methods. The method then may include comparing the protein's structure with the structure of the version obtained from the body and identifying the presence of a protein misfolding disease based on the comparison result. That is, misfolding of the protein version from within the body can be determined by comparing it with a structure determined by a computer.

[0031] Generally, identifying the presence of a protein misfolding disorder may involve obtaining the amino acid sequence of the protein, using the amino acid sequence to determine the protein's structure, as described herein, and comparing the protein's structure with the structure of a baseline version of the protein. The presence of the protein misfolding disorder is then identified based on the comparison results. For example, the structures being compared may be those of a mutant and a wild-type protein. In implementation, a wild-type protein can be used as the baseline version, but in principle, it can also be used as such.

[0032] In some other respects, computer-implemented methods, as described above or herein, can be used to identify active / binding / blocking sites on target proteins from their amino acid sequences.

[0033] According to one aspect, a method executed by one or more computers is provided for predicting the structure of a protein comprising one or more chains, each chain comprising an amino acid sequence, the method comprising: in a first iteration of an iterative sequence including a first iteration and one or more subsequent iterations: obtaining network input characterizing the protein for the first iteration; processing the network input of the first iteration using a protein structure prediction neural network to generate structural parameters for the first iteration defining an initially predicted structure of the protein; in each subsequent iteration of the iterative sequence: obtaining network input characterizing the protein for subsequent iterations; generating features for subsequent iterations from (i) structural parameters generated in previous iterations prior to subsequent iterations in the sequence, (ii) one or an intermediate output generated by the protein structure prediction neural network when generating structural parameters in the last iteration, or (iii) both; and processing the features and network input for subsequent iterations using the protein structure prediction neural network to generate structural parameters for subsequent iterations defining another predicted structure of the protein; and determining the final predicted structure of the protein based on the structural parameters of the last iteration in the sequence.

[0034] In some implementations, the network input for each iteration in the iteration sequence is the same network input.

[0035] In some implementations, a protein structure prediction neural network is used to process features and network inputs for subsequent iterations to generate structural parameters for subsequent iterations that define the predicted structure of the protein. This includes: generating combined inputs based on features and network inputs for subsequent iterations; and processing the combined inputs using the protein structure prediction neural network to generate structural parameters for subsequent iterations.

[0036] In some implementations, the corresponding network input for each iteration includes the corresponding initial pair embedding for each pair of amino acids in the protein; in each iteration, the structure prediction neural network is configured to repeatedly update the initial pair embeddings as it generates the structure parameters for the iteration; the generated features include a set of transformations of the pair embeddings generated from the updated pair embeddings generated when generating the structure parameters in the previous iteration; and the input for generating the combination includes a set of transformations of the pair embeddings for combining the pair embeddings for subsequent iterations and the initial pair embeddings.

[0037] In some implementations, the corresponding network input for each iteration includes an initial MSA representation representing the corresponding multiple sequence alignment (MSA) for each strand in the protein; the structure prediction neural network is configured to generate one or more sets of single embeddings, each set of single embeddings including the corresponding single embedding for each amino acid of the protein when generating the structural parameters of the iteration; generating features includes a set of transformations of single embeddings generated from one of the sets of single embeddings generated when generating the structural parameters in the previous iteration; and the input for generating combinations includes a set of transformations of single embeddings combined for subsequent iterations and the initial MSA representation.

[0038] In some implementations, combining the set of transformations of individual embeddings with the initial MSA representation for subsequent iterations involves adding the set of transformations of individual embeddings to the first row of the initial MSA representation.

[0039] In some implementations, the corresponding network input for each iteration includes the corresponding initial pair embedding for each pair of amino acids in the protein; in each iteration, structural parameters specify the predicted 3-D spatial location of a specific atom in the amino acid in the protein structure for each amino acid; generating features includes: generating a distance map based on the predicted 3-D spatial locations of the amino acids specified by the structural parameters in previous iterations, the distance map representing the corresponding estimated distance between amino acid pairs in the protein structure for each pair of amino acids; generating a distance map with a transformation having the same dimension as the initial pair embedding based on the distance map; and the input for generating the combination includes the distance map and the initial pair embedding transformed for subsequent iterations.

[0040] In some implementations, the corresponding network input for each iteration includes the corresponding initial pair embedding for each pair of amino acids in the protein; in each iteration, structural parameters specify a distance map that represents the estimated distance between amino acid pairs in the protein structure for each pair of amino acids; generating features includes: generating a distance map with the same dimensional transformation as the initial pair embedding based on the distance map specified by the structural parameters in the previous iteration; and the input for generating the combination includes the distance map and the initial pair embedding transformed for subsequent iterations.

[0041] In some implementations, the corresponding network input for each iteration includes the initial pair embeddings of each amino acid pair in the protein generated using a set of one or more template sequences and the corresponding known structure for each template sequence; and the input for generating the combination includes modifying the initial pair embeddings for subsequent iterations by adding the protein and the structure predictions defined by the embeddings in the previous iterations to a set of one or more template sequences and the corresponding known structure.

[0042] According to another aspect, a method for training a structure prediction neural network is provided, the method comprising: selecting a number of iterations to be performed; obtaining a corresponding training network input for each iteration, each training network input representing a training protein; in the first iteration of an iteration sequence including the selected number of iterations: processing the network input of the first iteration using the protein structure prediction neural network to generate structure parameters for the first iteration that define an initial predicted structure of the training protein; when the selected number of iterations is greater than 1, in each subsequent iteration of the iteration sequence, until the selected number of iterations has been performed: generating (i) structure parameters generated in previous iterations before subsequent iterations in the sequence, (ii) generating in the last iteration The structural parameters are generated by a protein structure prediction neural network as one or an intermediate output, or (iii) both, to generate features for subsequent iterations; and the protein structure prediction neural network is used to process the features for subsequent iterations and the input of the training network to generate structural parameters for subsequent iterations that define another predicted structure of the training protein; and the gradient of an objective function including a structural loss term, which characterizes the similarity between (i) the predicted structure of the target protein specified by the structural parameters generated by the last iteration in the sequence and (ii) the target protein structure of the training protein; and the gradient is used to update the parameter values ​​of the structural prediction neural network to obtain the final predicted structure of the protein from the structural parameters of the last iteration in the sequence.

[0043] In some implementations, determining the gradient involves applying a stopping gradient to the features that the structured predictive neural network processes as input in the final iteration.

[0044] In some implementations, selecting the number of iterations includes sampling the number of iterations from a probability distribution that assigns a probability to each iteration, the number of iterations ranging from 1 to the total number of iterations used after the structure prediction neural network has been trained.

[0045] In some implementations, the method further includes: when the selected number of iterations is equal to 1: determining the gradient of the objective function including a structural loss term, the structural loss term representing the similarity between (i) the initially predicted structure of the target protein defined by the structural parameters generated by the first iteration in the sequence and (ii) the target protein structure of the training protein; and using the gradient to update the parameter values ​​of the structural prediction neural network.

[0046] Specific embodiments of the subject matter described in this specification may be implemented in order to achieve one or more of the following advantages.

[0047] This specification describes a technique for predicting protein structures through multiple iterations. Specifically, in each iteration, the system uses a neural network to predict the protein structure, and in each specific iteration after the first iteration, when generating predictions for that specific iteration using the neural network, the system uses features from previous iterations (“recycles”). This method of generating predictions is commonly referred to as “recycles”. This effectively generates “deeper” neural networks, i.e., neural networks with a larger number of layers, without requiring modifications to the network's structure, and can significantly increase the accuracy of predictions made by the neural network. Furthermore, this specification describes a technique for training neural networks using recycles that does not increase the memory consumption of the training process used to train the neural network (compared to training the neural network with the same architecture that generates the final prediction in a single iteration) and minimizes the increase in training time required for additional iterations.

[0048] The structure of a protein determines its biological function. Therefore, determining protein structure can aid in understanding life processes (e.g., the mechanisms of many diseases) and designing proteins (e.g., as drugs, or as enzymes for industrial processes). For example, which molecules (e.g., drugs) will bind to a protein (and where this binding will occur) depends on the protein's structure. Since drug efficacy is affected by the degree to which they bind to proteins (e.g., in the blood), determining the structure of different proteins can be an important aspect of drug development. However, determining protein structure using physical experiments (e.g., by X-ray crystallography) can be time-consuming and very expensive. Therefore, the protein prediction system described in this specification can facilitate biochemical research and engineering fields involving proteins (e.g., drug development).

[0049] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the specification, drawings, and claims. Attached Figure Description

[0050] Figure 1A An example of a protein structure prediction system is shown.

[0051] Figure 1B An example structure for predictive neural networks is shown.

[0052] Figure 1C This is a flowchart of an example process for predicting protein structure.

[0053] Figure 2 An example architecture for embedding a neural network is shown.

[0054] Figure 3 An example architecture for an update block embedded in a neural network is shown.

[0055] Figure 4 An example schema for the MSA update block is shown.

[0056] Figure 5 An example schema of a pair of update blocks is shown.

[0057] Figure 6 An example architecture of a folded neural network is shown.

[0058] Figure 7 This explains the twist angle between bonds in amino acids.

[0059] Figure 8 This is a schematic diagram of unfolded and folded proteins.

[0060] Figure 9 This is a flowchart of another example process used to predict protein structure.

[0061] Figure 10 An example procedure for generating an MSA representation of the amino acid chain in a protein is shown.

[0062] Figure 11 An example process for generating corresponding pair embeddings for each pair of amino acids in a protein is shown.

[0063] The same reference numerals and names in different figures denote the same elements. Detailed Implementation

[0064] Figure 1A An example protein structure prediction system 100 is shown. The protein structure prediction system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, wherein the following systems, components and techniques are implemented.

[0065] System 100 is configured to process data of one or more amino acid chains 102 that define protein 104 to generate a set 106 of structural parameters that define the predicted protein structure 108, i.e., the predicted structure of protein 104. In other words, the predicted structure 108 of protein 104 can be defined by the set 106 of structural parameters, which together define the predicted 3D structure of the protein after protein folding.

[0066] The structural parameters defining the predicted protein structure 108 can be as described above. For example, they can include, for instance, positional and rotational parameters of each amino acid in protein 104, a distance map characterizing the estimated distance between each pair of amino acids in the protein, the corresponding spatial position of each atom or main chain atom in each amino acid in the protein structure, or combinations thereof, as described above.

[0067] In order to generate structural parameters 106 that define the predicted protein structure 108, the system 100 generates network inputs 140 characterizing the protein 104 from at least the data of the amino acid chain 102 that defines the protein 104.

[0068] As a specific example, the network input 140 characterizing protein 104 may include: (i) a multiple sequence alignment (MSA) representation of the protein, and / or (ii) a set of "pair" embeddings of the protein. These will be referenced below. Figures 1B to 11 To describe in more detail.

[0069] System 100 then generates structure parameters, which are defined multiple (N+1, where N is a positive integer) iterations (i.e., in...). Figure 1A The example iterates from 0 to N to predict the protein structure 108. In each of the multiple iterations, the system 100 generates a set of structure parameters that define the predicted protein structure of protein 104. The system then uses the structure parameters generated in the last iteration as the final structure parameters to define the predicted protein structure 108.

[0070] In some implementations, the number of iterations by which System 100 generates the final prediction after training, i.e., during inference when predicting new proteins, can be fixed. That is, after training, N can be set to a fixed integer such that a total of N+1 iterations are performed, for example, in a way that balances the desired trade-off between computational efficiency (e.g., latency) and prediction quality.

[0071] In some other implementations, after training, System 100 can change the number of iterations executed. For example, System 100 can change the number of iterations based on a time or resource budget allocated to processing, such as by executing the maximum number of iterations that meets the budget. As another example, System 100 can employ confidence estimation techniques to estimate the confidence that the structural prediction is accurate in any given iteration and use the confidence estimate to determine how many iterations to execute. For example, the system can stop executing iterations once the confidence estimate exceeds a threshold or once the confidence estimate has stopped improving. See below for reference. Figure 6 Describe an example of generating confidence estimates.

[0072] During training, system 100 can change the number of iterations performed, that is, make the value of N vary for different network inputs. This will be referenced below. Figure 1B and Figure 1C To describe in more detail.

[0073] More specifically, in the first iteration (“iteration 0”), system 100 uses a structure prediction neural network 150 to process network input 140 to generate a set 106-0 of structure parameters for the first iteration.

[0074] The structure prediction neural network 150 is a neural network configured to process the network input 140 to generate a set of structure parameters defining the predicted structure of protein 104. That is, even if processing occurs in multiple iterations, in each iteration, the neural network 150 generates a complete set of structure parameters defining the complete structure of protein 104. See below for reference. Figures 1B-11 A more detailed description of a specific example of the structure prediction neural network 150.

[0075] In each subsequent iteration, that is, in each iteration after the first iteration, system 100 generates features from the results of the processing performed in the previous iteration.

[0076] Specifically, system 100 generates features from (i) structural parameters generated by structural prediction neural network 150 in previous iterations, (ii) intermediate outputs generated by structural prediction neural network 150 during the generation of structural parameters in previous iterations, or (iii) both.

[0077] The following will refer to Figure 1B The generation of these features will be described in more detail.

[0078] Then, system 100 uses structure prediction neural network 150 to process network input 140 and features generated from the processing performed in previous iterations to generate a set of structure parameters for iteration. Thus, in each iteration after the first iteration, system 100 generates a set of structure parameters that depends not only on network input 140 but also on features from previous iterations.

[0079] For example, in iteration N-1, system 100 uses structure prediction neural network 150 to process network input 140 and features from iteration N-2 142 to generate structure parameters 106-N-1 for the next iteration.

[0080] In the final iteration, i.e., iteration N, system 100 uses a structure prediction neural network 150 to process the features of network input 140 and penultimate iteration N-1 144 to generate the structure parameters 106-N for the final iteration.

[0081] System 100 then uses these structural parameters 106-N as the final structural parameters defining the predicted protein structure 108. Therefore, even though a complete prediction is generated in each iteration, only the prediction from the last iteration is used as the final prediction defining the predicted protein structure 108.

[0082] Figure 1B An example structure for predictive neural network 150 is shown. Specifically, Figure 1B The processing performed by neural network 150 in a specific iteration out of multiple iterations 0 to N is shown.

[0083] like Figure 1B As shown, the neural network 150 processes network input 140, which includes (i) multiple sequence alignment (MSA) representations of proteins 110, and / or (ii) a set of “pair” embeddings of proteins 110.

[0084] The MSA representation of a protein includes the corresponding representation of the MSA for each amino acid chain in the protein. The MSA representation of an amino acid chain in a protein can be represented as an M×N embedding array (i.e., a 2-D embedding array with M rows and N columns), where N is the number of amino acids in the amino acid chain. Each row of the MSA representation corresponds to the corresponding MSA sequence of the amino acid chain in the protein. Reference Figure 10 An example procedure for generating an MSA representation of the amino acid chain in a protein is described.

[0085] System 100 generates the MSA representation of the amino acid chain from protein 104 and the MSA representation of protein 110.

[0086] If a protein consists of only a single amino acid chain, then system 100 can identify the MSA representation 110 of protein 104 as the MSA representation of a single amino acid chain in the protein.

[0087] If a protein comprises multiple amino acid chains, system 100 can generate an MSA representation 110 of the protein by assembling the MSA representations of the amino acid chains in the protein into an embedded block diagonal 2-D array, i.e., where the MSA representations of the amino acid chains in the protein form blocks on the diagonal. System 100 can initialize the embeddings at each position in the 2-D array outside the blocks on the diagonal to default embeddings, such as zero vectors. The amino acid chains in the protein can be arbitrarily ordered, and the MSA representations of the amino acid chains in the protein can be ordered accordingly in the block diagonal matrix. For example, the MSA representation of the first amino acid chain (i.e., according to the order) could be the first block on the diagonal, the MSA representation of the second amino acid chain could be the second block on the diagonal, and so on.

[0088] Typically, a protein's MSA representation can be represented as an embedded 2D array. Throughout this specification, a "row" of a protein's MSA representation refers to the row that defines the 2D embedded array of the protein's MSA representation. Similarly, a "column" of a protein's MSA representation refers to the column that defines the 2D embedded array of the protein's MSA representation.

[0089] The set 112 of embeddings includes corresponding embeddings for each pair of amino acids in protein 104. Generally, embeddings represent information encoding the relationships between amino acid pairs in protein 104. An amino acid pair is an ordered tuple consisting of the first and second amino acids in a protein; that is, the set of possible amino acid pairs in a protein is given by the following formula:

[0090] {(A i A j ):1≤i,j≤N}(1)

[0091] Where N is the number of amino acids in the protein, i,j∈{1,…,N} indexes the amino acids in the protein, and A i It is the amino acid in the protein indexed by i, A j The amino acid in the protein is indexed by j. If the protein contains multiple amino acid chains, the amino acids in the protein can be indexed according to the order of the amino acid chains in the protein, {1,…,N}. That is, the amino acids of the first amino acid chain are indexed sequentially first, then the amino acids of the second amino acid chain, then the amino acids of the third amino acid chain, and so on. The set of embeddings 112 can be represented as a 2-DN×N array of embeddings, for example, where the rows of the 2-D array are indexed by i∈{1,…,N}, the columns of the 2-D array are indexed by j∈{1,…,N}, and the position (i,j) in the 2-D array is indexed by the amino acid pair (A i A j ) to embedding occupancy.

[0092] refer to Figure 11 An example process for generating (initializing) the corresponding pair embeddings for each pair of amino acids in a protein is described.

[0093] System 100 can determine the predicted structure of a protein based on its embeddings. However, in its implementation, system 100 uses both the MSA representation 110 and the embeddings 112 to generate the structural parameters 106 defining the predicted protein structure 108, as both have complementary properties. The structure of the MSA representation 110 can explicitly depend on the number of amino acid chains in the MSA corresponding to each amino acid chain in the protein. Therefore, the MSA representation 110 may not be suitable for directly predicting protein structure, since the protein structure 108 does not explicitly depend on the number of amino acid chains in the MSA. In contrast, the embeddings 112 characterize the relationships between pairs of amino acids in protein 104 and are expressed without explicit reference to the MSA, thus providing a convenient and efficient data representation for predicting the protein structure 108.

[0094] Neural network 150 includes embedded neural network 200 and folded neural network 600.

[0095] System 100 processes the MSA representation 110 and the pair of embeddings 112 using the set of parameter values ​​of the embedding neural network 200 to update the MSA representation 110 and the pair of embeddings 112. That is, the embedding neural network 200 processes the MSA representation 110 and the pair of embeddings 112 to generate an updated MSA representation 114 and an updated pair of embeddings 116.

[0096] The embedding neural network 200 updates the MSA representation 110 and the pair of embeddings 112 by sharing information between the MSA representation 110 and the pair of embeddings 112. More specifically, the embedding neural network 200 alternates between updating the current MSA representation 110 based on the current pair of embeddings 112 and updating the current pair of embeddings 112 based on the current MSA representation 110.

[0097] refer to Figure 2 An example architecture for embedding a neural network 200 is described in more detail.

[0098] More generally, although Figure 2 The description of the following figures illustrates the architecture of the embedding neural network 200, in which each block updates both the pair embeddings and the MSA representation. However, the techniques described for incorporating auxiliary folding neural networks into a structure prediction system can be used with any suitable architecture of the embedding neural network that includes multiple blocks for each updated pair embedding. As a specific example of an alternative architecture to the one described below, the embedding neural network 200 can adopt an architecture in which the pair embeddings 112 are initialized using the MSA representation 110, and where each block receives only the current pair embeddings and updates those embeddings by applying self-attention as described below, but without relying on the MSA representation.

[0099] Neural network 150 generates input to folded neural network 600 based on updated pair embedding 116, updated MSA representation 114, or both, and uses folded neural network 600 to process the input to generate structural parameters 106 that define the predicted protein structure.

[0100] In some implementations, the folded neural network 600 processes the updated pair embeddings 116 to generate a distance map that includes a probability distribution over the set of possible distances between amino acid pairs in the protein structure for each pair of amino acids. For example, to generate a probability distribution over the set of possible distances between amino acid pairs in the protein structure, the folded neural network can apply one or more fully connected neural network layers to the updated pair embeddings 116 corresponding to the amino acid pairs.

[0101] In some implementations, the folded neural network 600 generates structural parameters 106 by processing the inputs derived from both the updated MSA representation 114 and the updated pair embeddings 116 using a geometric attention operation. This geometric attention operation explicitly infers the 3D geometry of amino acids in the protein structure. (Reference) Figure 6 An example architecture of a folded neural network 600 that implements the geometric attention mechanism is described.

[0102] As described above, in the first iteration of N+1 iterations, neural network 150 only processes network input 140.

[0103] However, in each subsequent iteration, system 100 uses the output generated by neural network 150 in the previous iteration to generate feature 113.

[0104] Specifically, system 100 uses structural parameters generated by neural network 150 in previous iterations, intermediate outputs generated by neural network 150 in previous iterations, or both, to generate feature 113.

[0105] More specifically, in any given subsequent iteration, system 100 processes both features 113 and network input 140 using neural network 150 by first generating a combined input based on features 113 and network input 140, and then processing the combined input using neural network 150 to generate structural parameters 106 for subsequent iterations.

[0106] If a subsequent iteration is the final iteration, then system 100 uses the structural parameters 106 generated in the iteration as the final structural parameters for the predicted structure 108 of protein 104.

[0107] Generally, in the implementation where the intermediate output generated by the neural network 150 is used to generate feature 113, the system 100 can use the output of any layer of the neural network 150 (except for the output of the final layer of the folded neural network 600), the input of any layer of the neural network 150 (except for the input layer of the folded neural network 150), or both to generate features.

[0108] As a specific example, system 100 may use updated pair embeddings 116 generated by the last updated block of embedded neural network 200 or updated pair embeddings generated by one of the intermediate layers of folded neural network 600 as at least a portion of generated feature 113, which will be described in more detail below.

[0109] In this example, system 100 can generate features at least in part by applying a transformation to an updated pair of embeddings from a previous iteration, and then combining the features with the initial pair of embeddings 112, for example by addition, averaging, or cascading, as part of the input for generating combinations for subsequent iterations.

[0110] As an example, system 100 can transform the updated pair embeddings by applying a normalization operation, such as a layer normalization operation (e.g., as described in “Layer Normalization” by Jimi Barre et al., arXiv: 1607.06450).

[0111] As another example, system 100 can transform the updated pair embeddings by processing the updated pair embeddings using a multilayer perceptron (MLP) neural network to generate the initial transformed embeddings, and then apply a normalization operation to the initial transformed embeddings to generate the transformed embeddings.

[0112] As another specific example, system 100 may use a “single” embedding of each amino acid in a protein provided as input to folded neural network 600, or a single embedding generated by one of the intermediate layers of folded neural network 600, as part of generating feature 113. See below for reference. Figure 6 The generation of these individual embeddings for amino acids in a protein and the updating of individual embeddings using a folded neural network 600 are described in detail.

[0113] In this example, system 100 may transform a single embedding from a previous iteration, for example, as described above with reference to the embedding, and then combine the transformed single embedding with the initial MSA representation 110 as part of the input for generating subsequent iterations. For example, system 100 may combine the transformed single embedding with the first row of the initial MSA representation 110, for example, by addition or averaging.

[0114] In some implementations of system 100 that use structural parameters 106 generated in a previous iteration as part of the generated features 113, system 100 may transform the structural parameters 106 from the previous iteration into an embedding with the same dimensions as the pair embeddings, and then combine the transformed structural parameters with the initial pair embeddings 110 as part of the network input to generate the combined network.

[0115] In some implementations, system 100 directly combines the structure parameters of the transformation and the initial pair embeddings, for example, system 100 adds, averages, or cascades the structure parameters of the transformation and the initial pair embeddings 110.

[0116] In some other implementations, system 100 can combine the initial pair of embeddings 110 and the structural parameters of the transformation by applying a learned combination, namely, a combination learned together with the training of the embedding neural network 200 and the folded neural network 600. As a specific example, system 100 can apply a learned cross-attention mechanism to combine information from the structural parameters of the transformation. Unlike the self-attention mechanism described below, in the cross-attention mechanism, queries are generated from the updated pair of embeddings, while keys and values ​​are generated from the structural parameters of the transformation.

[0117] Specifically, as described above, in some implementations, structural parameter 106 is a distance map that characterizes the estimated distance between each pair of amino acids in the protein. In these cases, system 100 can project the distance map onto the same dimension as the pair embeddings to generate transformed structural parameters.

[0118] For example, the system can discretize each distance in the distance graph by assigning each distance to a bin within a plurality of bins, each bin corresponding to a distance value within a different range. The system can then embed each discretized distance, for example, by multiplying the one-hot encoded representation of the discretized distance by an embedding matrix. The one-hot encoded representation represents the discretized distance as a vector that includes "1" at the position corresponding to the discretized distance and "0" at all other positions corresponding to all other discretized distances.

[0119] As another example, the system can apply a kernel function to each distance in the distance graph to generate distance representations, which are then embedded. For instance, the kernel function could be equal to the function exp(-distance), or more generally, proportional to the function exp(-distance).

[0120] As another example, as described above, in some other implementations, the structural parameters include predicted 3-D spatial positions of specific atoms within the amino acids in the protein structure for each amino acid. System 100 can construct a distance map based on the positions defined by the structural parameters by calculating the distances between specific pairs of atoms within the amino acids in the protein. In these cases, System 100 can transform the distance map to the same dimension as the updated pair embeddings to generate the transformed structural parameters as described above.

[0121] As will be described in more detail below, in some implementations, system 100 generates pairs of embeddings 112 based at least in part on a set of one or more template sequences. Each template sequence is an MSA sequence of an amino acid chain in a protein, wherein the folding structure of the template sequence 1110 is known, for example, through physical experiments.

[0122] In some implementations of system 100 that use structural parameters 106 generated in previous iterations as part of the generated feature 113, system 100 may add protein 104 and the folded structure of the protein defined by structural parameters 106 from previous iterations as another in the template sequence for generating embedding 112, which is part of the combined network input in subsequent iterations. See below for reference. Figure 11 The description uses a template sequence to generate a pair of embeddings 112.

[0123] The above description generally describes using the same network input 140 for all iterations. However, in some cases, the system uses different network input 140 for different iterations. As a particular example, the system may sample a different set of MSAs from a larger set of MSA sequences for each iteration, resulting in different MSA representations 110 being generated for each iteration in multiple iterations. As another example, in an implementation that uses a template sequence to update the generated pair embeddings 112, for the first iteration (“iteration o”), the system may generate the pair embeddings 112 without using any template sequence.

[0124] The training engine can train the protein structure prediction neural network150 end-to-end to optimize the objective function, which includes a term referred to in this paper as the structure loss.

[0125] The training engine can train system 100 on a collection of training data that includes multiple training examples. Each training example can specify: (i) an initial MSA representation of the protein and initial training input for the embedding, and (ii) a target protein structure that should be generated by system 100 by processing the training input. The target protein structure used to train system 100 can be determined using experimental techniques such as X-ray crystallography or cryo-electron microscopy.

[0126] The structural loss can characterize the similarity between the following two: (i) the predicted protein structure generated by system 100, i.e., the predicted protein structure defined by the final structural parameters of the last iteration, and (ii) the target protein structure that should have been generated by the system.

[0127] For example, if the predicted structural parameters define the predicted position and rotation parameters for each amino acid in the protein, then the structural loss L structure It can be given by the following formula:

[0128]

[0129]

[0130]

[0131] Where N is the number of amino acids in the protein, t i R represents the predicted position parameter of amino acid i. i This represents a 3×3 rotation matrix specified by the predicted rotation parameters of amino acid i. It is the target position parameter of amino acid i. This represents a 3×3 rotation matrix specified by the target rotation parameter of amino acid i, where A is a constant. Represents the predicted rotation parameter R i The inverse of a specified 3×3 rotation matrix, Indicates the rotation parameters of the target The inverse of the specified 3×3 rotation matrix, and (·)+ denotes the corrected linear unit (ReLU) operation.

[0132] The structural loss defined by equations (2)-(4) can be understood as the average loss of each pair of amino acids in a protein. Item t ij Define the predicted spatial position of amino acid j in the reference frame for predicting amino acid i, and The actual spatial position of amino acid j in the reference frame of actual amino acid i is defined. These terms are sensitive to both the predicted and actual rotations of amino acids i and j, and therefore carry richer information than loss terms, which are only sensitive to the predicted and actual distances between amino acids.

[0133] As another example, if the predicted structural parameters define the predicted spatial location of each atom in each amino acid of a protein, then the structural loss can be the average error (e.g., squared error) between (i) the predicted spatial location of the atom and (ii) the target (e.g., ground reality) spatial location of the atom.

[0134] Optimizing structural loss enables the system to generate predicted protein structures that are accurate and close to the actual protein structures.

[0135] In addition to structural losses, the objective function may include one or more auxiliary losses. For example, auxiliary losses may be disadvantageous to predicted structures with features that are impossible to occur in nature, based on the bond angles and / or bond lengths between atoms in amino acids in the predicted structure, or based on the proximity of atoms in different amino acids in the predicted structure.

[0136] In some implementations, the auxiliary loss also includes an auxiliary loss that measures the difference between the structure prediction defined by the structure parameters generated in the penultimate iteration and the structure prediction defined by the structure parameters in the last iteration; that is, an auxiliary loss that penalizes the neural network 150 for generating similar predictions in subsequent iterations. This can incentivize the neural network 150 not only to improve the initial prediction in each subsequent iteration, but also to generate new predictions that more accurately represent the actual structure of the protein. In these implementations, after training, if multiple plausible structure predictions are needed, the system 100 can output structure predictions as different structure predictions for the same protein in multiple iterations.

[0137] During training, system 100 applies a stop gradient to feature 113, which is used as input in the last iteration. The stop gradient prevents gradients from flowing through feature 113 back to the processing performed in any earlier iteration.

[0138] More specifically, the structure of the prediction generated in the final iteration depends not only on the processing performed in the final iteration, but also on the processing performed in each earlier iteration, since feature 113 is also provided as input in the final iteration. Without stopping gradients, the training engine could backpropagate gradients through feature 113 and compute the gradient of the objective function with respect to each parameter of the neural network 150, not only for the final iteration but also for each previous iteration. Because stopping gradients are applied, the system does not backpropagate any gradients through feature 113 and only computes a single gradient with respect to each parameter, i.e., the gradient computed based on the processing that occurred in the final iteration. Applying stopping gradients prevents the re-encapsulation of the framework from leading to a significant increase in training time and computation (e.g., processor cycles), as the prediction generated in iteration 0 is always used as the framework for the final prediction.

[0139] Figure 1C This is a flowchart of an example process 1300 for predicting protein structure. For convenience, process 1300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a protein structure prediction system appropriately programmed according to this specification, such as... Figure 1AThe protein structure prediction system 100 can execute process 1300.

[0140] The system can execute process 1300 during inference, i.e. after training the neural network, to predict new proteins, or during training, use proteins with known target structures to determine updates to the parameter values ​​of the structure prediction neural network.

[0141] The system selects the number of iterations to perform in predicting the protein structure (step 1302).

[0142] Specifically, as described above, in some cases, when process 1300 is executed during inference, the system selects the same number of iterations for each new protein that requires a new prediction.

[0143] In these cases, when performing process 1300 during training, the system can select different numbers of iterations for different iterations of process 1300. In particular, selecting different numbers of iterations allows the system to ensure that gradient feedback is received from the processes performed in all iterations, not just the last iteration in inference, even if stopping gradients are applied as described above.

[0144] As a specific example, during training, the system can sample the number of iterations from a probability distribution that assigns a corresponding probability to each iteration from 1 total iterations to N+1 (i.e., the total number of iterations used during inference in an implementation where the number of iterations performed after training is fixed). In implementations where the number of iterations performed after training is not fixed, N+1 can be set to either the maximum number of iterations that can be performed during inference or some other fixed value representing the maximum possible number of iterations.

[0145] In some implementations, the probability distribution is uniform, which assigns the same probability to each iteration from 1 to N+1.

[0146] In some other implementations, the probability distribution assigns a higher probability to some iterations than to others. For example, the probability distribution might assign a higher probability of 1 than any other possible iteration number, meaning only the initial iteration is performed. This ensures that the neural network learns to generate high-quality output during the initial iteration, which can then be improved in subsequent iterations. This also saves training time, as performing only the initial iteration takes less time than performing multiple iterations.

[0147] As another example, as training progresses—that is, as the neural network learns to make relatively accurate predictions with relatively few iterations—the system can adjust the probability distribution to increasingly favor larger iterations.

[0148] In some implementations, the system distributes the training of the neural network across multiple computing devices, each executing process 1300 in parallel. In these implementations, the system can constrain the selection such that for any given iteration of process 1300, each computing device chooses the same number of iterations. Choosing the same number of iterations prevents devices choosing fewer iterations from being idle while devices choosing more iterations have finished processing.

[0149] In some other cases, as described above, the system can choose different numbers of iterations depending on the circumstances. In these cases, as described above, the system also chooses different numbers of iterations during training, except that a probability distribution assigns a corresponding probability to each iteration from 1 total iterations to a fixed number of iterations, which is determined as an upper limit representing the number of iterations that can be performed after training.

[0150] The system obtains the corresponding network input for each of the selected iterations (step 1304). Specifically, as mentioned above, in some implementations, the network input is the same for all iterations. In other implementations, the network input may differ between iterations.

[0151] In the first iteration, the system uses a structure prediction neural network to process the network input for the first iteration in order to generate the structure parameters for the first iteration (step 1306).

[0152] If the first iteration is the last iteration, i.e., the number of iterations is chosen to be equal to 1, then the system uses the structural parameters of the first iteration as the final structural parameters for defining the predicted structure of the protein (step 1312).

[0153] If the first iteration is not the last iteration, the system executes steps 1308 and 1310 until the structure parameters for the last iteration are generated.

[0154] Specifically, the system generates features based on the output of the structure prediction neural network generated in previous iterations (step 1308).

[0155] Because of the aforementioned stopping gradient, the system can then discard all outputs generated in previous iterations, i.e., instead of storing the outputs in memory, since these outputs are not needed to compute gradients even when process 1300 is executed during training.

[0156] Therefore, since the system does not store the output of any “internal” iterations in memory, the memory consumption of the training process does not increase when the final prediction is generated through multiple iterations rather than in a single iteration.

[0157] The system then processes the iterative features and network inputs to generate iterative structural parameters (step 1310). Typically, as described above, the system generates a combined input based on the iterative features and network inputs, and then processes the combined inputs using a structure prediction neural network to generate the iterative structural parameters.

[0158] If the iteration is not the last iteration, then process 1300 returns to step 1308.

[0159] If the iteration is the last iteration, the system uses the structure parameters generated in the last iteration as the final structure parameters (step 1312).

[0160] If process 1300 is executed during training, the system uses the final structural parameters to determine the gradient of the parameters of the structural prediction neural network relative to the objective function (step 1314), i.e., as referenced above. Figure 1B The objective function is described. Specifically, the system computes gradients through backpropagation, that is, it performs backpropagation through a neural network and then applies the stopping gradient to the features at the penultimate iteration.

[0161] Therefore, as mentioned above, the system does not backpropagate gradients to previous iterations.

[0162] Backpropagation (calculating the gradient of the objective function with respect to all parameters of the neural network) is the most computationally expensive part of the training process. Due to the stopping gradient, the system still performs the same number of backpropagations for each iteration of process 1300; that is, only a single backpropagation is performed for each iteration of process 1300 if the system always generates a prediction in a single iteration. Therefore, the system can achieve the benefits of additional iterations during inference (in terms of prediction accuracy) without correspondingly reducing the computational efficiency of the training process.

[0163] Once a threshold condition is met, such as when the gradients of the entire batch of training inputs have been computed, the system can compute updates based on the gradients according to the update rules of the optimizer used for training, such as Adam, rmsProp, or SGD. The system can then apply the updates to the current values ​​of the parameters, either by subtracting the updates from the current values ​​or by adding the updates to the current values.

[0164] Figure 2 An example architecture of an embedded neural network 200 is shown, which is configured to process MSA representation 110 and pair embedding 112 to generate updated MSA representation 114 and updated pair embedding 116.

[0165] The embedded neural network 200 includes a sequence of updates to block 202-AN. Throughout this specification, "block" refers to a portion of a neural network, such as a subnetwork of a neural network that includes one or more neural network layers.

[0166] Each update block in the embedded neural network is configured to receive a block input including an MSA representation and an embedding, and to process the block input to generate a block output including an updated MSA representation and an updated embedding.

[0167] The embedding neural network 200 provides the first update block (i.e., in the update block sequence) with the MSA representation 110 and the pair embeddings 112 included in the network input of the embedding neural network 200. The first update block processes the MSA representation 110 and the pair embeddings 112 to generate updated MSA representations and updated pair embeddings.

[0168] For each update block after the first update block, the embedding neural network 200 provides the update block with the MSA representation and pair embeddings generated by the previous update block, and for each update block except the last update block, it provides the next update block with the updated MSA representation and updated pair embeddings generated by the update block.

[0169] The embedded neural network 200 gradually enriches its information content by repeatedly updating the MSA representation 110 and the embedding 112 using the sequence of update blocks 202-AN.

[0170] The embedded neural network 200 can provide an updated MSA representation 114 and an updated pair embedding 116 generated by the final update block (i.e., in the update block sequence) as network outputs.

[0171] Figure 3 An example architecture of the update block 300 embedded in the neural network 200 is shown, i.e., as referenced Figure 2 As described.

[0172] Update block 300 receives block input including current MSA representation 302 and current pair embedding 304, and processes the block input to generate updated MSA representation 306 and updated pair embedding 308.

[0173] Update block 300 includes MSA update block 400 and update block 500.

[0174] MSA update block 400 updates the current MSA representation 302 with the current pair embedding 304, and update block 500 updates the current pair embedding 304 with the updated MSA representation 306 (i.e., generated by MSA update block 400).

[0175] Typically, MSA representations and pair embeddings can encode complementary information. For example, an MSA representation can encode information about the correlation between amino acid identifiers at different positions in a set of evolutionarily relevant amino acid chains, while a pair embedding can encode information about the relationships between amino acids in a protein. MSA update block 400 uses the complementary information encoded in the pair embeddings to enrich the information content of the MSA representation, and pair update block 500 uses the complementary information encoded in the MSA representation to enrich the information content of the pair embeddings. As a result of this enrichment, the updated MSA representation and the updated pair embeddings encode information that is more relevant to predicting protein structure.

[0176] Update block 300 is described herein as first updating the current MSA representation 302 with the current pair embedding 304, and then updating the current MSA representation 304 with the updated MSA representation 306. This description should not be construed as limiting the update block to performing operations in this order; for example, the update block could first update the current pair embedding with the current MSA representation, and then update the current MSA representation with the updated pair embedding.

[0177] Update block 300 is described herein as comprising MSA update block 400 (i.e., updating the current MSA representation) and pair update block 500 (i.e., updating the current pair embedding). This description should not be construed as limiting update block 300 to containing only one MSA update block or only one pair update block. For example, update block 300 may include multiple MSA update blocks that update the MSA representation multiple times before the MSA representation is provided to the pair update blocks for updating the current pair embedding. As another example, update block 300 may include multiple pair update blocks that update the pair embedding multiple times using the MSA representation.

[0178] MSA update block 400 and update block 500 can have any suitable architecture that enables them to perform the functions they describe.

[0179] In some implementations, the MSA update block 400, update block 500, or both include one or more “self-attention” blocks. As used throughout this paper, a self-attention block generally refers to a neural network block that updates a set of embeddings; that is, it receives a set of embeddings and outputs updated embeddings. To update a given embedding, a self-attention block can determine the corresponding “attention weights” between the given embedding and each of one or more selected embeddings, and then use (i) the attention weights and (ii) the selected embeddings to update the given embedding. For convenience, a self-attention block can be described as updating the given embedding using attention “beyond” the selected embeddings.

[0180] For example, a self-attention block can receive a set of input embeddings. Where N is the number of amino acids in the protein, and in order to update the embedding x iSelf-attention blocks can determine attention weights. Where a i，j x represents i and x j Attention weights between them, such as:

[0181]

[0182]

[0183] Among them W q and W k Here, is the parameter matrix being learned, softmax(·) denotes the soft maximum normalization operation, and c is a constant. Using attention weights, the self-attention layer can update the embedding x as follows: i :

[0184]

[0185] Among them W v It is the parameter matrix for learning. (For the input embedding x) i W q x i This can be called a query embedding, for the input embedding x i W k x j This can be called "keyword embedding," for input embedding x i W v x j This can be called "value embedding".

[0186] Parameter matrix W q ("Query Embedding Matrix"), W k ("Keyword Embedding Matrix") and W v The "value embedding matrix" is the trainable parameter of the self-attention block. The parameters of any self-attention block included in MSA update block 400 and update block 500 can be understood as the parameters of update block 300, which can be used as a reference. Figures 1A-1C A portion of the end-to-end training of the described protein structure prediction system 100 is trained. Typically, the (trained) parameters of the query, keyword, and value embedding matrices are different for different self-attention blocks, for example, such that the self-attention blocks included in the MSA update block 400 can have different query, keyword, and value embedding matrices with different parameters than those of the self-attention blocks included in the update block 500.

[0187] In some implementations, the MSA update block 400, the update block 500, or both include one or more self-attention blocks conditioned on the pair embeddings, i.e., implementing self-attention operations conditioned on the pair embeddings. To perform self-attention operations on the pair embeddings, the self-attention blocks can process the pair embeddings to generate a corresponding “attention bias” for each attention weight. For example, in addition to determining the attention weights according to equations (5)-(6), In addition, self-attention blocks can generate a set of corresponding attention biases. Where b i，j x represents i and x j Attention bias between them. The self-attention module can address this by applying the learned parameter matrix to the embedding h. i，j That is, to generate attention bias b for amino acid pairs in proteins indexed by (i, j). i，j .

[0188] Self-attention blocks can be determined, for example, by summing (or otherwise combining) attention weights and attention biases to determine the set of "biased attention weights". Where c i，j x represents i and x j Attention weights are assigned to the deviations between them. For example, a self-attention block can determine the embedding x. i and x j Attention weight c of the deviation between i，j :

[0189] C i，j =a i，j +b i，j

[0190] Where a i，j It is between x i and x j Attention weights between, b i，j It is between x i and x j The attention bias between inputs. Self-attention blocks can use the attention weights of the bias to update each input embedding x. i ,For example:

[0191]

[0192] Among them W v It is the parameter matrix for learning.

[0193] Typically, embeddings encode information characterizing protein structure and the relationships between amino acid pairs within that structure. Applying a self-attention operation conditioned on embeddings to a set of input embeddings allows the input embeddings to be updated in a manner informed by the protein structural information encoded in the embeddings. Update blocks in embedding neural networks can use self-attention blocks conditioned on embeddings to update and enrich the MSA representation and the embeddings themselves.

[0194] Optionally, the self-attention block can have multiple "heads," each generating a corresponding updated embedding for each input embedding, i.e., such that each input embedding is associated with multiple updated embeddings. For example, each head can be a parameter matrix W described by reference equations (5)-(7). q W k and W v Different values ​​of the input embedding are used to generate updated embeddings. Self-attention blocks with multiple heads can implement a "gating" operation to combine updated embeddings generated by the heads for the input embeddings; that is, to generate a single updated embedding corresponding to each input embedding. For example, a self-attention block can use one or more neural network layers (e.g., a fully connected neural network layer) to process the input embeddings to generate a corresponding gating value for each head. The self-attention block can then combine updated embeddings corresponding to the input embeddings based on the gating values. For example, a self-attention block can generate updated embeddings for the input embedding x. i Generate the updated embedding as follows:

[0195]

[0196] Where, k is the index head, α k It is the gating value of the head k. The head k is the input embedding x i The generated updated embedding.

[0197] refer to Figure 4 An example architecture for MSA update block 400 using self-attention blocks conditioned on embeddings is described. References Figure 4 The example MSA update block described updates the current MSA representation based on the current pair embedding by processing the rows of the current MSA representation using a self-attention block conditioned on the current pair embedding.

[0198] refer to Figure 5 An example architecture for updating block 500 using self-attention blocks conditioned on embeddings is described. (Reference) Figure 5 The example described describes updating the pair embedding by computing the outer product mean of the updated MSA representation, adding the result of the outer product mean to the current pair embedding, and processing the current pair embedding with a self-attention block conditioned on the current pair embedding, thereby updating the current pair embedding based on the updated MSA representation.

[0199] Figure 4 An example architecture of MSA update block 400 is shown. MSA update block 400 is configured to receive the current MSA representation 302 to update the current MSA representation 306 (at least in part) based on the current pair embedding.

[0200] To update the current MSA representation 302, the MSA update block 400 uses a self-attention operation conditioned on the current pair of embeddings (i.e., a "line-by-line" self-attention operation) to update the embeddings in each line of the current MSA representation. More specifically, the MSA update block 400 provides the embeddings in each line of the current MSA representation 302 to the "line-by-line" self-attention block 402 conditioned on the current pair of embeddings, for example, as referenced Figure 3 This is done so that updated embeddings are generated for each row of the current MSA representation 302. Optionally, the MSA update block can sum the inputs of the row-by-row self-attention block 402 to the output of the row-by-row self-attention block 402. Adjusting the row-by-row self-attention block 402 according to the current pair of embeddings allows the MSA update block 400 to enrich the current MSA representation 302 with information from the current pair of embeddings.

[0201] The MSA update block then uses a self-attention operation (i.e., “column-by-column” self-attention operation) that is not conditioned on the current pair of embeddings to update the embeddings in each column of the current MSA representation. More specifically, the MSA update block 400 feeds the embeddings in each column of the current MSA representation 302 to a “column-by-column” self-attention block 404 that is not conditioned on the current pair of embeddings to generate updated embeddings for each column of the current MSA representation 302. As a result of not being conditioned on the current pair of embeddings, the column-by-column self-attention block 404 uses attention weights (e.g., as described in reference equations (5)-(6)) instead of biased attention weights (e.g., as described in reference equation (8)) to generate updated embeddings for each column of the current MSA representation. Optionally, the MSA update block may sum the inputs to the column-by-column self-attention block 404 to the output of the column-by-column self-attention block 404.

[0202] The MSA update block then processes the current MSA representation 302 using a transformation block, for example, by applying one or more fully connected neural network layers to the current MSA representation 302. Optionally, the MSA update block 400 can add the input of the transformation block 406 to the output of the transformation block 406.

[0203] The MSA update block can output an updated MSA representation 306 generated by the operations performed by the row-by-row self-attention block 402, the column-by-column self-attention block 404, and the transformation block 406.

[0204] Figure 5An example architecture for update block 500 is shown. Update block 500 is configured to receive the current pair embedding 304 and update the current pair embedding 304 based on (at least partially) the updated MSA representation 306.

[0205] To update the current pair embedding 304, the update block 500 applies the outer product averaging operation 502 to the updated MSA representation 306 and adds the result of the outer product averaging operation 502 to the current pair embedding 304.

[0206] The outer product mean operation defines a series of operations that, when applied to an MSA representation of an M×N embedding array, generate an N×N embedding array, where N is the number of amino acids in the protein. The current embedding 304 can also be represented as an N×N embedding array, and adding the result of the outer product mean 502 to the current embedding 304 refers to summing two N×N embedding arrays.

[0207] To compute the mean of the outer product, a tensor A(·) is generated for the update block, for example, given by the following equation:

[0208]

[0209] Where res1, res2 ∈ {1, ..., N}, ch1, ch2 ∈ {1, ..., C}, where C is the number of channels in each embedding of the MSA representation, |rows| is the number of rows in the MSA representation, LeftAct(row, res1, ch1) is a linear operation (e.g., defined by matrix multiplication) applied to the channel ch1 of the MSA representation embedded at the row indexed by “row” and the column indexed by “res1”, and RightAct(row, res2, ch2) is a linear operation (e.g., defined by matrix multiplication) applied to the channel ch2 of the MSA representation embedded at the row indexed by “row” and the column indexed by “res2”. The result of the outer product mean is generated by flattening and linearly projecting the (ch1, ch2) dimension of the tensor A. Optionally, as part of the calculation of the outer product mean, one or more layer normalization operations (e.g., as described in “Layer Normalization” by Jimi Barre et al., arXiv: 1607.06450) can be performed on the update block.

[0210] Typically, the updated MSA representation 306 encodes information about the correlation between amino acid identifiers at different positions in a set of evolutionarily relevant amino acid chains. The information encoded in the updated MSA representation 306 is related to the predicted protein structure, and the updated block 500 can enhance the information content of the current pair embedding by incorporating the information encoded in the updated MSA representation into the current pair embedding (i.e., by means of the outer product 502).

[0211] After updating the current pair embeddings 304 using the updated MSA representation (i.e., via the outer product mean 502), the update block 500 is updated to the N×N array with the current pair embeddings in each row of the permutation of the current pair embeddings using a self-attention operation conditioned on the current pair embeddings (i.e., a "row-by-row" self-attention operation). More specifically, the update block 500 provides each row of the current pair embeddings to the "row-by-row" self-attention block 504, which is also conditioned on the current pair embeddings, for example, as referenced. Figure 3 This is done so that an updated pair embedding can be generated for each row. Optionally, the pair update block can add the input of the row-by-row self-attention block 504 to the output of the row-by-row self-attention block 504.

[0212] Then, update block 500 uses the same self-attention operation conditioned on the current pair embeddings (i.e., "column-by-column" self-attention operation) to update the current pair embeddings in each column of the N×N array of current pair embeddings. More specifically, update block 500 provides each column of the current pair embeddings to "column-by-column" self-attention block 506, which also generates updated pair embeddings for each column conditioned on the current pair embeddings. Optionally, update block can sum the inputs of column-by-column self-attention block 506 to the output of column-by-column self-attention block 506.

[0213] Then, the current pair of embeddings is processed using a transformation block on update block 500, for example, by applying one or more fully connected neural network layers to the current pair of embeddings. Optionally, update block 500 can add the input of transformation block 508 to the output of transformation block 508.

[0214] The update block can output the updated pair embedding 308 generated by the operations performed by the row-by-row self-attention block 504, the column-by-column self-attention block 506, and the transformation block 508.

[0215] Figure 6 An example architecture of a folded neural network 600 is shown, which generates a set 106 of structural parameters defining a predicted protein structure 108. For example, in an implementation, the folded neural network 600 generates values ​​for structural parameters 314 by processing inputs including corresponding pairs of embeddings 312 for each pair of amino acids in the protein, determining the predicted structure of the protein based on the embeddings. The folded neural network 600 can be included in a reference... Figures 1A-1C The protein structure prediction system 100 is described.

[0216] The folded neural network 600 generates structural parameters for each amino acid in a protein, which can include: (i) position parameters and (ii) rotation parameters. As previously described, the position parameters of an amino acid can specify the predicted 3-D spatial position of a particular atom within the amino acid in the protein structure. The rotation parameters of an amino acid can specify the predicted “orientation” of the amino acid in the protein structure. More specifically, the rotation parameters can specify a 3-D spatial rotation operation, which, when applied to the coordinate system of the position parameters, results in the three “main chain” atoms in the amino acid being in fixed positions relative to the rotation coordinate system.

[0217] In its implementation, the folded neural network 600 receives input from the final MSA representation, the final pair embeddings 116, or both, and generates final values ​​for structure parameters 106 that define the predicted structure of the protein. For example, the folded neural network 600 may receive input including: (i) the respective pair embeddings 116 for each pair of amino acids in the protein, (ii) initial values ​​for the “single” embeddings 602 for each amino acid in the protein, and (iii) initial values ​​for the structure parameters 604 for each amino acid in the protein. The folded neural network 600 processes the input to generate the final values ​​for the structure parameters 106, which collectively characterize the predicted structure 108 of the protein.

[0218] The protein structure prediction system 100 can provide the folded neural network 600 with pairs of embeddings generated as the output of the embedding neural network, as shown in the reference. Figures 1A-1C As stated above.

[0219] Protein structure prediction system 100 can generate an initial single embedding 602 for amino acids from MSA representation 114, i.e., as referenced. Figures 1A-1C As described, the output is generated as an embedding neural network. For example, as mentioned above, the MSA representation 114 can be represented as a 2-D array of embeddings having a number of columns equal to the number of amino acids in the protein, where each column is associated with a corresponding amino acid in the protein. The protein structure prediction system 100 can generate an initial single embedding for each amino acid in the protein by summing (or combining) the embeddings from the columns of the MSA representation 114 associated with the amino acid. As another example, the protein structure prediction system 100 can generate an initial single embedding for an amino acid in the protein by extracting the embeddings from the rows of the MSA representation 114 corresponding to the amino acid sequence of the protein whose structure is estimated.

[0220] The protein structure prediction system 100 can generate initial structure parameters 604 with default values, for example, where the position parameter of each amino acid is initialized to the origin (e.g., [0, 0, 0] in a Cartesian coordinate system), and the rotation parameter of each amino acid is initialized to a 3×3 identity matrix.

[0221] The folded neural network 600 generates the final structure parameter 106 by repeatedly updating the current values ​​of a single embedding 606 and the structure parameter 608, i.e., starting from their initial values. More specifically, the folded neural network 600 includes a sequence of updating neural network blocks 610, wherein each updating block 610 is configured to update the current single embedding 606 (i.e., generate an updated single embedding 612) and update the current structure parameter 608 (i.e., generate an updated structure parameter 614). In addition to the updating blocks, the folded neural network 600 may include other neural network layers or blocks, for example, which may be interleaved with the updating blocks.

[0222] Each update block 610 may include: (i) a geometric attention block 616 and (ii) a folding block 618, each of which will be described in more detail below.

[0223] The geometric attention block 616 updates the current single embedding using a “geometric” self-attention operation that explicitly infers the 3-D geometry of the amino acids in the protein structure, i.e., defined by the structural parameters. More specifically, to update a given single embedding, the geometric attention block 616 determines a corresponding attention weight between the given single embedding and each of one or more selected single embeddings, where the attention weight depends on the current single embedding, the current structural parameters, and the pair of embeddings. The geometric attention block 616 then updates the given single embedding using (i) the attention weights, (ii) the selected single embeddings, and (iii) the current structural parameters.

[0224] To determine the attention weights, the geometric attention block 616 processes each current individual embedding to generate corresponding "symbolic query" embeddings, "symbolic keyword" embeddings, and "symbolic value" embeddings. For example, the geometric attention block 616 can be a single embedding h corresponding to the i-th amino acid. i Generate symbol query embedding q i Symbolic keyword embedding k i and symbolic value embedding v i As shown below:

[0225] q i =Linear(h i (10)

[0226] k i =Linear(h i (11)

[0227] v i =Linear(h i (12)

[0228] Linear(·) refers to a linear layer with independently learned parameter values.

[0229] The geometric attention block 616 further processes each current individual embedding to generate corresponding "geometric query" embeddings, "geometric keyword" embeddings, and "geometric value" embeddings. The geometric query, geometric keyword, and geometric value embeddings for each individual embedding are initially generated as 3-D points in the local reference frame of the corresponding amino acid, then rotated and transformed to the global reference frame using the amino acid's structural parameters. For example, the geometric attention block 616 can be a single embedding h corresponding to the i-th amino acid. i Generate geometric query embeddings Geometric Keyword Embedding and geometric value embedding As shown below:

[0230]

[0231]

[0232]

[0233] Linear p (·) indicates that the projection h is present. i A linear layer with independently learned parameter values ​​up to 3-D points (the superscript indicates that the quantity is at a 3-D point), R i Let represent the rotation matrix specified by the rotation parameter of the i-th amino acid, and t i This represents the position parameter of the i-th amino acid.

[0234] In order to update the single embedding h corresponding to amino acid i i Geometric attention block 616 can generate attention weights. Where N is the total number of amino acids in the protein, and a j The attention weights between amino acid i and amino acid j are as follows:

[0235]

[0236] Where q i This represents the symbolic query embedding used for amino acid i, k j This indicates the symbolic keyword embedding for amino acid j, where m represents q. i and k j The dimension, α represents the learning parameters. This represents the geometric query embedding used for amino acid i. Let |·|2 represent the geometric keyword embedding for amino acid j, where |·|2 is the L2 norm, and b i，jIt is the pair embedding 116 corresponding to the amino acid pair including amino acid i and amino acid j, and w is the learned weight vector (or some other learned projection operation).

[0237] Typically, the pair embeddings of amino acid pairs implicitly encode information related to the relationship between the amino acids in that pair, such as the distance between the amino acids in the pair. By determining the attention weights between amino acids i and j in part based on the pair embeddings of amino acids i and j, the folding neural network 600 enriches the attention weights with information from the pair embeddings, thereby improving the accuracy of the predicted folded structures.

[0238] In some implementations, the geometric attention block 616 generates multiple sets of geometric query embeddings, geometric keyword embeddings, and geometric value embeddings, and uses the set of generated geometric embeddings to determine the attention weights.

[0239] In generating a single embedding h corresponding to amino acid i i After the attention weights, the geometric attention block 616 uses the attention weights to update the individual embedding h. i Specifically, geometric attention block 616 uses attention weights to generate "symbolic return" embeddings and "geometric return" embeddings, and then uses the symbolic return embeddings and geometric return embeddings to update individual embeddings. Geometric attention block 124 can generate a symbolic return embedding o for amino acid i. i ,For example:

[0240]

[0241] in Represents the attention weights (e.g., as defined in equation (16), and each v j This represents the symbolic value embedding of amino acid j. Geometric attention block 616 can generate a geometric return embedding for amino acid i. For example, as follows:

[0242]

[0243] In geometric return embedding In the case of 3-D points, This represents the attention weights (e.g., as defined in equation (16)). It is the inverse of the rotation matrix specified by the rotation parameter of amino acid i, and t i This is the position parameter of amino acid i. It can be understood that the geometry-returned embedding is initially generated in the global reference frame, then rotated and translated to the local reference frame of the corresponding amino acid.

[0244] Geometric attention block 616 can use the corresponding symbol to return the embedding o i(e.g., generated according to equation (17)) and geometry return embedding (For example, based on equation (18)) to update the single embedding h of amino acid i. i ,For example:

[0245]

[0246] in It is a single embedding used for updating amino acid i, |·| is a norm, such as the L2 norm, and LayerNorm(·) denotes the layer normalization operation, for example, as described in reference JLBa, JRKiros, GEHinton, “Layer Normalization”, arXiv:1607.06450 (2016).

[0247] Individual embeddings 606 of amino acids are updated using specific 3-D geometric embeddings, for example, as described in reference equations (13)-(15), such that the geometric attention block 616 is able to infer 3-D geometry while updating individual embeddings. Furthermore, each update block updates individual embeddings and structural parameters in a manner invariant to rotation and translation over the entire protein structure. For example, applying the same global rotation and translation operations to the initial structural parameters provided to the folded neural network 600 will cause the folded neural network 600 to generate a predicted structure that is globally rotated and translated in the same way, but otherwise identical. Therefore, the global rotation and translation operations applied to the initial structural parameters do not affect the accuracy of the predicted protein structure generated by the folded neural network 600 from the initial structural parameters. The rotation and translation invariance of the representation generated by the folded neural network 600 aids training, for example, because the folded neural network 600 automatically learns to generalize all rotations and translations of the protein structure.

[0248] Before being provided to fold block 618, a single embedding for updating amino acids can be further transformed by one or more additional neural network layers, such as linear neural network layers, in geometric attention block 616.

[0249] After the geometric attention block 616 updates the current single embedding 606 of the amino acid, the fold block 618 updates the current structural parameter 608 using the updated single embedding 612. For example, the fold block 618 can update the current position parameter t of amino acid i. i Updated to:

[0250]

[0251] in These are the updated position parameters; Linear(·) represents a linear neural network layer, and... This represents a single embedding of the update for amino acid i. In another example, the rotation parameter R of amino acid i... i A rotation matrix can be specified, and fold block 618 can display the current rotation parameter R. i Updated to:

[0252]

[0253]

[0254] Where w i It is a 3D vector, and Linear(·) is a linear neural network layer. It is a single embedding for updating amino acid i, 1+w i This indicates that it has a real part 1 and an imaginary part w. i The quaternion is defined by the operation of transforming the quaternion into an equivalent 3×3 rotation matrix. Updating the rotation parameters using equations (21)-(22) ensures that the updated rotation parameters define a valid rotation matrix, such as an orthogonal matrix with a determinant of 1.

[0255] The folded neural network 600 can provide updated structural parameters generated by the final update module 610 as final structural parameters 106 defining the predicted protein structure 108. The folded neural network 600 can contain any suitable number of update blocks, for example, 5 update blocks, 25 update blocks, or 125 update blocks. Optionally, each update block of the folded neural network can share a set of single parameter values ​​jointly updated during the training of the folded neural network. Sharing parameter values ​​among update blocks 610 reduces the number of trainable parameters of the folded neural network and can therefore facilitate efficient training of the folded neural network, for example, by stabilizing training and reducing the likelihood of overfitting.

[0256] During training, the training engine can train the parameters of the structure prediction system, including the parameters of the folded neural network 600, based on a structural loss that evaluates the accuracy of the final structural parameters 106, as described above. In some implementations, the training engine can further evaluate auxiliary structural losses (i.e., those that generate the final structural parameters) for one or more update blocks 61o preceding the final update block. The auxiliary structural loss of the update block evaluates the accuracy of the updated structural parameters generated by the update block.

[0257] Alternatively, during training, the training engine may apply a "stop gradient" operation to prevent gradients from backpropagating through certain neural network parameters of each update block, such as the neural network parameters used to compute the updated rotation parameters (as described in equations (21)-(22)). Applying these stop gradient operations can improve the numerical stability of gradients computed during training.

[0258] Typically, the similarity between the predicted protein structure 108 generated by the folded neural network 600 and the corresponding underlying real protein structure can be measured, for example, by a similarity metric that assigns a corresponding accuracy score to each of the multiple atoms in the predicted protein structure. For example, the similarity metric can assign an individual accuracy score to each carbon α atom in the predicted protein structure. The accuracy score of the atoms in the predicted protein structure can characterize how close the position of the atoms in the predicted protein structure is to the actual position of the atoms in the real protein structure. An example of a similarity metric that can be used to compare the predicted protein structure with the underlying real protein structure to generate the accuracy score of the atoms in the predicted protein structure is the lDDT similarity metric, see: V. Mariani et al., “lDDT: Comparing Locally Unsuperimposed Scores of Protein Structures and Models Using Distance Difference Tests”, Bioinformatics, 1 Nov 2013; 29(21)2722-2728.

[0259] The folded neural network 600 can be configured to generate a corresponding confidence estimate 650 for each of one or more atoms in the predicted protein structure 108. The confidence estimate 650 for an atom in the predicted protein structure characterizes the predicted accuracy score (e.g., 1DDT accuracy score) of the atom in the predicted protein structure, i.e., the score generated by a similarity measure comparing the predicted protein structure to the (potentially unknown) underlying true protein structure. In one example, the confidence estimate 650 for an atom in the predicted protein structure can define a discrete probability distribution over a set of intervals that form a partition of the range of possible values ​​for the accuracy score of that atom. The discrete probability distribution can associate a corresponding probability with each interval, defining the likelihood that the actual accuracy score is included in that interval. For example, the range of possible values ​​for the accuracy score could be [0, 100], and the confidence estimate 650 can define a probability distribution over the set of intervals {[0, 2), [2, 4), ..., [98, 100]}. In another example, the confidence estimate of 650 for the predicted atom in the protein structure can be a numerical value, i.e., the accuracy score for directly predicting that atom.

[0260] In some implementations, the folded neural network 600 generates a corresponding confidence estimate 650 for a specific atom (e.g., α-carbon atom) in each amino acid of the protein. The folded neural network 600 can generate confidence estimates 650 for a specific atom in an amino acid of the protein, for example, by processing a single embedding of an updated amino acid generated by the last update block in the folded neural network using one or more neural network layers (e.g., fully connected layers).

[0261] The structure prediction system can generate a corresponding confidence score for each amino acid in the protein based on a confidence estimate of 650 for the atoms in the predicted protein structure. For example, the structure prediction system can generate the confidence score for an amino acid as the expected value of the probability distribution of the possible values ​​of the accuracy score for the α-carbon atom in the amino acid.

[0262] The structure prediction system can generate a confidence score for the entire predicted structure, for example, as the average confidence score of amino acids in a protein.

[0263] During the training of the structure prediction system, the training engine can adjust the parameter values ​​of the structure prediction system by backpropagating the gradient of the auxiliary loss, which measures (i) the confidence estimate generated by the folded neural network 600 and (ii) the error between the predicted protein structure and the underlying true protein structure. This error can be, for example, the cross-entropy error.

[0264] Confidence estimates generated by structure prediction systems can be used in a variety of ways. For example, confidence estimates of atoms in a predicted protein structure can indicate which parts of the structure have been reliably estimated and are therefore suitable for further downstream processing or analysis. As another example, the confidence score for each protein can be used to rank a set of predictions of protein structures, such as predictions generated by the same structure prediction system by processing different inputs characterizing the same protein, or predictions generated by different structure prediction systems.

[0265] The position and rotation parameters specified by structural parameter 106 define the spatial positions (e.g., [x,y,z] in Cartesian coordinates) of the main chain atoms in a protein's amino acids. However, structural parameter 106 does not necessarily limit the spatial positions of the remaining atoms in the protein's amino acids, such as atoms in the side chains. Specifically, the spatial positions of the remaining atoms in an amino acid depend on the values ​​of the twist angles between bonds in the amino acid, such as the omega-angle, phi-angle, psi-angle, chi1-angle, chi2-angle, chi3-angle, and chi4-angle, as referenced. Figure 7 As shown.

[0266] Optionally, one or more update blocks 610 of the folded neural network 600 can generate outputs that define the corresponding predicted spatial location of each atom in each amino acid of the protein. To generate the predicted spatial location of the atom in the amino acid, the update block can use one or more neural network layers to process the updated individual embeddings of the amino acid to generate predicted values ​​of the twist angles of the bonds between atoms in the amino acid. The neural network layers can be, for example, fully connected neural network layers with residual connections embedded. Each twist angle can be represented, for example, as a 2-D vector.

[0267] The update block can determine the spatial positions of atoms in an amino acid based on: (i) the torsion angle value of the amino acid, and (ii) the updated structural parameters of the amino acid (e.g., position and rotation parameters). For example, the update block can process the torsion angle according to a predefined function to generate the spatial positions of atoms in the amino acid in the local reference frame of the amino acid. By rotating and translating the spatial positions of atoms according to the updated structural parameters of the amino acid, the update block can generate the spatial positions of atoms in the amino acid in the global reference frame (i.e., the reference frame common to all amino acids in the protein). For example, the update block can determine the spatial positions of atoms in the global reference frame by applying a rotation operation defined by the updated rotation parameters to the spatial positions of atoms in the local reference frame to generate rotated spatial positions, and then applying a translation operation defined by the updated position parameters to the rotated spatial positions.

[0268] In some implementations, the folded neural network 600 outputs the predicted spatial locations of atoms in the amino acids of the protein generated by the final update block, either as an alternative to or in combination with the final structure parameters.

[0269] refer to Figure 6 The folded neural network 600 described herein is characterized as receiving input based on MSA representation 114 and embeddings 116 generated by an embedding neural network, for example, as referenced. Figure 2 As described. However, in general, the inputs to a folded neural network (e.g., a single embedding 602 and a pair of embeddings 116) can be generated using any suitable technique. Furthermore, aspects of the operations performed by a folded neural network (e.g., predicting the spatial location of atoms in each amino acid of a protein) can be performed by other folded neural networks, for example, with different architectures that receive different inputs.

[0270] Figure 7 This describes the torsion angles between bonds in amino acids, such as ω angle, φ angle, ψ angle, χ1 angle, χ2 angle, χ3 angle, χ4 angle, and χ5 angle.

[0271] Figure 8This is a schematic diagram of an unfolded protein and a folded protein. An unfolded protein is a random coil of amino acids. An unfolded protein undergoes protein folding and folds into a 3D conformation. Protein structures typically include stable local folding patterns, such as α-helices (as shown in 802) and β-sheets.

[0272] Figure 9 This is a flowchart of another example process 900 for predicting the structure of a protein comprising one or more chains, where each chain is specified as an amino acid sequence. For convenience, process 900 will be described as being performed by a system of one or more computers located in one or more locations. For example, a protein structure prediction system appropriately programmed according to this specification, such as... Figure 1A The protein structure prediction system 100 can execute process 900.

[0273] The system obtains an initial MSA representation (902) representing the corresponding multiple sequence alignment (MSA) for each strand in the protein.

[0274] The system obtains the corresponding initial pairing of each amino acid pair in the protein (904).

[0275] The system uses an embedded neural network to process inputs including an initial MSA representation and initial pair embeddings to generate an output including a final MSA representation and the corresponding final pair embeddings for each pair of amino acids in the protein.

[0276] The embedding neural network comprises a series of update blocks. Each update block has a set of corresponding update block parameters and is configured to receive the current MSA representation and the corresponding current pair embedding for each pair of amino acids in the protein. Each update block: (i) updates the current MSA representation based on the current pair embedding, and (ii) updates the current pair embedding based on the updated MSA representation (906).

[0277] The system is based on using the final MSA representation, the final embedding, or both to determine the predicted structure of the protein (908).

[0278] Figure 10 An example procedure 1000 for generating the MSA representation 1010 of the amino acid chain in a protein is shown. Reference Figure 1A The protein structure prediction system 100 described can implement the operation of process 1000.

[0279] In order to generate an MSA representation 1010 of the amino acid chain in a protein, system 100 obtains an MSA 1002 of the protein, which may include, for example, thousands of MSA sequences.

[0280] System 100 divides the set of MSA sequences into a set of "core" MSA sequences 1004 and a set of "extra" MSA sequences 1006. The set of core MSA sequences can be smaller than the set of extra MSA sequences 1006 (e.g., an order of magnitude smaller). System 100 can divide the set of MSA sequences into core MSA sequences 1004 and extra MSA sequences 1006, for example, by randomly selecting a predetermined number of MSA sequences as core MSA sequences and identifying the remaining MSA sequences as extra MSA sequences 1006.

[0281] For each additional MSA sequence 1006, system 100 can determine a corresponding similarity metric (e.g., based on Hamming distance) between the additional MSA sequence and each core MSA sequence 1004. System 100 can then associate each additional MSA sequence 1006 with the corresponding core MSA sequence 1004 that is most similar to the additional MSA sequence 1006 (i.e., according to the similarity metric). The set 1006 of additional MSA sequences associated with the core MSA sequence 1004 can be referred to as an "MSA sequence cluster" 1008. That is, system 100 determines a corresponding MSA sequence cluster 1008 corresponding to each core MSA sequence 1004, wherein the MSA sequence cluster 1008 corresponding to the core MSA sequence 1004 includes the set 1006 of additional MSA sequences that are most similar to the core MSA sequence 1004.

[0282] System 100 can generate an MSA representation of the amino acid chain in a protein based on a core MSA sequence and an MSA sequence cluster 1008. The MSA representation 1010 can be represented by an M×N embedding array, where M is the number of core MSA sequences (i.e., such that each core MSA sequence is associated with a corresponding row of the MSA representation), and N is the number of amino acids in the amino acid chain. The embeddings in the MSA representation can be indexed by (i, j) ∈ {(i, j): i = 1, ..., M, j = 1, ..., N}.

[0283] To generate the embedding at position (i, j) in the MSA representation 1010, system 100 can obtain the embedding (e.g., a one-hot embedding) that defines the amino acid identifier at position j in the core MSA sequence i. System 100 can also determine the probability distribution over the set of possible amino acids based on the relative frequency of each possible amino acid at position j in the additional MSA sequence 1006 within the MSA sequence cluster 1008 corresponding to the core MSA sequence i. System 100 can then determine the embedding at position (i, j) in the MSA representation by combining (e.g., linking) the embedding of the amino acid identifier at position j in the core MSA sequence i, and (ii) the probability distribution of the possible amino acids corresponding to position j in the core MSA sequence i.

[0284] In some cases, the (basic fact) protein structure of one or more core MSA sequences may be known. Specifically, for one or more core MSA sequences, the values ​​of the torsion angles (e.g., ω, φ, ψ, etc.) between bonds in the amino acids of the core MSA sequence may be known. If the torsion angle value of an amino acid in core MSA sequence i is known, system 100 can generate an embedding at position (i, j) in the MSA representation based at least in part on the torsion angle value of amino acid j in core MSA sequence i. For example, the system can use one or more neural network layers to generate the embedding of the torsion angle value and concatenate the embedding of the torsion angle value to the embedding at position (i, j) in the MSA representation.

[0285] Figure 11 An example process 1100 is shown for generating (initializing) the corresponding pair embedding 112 for each pair of amino acids in a protein. (Reference) Figure 1A The protein structure prediction system 100 described can implement the operation of process 1100.

[0286] System 100 can use the MSA representation 1102 of a protein to generate pairs of embeddings 112. (Reference) Figures 1A-1C and Figure 10 The generation of the MSA representation of the protein is described in more detail. System 100 can generate the MSA representation of the protein based on the corresponding MSA representation of each amino acid chain in the protein 1102. In order to generate the MSA representation of the amino acid chain in the protein, system 100 can use a method compared to that used for generating the reference. Figures 1A-1C The described MSA representation 110 contains more MSA sequences (e.g., orders of magnitude). Therefore, the MSA representation 1102 of the system 100 used to generate the embedding 112 can have a greater number of MSA sequences than the reference. Figures 1A-1C The described MSA represents 110 or more rows (e.g., orders of magnitude). In some implementations, system 100 can use references... Figure 10 The additional MSA sequence 1006 described is used to generate the MSA representation 1102.

[0287] After generating MSA representation 1102, system 100 processes MSA representation 1102 to generate embedding 1104 from MSA representation 1102, for example, by applying an outer product mean operation to MSA representation 1102 and identifying embedding 1104 as the result of the outer product mean operation.

[0288] System 100 uses an embedded neural network 1106 to process the MSA representation 1102 and the pair embedding 1104. The embedded neural network 1106 can update the MSA representation 1102 and the pair embedding 1104 by sharing information between them. More specifically, the embedded neural network 1106 can alternate between updating the MSA representation 1102 based on the pair embedding 1104 and updating the pair embedding 1104 based on the MSA representation 1102.

[0289] Embedded neural network 1106 can have reference-based Figures 2-5 The architecture described is an embedded neural network architecture that updates the embedding 1104 and MSA representation 1102 using row-by-row and column-by-column self-attention blocks.

[0290] In some implementations, the embedding neural network 1106 can use a column-wise "global" self-attention operation to update the embeddings in each column of the MSA representation 1102. More specifically, the embedding neural network 1106 can provide the embeddings in each column of the MSA representation to the column-wise global self-attention block to generate updated embeddings for each column of the current MSA representation. To achieve global column-wise self-attention, the self-attention block can generate a corresponding query embedding for each embedding in the column, and then average the query embeddings to generate a single "global" query embedding for that column. The column-wise self-attention block then performs the self-attention operation using the single global query embedding, which reduces the complexity of the self-attention operation from quadratic (i.e., the number of embeddings per column) to linear. Using a global self-attention operation reduces the computational complexity of the column-wise self-attention operation, enabling it to be performed on columns of the MSA representation 1102 that include a large number (e.g., thousands) of embeddings.

[0291] After updating the pair embedding 1104 and MSA representation 1102 using the embedding neural network 1106, the system 100 can identify the pair embedding 112 as the updated pair embedding generated by the embedding neural network 1106. The system 100 can discard the updated MSA representation generated by the embedding neural network 1106, or use it in any appropriate manner.

[0292] As part of generating pair embeddings 112, system 100 may include relative position encoding information in the corresponding pair embeddings for each pair of amino acids in the protein. The system may include the relative position encoding information of amino acid pairs included in the same amino acid chain in the corresponding pair embeddings by: calculating a signed difference representing the number of amino acids separating the amino acid pair in the amino acid chain; trimming the result to a predetermined interval; representing the trimmed value using a one-hot encoding vector; applying a linear transformation to the one-hot encoding vector; and summing the results of the linear transformation into the corresponding pair embeddings. The system may also include the relative position encoding information of amino acid pairs not included in the same amino acid chain in the corresponding pair embeddings by adding a default encoding vector to the corresponding pair embeddings; this default encoding vector indicates that the amino acid pair is not included in the same amino acid chain.

[0293] System 100 can also generate pairs of embeddings 112 based at least in part on a set 1110 of one or more template sequences. Each template sequence 1110 is an MSA sequence of an amino acid chain in a protein, wherein the folded structure of the template sequence 1110 is known, for example, through physical experiments.

[0294] System 100 can generate a corresponding template representation 1112 for each template sequence 1110. The template representation 1112 of template sequence 1110 includes a corresponding embedding for each pair of amino acids in the template sequence, for example, such that the template representation 1112 of a template sequence of length n (i.e., having n amino acids) can be represented as an n×n array of embeddings. System 100 generates the embedding at position (i, j) in the template representation 1112 of template sequence 1110 based on, for example: (i) the corresponding embedding representing the amino acid identifiers at positions i and j in the template sequence (e.g., uniquely heated embedding), (ii) a unit vector defined by the spatial position difference of the corresponding carbon α atom in the amino acid at positions i and j in the template sequence, i.e., in the folded structure of the template sequence, where the unit vector is calculated in the reference frame of amino acid i or amino acid j, and (iii) a discretized / lattice representation of the distance between the spatial positions of the corresponding carbon α atom in the amino acid at positions i and j in the template sequence.

[0295] System 100 can process each template representation using a sequence of one or more template update blocks 1114 to generate a corresponding updated template representation 1116 for each template sequence 1110. The template update blocks may include, for example, row-by-row self-attention blocks (e.g., updating the embeddings in each row of the template representation), column-by-column self-attention blocks (e.g., updating the embeddings in each column of the template representation), and transformation blocks (e.g., applying one or more neural network layers to each embedding in the template representation).

[0296] After generating the updated template representation 1116, system 100 uses the updated template representation 1116 to update the pair embeddings 112. For example, system 100 can use the "cross-attention" of the embeddings at the corresponding (i, j) positions of the updated template representation 1116 to update the corresponding pair embeddings 112 at each position (i, j). In the cross-attention operation of updating the pair embeddings 112 at position (i, j), a query embedding is generated from the pair embedding at position (i, j), and key and value embeddings are generated from the embeddings at the corresponding positions (i, j) of the updated template representation 1116.

[0297] Updating the embedding 112 using the template sequence 1110 enables the system 100 to enrich the embedding with information characterizing the protein structure of the evolution-related template sequence 1110, thereby enhancing the information content of the embedding 112 and improving the accuracy of the protein structure predicted using the embedding.

[0298] This specification uses the term "configuration" in conjunction with system and computer program components. For a system of one or more computers configured to perform specific operations or actions, this means that software, firmware, hardware, or a combination thereof are installed on the system, which, in operation, causes the system to perform those operations or actions. One or more computer programs configured to perform specific operations or actions means that one or more programs include instructions that, when executed by a data processing device, cause that device to perform those operations or actions.

[0299] Embodiments of the subject matter and functional operation described in this specification may be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware, including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-volatile storage medium, for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or combinations thereof. Alternatively or additionally, the program instructions may be encoded on artificially generated propagated signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus.

[0300] The term "data processing apparatus" refers to data processing hardware and includes all kinds of devices, apparatuses, and machines for processing data, such as programmable processors, computers, or multiple processors or computers. The apparatus may also be or further include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0301] A computer program can also be called or described as a program, software, software application, application, module, software module, script, or code. It can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for a computing environment. A program may, but does not need to, correspond to a file in a file system. A program may be stored as a portion of a file that holds other programs or data, for example, as one or more scripts stored in a markup language document, as a single file dedicated to the program in question, or as multiple collaborative files, for example, as a file storing one or more modules, subroutines, or code portions. A computer program can be deployed to execute on a single computer or on multiple computers located in one place or distributed across multiple locations and interconnected via a data communication network.

[0302] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines may be installed and run on the same one or more computers.

[0303] The processes and logic flows described in this specification can be executed by one or more programmable computers, which execute one or more computer programs to perform functions by manipulating input data and generating output. These processes and logic flows can also be executed by special-purpose logic circuitry (e.g., FPGA or ASIC), or by a combination of special-purpose logic circuitry and one or more programmable computers.

[0304] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for executing instructions and one or more storage devices for storing instructions and data. The central processing unit and memory may be supplemented or incorporated therein by special-purpose logic circuitry. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, to receive data from or transfer data to, or both. However, a computer does not necessarily need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name just a few.

[0305] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor storage devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CDROMs and DVD-ROMs.

[0306] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, that the user can use to provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's device in response to a request received from a web browser. Additionally, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and receiving response messages from the user.

[0307] The data processing apparatus used to implement machine learning models may also include, for example, dedicated hardware accelerator units for processing the general and computationally intensive parts of machine learning training or production, namely inference and workloads.

[0308] Machine learning models can be implemented and deployed using machine learning frameworks, such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0309] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes backend components, such as a data server, or middleware components, such as an application server, or frontend components, such as a client computer with a graphical user interface, a web browser, or an application through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0310] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship arises from computer programs running on their respective computers, and they have a client-server relationship. In some embodiments, the server sends data, such as HTML pages, to a user device, for example, to display data to a user interacting with the device acting as a client and to receive user input from that user. Data generated on the user device, such as the result of user interaction, may be received at the server from the device.

[0311] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features characteristic of particular embodiments of a particular invention. Certain features described in the context of independent embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations, and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and the claimed combination may be for sub-combinations or variations thereof.

[0312] Similarly, although the operations are described in a specific order in the accompanying drawings and recited in a specific order in the claims, this should not be construed as requiring these operations to be performed in the specific order or sequence shown, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0313] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions described in the claims can be performed in different orders and the desired results can still be obtained. As an example, the processes depicted in the figures do not necessarily require the specific order or sequential order shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method executed by one or more computers for predicting the structure of a protein comprising one or more chains, wherein each chain contains an amino acid sequence, the method comprising: In the first iteration of an iterative sequence that includes the first iteration and one or more subsequent iterations: Obtain the network input for the first iteration to characterize the protein. The network input includes the corresponding initial pairing of each amino acid pair in the protein and / or the initial MSA representation representing the corresponding multiple sequence alignment (MSA) corresponding to each strand in the protein. The network input for the first iteration is processed using a protein structure prediction neural network to generate structure parameters for the first iteration that define the initial predicted structure of the protein. The structure parameters include at least one of the following: each amino acid specifies the predicted 3-D spatial location of a specified atom in the protein structure; and each pair of amino acids in the protein characterizes the estimated distance between the corresponding amino acid pairs in the protein structure. In each subsequent iteration of the iterative sequence: Obtain the network input for subsequent iterations to characterize the protein; Features for subsequent iterations are generated from (i) structural parameters generated in previous iterations prior to subsequent iterations in the sequence, (ii) a single or intermediate output generated by the protein structure prediction neural network when generating structural parameters in the final iteration, or (iii) both, wherein the features for subsequent iterations include at least one of a set of transformations to the embeddings, a set of transformations to a single embedding, and a distance graph of the transformations; and The protein structure prediction neural network is used to process features and network inputs for subsequent iterations to generate structural parameters that define another predicted structure of the protein for subsequent iterations; and The final predicted structure of the protein is determined based on the structural parameters of the last iteration in the sequence. The features that generate subsequent iterations include one or more of the following: (i) Generate a set of transformations of the embedded pairs from the updated embedded pairs generated when generating the structural parameters in the previous iteration; (ii) Generate a set of transformations for a single embedding from one of the single embedding sets generated when generating the structure parameters in a previous iteration; (iii) Generate a transformed distance map based on the predicted 3-D spatial locations or distance maps of amino acids specified by the structural parameters of the previous iteration.

2. The method according to claim 1, wherein, The network input is the same for each iteration in the iterative sequence.

3. The method according to claim 1, wherein, The protein structure prediction neural network processes features and network inputs for subsequent iterations to generate structure parameters for subsequent iterations that define the predicted structure of the protein. Generate combined inputs based on the features and network inputs used for subsequent iterations; and The protein structure prediction neural network processes the combined inputs to generate structural parameters for subsequent iterations.

4. The method according to claim 1, wherein: The corresponding network input for each iteration includes the initial pair embeddings for each pair of amino acids in the protein. In each iteration, the structure prediction neural network is configured to repeatedly update the initial pair embeddings as it generates the structure parameters for the iteration. The generated features include an updated set of transformations of the generated embeddings, which were generated when the structural parameters were generated in the previous iteration. and The inputs for generating the combination include a set of transformations of the embeddings for subsequent iterations of the combination and the initial pair of embeddings.

5. The method according to claim 1, wherein: The corresponding network input for each iteration includes the initial MSA representation. In each iteration, the structure prediction neural network is configured to generate one or more sets of single embeddings, each set of single embeddings including the corresponding single embedding for each amino acid of the protein at the time of generating the structure parameters for the iteration. The generated features include one set of individual embeddings generated during the generation of structural parameters in previous iterations, and a set of transformations that generate individual embeddings; and The inputs for generating the combination include a set of transformations that combine individual embeddings for subsequent iterations and an initial MSA representation.

6. The method according to claim 5, wherein, For subsequent iterations, combining the set of transformations of individual embeddings with the initial MSA representation involves adding the set of transformations of individual embeddings to the first row of the initial MSA representation.

7. The method according to claim 1, wherein: The corresponding network input for each iteration includes the initial pair embeddings for each pair of amino acids in the protein. In each iteration, the structural parameters specify the predicted 3-D spatial location of the specified atom in the protein structure for each amino acid; The generated features include: A distance map is generated based on the predicted 3-D spatial locations of amino acids specified by structural parameters in previous iterations. The distance map represents the estimated distance between amino acid pairs in the protein structure for each pair of amino acids. and Generate a distance map from the distance map with a transformation having the same dimensions as the initial pair of embeddings; and The inputs for generating the combination include the distance map transformed for subsequent iterative combination and the initial pair embeddings.

8. The method according to claim 1, wherein: The corresponding network input for each iteration includes the initial pair embeddings for each pair of amino acids in the protein. In each iteration, the structure parameters specify the distance map; The generated features include: Based on the distance map specified by the structural parameters in the previous iteration, generate a distance map with the same dimensional transformation as the initial pair of embeddings; and The inputs for generating the combination include the distance map transformed for subsequent iterative combination and the initial pair embeddings.

9. The method according to claim 1, wherein: The corresponding network input for each iteration includes a set of one or more template sequences and the corresponding initial pair embedding of each amino acid pair in the protein generated using the corresponding known structure of each template sequence; and The input for generating the combination includes modifying the initial pair of embeddings for subsequent iterations by adding the protein and the structure predictions defined by the embeddings in the previous iterations to a set of one or more template sequences and the corresponding known structures.

10. A method for training a structure prediction neural network according to any of the preceding claims, the method comprising: Select the number of iterations to perform; To obtain the corresponding training network input for each iteration, each training network input represents the training protein; In the first iteration of the iteration sequence that includes the selected number of iterations: The protein structure prediction neural network is used to process the network input for the first iteration to generate structure parameters for the first iteration that define the initial predicted structure of the trained protein. When the selected number of iterations is greater than 1, in each subsequent iteration of the iteration sequence, until the selected number of iterations has been executed: Features for subsequent iterations are generated from (i) structural parameters generated in previous iterations before subsequent iterations in the sequence, (ii) a single or intermediate output generated by a protein structure prediction neural network when generating structural parameters in the last iteration, or (iii) both. and The protein structure prediction neural network is used to process features and training network inputs for subsequent iterations to generate structural parameters for subsequent iterations that define another predicted structure of the trained protein. and Determine the gradient of the objective function, which includes a structural loss term that characterizes (i) the predicted structure of the target protein specified by the structural parameters generated by the last iteration in the sequence and (ii) the similarity between the target protein structure of the training protein. and The parameter values ​​of the structure prediction neural network are updated using gradients, and the final predicted structure of the protein is determined from the structural parameters of the last iteration in the sequence.

11. The method according to claim 10, wherein, Determining the gradient involves applying the stopping gradient to the features that are processed as input by the structure prediction neural network in the final iteration.

12. The method according to claim 10, wherein, The number of iterations can be selected as follows: The number of iterations is sampled from a probability distribution that assigns a probability to each iteration, the number of iterations ranging from 1 to the total number of iterations used after the structure prediction neural network is trained.

13. The method according to any one of claims 10, further comprising: When the selected number of iterations is equal to 1: Determine the gradient of the objective function, which includes a structural loss term that characterizes (i) the initial predicted structure of the target protein defined by the structural parameters generated by the first iteration in the sequence and (ii) the similarity between the target protein structure of the training protein; and Gradients are used to update the parameter values ​​of a structural prediction neural network.

14. A system comprising: One or more computers; and One or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operation of the corresponding method according to any one of claims 1-13.

15. A non-transitory computer storage medium storing one or more instructions, which, when executed by one or more computers, cause the one or more computers to perform the operation of the corresponding method according to any one of claims 1-13.

16. A method for obtaining a ligand, wherein, The ligand is a drug or an industrial enzyme ligand, and the method includes: Perform the method according to any one of claims 1-13 to determine the predicted structure of the target protein; Evaluate the interaction between one or more candidate ligands and the predicted structure of the target protein; And select one or more candidate ligands as ligands based on the evaluation results.

17. The method according to claim 16, wherein, The target proteins include receptors or enzymes, and the ligands are agonists or antagonists of the receptors or enzymes.

18. The method according to claim 16, wherein, The ligand is a drug, and the method includes: Perform the method of any one of claims 1-13 to determine the predicted structure of each of a plurality of target proteins; Evaluate the interaction between one or more candidate ligands and the predicted structure of each target protein; and Select one or more candidate ligands as ligands to i) obtain ligands that interact with each target protein, or ii) obtain ligands that interact with only one target protein.

19. The method of claim 16, wherein, Evaluating the interaction of one of the candidate ligands involves determining the interaction score between the candidate ligand and the target protein, where the interaction score is a measure of the interaction between the candidate ligand and the target protein.

20. The method of claim 16, further comprising synthesizing a ligand.

21. The method of claim 20, further comprising testing the bioactivity of the ligand in vitro.

22. A method for obtaining polypeptide ligands, wherein, The ligand is a drug or an industrial enzyme ligand, and the method includes: For each of one or more candidate polypeptide ligands, the method of any one of claims 1-13 is performed to determine the predicted structure of the candidate polypeptide ligand; Obtain the target protein structure; Evaluate the interaction between the predicted structure of each of the one or more candidate polypeptide ligands and the target protein structure; and Based on the evaluation results, one of the one or more candidate polypeptide ligands is selected as the polypeptide ligand.

23. The method according to claim 22, wherein, The target protein includes a receptor or an enzyme, and wherein the ligand is an agonist or antagonist of the receptor or enzyme.

24. The method according to claim 22, wherein, Peptide ligands contain antibodies, and target proteins contain antibody targets.

25. The method according to claim 24, wherein, Antibody targets include viral or cancer cell proteins, and antibodies bind to antibody targets to provide therapeutic effects.

26. The method according to claim 22, wherein, Evaluating the interaction of one of the candidate ligands involves determining the interaction score between the candidate ligand and the target protein, where the interaction score is a measure of the interaction between the candidate ligand and the target protein.

27. The method of claim 22, further comprising synthesizing a ligand.

28. The method of claim 27, further comprising testing the bioactivity of the ligand in vitro.

29. A method for obtaining a diagnostic antibody marker for a disease, the method comprising: For each of one or more candidate antibodies, perform the method of any one of claims 1-13 to determine the predicted structure of the candidate antibody; Obtain the target protein structure; Evaluate the interaction between the predicted structure of each of the one or more candidate antibodies and the target protein structure; and Based on the evaluation results, one of the one or more candidate antibodies is selected as the diagnostic antibody marker.

30. The method according to claim 29, wherein, Assessing the interaction of one of the candidate antibodies involves determining the interaction score between the candidate antibody and the target protein, where the interaction score is a measure of the interaction between the candidate antibody and the target protein.

31. The method of claim 29 further comprises synthesizing antibodies.

32. The method of claim 31, further comprising testing the bioactivity of the antibody in vitro.

Images