Predicting complete protein expressions from masked protein expressions

Abstract

A method, system, and apparatus, including a computer program encoded on a computer storage medium, for unmasking a masked representation of a protein using a protein reconstruction neural network. In one aspect, the method includes receiving a masked representation of the protein, and processing the masked representation of the protein using the protein reconstruction neural network to generate respective predicted embeddings corresponding to one or more masked embeddings included in the masked representation of the protein, wherein the predicted embeddings corresponding to the masked embeddings in a representation of the amino acid sequence of the protein define predictions of the identity of amino acids at corresponding positions in the amino acid sequence, and the predicted embeddings corresponding to the masked embeddings in a representation of the structure of the protein define predictions of corresponding structural features of the protein.

Description

[Technical field]

[0001] The present specification relates to predicting complete protein representations from masked protein representations. [Background technology]

[0002] Proteins are defined by one or more sequences of amino acids, which are organic compounds that contain amino and carboxyl functional groups as well as a side chain (i.e., group of atoms) characteristic of amino acids.

[0003] Protein folding refers to the physical process by which a sequence of amino acids folds into a three-dimensional configuration. The structure of a protein defines the three-dimensional configuration of atoms in the amino acid sequence of the protein after the protein has undergone protein folding. When in a sequence linked by peptide bonds, the amino acids may be referred to as amino acid residues.

[0004] Machine learning models can be used to make predictions. Machine learning models receive inputs and generate outputs, e.g., predicted outputs, based on the received inputs. Some machine learning models are parametric models, generating outputs based on the received inputs and the values ​​of the parameters of the model. Some machine learning models are deep models that use multiple layers of the model to generate outputs related to the received inputs. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a nonlinear transformation to the received inputs to generate an output. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] A. Vaswani et al., "Attention is all you need", 21st Conference on Neural Informational Processing Systems (NIPS 2017) [Non-Patent Document 2] Jimmy Lei Ba et al., "Layer Normalization", arXiv:1607.06450 Summary of the Invention [Means for solving the problem]

[0006] This specification describes a protein reconstruction system implemented as a computer program on one or more computers in one or more locations that can unmask a masked representation of a protein using a protein reconstruction neural network. The protein reconstruction neural network is not limited to having a particular architecture, and as described below, the system can improve the accuracy of the protein representation by processing both the amino acid sequence and the structural representation of the protein together.

[0007] As used throughout this specification, the term "protein" may be understood to refer to any biological molecule defined by one or more sequences of amino acids. For example, the term protein may be understood to refer to a protein domain (i.e., a portion of an amino acid sequence that can undergo protein folding largely independently of the remainder of the amino acid sequence) or a protein complex (i.e., defined by multiple related amino acid sequences).

[0008] Throughout this specification, an embedding refers to an ordered collection of numbers, for example a vector or matrix of numbers.

[0009] According to a first aspect, there is provided a method, executed by one or more data processing devices, for unmasking a masked representation of a protein using a protein reconstruction neural network, comprising the steps of: receiving a masked representation of the protein, the masked representation of the protein comprising (i) a representation of the amino acid sequence of the protein comprising a plurality of embeddings, each corresponding to a respective position in the amino acid sequence of the protein, and (ii) a representation of the structure of the protein comprising a plurality of embeddings, each corresponding to a respective structural feature of the protein, at least one of the embeddings included in the masked representation of the protein is masked; and processing the masked representation of the protein using the protein reconstruction neural network to generate respective predicted embeddings corresponding to one or more masked embeddings included in the masked representation of the protein, wherein the predicted embeddings corresponding to the masked embeddings in the representation of the amino acid sequence of the protein define a prediction of the identity of the amino acid at the corresponding position in the amino acid sequence and the predicted embeddings corresponding to the masked embeddings in the representation of the structure of the protein define a prediction of the corresponding structural feature of the protein.

[0010] In some implementations, the method further includes updating the masked representation of the protein by replacing a suitable subset of the masked embeddings in the masked representation of the protein with corresponding predicted embeddings, and processing the updated masked representation of the protein using a protein reconstruction neural network to generate respective predicted embeddings corresponding to one or more remaining masked embeddings included in the masked representation of the protein.

[0011] In some implementations, the representation of the amino acid sequence of the protein includes one or more masked embeddings, and the method further includes processing the predicted amino acid sequence of the protein defined by replacing each masked embedding in the representation of the amino acid sequence with a corresponding predicted embedding using a protein folding neural network to generate data defining a predicted protein structure of the predicted amino acid sequence, and processing both (i) the masked representation of the protein and (ii) the predicted protein structure of the predicted amino acid sequence using a protein reconstruction neural network to generate new predicted embeddings that correspond to the one or more masked embeddings included in the masked representation of the protein.

[0012] In some implementations, each masked embedding included in the masked representation of a protein is a default embedding.

[0013] In some implementations, the default embedding includes a vector of zeros.

[0014] In some implementations, each predicted embedding that corresponds to a masked embedding in the representation of the protein's structure defines a prediction of the spatial distance between a corresponding pair of amino acids in the protein's structure.

[0015] In some implementations, at least one of the embeddings of the representation of the amino acid sequence of the protein is masked.

[0016] In some implementations, at least one of the embeddings of the representation of the protein's structure is masked.

[0017] In some implementations, the representation of the amino acids of the protein includes a plurality of single embeddings, each corresponding to a respective position in the amino acid sequence of the protein, the representation of the structure of the protein includes a plurality of pair embeddings, each corresponding to a respective pair of positions in the amino acid sequence of the protein, and the protein reconstruction neural network includes a sequence of update blocks, each update block having a respective set of update block parameters, and performs operations including receiving a current pair embedding and a current single embedding, updating the current single embedding based on the current pair embedding in accordance with values ​​of the update block parameters of the update block, and updating the current pair embedding based on the updated single embedding in accordance with values ​​of the update block parameters of the update block, and a final update block in the sequence of update blocks generates a final pair embedding and a final single embedding.

[0018] In some implementations, the protein reconstruction neural network performs further operations including, for each of one or more masked single embeddings in the representation of the amino acid sequence of the protein, generating a predicted embedding for the masked single embedding based on the corresponding final single embedding generated by the last update block.

[0019] In some implementations, the protein reconstruction neural network performs further operations including, for each of one or more masked pair embeddings in the representation of the amino acid sequence of the protein, generating a predicted embedding for the masked pair embedding based on the corresponding final pair embedding generated by the last update block.

[0020] In some implementations, the act of updating the current single embedding based on the current pair embedding includes updating the current single embedding using attention on the current single embedding, where the attention is conditioned on the current pair embedding.

[0021] In some implementations, updating the current single embedding using attention on the current single embedding includes generating a plurality of attention weights based on the current single embedding, generating a respective attention bias corresponding to each of the attention weights based on the current pair embedding, generating a plurality of biased attention weights based on the attention weights and the attention biases, and updating the current single embedding using attention on the current single embedding based on the biased attention weights.

[0022] In some implementations, updating the current pair embedding based on the updated single embedding includes updating the current pair embedding by applying a transform operation to the updated single embedding and adding a result of the transform operation to the current pair embedding.

[0023] In some implementations, the conversion operation includes a cross product operation.

[0024] In some implementations, the operation of updating the current pair embedding based on the updated single embedding further includes updating the current pair embedding using attention on the current pair embedding after adding the result of the transform operation to the current pair embedding, where the attention is conditioned on the current pair embedding.

[0025] According to another aspect, there is provided a method for obtaining a ligand, the ligand being a ligand of a drug or an industrial enzyme, the method comprising the steps of determining a predicted structure of a target protein by generating a predicted embedding defining a complete protein structural representation of the target protein, the masked representation of the protein comprising a complete representation of the amino acid sequence of the target protein, and the representation of the protein structural comprises a fully masked representation of the target protein structural; evaluating the interaction of one or more candidate ligands with the predicted structure of the target protein; and selecting one or more of the candidate ligands as the ligand depending on the result of the evaluation.

[0026] According to another aspect, there is provided a method for obtaining a ligand, the ligand being a ligand of a drug or an industrial enzyme, the method comprising the steps of: determining a predicted structure of each of a plurality of target proteins by generating a predicted embedding defining a complete protein structural representation of each target protein, wherein for each target protein, the masked representation of the protein comprises a complete representation of the amino acid sequence of the target protein and the representation of the protein's structure comprises a fully masked representation of the target protein's structure; evaluating the interaction of one or more candidate ligands with the predicted structure of each of the target proteins; and selecting one or more of the candidate ligands as ligands to either i) obtain a ligand that interacts with each of the target proteins or ii) obtain a ligand that interacts with only one of the target proteins.

[0027] In some implementations, the target protein comprises a receptor or an enzyme, and the ligand is an agonist or antagonist of the receptor or enzyme.

[0028] According to another aspect, there is provided a method for obtaining a polypeptide ligand, the ligand being a ligand of a pharmaceutical or industrial enzyme, the method comprising the steps of: determining, for each of one or more candidate polypeptide ligands, a predicted structure of the candidate polypeptide ligand by generating a predicted embedding defining a complete protein structure representation of the candidate polypeptide ligand, wherein, for each of the one or more candidate polypeptide ligands, the masked representation of the protein comprises a complete representation of the amino acid sequence of the candidate polypeptide ligand and the representation of the protein structure comprises a fully masked representation of the structure of the candidate polypeptide ligand; obtaining a target protein structure of a target protein; evaluating an interaction between the predicted structure of each of the one or more candidate polypeptide ligands and the target protein structure; and depending on the result of the evaluation, selecting one of the one or more of the candidate polypeptide ligands as the polypeptide ligand.

[0029] In some implementations, the target protein comprises a receptor or enzyme and the ligand is an agonist or antagonist of the receptor or enzyme, or the polypeptide ligand comprises an antibody and the target protein comprises an antigen and the antibody binds to the antigen to provide a therapeutic effect.

[0030] According to another aspect, there is provided a method for obtaining an antibody to an antigen, comprising determining a predicted structure and amino acid sequence of the antibody by generating i) a complete amino acid sequence representation of the antibody, and ii) a predicted embedding that defines a complete protein structure representation of the antibody, wherein the masked representation of the protein includes a representation of a paratope of the antibody that binds to the antigen, the masked representation including i) a partially masked representation of the antibody's amino acid sequence, and ii) a partially masked representation of the antibody's structure.

[0031] In some implementations, the antigen comprises a viral protein or a cancer cell protein.

[0032] According to another aspect, there is provided a method for obtaining a diagnostic antibody marker for a disease, comprising the steps of: determining, for each of one or more candidate antibodies, a predicted structure of the candidate antibody by generating a predicted embedding defining a complete protein structural representation of the candidate antibody, wherein for each of the one or more candidate antibodies, the masked representation of the protein comprises a complete representation of the amino acid sequence of the candidate antibody and the representation of the protein structure comprises a fully masked representation of the candidate antibody's structure; obtaining a target protein structure of a target protein; evaluating the interaction between the predicted structure and the target protein structure of each of the one or more candidate antibodies; and depending on the result of the evaluation, selecting one of the one or more of the candidate antibodies as a diagnostic antibody marker.

[0033] According to another aspect, there is provided a method for designing mutated proteins with optimized properties, comprising the steps of: i) obtaining a complete representation of an amino acid sequence of a known protein; and ii) a complete protein structural representation of the known protein; determining, for each of one or more candidate mutated proteins, a predicted amino acid sequence of the candidate mutated protein by generating a predicted embedding defining the complete amino acid sequence of the candidate mutated protein, wherein generating the predicted embedding comprises: generating a partially masked representation of the candidate mutated protein by masking one or more of the embeddings in the representation of the amino acid sequence of the candidate mutated protein; generating, for each masked amino acid embedding, a respective score distribution defining a score for each amino acid type of a set of possible amino acid types; generating the predicted embedding by sampling, for each masked amino acid, each type according to the score distribution for the amino acids; and selecting one of the candidate mutated proteins as the mutated protein by identifying from among the candidate mutated proteins a predicted amino acid sequence that predicts optimal properties for the candidate mutated protein.

[0034] In some implementations, the method further comprises synthesizing the mutated protein.

[0035] According to another aspect, there is provided a method for identifying the presence of a protein mis-folding disease, the method comprising the steps of: determining a predicted structure of a protein by generating a predicted embedding defining a complete protein structural representation of the protein, wherein the masked representation of the protein comprises a complete representation of the amino acid sequence of the protein, and wherein the representation of the structure of the protein comprises a fully masked representation of the structure of the protein; obtaining a structure of a version of the protein obtained from a human or animal body; comparing the predicted structure of the protein with the structure of a version of the protein obtained from the human or animal body; and identifying the presence of a protein mis-folding disease depending on a result of the comparison.

[0036] According to another aspect, there is provided a method for obtaining an amino acid sequence of a protein, the method comprising the steps of receiving a structure of the protein, the structure of the protein being obtained by experiment, determining a complete protein structure representation of the protein from the structure, and determining a predicted amino acid sequence of the protein by generating a predicted embedding that defines a complete amino acid sequence representation of the protein, wherein the masked representation of the protein comprises a complete representation of the structure of the protein, the representation of the amino acid sequence of the protein comprises a completely masked representation of the amino acid sequence of the protein, and the predicted amino acid sequence of the protein is the obtained amino acid sequence of the protein.

[0037] According to another aspect, a system is provided that includes one or more computers and one or more storage devices communicatively coupled to the one or more computers, the one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of the methods described herein.

[0038] According to another aspect, one or more non-transitory computer storage media are provided that store instructions that, when executed by one or more computers, cause the one or more computers to perform operations of the methods described herein.

[0039] Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.

[0040] In general, protein folding (i.e., predicting a protein structure from an amino acid sequence) and protein design (i.e., predicting an amino acid sequence from a protein structure) are closely related tasks. The system described herein can be trained to perform both of these tasks in parallel. In particular, the system can be provided with a masked representation of a protein, including a representation of the protein's amino acid sequence and a representation of the protein's structure, one or both of which are at least partially masked. The system then processes the masked protein representation to generate a "complete" (i.e., unmasked) representation of the protein, i.e., a complete representation that includes predictions of the masked portions of the amino acid sequence representation and the protein structure representation. As a result of being trained to perform both protein folding and protein design in parallel, the system can achieve higher prediction accuracy for each of these tasks than if the system had been trained to perform either of these tasks independently of the other. The system can achieve acceptable prediction accuracy for protein folding tasks, protein design tasks, or both, in some cases while consuming less computational resources (e.g., memory and computing power) than other systems that perform either of these tasks independently of the other.

[0041] The systems described herein can unmask a masked protein representation by progressively replacing masked embeddings in the masked protein representation with corresponding predicted embeddings over a series of iterations. Replacing masked embeddings in the masked protein representation with corresponding predicted embeddings over a series of iterations, e.g., rather than all at once in one iteration, can allow the system to progressively accumulate contextual information, thereby unmasking the masked protein representation with greater accuracy.

[0042] The system described herein can predict, in each of one or more iterations, a protein structure of a current amino acid sequence defined by replacing each masked embedding in the amino acid sequence representation with a corresponding predicted embedding generated in the current iteration. The system can then process both the predicted protein structure and the masked protein representation in the next iteration, which can enable the system to adaptively correct errors in the predicted embeddings that cause the corresponding predicted protein structure to deviate from the target protein structure representation. In particular, in each iteration after the first iteration, the system can generate new (and potentially corrected) predicted embeddings in the iteration based at least in part on the predicted protein structure generated in the previous iteration.

[0043] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief description of the drawings]

[0044] [Figure 1] FIG. 1 illustrates an exemplary protein remodeling system. [Diagram 2]FIG. 1 illustrates an exemplary architecture of a protein reconstructing neural network. [Diagram 3] FIG. 1 illustrates an exemplary architecture of the update block of a protein reconstruction neural network. [Figure 4] FIG. 2 illustrates an example architecture of a single embedded update block. [Diagram 5] FIG. 2 illustrates an example architecture of a paired embedded update block. [Figure 6] 1 is a flow diagram of an exemplary process for unmasking a masked representation of a protein using a protein reconstruction neural network. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0045] Like reference numbers and designations in the various drawings indicate like elements.

[0046] 1 illustrates an exemplary protein remodeling system 100. Protein remodeling system 100 is an example of a system in which the systems, components, and techniques described below are implemented, implemented as a computer program on one or more computers at one or more locations.

[0047] The system 100 is configured to receive a masked representation 102 of a protein that includes (i) a representation of the amino acid sequence of the protein (i.e., amino acid sequence representation 104), and (ii) a representation of the structure of the protein (i.e., protein structure representation 106). The amino acid sequence representation 104 and the protein structure representation 106 are each represented by a respective set of embeddings, and at least one of the embeddings of the amino acid sequence representation 104, the protein structure representation 106, or both, is masked. An embedding may be referred to as "masked", for example, if the embedding is a default (e.g., predefined) embedding, e.g., an embedding represented as a vector of zeros.

[0048] The amino acid sequence representation 104 may include a respective embedding corresponding to each position of the amino acid sequence of the protein. Each embedding of the amino acid sequence representation 104 that is not a masked embedding may represent the amino acid at the corresponding position in the amino acid sequence, for example, by a one-hot embedding that identifies an amino acid from a set of possible amino acids. The set of possible amino acids may include, for example, alanine, arginine, asparagine, etc., and the total number of amino acids in the set of possible amino acids may be, for example, 20.

[0049] The protein structure representation 106 may include a respective embedding corresponding to each "structural feature" of a set of structural features that characterize the protein structure.

[0050] For example, each structural feature of a set of structural features characterizing a protein structure can define a spatial distance (e.g., measured in angstroms) that separates specified atoms (e.g., alpha carbon atoms) of a corresponding pair of amino acids in the protein structure. In this example, the embedding representing the spatial distance between the pair of amino acids in the protein structure can be a one-hot embedding that identifies the spatial distance between the pair of amino acids as falling within one distance interval from a set of possible distance intervals. The set of possible distance intervals can be, for example, 0 to 2 angstroms, 2 to 4 angstroms, 4 to 6 angstroms, etc.

[0051] As another example, each structural feature of a set of structural features characterizing a protein structure may define a spatial location of an atom (e.g., an alpha carbon atom) of a corresponding amino acid in the protein structure. Each embedding of a protein structure representation that is not a masked embedding may represent the spatial location of an atom of a corresponding amino acid in the protein structure, for example, as an xyz coordinate in a predefined Cartesian coordinate system. As a further example, a structural feature may define a torsion angle of a backbone atom of an amino acid residue in a protein.

[0052] Certain embeddings in the amino acid sequence representation 104 and the protein structure representation 106 may be masked, for example, because those embeddings represent information about the protein that is not known. For example, if the amino acid sequence of a protein is known, but the structure of the protein is not known, the amino acid sequence representation may be "complete" (i.e., none of the embeddings are masked), while all of the embeddings in the protein structure representation may be masked. As another example, if the structure of a protein is known, but the amino acid sequence of the protein is not known, the protein structure representation may be complete, while all of the embeddings in the amino acid sequence representation may be masked. As another example, if both the amino acid sequence of a protein and the structure of the protein are only partially known, both the amino acid sequence representation and the protein structure representation may include some embeddings that are masked and other embeddings that are not masked.

[0053] The system 100 processes the amino acid sequence representation 104 and the protein structure representation 106 using a protein reconstruction neural network 200 to generate respective predicted embeddings corresponding to respective masked embeddings in the masked protein representation 102. The predicted embeddings 108 corresponding to the masked embeddings in the amino acid sequence representation 104 may define a prediction of the identity of the amino acid at the corresponding position in the amino acid sequence of the protein. The predicted embeddings 108 corresponding to the masked embeddings in the protein structure representation 106 may define a prediction of the corresponding structural feature of the protein, e.g., the spatial distance between each atom of a corresponding pair of amino acids in the protein. Generating the predicted embeddings 108 may be understood as reconstructing the masked embeddings in the masked protein representation 102 using context information available from the unmasked embeddings in the masked protein representation 102.

[0054] The protein reconstruction neural network 200 can have any suitable neural network architecture that enables the protein reconstruction neural network 200 to perform its described functions, including any suitable neural network layers (e.g., fully connected layers, convolutional layers, attention layers, etc.) arranged in any suitable configuration (e.g., as a consecutive sequence of layers). An exemplary architecture of the protein reconstruction neural network 200 is described in more detail with reference to Figures 2-5. However, existing protein reconstruction neural networks may be adapted to use the described techniques, i.e., to process both amino acid sequence and protein structure representations together, e.g., iteratively.

[0055] Replacing the masked embeddings in the masked protein representation 102 with the corresponding predicted embeddings 108 results in a complete protein representation 110, i.e., a complete protein representation 110 in which none of the embeddings in the complete protein representation 110 are masked. That is, a complete protein representation can define a complete reconstruction of the amino acid sequence of the protein (i.e., the identity of the amino acid at each position in the amino acid sequence is specified and unmasked) and a complete reconstruction of the protein structure (i.e., each structural feature of a set of structural features that characterize the protein structure is specified and unmasked). The system 100 can then provide as output the complete protein representation 110 or a portion thereof (e.g., only the complete amino acid sequence representation or only the complete protein structure representation).

[0056] In some implementations, the system 100 progressively replaces masked embeddings in the masked protein representation 102 by corresponding predicted embeddings 108 over a series of iterations. More specifically, in each iteration, the system 100 processes the current masked protein representation 102 using the protein reconstruction neural network 200 to generate predicted embeddings 108, and updates the current masked protein representation 102 by replacing one or more of the remaining masked embeddings by the corresponding predicted embeddings 108. The number of remaining masked embeddings in the masked protein representation 102 is reduced in each iteration, and in the final iteration, the system 100 replaces all remaining masked embeddings in the masked protein representation 102 by the corresponding predicted embeddings 108 generated in the final iteration.

[0057] The system 100 can determine, at each iteration, which masked embeddings in the masked protein representation 102 should be replaced by the corresponding predicted embeddings 108 in any of a variety of ways, some examples of which are provided below.

[0058] In one example, at each iteration, the system 100 may randomly select a predefined percentage (e.g., 15%) of the remaining masked embeddings in the masked protein representation 102 to be replaced by corresponding predicted embeddings 108. When the system 100 determines that less than a predefined threshold number of masked embeddings remain in the masked protein representation 102, it may replace all remaining masked embeddings with the corresponding predicted embeddings 108 and terminate the iterative process.

[0059] In another example, at each iteration, the system 100 may determine which masked embeddings in the amino acid sequence representation 104 to replace with the corresponding predicted embeddings 108 based on the alignment of the embeddings of the amino acid sequence representation 104 into the sequence. More specifically, the embeddings of the amino acid sequence representation 104 may be associated with an alignment into a one-dimensional (1D) sequence, where an embedding at position i in the sequence corresponds to the amino acid at position i in the amino acid sequence of the protein. At each iteration, the system 100 may determine that a masked embedding should be replaced with the corresponding predicted embedding 108 if the masked embedding of the amino acid sequence representation 104 is adjacent to an unmasked embedding in the 1D alignment of the embeddings of the amino acid sequence representation.

[0060] In another example, at each iteration, the system 100 may determine which masked embeddings in the protein structure representation 106 to replace with the corresponding predicted embeddings 108 based on the arrangement of the embeddings of the protein structure representation into an array. More specifically, the embeddings of the protein structure representation 106 may be associated with an arrangement into a two-dimensional (2D) array, where an embedding at a position (i,j) in the array corresponds to a pair of amino acids at positions i and j in the amino acid sequence of the protein. At each iteration, the system 100 may determine that a masked embedding of the protein structure representation 106 should be replaced with a corresponding predicted embedding 108 if the masked embedding is adjacent to an unmasked embedding in the 2D array of the embeddings of the protein structure representation 106. An embedding may be understood to be "adjacent" to another embedding in the 2D array of the embeddings, for example, if the embedding and the other embedding are adjacent in the same row of the 2D array or adjacent in the same column of the 2D array.

[0061] Replacing the masked embeddings in the masked protein representation 102 with the corresponding predicted embeddings 108 over a series of iterations (e.g., rather than all at once) can enable the system 100 to progressively accumulate contextual information, thereby generating more accurate predicted embeddings 108.

[0062] In some implementations, the amino acid sequence representation 104 includes at least one masked embedding, and in each of one or more iterations, the system 100 generates a respective predicted embedding 108 corresponding to the respective masked embedding in the amino acid sequence representation 104. For convenience, the amino acid sequence defined by replacing each masked embedding in the amino acid sequence representation 104 with the corresponding predicted embedding 108 generated in the current iteration is referred to as the "current amino acid sequence." In each iteration, the system 100 can process the current amino acid sequence using a protein folding neural network to generate a predicted structure of a protein having the current amino acid sequence. The system can then provide the predicted protein structure as an additional input to the protein reconstruction neural network 200 in the next iteration.

[0063] To provide the predicted protein structure as an additional input to the protein reconstruction neural network 200 in the next iteration, the system 100 can generate a predicted protein structure representation. The predicted protein structure representation can include a respective embedding corresponding to each structural feature of the set of structural features characterizing the predicted protein structure. For example, the predicted protein structure representation can include a respective embedding representing a spatial distance between a pair of amino acids in the predicted protein structure, as described above. The protein reconstruction neural network 200 can process the additional input defined by the predicted protein structure representation in any suitable manner. For example, the protein reconstruction neural network 200 can sum, average, or otherwise combine the predicted protein structure representation with the protein structure representation 106. The protein reconstruction neural network 200 can then process the resulting combined protein structure and amino acid sequence representation 104 according to parameter values ​​of the protein reconstruction neural network 200 to generate a predicted embedding 108 for the next iteration, as described above.

[0064] The protein folding neural network can have any suitable neural network architecture that enables the protein folding neural network to perform its described function, i.e., processing inputs including representations of amino acid sequences to generate predicted structures having the amino acid sequences. In particular, the protein folding neural network can include any suitable neural network layers (e.g., fully connected layers, convolutional layers, attention layers, etc.) arranged in any suitable configuration (e.g., as a sequence of layers).

[0065] Providing the protein reconstruction neural network 200 with a predicted protein structure corresponding to the current amino acid sequence can enable the system 100 to implicitly compare the predicted protein structure with the protein structure representation 106. This comparison can enable the protein reconstruction neural network 200 to correct potential errors in the current amino acid sequence that cause the corresponding predicted protein structure to deviate from the protein structure representation 106, thereby improving the performance (e.g., prediction accuracy) of the system 100.

[0066] As an alternative or in combination with progressively replacing masked embeddings in the masked protein representation at each iteration, the system 100 can generate a predicted protein structure corresponding to the current amino acid sequence at each iteration and provide the predicted protein structure to the reconstructed neural network at the next iteration. That is, at each iteration, the system can either or both: (i) process the (tentative) amino acid sequence defined by replacing each masked embedding in the amino acid sequence representation 104 with a corresponding predicted embedding 108 generated at the iteration to generate a corresponding predicted protein structure that is provided to the reconstructed neural network at the next iteration; and (ii) use one or more predicted embeddings generated at the iteration to replace corresponding masked embeddings in the masked protein representation (e.g., masked embeddings in the amino acid sequence representation 104, the protein structure representation 106, or both).

[0067] Some examples of possible applications of the system 100 are now described in more detail.

[0068] In one example, system 100 can be used to predict a protein structure corresponding to a known amino acid sequence by processing the complete amino acid sequence representation and a fully masked protein structure representation to "unmask" the protein structure representation. Unmasking the protein structure representation refers to generating a predicted embedding that defines the complete protein structure representation.

[0069] In another example, system 100 can be used to predict amino acid sequences corresponding to known protein structures by processing the complete protein structure representation and a fully masked amino acid sequence representation to "unmask" the amino acid sequence representation. Unmasking the amino acid sequence representation refers to generating a predicted embedding that defines the complete amino acid sequence representation. The known protein structure may be obtained by experiments using conventional physical techniques, such as X-ray crystallography, magnetic resonance techniques, or cryo-electron microscopy (cryo-EM).

[0070] In another example, the system 100 can be used to generate a complete protein representation of a protein with a partially known amino acid sequence and a partially known protein structure. In particular, the system can process a partially masked amino acid sequence representation representing the partially known amino acid sequence and a partially masked protein structure representation representing the partially known protein structure to unmask the amino acid sequence representation and the protein structure representation. Generating a complete protein representation from a partially masked amino acid sequence and a partially masked protein structure can be performed, for example, to design a complete antibody starting from a paratope known to selectively bind to a particular antigen, for example, to provide a therapeutic effect, among other things. For example, the antigen may include a viral protein or a cancer cell protein. The designed antibody can then be synthesized.

[0071] To design a complete antibody starting from a known paratope, system 100 can be used to process a partially masked representation of the antibody's amino acid sequence and a partially masked representation of the antibody's structure to generate a complete representation of the antibody. The representation of the antibody's amino acid sequence can include a one-hot embedding representing the known amino acids of the paratope and a masked amino acid embedding for each other amino acid in the antibody. The representation of the antibody's protein structure can include an embedding representing the structure of the paratope and a masked embedding representing the structure of the remainder of the antibody (i.e., outside the paratope). The complete representation of the antibody can define the respective types of each amino acid in the antibody and the structure of the antibody.

[0072] In another example, system 100 can be used to generate a complete protein representation of a protein with (i) a partially known amino acid sequence and a completely known protein structure, or (ii) a completely known amino acid sequence and a partially known protein structure. For example, the system can process a partially masked amino acid sequence representation and a complete protein structure representation to unmask the amino acid sequence representation.

[0073] Generating a complete protein representation from a partially masked amino acid sequence of a protein and the complete structure of the protein can be performed, for example, to optimize a particular property of the protein, such as binding affinity, solubility, stability, aggregation propensity, or any other suitable property. For example, starting from a protein with a known amino acid sequence and a known protein structure, a masked representation of the amino acid sequence of the protein can be generated, i.e., a masked representation in which the identity of one or more amino acids in the protein is masked. The system 100 can process the masked amino acid sequence representation and the complete structure representation of the protein to generate, for each masked amino acid, a respective score distribution that defines a score for each amino acid type of a set of possible amino acid types. An example of generating a score distribution over amino acid types is described later. The system 100 can then generate a number of "candidate" proteins, the amino acid sequence of each candidate protein being determined by sampling each type of each masked amino acid according to the score distribution for the amino acid. A value for each property (e.g., solubility, stability, binding affinity, or aggregation tendency) can be predicted for each candidate protein, and the candidate protein having the most desirable (e.g., highest or lowest) value for each property can be selected. The value for each property may be predicted from the amino acid sequence of the candidate protein, for example, using published techniques or available software tools. The selected candidate protein can be understood to "mutate" the original protein to optimize the desired property (e.g., solubility, stability, or binding affinity) of the protein. Thus, a mutated protein with the desired property may be synthesized by synthesizing a protein having the amino acid sequence of the selected candidate protein.

[0074] The system 100 can receive the masked protein representation 102 from a remotely located user of the protein reconstruction system 100, for example, through an interface (e.g., an application programming interface (API)) made available by the protein reconstruction system 100 over a data communications network (e.g., the Internet). After generating the complete protein representation 110, the system 100 can provide the complete protein representation 110 (or a portion thereof) to the remotely located user over the data communications network.

[0075] A training engine can train the parameters of the protein reconstruction neural network 200 on a set of training examples over multiple training iterations. Each training example can define a complete protein representation, i.e., a complete protein representation including a complete amino acid sequence representation of the protein and a complete protein structure representation.

[0076] In each training iteration, the training engine may sample one or more complete protein representations and generate a masked protein representation corresponding to each complete protein representation, e.g., by randomly masking a portion of the complete protein representation. The training engine may process each masked protein representation using the system 100 according to the current parameter values ​​of the protein reconstruction neural network (as described above) to generate a respective predicted embedding for each masked embedding of the masked protein representation. The training engine may then determine a gradient with respect to the parameters of the protein reconstruction neural network of an objective function that measures the error between (i) the predicted embedding generated by the system 100 and (ii) the corresponding embedding defined by the complete protein representation. The training engine may measure the error between the predicted embedding generated by the system 100 and the corresponding embedding from the complete protein representation, e.g., by cross-entropy loss or squared error loss. The training engine uses the gradient of the objective function to update the parameter values ​​of the protein reconstruction neural network using any suitable gradient descent optimization technique, e.g., any suitable update rule of RMSprop or Adam.

[0077] 2 illustrates an exemplary architecture of a protein reconstruction neural network 200. The protein reconstruction neural network 200 is configured to process a masked representation of a protein that includes (i) an amino acid sequence representation 104 and (ii) a protein structure representation 106, where one or more of the embeddings of the masked protein representation are masked.

[0078] The amino acid sequence representation 104 includes a respective "single" embedding corresponding to each position in the amino acid sequence of the protein. Each embedding of the amino acid sequence representation 104 that is not a masked embedding may represent the amino acid at the corresponding position in the amino acid sequence, for example, by a one-hot embedding that identifies the amino acid from a set of possible amino acids. The protein reconstruction neural network 200 may optionally apply position encoding data to each single embedding, where the position encoding data applied to the single embedding is a function of the index of the position in the amino acid sequence corresponding to the single embedding. For example, the protein reconstruction neural network 200 may apply sinusoidal positional encoding data to each single embedding, as described in connection with A. Vaswani et al., "Attention is all you need", 21st Conference on Neural Information Processing Systems (NIPS 2017).

[0079] The protein structure representation 106 includes a respective "pair" embedding corresponding to each pair of amino acids in the protein (e.g., NxN pairs). Each pair embedding that is not a masked embedding may represent the spatial distance between the corresponding pair of amino acids, for example, by a one-hot embedding that specifies the spatial distance between the pair of amino acids as falling within one distance interval from a set of possible distance intervals.

[0080] The protein reconstructing neural network 200 includes a sequence of update blocks 206-A to 206-N. Throughout this specification, a "block" refers to a portion of a neural network, e.g., a sub-network of a neural network that includes one or more neural network layers.

[0081] Each update block of the protein reconstruction neural network is configured to receive a block input including a set of single embeddings and a set of paired embeddings, and to process the block input to generate a block output including updated single embeddings and updated paired embeddings.

[0082] The protein reconstruction neural network 200 provides to a first update block (i.e., in a sequence of update blocks) the single embeddings 202 and pair embeddings 204 contained in a network input of the protein reconstruction neural network 200. The first update block processes the single embeddings 202 and pair embeddings 204 to generate updated single embeddings and updated pair embeddings.

[0083] For each update block after the first update block, the protein reconstruction neural network 200 provides the single embeddings and pair embeddings generated by the previous update block to the update block, and provides the updated single embeddings and updated pair embeddings generated by the update block to the next update block.

[0084] The protein reconstruction neural network 200 gradually enriches the information content of the single and pair embeddings 202, 204 by iteratively updating them using a sequence of update blocks 206-A to 206-N.

[0085] The final update block in the sequence of update blocks outputs a set of updated single embeddings 208 and a set of updated pair embeddings 210. Each updated single embedding 208 may include a respective "soft" score for each amino acid in the set of possible amino acids, and each updated pair embedding may include a respective "soft" score for each distance interval in the set of possible distance intervals.

[0086] The protein reconstruction neural network 200 can identify a predicted embedding 108 for a masked single embedding from the amino acid sequence representation 104 as being a one-hot embedding that represents the amino acid associated with the highest soft score by the corresponding updated single embedding 208. Similarly, the protein reconstruction neural network 200 can identify a predicted embedding 108 for a masked pair embedding from the protein structure representation 106 as being a one-hot embedding that represents the distance interval associated with the highest soft score by the corresponding updated pair embedding 210.

[0087] FIG. 3 shows an exemplary architecture of the protein reconstruction neural network 200, namely of the update block 300 described with reference to FIG.

[0088] Update block 300 receives block inputs including a current single embedding 302 and a current pair embedding 304 and processes the block inputs to generate an updated single embedding 306 and an updated pair embedding 308 .

[0089] The update blocks 300 include a single padded update block 400 and a pair padded update block 500 .

[0090] The single embedding update block 400 updates the current single embedding 302 using the current pair embedding 304, and the pair embedding update block 500 updates the current pair embedding 304 using the updated single embedding (i.e., generated by the single embedding update block 400).

[0091] In general, the single embedding and the pair embedding can encode complementary information. The single embedding update block 400 enriches the information content of the single embedding with the complementary information encoded in the pair embedding, and the pair embedding update block 500 enriches the information content of the pair embedding with the complementary information encoded in the single embedding. As a result of this enrichment, the updated single embedding and the updated pair embedding encode relevant information by accurately unmasking the masked embedding of the masked protein representation.

[0092] The update block 300 is described herein as first updating the current single embedding 302 with the current pair embedding 304, and then updating the current pair embedding 304 with the updated single embedding 306. The description should not be understood to limit the update block to performing operations in this order, for example, the update block could first update the current pair embedding with the current single embedding, and then update the current single embedding with the updated pair embedding.

[0093] Update block 300 is described herein as including a single embedding update block 400 (i.e., updating a current single embedding) and a pair embedding update block 500 (i.e., updating a current pair embedding). The description should not be understood to limit update block 300 to including only one single embedding update block or only one pair embedding update block. For example, update block 300 may include multiple single embedding update blocks that update a single embedding multiple times before a single embedding is provided to a pair update block for use in updating a current pair embedding. As another example, update block 300 may include multiple pair update blocks that update a pair embedding multiple times using a single embedding.

[0094] The single embedded update block 400 and the pair embedded update block 500 may have any suitable architecture that enables them to perform their described functions.

[0095] In some implementations, the single embedding update block 400, the pair embedding update block 500, or both, include one or more "self-attention" blocks. As used throughout this specification, a self-attention block generally refers to a neural network block that updates a set of embeddings, i.e., receives a set of embeddings and outputs updated embeddings. To update a given embedding, the self-attention block may determine respective "attention weights", e.g., similarity measures, between the given embedding and one or more selected embeddings, e.g., each of the received set of embeddings, and then update the given embedding using (i) the attention weights and (ii) the selected embedding. For example, the updated embedding may include a sum of values, each derived from one of the selected embeddings, each weighted by a respective attention weight. For convenience, the self-attention block may be said to update the given embedding using attention "on" the selected embedding.

[0096] For example, the self-attention block is a set of input embeddings

number

number

number

number

[0097] Parameter matrix W q ("Query Embedding Matrix"), W k (the "key embedding matrix"), and W v("value embedding matrix") is a trainable parameter of the self-attention block. The parameters of all self-attention blocks included in the single-embedded update block 400 and the pair-embedded update block 500 can be understood to be parameters of the update block 300 that can be trained as part of the end-to-end training of the protein protein reconstruction system 100 described with reference to FIG. 1. In general, the (trained) parameters of the query, key, and value embedding matrices are different for different self-attention blocks, such that, for example, the self-attention block included in the single-embedded update block 400 may have different query, key, and value embedding matrices with different parameters than the self-attention block included in the pair-embedded update block 500.

[0098] In some implementations, the single embedding update block 400, the pair embedding update block 500, or both, include one or more self-attention blocks that perform self-attention operations that are conditioned (dependent) on the pair embeddings, i.e., conditioned on the pair embeddings. To condition the self-attention operations on the pair embeddings, the self-attention blocks can process the pair embeddings to generate respective "attention biases" corresponding to each attention weight, where each attention weight may then be biased by the corresponding attention bias. For example, attention weights ≠ ≠ ≠ ≠ ≠ ≠ ≠ ≠

number

number

[0099] The self-attention block is a set of "biased attention weights"

number

number

[0100] In general, pairwise embeddings encode information that characterizes the structure of a protein and the relationships between pairs of amino acids within the protein's structure. Applying a self-attention operation conditioned on the pairwise embeddings to a set of input embeddings allows the input embeddings to be updated in a manner that is given the protein structure information encoded in the pairwise embeddings. The update block of the protein reconstruction neural network can use a self-attention block conditioned on the pairwise embeddings to update and enrich the single embeddings and the pairwise embeddings themselves.

[0101] Optionally, the self-attention block can have multiple "heads," each generating a respective updated embedding corresponding to each input embedding, i.e., each input embedding is associated with multiple updated embeddings. For example, each head may be associated with a parameter matrix W q , W k , and W v A self-attention block with multiple heads may perform a "gating" operation to combine the updated embeddings generated by the heads with respect to the input embeddings, i.e., to generate a single updated embedding corresponding to each input embedding. For example, the self-attention block may process the input embeddings using one or more neural network layers (e.g., fully connected neural network layers) to generate a respective gating value for each head. The self-attention block may then combine the updated embeddings corresponding to the input embeddings according to the gating values. For example, the self-attention block may perform a gating operation to combine the updated embeddings corresponding to the input embeddings for the input embeddings x i The updated embedding of

number

number

[0102] An exemplary architecture of a single-embedded update block 400 using a self-attention block conditioned on pair-embedding is described with reference to FIG.

[0103] An exemplary architecture of a pair embedding update block 500 using a self-attention block conditioned on the pair embedding is described with reference to Figure 5. The exemplary pair embedding update block described with reference to Figure 5 updates the current pair embedding based on the updated single embedding by computing the cross product (hereafter referred to as the outer product mean) of the updated single embedding, adding the result of the cross product mean to the current pair embedding (projected to the dimensions of the pair embedding if necessary), and processing the current pair embedding using a self-attention block conditioned on the current pair embedding.

[0104] 4 illustrates an example architecture of a unimodal embedding update block 400. The unimodal embedding update block 400 is configured to receive the current unimodal embedding 302 and update the current unimodal embedding 302 based (at least in part) on the current pair embedding.

[0105] To update the current uni-embedding 302, the uni-embedding update block 400 updates the uni-embedding using a self-attention operation conditioned on the current pair embedding. More specifically, the uni-embedding update block 400 provides a uni-embedding to a self-attention block 402 conditioned on the current pair embedding, e.g., as described with reference to FIG. 3, to generate an updated uni-embedding. Optionally, the uni-embedding update block can add an input to the self-attention block 402 to an output of the self-attention block 402. Conditioning the self-attention block 402 with the current pair embedding allows the uni-embedding update block 400 to enrich the current uni-embedding 302 using information from the current pair embedding.

[0106] The unimodal embedding update block then processes the current unimodal embedding 302 using, for example, a transition block that applies one or more fully connected neural network layers to the current unimodal embedding. Optionally, the unimodal embedding update block 400 can add the input to the transition block 404 to the output of the transition block 404.

[0107] The update single embedding block may output an updated single embedding 306 resulting from the operations performed by the self-attention block 402 and the transition block 404 .

[0108] 5 illustrates an example architecture of the pair embedding update block 500. The pair embedding update block 500 is configured to receive the current pair embedding 304 and update the current pair embedding 304 based (at least in part) on the updated single embedding 306.

[0109] In the following description, pairwise embeddings may be understood to be arranged in an NxN array, i.e. such that an embedding at position (i, j) in the array is a pairwise embedding of the amino acids at positions i and j in the amino acid sequence.

[0110] To update the current pair embedding 304 , the update pair embedding block 500 applies a cross-product average operation 502 to the updated single embedding 306 and adds the result of the cross-product average operation 502 to the current pair embedding 304 .

[0111] The cross product average operation defines a sequence of operations that, when applied to a set of single embeddings represented as a 1×N array of embeddings, produces an N×N array of embeddings, i.e., N is the number of amino acids in the protein. The current paired embedding 304 can also be represented as an N×N array of paired embeddings, and adding the result of the cross product average 502 to the current paired embedding 304 refers to summing the two N×N arrays of embeddings.

[0112] To calculate the cross product average, the pair embedding update block can be, for example, A(res1, res2, ch1, ch2) = LeftAct(res1, ch1) RightAct(res2, ch2) (6) where res1, res2∈{1, ..., N}, ch1, ch2∈{1, ..., C}, C is the number of channels in each single embedding, and LeftAct(res1, ch1) is a linear operation (e.g. defined by matrix multiplication, e.g. projection) applied to channel ch1 of the single embedding indexed by “res1”, and RightAct(res2, ch2) is a linear operation (e.g. defined by matrix multiplication, e.g. projection) applied to channel ch2 of the single embedding indexed by “res2”. The cross-average result is produced by flattening the (ch1, ch2) dimension of tensor A and linearly projecting it. Optionally, the Pair Embedding Update block may perform one or more Layer Normalization operations (e.g., as described in connection with Jimmy Lei Ba et al., "Layer Normalization", arXiv:1607.06450) as part of computing the cross-product average.

[0113] In general, the updated single embedding 306 encodes information about amino acids in the amino acid sequence of the protein. By incorporating the information encoded in the updated single embedding into the current pair embedding (i.e., via cross-product average 502), the pair embedding update block 500 can enhance the information content of the current pair embedding.

[0114] After updating the current pair embedding 304 with the updated single embedding (i.e., by cross-product average 502), the pair embedding update block 308 updates the current pair embedding of each row of the array of the current pair embedding into an N×N array using a self-attention operation conditioned on the current pair embedding (i.e., a "row-wise" self-attention operation). More specifically, the pair embedding update block 500 provides each row of the current pair embedding to a "row-wise" self-attention block 504, which is also conditioned on the current pair embedding, e.g., as described with reference to FIG. 3, to generate an updated pair embedding for each row. Optionally, the pair embedding update block can add the input to the row-wise self-attention block 504 to the output of the row-wise self-attention block 504.

[0115] Next, the pair embedding update block 500 updates the current pair embedding of each column of the N×N array of the current pair embedding using a self-attention operation that is also conditioned on the current pair embedding (i.e., a "column-wise" self-attention operation). More specifically, the pair embedding update block 500 provides each column of the current pair embedding to a "column-wise" self-attention block 506 that is also conditioned on the current pair embedding to generate an updated pair embedding for each column. Optionally, the pair embedding update block can add the input to the column-wise self-attention block 506 to the output of the column-wise self-attention block 506.

[0116] The pair embedding update block 500 then processes the current pair embedding using, for example, a transition block 508 that applies one or more fully connected neural network layers to the current pair embedding. Optionally, the pair embedding update block 500 can add the input to the transition block 508 to the output of the transition block 508.

[0117] The pair embedding update block may output updated pair embeddings 308 resulting from the operations performed by the row-wise self-attention block 504, the column-wise self-attention block 506, and the transition block 508.

[0118] 6 is a flow diagram of an exemplary process 600 for unmasking a masked representation of a protein using a protein reconstruction neural network. For convenience, the process 600 is described as being performed by one or more computer systems located at one or more locations. For example, a protein reconstruction system suitably programmed in accordance with the present specification, such as the protein reconstruction system 100 of FIG. 1, can perform the process 600.

[0119] The system receives 602 a masked representation of a protein. The masked representation of the protein includes (i) a representation of the amino acid sequence of the protein that includes a set of embeddings, each of which corresponds to a respective position in the amino acid sequence of the protein, and (ii) a representation of the structure of the protein that includes a set of embeddings, each of which corresponds to a respective structural feature of the protein. At least one of the embeddings included in the masked representation of the protein is masked.

[0120] Steps 604-610, described below, may be performed in one or more iterations each.

[0121] The system processes the masked representation of the protein using a protein reconstruction neural network to generate respective predicted embeddings corresponding to one or more masked embeddings included in the masked representation of the protein (604). The predicted embeddings corresponding to the masked embeddings in the representation of the amino acid sequence of the protein define predictions of the identity of amino acids at corresponding positions in the amino acid sequence. The predicted embeddings corresponding to the masked embeddings in the representation of the structure of the protein define predictions of corresponding structural features of the protein.

[0122] Optionally, if the current iteration is after the first iteration, the system can provide the predicted protein structure generated in the previous iteration (as described in more detail in steps 608-610) as an additional input to the protein reconstruction neural network, i.e., in addition to the masked representation of the protein.

[0123] In some implementations, the system may update the masked representation of the protein by replacing a so-called proper subset of masked embeddings in the masked representation of the protein (i.e., a subset that does not include all masked embeddings) with corresponding predicted embeddings (606). The system may then proceed to a next iteration (e.g., by returning to step 604), in which the system may process the updated masked representation of the protein using the protein reconstruction neural network to generate respective predicted embeddings that correspond to one or more remaining masked embeddings included in the masked representation of the protein.

[0124] In some implementations, the representation of the amino acid sequence of the protein includes one or more masked embeddings, and the system identifies a predicted amino acid sequence of the protein in which each masked embedding in the representation of the amino acid sequence is replaced by a corresponding predicted embedding. The system can process the predicted amino acid sequence using a protein folding neural network to generate data defining a predicted protein structure for the predicted amino acid sequence (608). For example, any protein folding neural network based on published techniques or software such as AlphaFold2 (available in open source) may be used. The system can then proceed to a next iteration (i.e., by returning to step 604), in which the system can provide the predicted protein structure as an additional input (i.e., in addition to the masked protein representation) to the protein reconstruction neural network (610). The protein reconstruction neural network can then process the predicted protein structure and the masked protein representation to generate new predicted embeddings in the next iteration.

[0125] In some implementations, the system may perform both step 606 (i.e., updating the masked representation of the protein using the predicted embedding) and steps 608-610 (i.e., processing the predicted amino acid sequence to generate a predicted protein structure, as described above, and providing the predicted protein structure as an additional input to the protein reconstruction neural network) in one or more iterations.

[0126] The system may, for example, determine that the iterative process is complete after each masked embedding in the masked protein representation has been replaced by a corresponding predicted embedding, and the system may provide as output a complete protein representation, i.e., a complete protein representation in which all masked embeddings of the masked protein representation have been replaced by corresponding predicted embeddings generated over the series of iterations.

[0127] In general, the system can be used to determine a predicted structure of a (target) protein, polypeptide ligand, or antibody by generating a predicted embedding that defines a complete protein structural representation of the (target) protein, polypeptide ligand, or antibody. This can be achieved, for example, when the masked representation of the protein includes a complete representation of the amino acid sequence of the (target) protein, polypeptide ligand, or antibody, and the representation of the protein's structure includes a fully masked representation of the (target) protein, polypeptide ligand, or antibody's structure.

[0128] Some further applications of the system are described below.

[0129] The system may be used to obtain a ligand, such as a ligand for a drug or an industrial enzyme. For example, a method for obtaining a ligand may include obtaining a target amino acid sequence of a target protein and using the target amino acid sequence to determine the (tertiary) structure of the target protein. The method may include evaluating the interaction of one or more candidate ligands with the structure of the target protein and selecting one or more of the candidate ligands as ligands depending on the results. Evaluating the interaction may include evaluating binding of the candidate ligands with the structure of the target protein, for example, to identify a ligand that binds with sufficient affinity for a biological effect. The candidate ligand may be an enzyme. Evaluating may include evaluating the affinity between the candidate ligand and the target protein, or the selectivity of the interaction. The candidate ligand may be derived from a database of candidate ligands, or by modifying a ligand in the database of candidate ligands, or by stepwise or iterative assembly or optimization of the candidate ligands. The evaluation may be performed, for example, using a computer-assisted approach in which graphical models of the candidate ligand and target protein structures are displayed for user manipulation, or the evaluation may be performed partially or fully automatically, for example, using standard protein-ligand docking software. The evaluation may include, for example, determining an interaction score for the candidate ligand that depends on the strength or specificity of the interaction, for example, a score that depends on the binding free energy. Ligand candidates may be selected according to their scores.

[0130] In some implementations, the target protein includes a receptor or enzyme, and the ligand is an agonist or antagonist of the receptor or enzyme. In some implementations, the method may be used to identify the structure of a cell surface marker. This may then be used to identify a ligand, e.g., an antibody or a label, such as a fluorescent label, that binds to the cell surface marker. This may be used to identify and / or treat cancer cells. In some implementations, the candidate ligand may include a small molecule ligand, e.g., an organic compound with a molecular weight <900 Daltons. In some other implementations, the candidate ligand may include a polypeptide ligand, i.e., a polypeptide ligand defined by an amino acid sequence.

[0131] Some implementations of the system may be used to determine the structure of a candidate polypeptide ligand, for example a ligand of a pharmaceutical drug or an industrial enzyme, and then its interaction with a target protein structure may be assessed, the target protein structure may have been determined using the computer-implemented methods described herein or using conventional physical investigation techniques such as X-ray crystallography and / or magnetic resonance techniques.

[0132] Thus, the system may be used to obtain a polypeptide ligand, e.g., a molecule or sequence thereof. This may include obtaining an amino acid sequence of one or more candidate polypeptide ligands, and performing the method as described above using the amino acid sequence of the candidate polypeptide ligand as the sequence of amino acids to determine the (tertiary) structure of the candidate polypeptide ligand. The structure of the target protein may be obtained, e.g., in silico or by physical investigation, and the interaction between the structure of each of the one or more candidate polypeptide ligands and the target protein structure may be evaluated. One of the one or more of the candidate polypeptide ligands may be selected as the polypeptide ligand depending on the result of the evaluation. As mentioned above, evaluating the interaction may include evaluating the binding of the candidate polypeptide ligand with the structure of the target protein, e.g., identifying a ligand that binds with sufficient affinity for a biological effect, and / or evaluating the connection of the candidate polypeptide ligand with the structure of the target protein that affects the function of the target protein, e.g., an enzyme, and / or evaluating the affinity between the candidate polypeptide ligand and the structure of the target protein, or evaluating the selectivity of the interaction. In some implementations, the polypeptide ligand may be an aptamer. Again, polypeptide candidate ligands may be selected according to which one has the highest affinity.

[0133] As previously mentioned, the selected polypeptide ligand may include a receptor or enzyme, and the ligand may be an agonist or antagonist of the receptor or enzyme. In some implementations, the polypeptide ligand may include an antibody, and the target protein includes an antibody target, i.e., an antigen, for example, a virus, particularly a viral coat protein, or a protein expressed on a cancer cell. In these implementations, the antibody binds to the antigen to provide a therapeutic effect. For example, an antibody may bind to an antigen and act as an agonist of a particular receptor, or the antibody may inhibit the binding of another ligand to the target, thus inhibiting the activation of a relevant biological pathway.

[0134] Such methods may include synthesizing, i.e., making, a small molecule or polypeptide ligand. The ligand may be synthesized by any conventional chemical technique and / or may be already available, for example from a compound library, or may have been synthesized using combinatorial chemistry.

[0135] The method may further include testing the ligand for biological activity in vitro and / or in vivo. For example, the ligand may be tested for ADME (absorption, distribution, metabolism, excretion) and / or toxicological properties to screen out unsuitable ligands. Testing may include, for example, contacting a candidate small molecule or polypeptide ligand with a target protein and measuring a change in expression or activity of the protein.

[0136] In some implementations, the candidate (polypeptide) ligand may include an isolated antibody, a fragment of an isolated antibody, a single variable domain antibody, a bi- or multispecific antibody, a multivalent antibody, a dual variable domain antibody, an immunoconjugate, a fibronectin molecule, an adnectin, a DARPin, an avimer, an affibody, an anticalin, an affilin, a protein epitope mimetic, or a combination thereof. The candidate (polypeptide) ligand may include, for example, an antibody with a mutated or chemically modified amino acid Fc region that inhibits or reduces ADCC (antibody-dependent cellular cytotoxicity) activity and / or extends half-life when compared to a wild type Fc region.

[0137] Misfolded proteins are associated with many diseases. The system can be used to identify the presence of a protein misfolding disease. This may include obtaining an amino acid sequence of a protein, performing a method as described above using the amino acid sequence of the protein to determine the structure of the protein, obtaining, for example, by conventional (physical) methods, the structure of a version of the protein obtained from a human or animal body, and comparing the structure of the protein with the structure of the version obtained from the body and identifying the presence of a protein misfolding disease depending on the result. That is, by comparison with the determined structure, the misfolding of the version of the protein from the body may be determined. In general, identifying the presence of a protein misfolding disease may include obtaining an amino acid sequence of a protein, determining the structure of the protein using the amino acid sequence of the protein as described herein, comparing the structure of the protein with the structure of a reference version of the protein and identifying the presence of a protein misfolding disease depending on the result of the comparison. For example, the structures compared may be the structures of a mutant and a wild type protein. In implementations, the wild type protein may be used as the baseline version, but in principle either may be used as the reference version.

[0138] In some implementations, the system can be used to identify active sites / binding sites / blocking sites on a target protein from the amino acid sequence of the target protein.

[0139] This specification uses the term "configured" in reference to systems and computer program components. One or more computer systems being configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that, during operation, causes the system to perform the operation or action. One or more computer programs being configured to perform a particular operation or action means that one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0140] The embodiments and functional operations of the subject matter described herein can be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware, including the structures disclosed herein and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by or for controlling the operation of a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a receiver device suitable for execution by a data processing device.

[0141] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. The apparatus may also be or further include special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Optionally, in addition to hardware, the apparatus may include code that creates an execution environment for a computer program, such as code that constitutes a processor's firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0142] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may correspond to a file in a file system, but this is not necessarily the case. A program may be stored as part of a file that holds other programs or data, for example one or more scripts stored in a markup language document, a single file dedicated to the program in question, or in multiple organized files, for example files that store one or more modules, subprograms, or code portions. A computer program may be deployed to be executed on one computer or on multiple computers that are located in one location or distributed across multiple locations and connected together by a data communications network.

[0143] The term "engine" is used broadly herein to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers are dedicated to a particular engine, and in other cases, multiple engines can be installed and running on the same computer or computers.

[0144] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by performing operations on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., FPGAs or ASICs, or a combination of special purpose logic circuitry and one or more programmed computers.

[0145] A computer suitable for executing a computer program can be based on a general-purpose or dedicated microprocessor or both, or any other type of central processing unit. In general, the central processing unit receives instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for executing or executing instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory can be supplemented by or incorporated in special-purpose logic circuitry. In general, a computer also includes one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively coupled to receive data from or transfer data to the mass storage devices, or both. However, a computer may not have such devices. Additionally, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, such as a Universal Serial Bus (USB) flash drive, to name just a few.

[0146] Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media, and memory devices including semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM disks and DVD-ROM disks.

[0147] To provide interaction with a user, embodiments of the subject matter described herein can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, e.g., a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user, e.g., the feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and the input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, the computer can interact with the user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to the web browser of the user's device in response to a request received from the web browser. The computer can also interact with the user by sending text messages or other forms of messages to a personal device, e.g., a smartphone, running a messaging application, and receiving a response message from the user in return.

[0148] A data processing apparatus for implementing machine learning models may also include dedicated hardware accelerator units for handling the often computationally intensive parts of the machine learning training or generation, i.e., inference, workload.

[0149] The machine learning model can be implemented and deployed using a machine learning framework, for example, the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0150] Embodiments of the subject matter described herein can be implemented in a computing system that includes a back-end component, e.g., as a data server, or includes a middleware component, e.g., an application server, or includes a front-end component, e.g., a client computer having a graphical user interface, web browser, or app through which a user can interact with an implementation of the subject matter described herein, or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.

[0151] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data, e.g., HTML pages, to a user device for the purpose of, e.g., displaying data to a user interacting with the device acting as a client and receiving user input from such a user. Data generated at the user device, e.g., results of user interaction, can be received from the device at the server.

[0152] Although this specification contains many specific implementation details, these should not be considered as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to certain embodiments of a particular invention. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as working in a particular combination, and may even be initially claimed as such, one or more features of the claimed combination may in some cases be deleted from the combination, and the claimed combination may be directed to a subcombination, or a variation of the subcombination.

[0153] Similarly, although operations are illustrated in the figures and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown, or in sequential order, or that all of the operations shown be performed to achieve a desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the division of various system modules and components in the above-described embodiments should not be understood as requiring such division in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products.

[0154] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. By way of example, the processes depicted in the accompanying figures do not necessarily require the particular order depicted, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0155] 100 Protein Reconstitution System 102 Masked Representation of Proteins, Masked Protein Representation 104 Amino acid sequence representation 106 Protein Structure Representation 108 Predicted Embeddings 110 Complete Protein Expression 200 Protein Reconstruction Neural Network 202 Single Embed 204 Pair Embedding 206-A~N Update block 208 Updated Single Embed 210 Updated Pair Embeddings 300 Update Block 302 Current Single Embed 304 Current Pair Embedding 306 Updated Single Embed 308 Updated pair embedding, pair embedding update block 400 Single Embedded Update Block 402 Self-Attention Block 404 Transition Block 500 pair embedded update block 502 Cross product average calculation, cross product average 504 Line-by-line self-attention blocks 506 Column-wise self-attention block 508 Transition Block 600 processes

Claims

1. 1. A method executed by one or more data processing devices for unmasking a masked representation of a protein using a protein reconstruction neural network, comprising: receiving the masked representation of the protein, the masked representation of the protein comprises: (i) a representation of the amino acid sequence of the protein comprising a plurality of embeddings, each of which corresponds to a respective position within the amino acid sequence of the protein; and (ii) a representation of the structure of the protein comprising a plurality of embeddings, each of which corresponds to a respective structural feature of the protein; At least one of the embeddings included in the representation of the amino acid sequence in the masked representation of the protein is masked. Steps and processing the masked representation of the protein using the protein reconstruction neural network to generate respective predicted embeddings corresponding to one or more masked embeddings contained in the masked representation of the protein, a predicted embedding corresponding to a masked embedding in said representation of said amino acid sequence of said protein defines a prediction of the identity of an amino acid at a corresponding position in said amino acid sequence; a predicted embedding corresponding to the masked embedding in the representation of the structure of the protein defines a prediction of a corresponding structural feature of the protein. Steps and A method comprising:

2. updating the masked representation of the protein by replacing a suitable subset of the masked embeddings in the masked representation of the protein with corresponding predicted embeddings; processing the updated masked representation of the protein using the protein reconstruction neural network to generate respective predicted embeddings corresponding to one or more remaining masked embeddings included in the masked representation of the protein; The method of claim 1, further comprising:

3. wherein the representation of the amino acid sequence of the protein comprises one or more masked embeddings, the method comprising: processing the predicted amino acid sequence of the protein defined by replacing each masked embedding in the representation of the amino acid sequence with a corresponding predicted embedding using a protein folding neural network to generate data defining a predicted protein structure of the predicted amino acid sequence; using the protein reconstruction neural network to process both (i) the masked representation of the protein and (ii) the predicted protein structure of the predicted amino acid sequence to generate new predicted embeddings corresponding to one or more masked embeddings contained in the masked representation of the protein; 3. The method of claim 1 or 2, further comprising:

4. 4. The method of claim 1, wherein each masked embedding included in the masked representation of the protein is a default embedding.

5. The method of claim 4 , wherein the default embedding comprises a vector of zeros.

6. 6. The method of claim 1, wherein each predicted embedding corresponding to a masked embedding in the representation of the structure of the protein defines a prediction of a spatial distance between a corresponding pair of amino acids in the structure of the protein.

7. 7. The method of claim 1, wherein at least one of the embeddings of the representation of the amino acid sequence of the protein is masked.

8. 8. The method of claim 1 , wherein at least one of the embeddings of the representation of the structure of the protein is masked.

9. the representation of the amino acid sequence of the protein comprises a plurality of single embeddings, each corresponding to a respective position within the amino acid sequence of the protein; the representation of the structure of the protein comprises a plurality of pair embeddings, each pair corresponding to a respective pair of positions in the amino acid sequence of the protein; the protein reconstruction neural network includes a sequence of update blocks; each update block having a respective set of update block parameters; receiving a current paired embedding and a current single embedding; updating the current single embedding based on the current pair embedding according to values ​​of the update block parameters of the update block; updating the current pair embedding based on an updated single embedding according to the value of the update block parameter of the update block; Performing an action including The method of claim 1 , wherein a last update block in the sequence of update blocks generates a final pair embedding and a final single embedding.

10. said protein reconstruction neural network, for each of one or more masked single embeddings within said representation of the amino acid sequence of said protein, 10. The method of claim 9, further comprising: generating the predicted embedding for the masked single embedding based on a corresponding final single embedding generated by the last update block.

11. said protein reconstruction neural network, for each of one or more masked pair embeddings in said representation of the amino acid sequence of said protein, The method of claim 9 or 10, further comprising: generating the predicted embedding for the masked pair embedding based on a corresponding final pair embedding generated by the last update block.

12. updating the current single embedding based on the current pair embedding; 12. The method of claim 9, further comprising: updating the current single embedding using attention on the current single embedding, the attention being conditioned on the current pair embedding.

13. updating the current single embedding using attention on the current single embedding; generating a plurality of attention weights based on the current single embedding; generating a respective attention bias corresponding to each of the attention weights based on the current pair embedding; generating a plurality of biased attention weights based on the attention weights and the attention bias; updating the current single embedding using attention on the current single embedding based on the biased attention weights; 13. The method of claim 12, comprising:

14. updating the current pair embedding based on the updated single embedding; applying a transformation operation to the updated single embedding; updating the current pair embedding by adding a result of the transform operation to the current pair embedding; 14. The method according to any one of claims 9 to 13, comprising:

15. The method of claim 14 , wherein the transform operation comprises a cross product operation.

16. updating the current pair embedding based on the updated single embedding, after adding the result of the transform operation to the current pair embedding; 16. The method of claim 14 or 15, further comprising: updating the current pair embedding using attention on the current pair embedding, the attention being conditioned on the current pair embedding.

17. 1. A method for obtaining a ligand, implemented by one or more data processing devices, wherein the ligand is a ligand of a drug or an industrial enzyme, the method comprising:

17. Carrying out the method of any one of claims 1 to 16 to determine a predicted structure of a target protein by generating a predicted embedding that defines a complete protein structural representation of the target protein, wherein the masked representation of the protein comprises a complete representation of the amino acid sequence of the target protein and the representation of the structure of the protein comprises a fully masked representation of the structure of the target protein; evaluating the interaction of one or more candidate ligands with the predicted structure of the target protein; selecting one or more of the candidate ligands as the ligand depending on the result of the evaluation; A method comprising:

18. 1. A method for obtaining a ligand, implemented by one or more data processing devices, wherein the ligand is a ligand of a drug or an industrial enzyme, the method comprising:

17. Carrying out the method of any one of claims 1 to 16 to determine a predicted structure for each of a plurality of target proteins by generating a predicted embedding defining a complete protein structural representation for each target protein, wherein for each target protein, the masked representation of the protein comprises a complete representation of the amino acid sequence of the target protein and the representation of the structure of the protein comprises a fully masked representation of the structure of the target protein; evaluating the interaction of one or more candidate ligands with the predicted structure of each of the target proteins; selecting one or more of the candidate ligands as said ligand, either to obtain a ligand that interacts with each of said target proteins, or to obtain a ligand that interacts with only one of said target proteins; A method comprising:

19. 19. The method of claim 17 or 18, wherein the target protein comprises a receptor or an enzyme and the ligand is an agonist or antagonist of the receptor or enzyme.

20. 1. A method for obtaining a polypeptide ligand, implemented by one or more data processing devices, said polypeptide ligand being a ligand of a drug or an industrial enzyme, said method comprising:

17. Carrying out the method of any one of claims 1 to 16 to determine a predicted structure of a candidate polypeptide ligand by generating, for each of one or more candidate polypeptide ligands, a predicted embedding that defines a complete protein structural representation of the candidate polypeptide ligand, wherein, for each of the one or more candidate polypeptide ligands, the masked representation of the protein comprises a complete representation of the amino acid sequence of the candidate polypeptide ligand and the representation of the structure of the protein comprises a fully masked representation of the structure of the candidate polypeptide ligand; obtaining a target protein structure of a target protein; evaluating the interaction between the predicted structure of each of the one or more candidate polypeptide ligands and the target protein structure; selecting one of the one or more of the candidate polypeptide ligands as the polypeptide ligand depending on the result of the evaluation; A method comprising:

21. 21. The method of claim 20, wherein the target protein comprises a receptor or enzyme and the ligand is an agonist or antagonist of the receptor or enzyme, or the polypeptide ligand comprises an antibody and the target protein comprises an antigen and the antibody binds to the antigen to provide a therapeutic effect.

22. 1. A method for obtaining an antibody of an antigen, the method being carried out by one or more data processing devices, comprising:

17. The method of claim 1, further comprising: performing a method for determining a predicted structure and amino acid sequence of the antibody by generating: i) a complete amino acid sequence representation of the antibody; and ii) a predicted embedding that defines a complete protein structural representation of the antibody; The method of claim 1, wherein the masked representation of the protein comprises a representation of a paratope of the antibody that binds to the antigen, and comprises i) a partially masked representation of the amino acid sequence of the antibody, and ii) a partially masked representation of the structure of the antibody.

23. 23. The method of claim 21 or 22, wherein the antigen comprises a viral protein or a cancer cell protein.

24. A method for obtaining diagnostic antibody markers for a disease, which is carried out by one or more data processing devices, comprising:

17. Carrying out the method of any one of claims 1 to 16 to determine a predicted structure of a candidate antibody by generating, for each of one or more candidate antibodies, a predicted embedding that defines a complete protein structural representation of the candidate antibody, wherein, for each of the one or more candidate antibodies, the masked representation of the protein comprises a complete representation of the amino acid sequence of the candidate antibody and the representation of the structure of the protein comprises a fully masked representation of the structure of the candidate antibody; obtaining a target protein structure of a target protein; evaluating the interaction between the predicted structure of each of the one or more candidate antibodies and the target protein structure; Selecting one of the one or more candidate antibodies as the diagnostic antibody marker depending on the results of the evaluation. A method comprising:

25. 1. A method for designing mutated proteins with optimized properties, performed by one or more data processing devices, comprising: obtaining i) a complete representation of the amino acid sequence of a known protein, and ii) a complete protein structural representation of said known protein; 17. Carrying out the method of any one of claims 1 to 16 to determine a predicted amino acid sequence of one or more candidate mutated proteins by generating, for each of said candidate mutated proteins, a predicted embedding defining a complete amino acid sequence of said candidate mutated protein, said generating said predicted embedding comprising: generating a partially masked representation of the candidate mutated protein by masking one or more of the embeddings within a representation of the amino acid sequence of the candidate mutated protein; generating, for each masked amino acid embedding, a respective score distribution defining a score for each amino acid type in the set of possible amino acid types; generating the predicted embeddings by sampling each type for each masked amino acid according to the score distribution for that amino acid; and selecting one of the candidate mutated proteins as the mutated protein by identifying, from among the candidate mutated proteins, the predicted amino acid sequence that predicts an optimal property for the candidate mutated protein; A method comprising:

26. 26. The method of claim 25, further comprising the step of synthesizing the mutated protein.

27. 1. A method for identifying the presence of a protein misfolding disease, performed by one or more data processing devices, comprising:

17. Carrying out the method of any one of claims 1 to 16 to determine a predicted structure of a protein by generating a predicted embedding that defines a complete protein structure representation of the protein, wherein the masked representation of the protein comprises a complete representation of the amino acid sequence of the protein and the representation of the structure of the protein comprises a fully masked representation of the structure of the protein; obtaining a structure of a version of said protein obtained from the human or animal body; comparing the predicted structure of the protein to the structure of a version of the protein obtained from the human or animal body; determining the presence of a protein misfolding disease depending on the result of said comparison; A method comprising:

28. 1. A method for obtaining an amino acid sequence of a protein, the method being carried out by one or more data processing devices, comprising: receiving a structure of the protein, the structure of the protein being obtained by experiment; determining a complete protein structural representation of the protein from said structure; 17. Carrying out the method of any one of claims 1 to 16 to determine a predicted amino acid sequence of the protein by generating a predicted embedding that defines a complete amino acid sequence representation of the protein, wherein the masked representation of the protein comprises a complete representation of the structure of the protein, the representation of the amino acid sequence of the protein comprises a completely masked representation of the amino acid sequence of the protein, and the predicted amino acid sequence of the protein is the obtained amino acid sequence of the protein; A method comprising:

29. One or more computers; one or more storage devices communicatively coupled to the one or more computers, the one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 28; Including, the system.

30. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 28.

Images