predicting complete protein representation from masked protein representation

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for de-masking masked representations of proteins using a protein reconstruction neural network. In one aspect, a method includes receiving a masked representation of a protein; and processing the masked representation of the protein using a protein reconstruction neural network to generate respective predicted embeddings corresponding to one or more masked embeddings included in the masked representation of the protein, wherein a predicted embedding corresponding to a masked embedding in a representation of an amino acid sequence of the protein defines a prediction of identity of an amino acid at a corresponding position in the amino acid sequence, wherein a predicted embedding corresponding to a masked embedding in a representation of a structure of the protein defines a prediction of a respective structural feature of the protein.

Description

Background Technology

[0001] This specification relates to predicting complete protein representations from masked protein representations.

[0002] A protein is defined by a sequence of one or more amino acids. Amino acids are organic compounds that include an amino functional group and a carboxyl functional group, as well as amino acid-specific side chains (i.e., atomic groups).

[0003] Protein folding refers to the physical process by which an amino acid sequence folds into a three-dimensional conformation. The structure of a protein defines the three-dimensional configuration of the atoms in its amino acid sequence after folding. When amino acids are linked by peptide bonds, they are referred to as amino acid residues.

[0004] Machine learning models can be used for prediction. A machine learning model takes input and generates an output, such as a predicted output, based on the received input. Some machine learning models are parametric models and generate outputs based on the received input and the model's parameter values. Some machine learning models are deep models, which employ multiple layers to generate outputs for received inputs. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers, each of which applies a non-linear transformation to the received input to generate an output. Summary of the Invention

[0005] This specification describes a protein reconstruction system implemented as a computer program on one or more computers in one or more locations, which can use a protein reconstruction neural network to demask a masked representation of a protein. The protein reconstruction neural network is not limited to having a specific architecture, and as described later, the system can improve the accuracy of protein representation by jointly processing the representations of the protein's amino acid sequence and structure.

[0006] As used throughout this specification, the term "protein" can be understood as any biomolecule specified by one or more amino acid sequences. For example, the term protein can be understood as a protein domain (i.e., a portion of an amino acid sequence that can fold almost independently of the rest of the amino acid sequence) or a protein complex (i.e., specified by a plurality of related amino acid sequences).

[0007] Throughout the specification, embedding refers to an ordered collection of numerical values, such as a vector or matrix of numerical values.

[0008] According to a first aspect, a method for demasking a masked representation of a protein using a protein reconstruction neural network, executed by one or more data processing devices, is provided. The method includes: receiving a masked representation of a protein, wherein the masked representation of the protein includes: (i) a representation of the amino acid sequence of the protein, comprising a plurality of embeddings, each embedding corresponding to a corresponding position in the amino acid sequence of the protein; and (ii) a representation of the structure of the protein, comprising a plurality of embeddings, each embedding corresponding to a corresponding structural feature of the protein, including at least one of the embeddings in the masked representation of the protein being masked; and processing the masked representation of the protein using a protein reconstruction neural network to generate a corresponding predicted embedding corresponding to one or more masking embeddings included in the masked representation of the protein, wherein the predicted embedding corresponding to the masking embedding in the representation of the amino acid sequence of the protein defines a prediction of the identity of an amino acid at a corresponding position in the amino acid sequence. The predicted embedding corresponding to the masking embedding in the representation of the structure of the protein defines a prediction of a corresponding structural feature of the protein.

[0009] In some embodiments, the method further includes: updating the masking representation of a protein by replacing an appropriate subset of the masking embeddings in the masking representation of the protein with corresponding predictive embeddings; and processing the updated masking representation of the protein using a protein reconstruction neural network to generate corresponding predictive embeddings that correspond to one or more remaining masking embeddings included in the masking representation of the protein.

[0010] In some embodiments, the representation of the amino acid sequence of a protein includes one or more masking embeddings, and the method further includes: processing the predicted amino acid sequence of the protein using a protein folding neural network to generate data defining a predicted protein structure that defines the predicted amino acid sequence, the predicted amino acid sequence of the protein being defined by replacing each masking embedding in the representation of the amino acid sequence with a corresponding predicted embedding; and processing both (i) the masking representation of the protein and (ii) the predicted protein structure that defines the predicted amino acid sequence using a protein reconstruction neural network to generate new predicted embeddings corresponding to one or more masking embeddings included in the masking representation of the protein.

[0011] In some implementations, each masking embedding included in the masking representation of a protein is a default embedding.

[0012] In some implementations, the default embedding includes a zero vector.

[0013] In some implementations, each predicted embedding corresponding to the masking embedding in the representation of the protein structure defines a prediction of the spatial distance between corresponding amino acid pairs in the protein structure.

[0014] In some embodiments, at least one of the representations of the protein's amino acid sequence in the embedding is masked.

[0015] In some implementations, at least one of the embeddings representing the protein structure is masked.

[0016] In some implementations, the representation of the protein's amino acid sequence includes multiple single embeddings, each corresponding to a specific position in the protein's amino acid sequence; the representation of the protein's structure includes multiple paired embeddings, each corresponding to a specific position pair in the protein's amino acid sequence; the protein reconstruction neural network includes an update block sequence; each update block has a set of corresponding update block parameters and performs the following operations: receiving a current paired embedding and a current single embedding; updating the current single embedding based on the current paired embedding and according to the values ​​of the update block parameters of the update block; and updating the current paired embedding based on the updated single embedding and according to the values ​​of the update block parameters of the update block; and the final update block in the update block sequence generates a final paired embedding and a final single embedding.

[0017] In some implementations, the protein reconstruction neural network performs further operations, which include masking each of one or more single embeddings in the representation of the protein's amino acid sequence: generating a predicted embedding for the masked single embedding based on the corresponding final single embedding generated by the final update block.

[0018] In some implementations, the protein reconstruction neural network performs further operations, which include generating a predicted embedding for each of one or more masked pair embeddings in the representation of the protein's amino acid sequence, based on the corresponding final pair embedding generated by the final update block.

[0019] In some implementations, updating the current single embedding based on the current pairwise embeddings includes updating the current single embedding using attention to the current single embedding, wherein the attention is conditioned on the current pairwise embeddings.

[0020] In some implementations, updating the current single embedding using attention to the current single embedding includes: generating multiple attention weights based on the current single embedding; generating a corresponding attention bias based on the current pairwise embeddings for each attention weight; generating multiple attention weights with biases based on the attention weights and attention biases; and updating the current single embedding using attention to the current single embedding based on the attention weights with biases.

[0021] In some implementations, updating the current pairwise embedding based on the updated single embedding includes: applying a transformation operation to the updated single embedding; and updating the current pairwise embedding by adding the result of the transformation operation to the current pairwise embedding.

[0022] In some implementations, the transformation operation includes an outer product operation.

[0023] In some implementations, updating the current pairwise embedding based on the updated single embedding further includes: after adding the result of the transformation operation to the current pairwise embedding, updating the current pairwise embedding using attention to the current pairwise embedding, wherein the attention is conditioned on the current pairwise embedding.

[0024] According to another aspect, a method for obtaining a ligand is provided, wherein the ligand is a ligand for a drug or an industrial enzyme, the method comprising: determining a predicted structure of a target protein by generating a predicted embedding that defines a complete protein structure representation of the target protein, wherein the masked representation of the protein includes a complete representation of the amino acid sequence of the target protein, and wherein the representation of the protein structure includes a completely masked representation of the structure of the target protein; evaluating the interaction between one or more candidate ligands and the predicted structure of the target protein; and selecting one or more candidate ligands as ligands based on the evaluation results.

[0025] According to another aspect, a method for obtaining a ligand is provided, wherein the ligand is a ligand for a drug or an industrial enzyme, the method comprising: determining a predicted structure of each of a plurality of target proteins by generating a predicted embedding that defines a complete protein structure representation of each target protein, wherein for each target protein, the masked representation of the protein includes a complete representation of the amino acid sequence of the target protein, and wherein the representation of the structure of the protein includes a completely masked representation of the structure of the target protein; evaluating the interaction of one or more candidate ligands with the predicted structure of each target protein; and selecting one or more candidate ligands as ligands to i) obtain a ligand that interacts with each target protein, or ii) obtain a ligand that interacts with only one target protein.

[0026] In some implementations, the target protein includes a receptor or an enzyme, and the ligand is an agonist or antagonist of the receptor or enzyme.

[0027] According to another aspect, a method for obtaining a polypeptide ligand is provided, wherein the ligand is a ligand for a drug or an industrial enzyme, the method comprising: for each of one or more candidate polypeptide ligands, determining a predicted structure of the candidate polypeptide ligand by generating a predicted embedding that defines a complete protein structure representation of the candidate polypeptide ligand, wherein for each of the one or more candidate polypeptide ligands, a masked representation of the protein contains a complete representation of the amino acid sequence of the candidate polypeptide ligand, and wherein a representation of the structure of the protein contains a completely masked representation of the structure of the candidate polypeptide ligand; obtaining a target protein structure of a target protein; evaluating the interaction between the predicted structure of each of the one or more candidate polypeptide ligands and the target protein structure; and selecting one of the one or more candidate polypeptide ligands as the polypeptide ligand based on the evaluation results.

[0028] In some embodiments, the target protein includes a receptor or an enzyme, and the ligand is an agonist or antagonist of the receptor or enzyme, or the polypeptide ligand includes an antibody, and the target protein includes an antigen, and the antibody binds to the antigen to provide a therapeutic effect.

[0029] According to another aspect, a method for obtaining an antibody against an antigen is provided, the method comprising: determining a predicted structure and amino acid sequence of an antibody by generating a predicted embedding, the predicted embedding defining i) a complete amino acid sequence representation of the antibody, and ii) a complete protein structure representation of the antibody, wherein the masked representation of the protein includes a representation of the complement of the antibody binding to the antigen, and includes i) a partially masked representation of the amino acid sequence of the antibody, and ii) a partially masked representation of the antibody structure.

[0030] In some implementations, the antigen includes viral proteins or cancer cell proteins.

[0031] According to another aspect, a method for obtaining a diagnostic antibody marker for a disease is provided, the method comprising: for each of one or more candidate antibodies, determining a predicted structure of the candidate antibody by generating a predicted embedding that defines a complete protein structure representation of the candidate antibody, wherein for each of the one or more candidate antibodies, the masked representation of the protein includes a complete representation of the amino acid sequence of the candidate antibody, and wherein the representation of the protein structure includes a completely masked representation of the structure of the candidate antibody; obtaining a target protein structure of a target protein; evaluating the interaction between the predicted structure of each of the one or more candidate antibodies and the target protein structure; and selecting one of the one or more candidate antibodies as a diagnostic antibody marker based on the evaluation results.

[0032] According to another aspect, a method for designing mutant proteins with optimized properties is provided, comprising: obtaining i) a complete representation of the amino acid sequence of a known protein, and ii) a complete protein structure representation of a known protein; and for each of one or more candidate mutant proteins, determining a predicted amino acid sequence of the candidate mutant protein by generating a predicted embedding that defines the complete amino acid sequence of the candidate mutant protein, wherein generating the predicted embedding comprises: generating a partially masked representation of the candidate mutant protein by masking one or more embeddings in the representation of the amino acid sequence of the candidate mutant protein; for each masked amino acid embedding, generating a corresponding score distribution that defines a score for each amino acid type in a set of possible amino acid types; generating a predicted embedding by sampling the corresponding type of each masked amino acid according to the score distribution of the amino acids; and selecting one of the candidate mutant proteins as the mutant protein by identifying a predicted amino acid sequence that predicts the optimal properties of the candidate mutant protein from the candidate mutant proteins.

[0033] In some embodiments, the method further includes synthesizing mutant proteins.

[0034] According to another aspect, a method for identifying the presence of a protein misfolding disease is provided, comprising: determining a predicted structure of a protein by generating a predicted embedding that defines a complete protein structure representation of the protein, wherein the masked representation of the protein includes a complete representation of the protein's amino acid sequence, and wherein the representation of the protein structure includes a completely masked representation of the protein structure; obtaining the structure of a version of the protein obtained from a human or animal body; comparing the predicted structure of the protein with the structure of the version of the protein obtained from a human or animal body; and identifying the presence of a protein misfolding disease based on the comparison result.

[0035] According to another aspect, a method for obtaining the amino acid sequence of a protein is provided, comprising: receiving the structure of a protein, wherein the structure of the protein has been obtained experimentally; determining a complete protein structure representation of the protein from the structure; and determining a predicted amino acid sequence of the protein by generating a predicted embedding that defines a complete amino acid sequence representation of the protein, wherein the masked representation of the protein includes a complete representation of the protein structure, wherein the representation of the amino acid sequence of the protein includes a completely masked representation of the amino acid sequence of the protein, and wherein the predicted amino acid sequence of the protein is the obtained amino acid sequence of the protein.

[0036] According to another aspect, a system is provided, comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the methods described herein.

[0037] According to another aspect, one or more non-transitory computer storage media are provided that store instructions, which, when executed by one or more computers, cause the one or more computers to perform the operations of the methods described herein.

[0038] Specific embodiments of the subject matter described in this specification may be implemented in order to achieve one or more of the following advantages.

[0039] Typically, protein folding (i.e., predicting protein structure from amino acid sequence) and protein design (i.e., predicting amino acid sequence from protein structure) are closely related tasks. The system described in this specification can be trained to perform these two tasks in parallel. Specifically, a masked representation of a protein can be provided to the system, comprising a representation of the protein's amino acid sequence and a representation of the protein's structure, wherein one or both of these representations are at least partially masked. The system then processes the masked protein representation to generate a "complete" (i.e., unmasked) representation of the protein, which includes predictions of the masked portions of the amino acid sequence and protein structure representations. As a result of being trained to perform both protein folding and protein design in parallel, the system can achieve higher prediction accuracy on each of these tasks than if the system had been trained to perform either of these tasks independently of the other. In some cases, the system can achieve acceptable prediction accuracy on the protein folding task, the protein design task, or both, while consuming fewer computational resources (e.g., memory and computing power) than other systems that perform either of these tasks independently of the other.

[0040] The system described in this specification can demask a masked protein representation by incrementally replacing the masking embedding in the masked protein representation with the corresponding predicted embedding over a series of iterations. Instead of replacing all the masking embeddings at once in a single iteration, for example, replacing them with the corresponding predicted embeddings over a series of iterations allows the system to incrementally accumulate contextual information, thereby demasking the masked protein representation with greater accuracy.

[0041] The system described in this specification can predict the protein structure of the current amino acid sequence in each of one or more iterations. This protein structure is defined by replacing each masking embedding in the amino acid sequence representation with the corresponding predicted embedding generated in the current iteration. The system can then process all predicted protein structures and masking protein representations in the next iteration, allowing the system to adaptively correct errors in the predicted embeddings that cause the corresponding predicted protein structure to deviate from the target protein structure representation. Specifically, in each iteration after the first iteration, the system can generate new (and potentially corrected) predicted embeddings in the iteration, at least in part, based on the predicted protein structures generated in previous iterations.

[0042] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become more apparent from the specification, drawings, and claims. Attached Figure Description

[0043] Figure 1 An exemplary protein reconstruction system is shown.

[0044] Figure 2 An example architecture for protein-reconstructed neural networks is shown.

[0045] Figure 3 An example architecture for the update block of a protein-reconstructed neural network is shown.

[0046] Figure 4 An example architecture for a single embedded update block is shown.

[0047] Figure 5 An example architecture of paired embedded update blocks is shown.

[0048] Figure 6 This is a flowchart illustrating an example process of demasking a protein's masked representation using a protein reconstruction neural network.

[0049] The same reference numerals and names in the various figures indicate the same elements. Detailed Implementation

[0050] Figure 1 An exemplary protein reconstruction system 100 is shown. The protein reconstruction system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations that implement the systems, components and techniques described below.

[0051] System 100 is configured to receive a masked representation of protein 102, comprising: (i) a representation of the protein's amino acid sequence (i.e., amino acid sequence representation 104), and (ii) a representation of the protein's structure (i.e., protein structure representation 106). Each of the amino acid sequence representation 104 and the protein structure representation 106 is represented by a corresponding set of embeddings, and at least one of the embeddings of amino acid sequence representation 104, protein structure representation 106, or both is masked. An embedding may be referred to as being "masked," for example, if the embedding is a default (e.g., predefined) embedding, such as an embedding represented as a vector of zeros.

[0052] The amino acid sequence representation 104 may include a corresponding embedding at each position in the amino acid sequence of a protein. Each embedding of the amino acid sequence representation 104 that is not a masked embedding can represent an amino acid at the corresponding position in the amino acid sequence, for example, by identifying a one-hot embedding of an amino acid from a set of possible amino acids. The set of possible amino acids may include, for example, alanine, arginine, asparagine, etc., and the total number of amino acids in the set of possible amino acids may be, for example, 20.

[0053] The protein structure representation 106 may include a corresponding embedding for each of a set of "structural features" that characterize the protein structure.

[0054] For example, each structural feature in a set of structural features characterizing a protein structure can define the spatial distance (e.g., measured in angstroms) between specified atoms (e.g., α-carbon atoms) in the corresponding amino acid pair within the protein structure. In this example, the embedding representing the spatial distance between a pair of amino acids in the protein structure can be a one-hot embedding, which identifies the spatial distance between the pair of amino acids as included in a distance interval from a set of possible distance intervals. The set of possible distance intervals can be, for example, 0–2 angstroms, 2–4 angstroms, 4–6 angstroms, etc.

[0055] As another example, each structural feature in a set of structural features characterizing a protein structure can define the spatial location of an atom (e.g., an α-carbon atom) in the corresponding amino acid within the protein structure. Each embedding of a non-masking embedding in the protein structure representation can represent the spatial location of an atom in the corresponding amino acid within the protein structure, for example, as xyz coordinates in a predefined Cartesian coordinate system. As another example, structural features can define the backbone atom torsion angle of an amino acid residue in the protein.

[0056] Some embeddings in the amino acid sequence representation 104 and the protein structure representation 106 can be masked, for example, because they represent unknown information about the protein. For instance, if the protein's amino acid sequence is known but its structure is unknown, the amino acid sequence representation can be "complete" (i.e., no embeddings are masked), while all embeddings in the protein structure representation can be masked. As another example, if the protein's structure is known but its amino acid sequence is unknown, the protein structure representation can be complete, while all embeddings in the amino acid sequence representation can be masked. As yet another example, if both the protein's amino acid sequence and structure are only partially known, both the amino acid sequence representation and the protein structure representation can include some masked embeddings and others that are not masked.

[0057] System 100 uses a protein reconstruction neural network 200 to process amino acid sequence representation 104 and protein structure representation 106 to generate a corresponding predicted embedding for each masking embedding in masked protein representation 102. The predicted embedding 108 corresponding to the masking embedding in amino acid sequence representation 104 defines a prediction of the identity of amino acids at corresponding positions in the protein's amino acid sequence. The predicted embedding 108 corresponding to the masking embedding in protein structure representation 106 defines a prediction of corresponding structural features of the protein, such as the spatial distance between corresponding atoms in corresponding amino acid pairs within the protein. Generating the predicted embedding 108 can be understood as reconstructing the masking embedding in masked protein representation 102 using contextual information obtainable from the unmasked embedding in masked protein representation 102.

[0058] The protein reconstruction neural network 200 can have any suitable neural network architecture that enables it to perform the functions described herein, including any suitable neural network layers (e.g., fully connected layers, convolutional layers, attention layers, etc.) configured in any suitable configuration (e.g., as a sequential sequence of layers). Reference Figures 2-5 An example architecture of the protein reconstruction neural network 200 is described in more detail. However, existing protein reconstruction neural networks can also be adapted to use the described techniques, i.e., jointly processing the representation of amino acid sequences and protein structures, for example, iteratively.

[0059] Replacing the masking embedding in the masked protein representation 102 with the corresponding predicted embedding 108 yields the complete protein representation 110, i.e., ensuring that none of the embeddings in the complete protein representation 110 are masked. In other words, the complete protein representation defines a complete reconstruction of the protein's amino acid sequence (i.e., the identity of amino acids at each position in the amino acid sequence is specified and not masked), and a complete reconstruction of the protein structure (i.e., each structural feature in a set of structural features characterizing the protein structure is specified and not masked). The system 100 can then provide the complete protein representation 110 or a portion thereof (e.g., only the complete amino acid sequence representation, or only the complete protein structure representation) as output.

[0060] In some implementations, the system 100 incrementally replaces the masking embeddings in the masked protein representation 102 with corresponding predictive embeddings 108 in a series of iterations. More specifically, in each iteration, the system 100 processes the current masked protein representation 102 using a protein reconstruction neural network 200 to generate predictive embeddings 108, and updates the current masked protein representation 102 by replacing one or more remaining masking embeddings with corresponding predictive embeddings 108. The number of remaining masking embeddings in the masked protein representation 102 decreases in each iteration, and in the last iteration, the system 100 replaces all remaining masking embeddings in the masked protein representation 102 with the corresponding predictive embeddings 108 generated in the last iteration.

[0061] System 100 can determine, in any of a variety of ways, which masking embeddings in the masked protein representation 102 will be replaced by the corresponding predicted embeddings 108 in each iteration; several examples are below.

[0062] In one example, in each iteration, the system 100 may randomly select a predefined portion (e.g., 15%) of the remaining masking embeddings in the masking protein representation 102 to be replaced by the corresponding predicted embeddings 108. When the system 100 determines that there are fewer than a predefined threshold number of remaining masking embeddings in the masking protein representation 102, the system 100 may replace all remaining masking embeddings with the corresponding predicted embeddings 108 and terminate the iteration process.

[0063] In another example, in each iteration, the system 100 can determine which masked embeddings in the amino acid sequence representation 104 should be replaced by the corresponding predicted embeddings 108 based on the arrangement of the embeddings in the array. More specifically, the embeddings of the amino acid sequence representation 104 can be associated with an arrangement in a one-dimensional (1D) array, where the embedding at position i in the array corresponds to the amino acid at position i in the sequence of amino acids of the protein. In each iteration, if a masked embedding is adjacent to a non-masked embedding in the 1D array of embeddings of the amino acid sequence representation, the system 100 can determine that the masked embedding of the amino acid sequence representation 104 should be replaced by the corresponding predicted embedding 108.

[0064] In another example, in each iteration, the system 100 can determine which masking embeddings in the protein structure representation 106 should be replaced by the corresponding predicted embeddings 108 based on the arrangement of the embeddings in the array. More specifically, the embeddings of the protein structure representation 106 can be associated with an arrangement in a two-dimensional (2D) array, where the embedding at position (i,j) in the array corresponds to the amino acid pair at positions i and j in the amino acid sequence of the protein. In each iteration, if a masking embedding is adjacent to a non-masked embedding in the 2D array of the embeddings of the protein structure representation 106, the system 100 can determine that the masking embedding of the protein structure representation 106 should be replaced by the corresponding predicted embedding 108. An embedding can be understood as being "adjacent" to another embedding in the 2D array of the embeddings, for example, if they are adjacent in the same row or column of the 2D array.

[0065] Replacing the masking embedding in the masking protein representation 102 with the corresponding predictive embedding 108 in a series of iterations (rather than all at once, for example) allows the system 100 to incrementally accumulate contextual information, thereby generating a more accurate predictive embedding 108.

[0066] In some embodiments, the amino acid sequence representation 104 includes at least one masking embedding, and in each iteration of one or more iterations, the system 100 generates a corresponding predicted embedding 108 corresponding to each masking embedding in the amino acid sequence representation 104. For convenience, the amino acid sequence defined by replacing each masking embedding in the amino acid sequence representation 104 with the corresponding predicted embedding 108 generated in the current iteration will be referred to as the "current amino acid sequence". In each iteration, the system 100 can process the current amino acid sequence using a protein folding neural network to generate a predicted structure of a protein having the current amino acid sequence. The system can then provide the predicted protein structure as additional input to the protein reconstruction neural network 200 in the next iteration.

[0067] To provide the predicted protein structure as additional input to the protein reconstruction neural network 200 in the next iteration, the system 100 can generate a representation of the predicted protein structure. The representation of the predicted protein structure may include a corresponding embedding corresponding to each structural feature in a set of structural features characterizing the predicted protein structure. For example, as described above, the representation of the predicted protein structure may include a corresponding embedding representing the spatial distance between amino acid pairs in the predicted protein structure. The protein reconstruction neural network 200 can process the additional input defined by the representation of the predicted protein structure in any suitable manner. For example, the protein reconstruction neural network 200 may sum, average, or otherwise combine the representation of the predicted protein structure with the protein structure representation 106. The protein reconstruction neural network 200 can then process the resulting combined protein structure representation and amino acid sequence representation 104 according to the parameter values ​​of the protein reconstruction neural network 200 to generate a prediction embedding 108 for the next iteration, as described above.

[0068] Protein folding neural networks can have any suitable neural network architecture that enables them to perform the functions they describe (i.e., processing inputs including representations of amino acid sequences to generate predicted structures of proteins having amino acid sequences). In particular, protein folding neural networks can include any suitable neural network layers (e.g., fully connected layers, convolutional layers, attention layers, etc.) arranged in any suitable configuration (e.g., as layer sequences).

[0069] Providing the predicted protein structure corresponding to the current amino acid sequence to the protein reconstruction neural network 200 enables the system 100 to implicitly compare the predicted protein structure with the protein structure representation 106. This comparison allows the protein reconstruction neural network 200 to correct potential errors in the current amino acid sequence that cause the corresponding predicted protein structure to deviate from the protein structure representation 106, thereby improving the performance of the system 100 (e.g., prediction accuracy).

[0070] System 100 can generate a predicted protein structure corresponding to the current amino acid sequence in each iteration and provide it to the reconstructing neural network in the next iteration, as an alternative to or in combination with incrementally replacing the masking embeddings in the masked protein representation in each iteration. That is, in each iteration, the system can perform one or both of the following: (i) process the (temporary) amino acid sequence defined by replacing each masking embedding in the amino acid sequence representation 104 with the corresponding predicted embedding 108 generated in the iteration to generate the corresponding predicted protein structure provided to the reconstructing neural network in the next iteration, and (ii) replace the corresponding masking embeddings in the masked protein representation (e.g., the masking embeddings in the amino acid sequence representation 104, the protein structure representation 106, or both) with one or more predicted embeddings generated in the iteration.

[0071] The following describes several examples of possible applications of System 100 in more detail.

[0072] In one example, the system 100 can be used to predict protein structures corresponding to known amino acid sequences by processing the full amino acid sequence representation and the fully masked protein structure representation to generate a "demasked" protein structure representation. A demasked protein structure representation refers to generating a predictive embedding that defines the full protein structure representation.

[0073] In another example, the system 100 can be used to predict amino acid sequences corresponding to known protein structures by processing the full protein structure representation and the fully masked amino acid sequence representation to generate a "demasked" amino acid sequence representation. The demasked amino acid sequence representation refers to generating a predicted embedding that defines the full amino acid sequence representation. The known protein structure can be obtained experimentally using conventional physical techniques, such as X-ray crystallography, magnetic resonance imaging, or cryo-electron microscopy (cryo-EM).

[0074] In another example, the system 100 can be used to generate a complete protein representation of a protein having a partially known amino acid sequence and a partially known protein structure. Specifically, the system can process a partially masked amino acid sequence representation representing a partially known amino acid sequence and a partially masked protein structure representation representing a partially known protein structure to demask the amino acid sequence representation and the protein structure representation. A complete protein representation can be generated from the partially masked amino acid sequence and the partially masked protein structure, for example, to design a complete antibody starting from a known complement, which selectively binds to a specific antigen, particularly providing a therapeutic effect. For example, the antigen may contain a viral protein or a cancer cell protein. The designed antibody can then be synthesized.

[0075] To design a complete antibody starting from a known complement, the system 100 can be used to process a partially masked representation of the antibody's amino acid sequence and a partially masked representation of the antibody's structure to generate a complete representation of the antibody. The representation of the antibody's amino acid sequence may include a uniquely heated intercalation representing the known amino acid of the complement, and a masked amino acid intercalation representing each other amino acid in the antibody. The representation of the antibody's protein structure may include an intercalation representing the complement structure, and a masked intercalation representing the structure of the rest of the antibody (i.e., outside the complement). The complete representation of the antibody can define the corresponding type of each amino acid in the antibody, as well as the antibody's structure.

[0076] In another example, the system 100 can be used to generate a complete protein representation of a protein having: (i) a partially known amino acid sequence and a fully known protein structure, or (ii) a fully known amino acid sequence and a partially known protein structure. For example, the system can process a partially masked amino acid sequence representation and a complete protein structure representation to demask the amino acid sequence representation.

[0077] Generating a complete protein representation from a partially masked amino acid sequence and the complete structure of a protein can be performed, for example, to optimize certain characteristics of the protein, such as binding affinity, solubility, stability, aggregation tendency, or any other suitable characteristic. For example, starting with a protein having a known amino acid sequence and a known protein structure, a masked representation of the protein's amino acid sequence can be generated, i.e., where the identity of one or more amino acids in the protein is masked. System 100 can process the masked amino acid sequence representation and the complete structure representation of the protein to generate a corresponding score distribution for each masked amino acid, which defines a score for each amino acid type in a set of possible amino acid types. An example of generating a score distribution over amino acid types is described later. System 100 can then generate multiple "candidate" proteins, where the amino acid sequence of each candidate protein is determined by sampling the corresponding type of each masked amino acid according to the score distribution of amino acids. The value of the corresponding property (e.g., solubility, stability, binding affinity, or aggregation tendency) for each candidate protein can be predicted, and the candidate protein with the most desired (e.g., highest or lowest) value for the corresponding property can be selected. Values ​​for corresponding properties can be predicted from the amino acid sequence of a candidate protein using, for example, publicly available techniques or software tools. The selected candidate protein can be understood as a "mutant" of the original protein to optimize the desired properties (e.g., solubility, stability, or binding affinity). Therefore, a mutant protein with the desired properties can be synthesized by synthesizing a protein having the amino acid sequence of the selected candidate protein.

[0078] System 100 can receive masked protein representation 102, for example, from a remotely located user of protein reconstruction system 100 via an interface (e.g., application programming interface (API)) provided by protein reconstruction system 100 through a data communication network (e.g., the Internet). After generating complete protein representation 110, system 100 can provide complete protein representation 110 (or a portion thereof) to the remotely located user via the data communication network.

[0079] The training engine can train the parameters of a protein reconstruction neural network 200 on a set of training examples through multiple training iterations. Each training example can define a complete protein representation, including both the complete amino acid sequence representation and the complete protein structure representation.

[0080] In each training iteration, the training engine can sample one or more complete protein representations and generate a masked protein representation corresponding to each complete protein representation, for example, by randomly masking portions of the complete protein representation. The training engine can process each masked protein representation using System 100 based on the current parameter values ​​of the protein reconstruction neural network (as described above) to generate a corresponding predicted embedding for each masked embedding of the masked protein representation. The training engine can then determine the gradient of an objective function relative to the parameters of the protein reconstruction neural network, which measures the error between (i) the predicted embedding generated by System 100 and (ii) the corresponding embedding defined by the complete protein representation. The training engine can measure the error between the predicted embedding generated by System 100 and the corresponding embedding from the complete protein representation, for example, through cross-entropy loss or squared error loss. The training engine uses the gradient of the objective function to update the parameter values ​​of the protein reconstruction neural network using any appropriate update rule of any suitable gradient descent optimization technique (e.g., RMSprop or Adam).

[0081] Figure 2 An example architecture of a protein reconstruction neural network 200 is shown. The protein reconstruction neural network 200 is configured to process masked representations of proteins, which include: (i) an amino acid sequence representation 104, and (ii) a protein structure representation 106, wherein one or more embeddings of the masked protein representation are masked.

[0082] The amino acid sequence representation 104 includes a corresponding “single” embedding for each position in the amino acid sequence of the protein. Each embedding of the amino acid sequence representation 104 that is not a masked embedding can represent an amino acid at the corresponding position in the amino acid sequence, for example, by identifying a one-hot embedding of an amino acid from a set of possible amino acids. The protein reconstruction neural network 200 can optionally apply position-encoded data to each single embedding, wherein the position-encoded data applied to the single embedding is a function of the index of the position in the amino acid sequence corresponding to the single embedding. For example, the protein reconstruction neural network 200 can apply sinusoidal position-encoded data to each single embedding, as described in A. Vaswani et al., “Attention is all you need,” 21st Conference on Neural Informational Processing Systems (NIPS2017).

[0083] The protein structure representation 106 includes corresponding “pair” embeddings (e.g., NxN pairs) for each pair of amino acids in the protein. Each pair of embeddings that is not masked can represent the spatial distance between the corresponding amino acid pairs, for example by a one-hot embedding, which identifies the spatial distance between amino acid pairs as included in a distance interval from a set of possible distance intervals.

[0084] The protein-reconstructed neural network 200 includes the sequence 206-AN of the updated block. Throughout the specification, "block" refers to a portion of a neural network, such as a subnetwork of a neural network that includes one or more neural network layers.

[0085] Each update block in the protein reconstruction neural network is configured to receive a block input comprising a set of single embeddings and a set of paired embeddings, and to process the block input to generate a block output comprising updated single embeddings and updated paired embeddings.

[0086] The protein reconstruction neural network 200 provides a first update block (i.e., in the update block sequence) with single embeddings 202 and paired embeddings 204 included in the network input of the protein reconstruction neural network 200. The first update block processes the single embeddings 202 and paired embeddings 204 to generate updated single embeddings and updated paired embeddings.

[0087] For each update block following the first update block, the protein reconstruction neural network 200 provides the update block with single and paired embeddings generated by the previous update block, and provides the updated single and paired embeddings generated by the update block to the next update block.

[0088] The protein reconstruction neural network 200 gradually enriches the information content of the single embedding 202 and the paired embedding 204 by repeatedly updating the sequence 206-AN of the update block.

[0089] The last update block in the update block sequence outputs a set of updated single embeddings 208 and a set of updated paired embeddings 210. Each updated single embedding 208 may include a corresponding "soft" score for each amino acid in the possible set of amino acids, and each updated paired embedding may include a corresponding "soft" score for each distance interval in the possible set of distance intervals.

[0090] The protein reconstruction neural network 200 can identify the predicted embedding 108 of the masked single embedding from the amino acid sequence representation 104 as a one-hot embedding representing the amino acid associated with the highest soft score through a corresponding updated single embedding 208. Similarly, the protein reconstruction neural network 200 can identify the predicted embedding 108 of the masked paired embedding from the protein structure representation 106 as a one-hot embedding, which represents the distance interval associated with the highest soft score of the corresponding updated paired embedding 210.

[0091] Figure 3 An example architecture of the update block 300 of the protein reconstruction neural network 200 is shown, i.e., as referenced Figure 2 As described.

[0092] Update block 300 receives block input including current single embedding 302 and current paired embedding 304, and processes the block input to generate updated single embedding 306 and updated paired embedding 308.

[0093] Update block 300 includes a single embedded update block 400 and a pair of embedded update blocks 500.

[0094] The single embedding update block 400 updates the current single embedding using the current paired embedding 304, and the paired embedding update block 500 updates the current paired embedding 304 using the updated single embedding (i.e., the single embedding generated by the single embedding update block 400).

[0095] Typically, single embeddings and paired embeddings can encode complementary information. Single embedding update block 400 uses the complementary information encoded in the paired embeddings to enrich the information content of the single embedding, and paired embedding update block 500 uses the complementary information encoded in the single embeddings to enrich the information content of the paired embeddings. As a result of this enrichment, the updated single embeddings and updated paired embeddings encode more relevant information compared to the masking embeddings of the masking protein representation.

[0096] Update block 300 is described herein as first updating the current single embedding 302 using the current paired embedding 304, and then updating the current paired embedding 304 using the updated single embedding 306. This description should not be construed as limiting the update block to performing operations in this order; for example, the update block may first update the current paired embedding using the current single embedding, and then update the current single embedding using the updated paired embedding.

[0097] Update block 300 is described herein as comprising a single-embedded update block 400 (i.e., which updates the current single embedding) and a pairwise-embedded update block 500 (i.e., which updates the current pairwise embedding). This description should not be construed as limiting update block 300 to containing only a single-embedded update block or only pairwise-embedded update blocks. For example, update block 300 may include multiple single-embedded update blocks that update the single embedding multiple times before providing it to pairwise update blocks for updating the current pairwise embedding. As another example, update block 300 may include multiple pairwise update blocks that update pairwise embeddings multiple times using a single embedding.

[0098] The single embedded update block 400 and the paired embedded update blocks 500 can have any suitable architecture that enables them to perform the functions they describe.

[0099] In some implementations, a single embedding update block 400, a paired embedding update block 500, or both include one or more “self-attention” blocks. As used throughout this document, a self-attention block generally refers to a neural network block that updates a set of embeddings (i.e., receives a set of embeddings and outputs updated embeddings). To update a given embedding, a self-attention block may determine a corresponding “attention weight,” such as a similarity measure, between the given embedding and each of one or more selected embeddings (e.g., the received set of embeddings), and then update the given embedding using (i) the attention weights and (ii) the selected embeddings. For example, the updated embedding may include a sum of values, each derived from one of the selected embeddings and each value weighted by a corresponding attention weight. For convenience, it can be said that a self-attention block updates a given embedding using attention “over” the selected embeddings.

[0100] For example, a self-attention block can receive a set of input embeddings. This refers to the number of amino acids in the N protein, and in order to update the embedded x i Self-attention blocks can determine attention weights. Where a i,j x represents i and x j The attention weights between them are as follows:

[0101]

[0102]

[0103] Its W q W k `x` and `c` are the parameter matrices learned, `softmax(·)` denotes the soft maximum normalization operation, and `c` is a constant. Using attention weights, the self-attention layer can embed `x`... i Updated to:

[0104]

[0105] Among them W v It is the parameter matrix for learning. (W) q x i It can be referred to as the input embedding x i "Query embedding", W k x j It can be referred to as the input embedding x i "Key embedding", and W v x j It can be referred to as the input embedding x i (value embedding).

[0106] Parameter matrix W q ("Query Embedding Matrix"), W k (“Key Embedding Matrix”) and W v The “value embedding matrix” is the trainable parameter of the self-attention block. The parameters of any self-attention block included in the single embedding update block 400 and the paired embedding update block 500 can be understood as the parameters of update block 300, which can be trained as a reference. Figure 1 This is part of the end-to-end training of the protein reconstruction system 100 described. Typically, the (trained) parameters of the query, key, and value embedding matrices are different for different self-attention blocks, for example, such that self-attention blocks included in a single embedding update block 400 can have different query, key, and value embedding matrices with parameters different from those included in a pairwise embedding update block 500.

[0107] In some implementations, the single embedding update block 400, the paired embedding update block 500, or both include one or more self-attention blocks conditioned on paired embeddings (depending on the paired embeddings), i.e., one or more self-attention blocks implementing self-attention operations conditioned on paired embeddings. To condition the self-attention operation on paired embeddings, the self-attention block can process the paired embeddings to generate a corresponding "attention bias" for each attention weight; then, each attention weight can be biased by the corresponding attention bias. For example, in addition to determining the attention weights according to equations (1)-(2)... In addition, self-attention blocks can also generate a corresponding set of attention biases. Where b i,j x represents i and x j Attention bias between pairs of embeddings. Self-attention blocks can be achieved by applying the learned parameter matrix to the pairwise embeddings h. i,j(i.e., the amino acid pairs in the protein indexed by (i,j)) to generate attention bias b i,j .

[0108] Self-attention blocks can define a set of "biased attention weights". Where c i,j Indicates in x i and x j The biased attention weights are determined, for example, by summing the attention weights and attention biases (or otherwise combining them). For example, a self-attention block can determine the embedding x. i and x j The bias attention weights c between i,j for:

[0109] c i,k =a i,j +b i,j

[0110] Where a i,j It is x i With x j The attention weights between them, and b i,j It is x i With x j Attention bias between inputs. Self-attention blocks can use the biased attention weights to update each input embedding x. i ,For example:

[0111]

[0112] Among them W v It is the parameter matrix for learning.

[0113] Typically, pairwise embeddings encode information characterizing protein structure and the relationships between amino acid pairs within that structure. Applying a self-attention operation conditioned on pairwise embeddings to a set of input embeddings allows the input embeddings to be updated in a manner notified by the protein structure information encoded in the pairwise embeddings. Update blocks in protein reconstruction neural networks can use self-attention blocks conditioned on pairwise embeddings to update and enrich both single embeddings and pairwise embeddings themselves.

[0114] Optionally, the self-attention block can have multiple "heads," each generating a corresponding updated embedding for each input embedding, i.e., such that each input embedding is associated with multiple updated embeddings. For example, each head can have a parameter matrix W described by reference equations (1)-(4). q W k and W vDifferent values ​​of the input embedding are used to generate updated embeddings. Self-attention blocks with multiple heads can implement a "gating" operation to combine updated embeddings generated by the heads from the input embeddings; that is, to generate a single updated embedding corresponding to each input embedding. For example, a self-attention block can use one or more neural network layers (e.g., a fully connected neural network layer) to process the input embeddings to generate corresponding gating values ​​for each head. The self-attention block can then combine updated embeddings corresponding to the input embeddings based on the gating values. For example, a self-attention block can generate an input embedding x... i The updated embedding:

[0115]

[0116] Where k is the index of the head, α k It is the gating value of the head k, and It is the embedding of x from the head k pairs of inputs i The generated updated embedding.

[0117] refer to Figure 4 Describe an example architecture for updating a single embedding block 400 using self-attention blocks conditioned on pairwise embeddings.

[0118] refer to Figure 5 Describes an example architecture for a pairwise embedding update block 500 using self-attention blocks conditioned on pairwise embeddings. References Figure 5 The example pairwise embedding update block described hereby computes the outer product of the updated single embedding (hereinafter referred to as the outer product mean), adds the result of the outer product mean to the current pairwise embedding (projected to the pairwise embedding dimension if necessary), and processes the current pairwise embedding using a self-attention block conditioned on the current pairwise embedding, updating the current pairwise embedding based on the updated single embedding.

[0119] Figure 4 An example architecture of a single embedding update block 400 is shown. The single embedding update block 400 is configured to receive the current single embedding 302 and update the current single embedding 302 (at least in part) based on the current paired embeddings.

[0120] To update the current single embedding 302, the single embedding update block 400 uses a self-attention operation conditioned on the current pair of embeddings to update the single embedding. More specifically, the single embedding update block 400 provides the single embedding to the self-attention block 402 conditioned on the current pair of embeddings, for example, as referenced. Figure 3The process is described above, to generate an updated single embedding. Optionally, the single embedding update block can add the input of the self-attention block 402 to the output of the self-attention block 402. The self-attention block 402 is adjusted for the current pair embeddings so that the single embedding update block 400 can use information from the current pair embeddings to enrich the current single embedding 302.

[0121] Then, the single embedding update block processes the current single embedding 302 using, for example, a transition block that applies one or more fully connected neural network layers to the current single embedding. Optionally, the single embedding update block 400 may add the input of the transition block 404 to the output of the transition block 404.

[0122] The single embedding update block can output the updated single embedding 306 obtained by the operations performed by the self-attention block 402 and the transition block 404.

[0123] Figure 5 An example architecture for a pairwise embedding update block 500 is shown. The pairwise embedding update block 500 is configured to receive the current pairwise embedding 304 and update the current pairwise embedding 304 (at least in part) based on the updated single embedding 306.

[0124] In the following description, pairwise embeddings can be understood as arranged in an N×N array, such that the embedding at position (i,j) in the array is a pairwise embedding of amino acids at positions i and j in the amino acid sequence.

[0125] To update the current pairwise embedding 304, the pairwise embedding update block 500 applies the outer product mean operation 502 to the updated single embedding 306 and adds the result of the outer product mean operation 502 to the current pairwise embedding 304.

[0126] The outer product mean operation defines a series of operations that, when applied to a set of single embeddings represented as a 1×N array, generate an N×N array of embeddings, i.e., where N is the number of amino acids in the protein. The current pairwise embedding 304 can also be represented as a pairwise embedding of an N×N array, and adding the result of the outer product mean 502 to the current pairwise embedding 304 means summing the embeddings of the two N×N arrays.

[0127] To compute the outer product mean, pairwise embedding update blocks generate tensor A(·), for example, given by:

[0128] A(res1,res2,ch1,ch2)

[0129] =LeftAct(res1,ch1)·RightAct(res2,ch2) (6)

[0130] Where res1, res2 ∈ {1,…,N}, ch1, ch2 ∈ {1,…,C}, where C is the number of channels in each single embedding, LeftAct(res1,ch1) is a linear operation (e.g., a projection defined by matrix multiplication) applied to channel ch1 of the single embedding indexed by res1, and RightAct(res2,ch2) is a linear operation (e.g., a projection defined by matrix multiplication) applied to channel ch2 of the single embedding indexed by res2. The result of the outer product mean is generated by flattening and linearly projecting the dimensions (ch1,ch2) of tensor A. Optionally, the pairwise embedding update block may perform one or more layer normalization operations (e.g., as described in “Layer Normalization” by Jimmy Lei Ba et al., arXiv:1607.06450) as part of the calculation of the outer product mean.

[0131] Typically, the updated single embedding 306 encodes information about the amino acids in the protein's amino acid sequence. The pairwise embedding update block 500 can enhance the information content of the current pairwise embedding by incorporating the information encoded in the updated single embedding into the current pairwise embedding (i.e., by means of the outer product 502).

[0132] After updating the current pairwise embeddings 304 with the updated single embeddings (i.e., via the outer product mean 502), the pairwise embedding update block 308 updates the current pairwise embeddings in each row of the current pairwise embeddings arrangement to an N×N array using a self-attention operation conditioned on the current pairwise embeddings (i.e., a "row-by-row" self-attention operation). More specifically, the pairwise embedding update block 500 feeds each row of the current pairwise embeddings to a "row-by-row" self-attention block 504, also conditioned on the current pairwise embeddings, for example, as referenced Figure 3 This is done so that updated pairwise embeddings are generated for each row. Optionally, the pairwise embedding update block can add the input of the row-by-row self-attention block 504 to the output of the row-by-row self-attention block 504.

[0133] Then, the pairwise embedding update block 500 uses a self-attention operation (i.e., a "column-by-column" self-attention operation) also conditioned on the current pairwise embeddings to update the current pairwise embeddings in each column of the N×N array. More specifically, the pairwise embedding update block 500 feeds each column of the current pairwise embeddings to a "column-by-column" self-attention block 506 also conditioned on the current pairwise embeddings, in order to generate updated pairwise embeddings for each column. Optionally, the pairwise embedding update block can add the input of the column-by-column self-attention block 506 to the output of the column-by-column self-attention block 506.

[0134] Then, the pairwise embedding update block 500 processes the current pairwise embedding using the transition block 508. For example, the transition block 508 applies one or more fully connected neural network layers to the current pairwise embedding. Optionally, the pairwise embedding update block 500 may add the input of the transition block 508 to the output of the transition block 508.

[0135] The pairwise embedding update block can output the updated pairwise embedding 308 obtained by the operations performed by the row-wise self-attention block 504, the column-wise self-attention block 506, and the transformation block 508.

[0136] Figure 6 This is a flowchart of an example process 600 for demasking a masked representation of a protein using a protein reconstruction neural network. For convenience, process 600 will be described as being executed by a system of one or more computers located in one or more locations. For example, a protein reconstruction system appropriately programmed according to this specification, such as... Figure 1 The protein reconstruction system 100 can execute process 600.

[0137] The system receives a masked representation of a protein (602). The masked representation of the protein includes: (i) a representation of the amino acid sequence of the protein, comprising a set of embeddings each corresponding to a corresponding position in the amino sequence of the protein, and (ii) a representation of the structure of the protein, comprising a set of embeddings each corresponding to a corresponding structural feature of the protein. At least one of the embeddings included in the masked representation of the protein is masked.

[0138] Steps 604-610, which are described below, can be performed in each iteration of one or more iterations.

[0139] The system uses a protein reconstruction neural network to process the masked representation of a protein to generate corresponding predictive embeddings (604) that correspond to one or more masking embeddings included in the masked representation of the protein. The predictive embeddings corresponding to the masking embeddings in the representation of the protein's amino acid sequence define predictions of the identity of amino acids at corresponding positions in the amino acid sequence. The predictive embeddings corresponding to the masking embeddings in the representation of the protein's structure define predictions of the corresponding structural features of the protein.

[0140] Optionally, if the current iteration is after the first iteration, the system can provide the predicted protein structure generated in the previous iteration (as will be described in more detail in steps 608-610) as additional input to the protein reconstruction neural network, i.e., in addition to the masked representation of the protein.

[0141] In some implementations, the system can update the protein masking representation by replacing a so-called proper subset (i.e., a subset excluding all masking embeddings) of the masking embeddings in the protein masking representation with the corresponding predictive embeddings (606). The system can then proceed to the next iteration (e.g., by returning to step 604), and in the next iteration, the system can process the updated protein masking representation using a protein reconstruction neural network to generate corresponding predictive embeddings corresponding to one or more remaining masking embeddings included in the protein masking representation.

[0142] In some implementations, the representation of the protein's amino acid sequence includes one or more masking embeddings, and the system identifies a predicted amino acid sequence of the protein, wherein each masking embedding in the amino acid sequence representation is replaced by a corresponding predicted embedding. The system can process the predicted amino acid sequence using a protein folding neural network to generate data defining a predicted protein structure that defines the predicted amino acid sequence (608). Any protein folding neural network can be used, for example, based on publicly available methods or software such as AlphaFold2 (available open source). The system can then proceed to the next iteration (i.e., by returning to step 604), and in the next iteration, the system can provide the predicted protein structure as additional input to the protein reconstruction neural network (i.e., in addition to the masked protein representation) (610). The protein reconstruction neural network can then process the predicted protein structure and the masked protein representation to generate new predicted embeddings in the next iteration.

[0143] In some implementations, the system may perform all of steps 606 (i.e., updating the masked representation of the protein using the predicted embedding) and steps 608-610 (i.e., processing the predicted amino acid sequence to generate the predicted protein structure and providing the predicted protein structure as additional input to the protein reconstruction neural network, as described above) in one or more iterations.

[0144] The system can determine when the iterative process is complete, for example, after each masking embedding in the masked protein representation has been replaced by the corresponding predicted embedding. The system can then provide a complete protein representation as output, where all masking embeddings of the masked protein representation have been replaced by the corresponding predicted embeddings generated during the series of iterations.

[0145] Typically, this system can be used to determine the predicted structure of a (target) protein, peptide ligand, or antibody by generating a predicted embedding that defines a complete representation of the protein's structure. This can be achieved, for example, when the masked representation of the protein includes a complete representation of the amino acid sequence of the (target) protein, peptide ligand, or antibody, and the representation of the protein's structure includes a completely masked representation of the structure of the (target) protein, peptide ligand, or antibody.

[0146] The following describes some other applications of this system.

[0147] This system can be used to obtain ligands, such as ligands for drugs or industrial enzymes. For example, methods for obtaining ligands may include obtaining the target amino acid sequence of a target protein and using the target amino acid sequence to determine the (tertiary) structure of the target protein. The method may include evaluating the interaction between one or more candidate ligands and the structure of the target protein, and selecting one or more candidate ligands as ligands based on the results. Evaluation of the interaction may include evaluating the binding of the candidate ligand to the target protein structure, for example, to identify ligands that bind with sufficient affinity for biological effects. Candidate ligands may be enzymes. Evaluation may include evaluating the affinity between the candidate ligand and the target protein, or evaluating the selectivity of the interaction. Candidate ligands may be derived from a database of candidate ligands, or by modifying ligands in the database, or by assembling or optimizing candidate ligands stepwise or iteratively. Evaluation may be performed, for example, using computer-aided methods, where a graphical model of the candidate ligand and the target protein structure is displayed for user operation, or evaluation may be performed partially or fully automatically, for example using standard protein-ligand docking software. The evaluation may include determining the interaction score of candidate ligands, for example, depending on the strength or specificity of the interaction, such as depending on the binding free energy score. Candidate ligands can be selected based on the score.

[0148] In some embodiments, the target protein includes a receptor or enzyme, and the ligand is an agonist or antagonist of the receptor or enzyme. In some embodiments, the method can be used to identify the structure of a cell surface marker. This can then be used to identify ligands that bind to the cell surface marker, such as antibodies or labels, such as fluorescent labels. This can be used to identify and / or treat cancer cells. In some embodiments, candidate ligands may include small molecule ligands, such as organic compounds with a molecular weight <900 Daltons. In some other embodiments, candidate ligands may include peptide ligands, i.e., peptide ligands defined by an amino acid sequence.

[0149] Some implementations of this system can be used to determine the structure of candidate peptide ligands (e.g., ligands for drugs or industrial enzymes). Their interaction with the target protein structure can then be evaluated; the target protein structure can be determined using computer-implemented methods as described herein or using conventional physical research techniques such as X-ray crystallography and / or magnetic resonance imaging.

[0150] Therefore, this system can be used to obtain peptide ligands, such as molecules or their sequences. This may include obtaining the amino acid sequences of one or more candidate peptide ligands and using the amino acid sequences of the candidate peptide ligands as the amino acid sequence to perform the methods described above to determine the (tertiary) structure of the candidate peptide ligands. The structure of the target protein can be obtained, for example, by electronic (silico) or physical adjustment, and the interaction between the structure of each of the one or more candidate peptide ligands and the target protein structure can be evaluated. One of the one or more candidate peptide ligands can be selected as the peptide ligand based on the evaluation results. As previously described, evaluating the interaction may include evaluating the binding of the candidate peptide ligand to the target protein structure, such as identifying ligands that bind with sufficient affinity for biological effects, and / or evaluating the association of the candidate peptide ligand with the target protein structure that affects the function of the target protein (e.g., an enzyme), and / or evaluating the affinity between the candidate peptide ligand and the target protein structure, or evaluating the selectivity of the interaction. In some embodiments, the peptide ligand may be an aptamer. Similarly, the peptide candidate ligand can be selected based on the peptide candidate ligand with the highest affinity.

[0151] As previously described, the selected polypeptide ligand may comprise a receptor or an enzyme, and the ligand may be an agonist or antagonist of the receptor or enzyme. In some embodiments, the polypeptide ligand may comprise an antibody, and the target protein may comprise an antibody target, i.e., an antigen, such as a virus, particularly a viral capsid protein, or a protein expressed on cancer cells. In these embodiments, the antibody binds to the antigen to provide a therapeutic effect. For example, the antibody may bind to the antigen and act as an agonist of a specific receptor; or, the antibody may prevent another ligand from binding to the target and thus prevent activation of the associated biological pathway.

[0152] Such methods may include synthesis, i.e., the preparation of small molecule or peptide ligands. The ligands can be synthesized using any conventional chemical technique and / or may be already available, for example, derived from a compound library or synthesized using combinatorial chemistry.

[0153] This method can further include testing the bioactivity of the ligand in vitro and / or in vivo. For example, the ADME (absorption, distribution, metabolism, excretion) and / or toxicological properties of the ligand can be tested to screen out unsuitable ligands. The tests may include, for example, contacting the candidate small molecule or peptide ligand with the target protein and measuring changes in protein expression or activity.

[0154] In some embodiments, the candidate (peptide) ligand may include: isolated antibodies, fragments of isolated antibodies, monovariable domain antibodies, bispecific or multispecific antibodies, multivalent antibodies, bivariable domain antibodies, immunoconjugates, fibronectin molecules, adnectin, DARPin, avimers, affibosomes, anticarriers, affilins, protein epitope mimics, or combinations thereof. The candidate (peptide) ligand may contain antibodies with mutated or chemically modified amino acid Fc regions, which, for example, inhibit or reduce ADCC (antibody-dependent cytotoxicity) activity and / or increase half-life compared to wild-type Fc regions.

[0155] Misfolded proteins are associated with many diseases. This system can be used to identify the presence of protein misfolding diseases. This may include obtaining the amino acid sequence of a protein and performing the method described above to use the protein's amino acid sequence to determine the protein's structure, for example, by obtaining the structure of a version of the protein obtained from a human or animal body using conventional (physical) methods, and then comparing the protein's structure with the structure of the version obtained from the body, identifying the presence of a protein misfolding disease based on the result. That is, misfolding of a version of a protein from the body can be determined by comparing it with a determined structure. Typically, identifying the presence of a protein misfolding disease may include obtaining the amino acid sequence of a protein, using the protein's amino acid sequence to determine the protein's structure, as described herein, and comparing the protein's structure with the structure of a baseline version of the protein, identifying the presence of a protein misfolding disease based on the comparison result. For example, the structures being compared may be the structures of mutant and wild-type proteins. In an embodiment, a wild-type protein may be used as the baseline version, but in principle either can be used as the baseline version.

[0156] In some implementations, the system can be used to identify active / binding / blocking sites on target proteins from their amino acid sequences.

[0157] This specification uses the term "configuration" in conjunction with system and computer program components. For a system of one or more computers configured to perform a specific operation or action, this means that software, firmware, hardware, or a combination thereof have been installed on the system, which, in operation, causes the system to perform the operation or action. For one or more computer programs configured to perform a specific operation or action, this means that one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0158] Embodiments of the subject matter and functional operation described in this specification may be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware (including the structures disclosed in this specification and their equivalents), or combinations thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or control of the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof. Alternatively or additionally, program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by the data processing device.

[0159] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of devices, apparatuses, and machines used for processing data, including programmable processors, computers, or multiple processors or computers. The apparatus may also be or further include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0160] A computer program (which may also be referred to or described as a program, software, software application, applet, module, software module, script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not necessarily, correspond to a file in a file system. A program may be stored as a portion of a file containing other programs or data, for example, as one or more scripts stored in a markup language document, as a single file dedicated to the program in question, or as multiple harmonized files, for example, as a file storing one or more modules, subroutines, or code portions. A computer program can be deployed to execute on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected via a data communication network.

[0161] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in others, multiple engines may be installed and run on the same or multiple computers.

[0162] The processes and logic flows described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform functions by manipulating input data and generating outputs. The processes and logic flows can also be executed by special-purpose logic circuitry (e.g., FPGA or ASIC), or by a combination of special-purpose logic circuitry and one or more programmable computers.

[0163] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into special-purpose logic circuitry. Typically, a computer will also include or be operatively coupled to one or more mass storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, for receiving data from, or for transferring data to, or both. However, a computer does not necessarily need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0164] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CDROM and DVD-ROM disks.

[0165] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's device in response to a request received from a web browser. Furthermore, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and receiving response messages from the user in return.

[0166] The data processing apparatus used to implement machine learning models may also include, for example, dedicated hardware accelerator units for processing the common and computationally intensive parts of machine learning training or production, namely inference and workloads.

[0167] Machine learning models can be implemented and deployed using machine learning frameworks such as TensorFlow, Microsoft Cognitive Toolkit, Apache Singa, or Apache MXNet.

[0168] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components (e.g., as a data server), or middleware components (e.g., an application server), or front-end components (e.g., a client computer having a graphical user interface, web browser, or applet through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0169] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship is established by means of computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server sends data (e.g., HTML pages) to a user of the device, for example, for the purpose of displaying data to and receiving user input from a user interacting with the device acting as a client. Data generated at the user's device, such as the result of user interaction, may be received at the server from the device.

[0170] While this specification contains numerous details of specific implementation, these should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described in the context of separate embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, one or more features from a claimed combination may be removed from the combination in some cases, and the claimed combination may be for sub-combinations or variations thereof.

[0171] Similarly, although operations are depicted in the accompanying drawings and described in a specific order in the claims, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or to perform all of the shown operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the division of various system modules and components in the above embodiments should not be construed as requiring such division in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0172] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method, executed by one or more data processing devices, for demasking a masked representation of a protein using a protein reconstruction neural network, the method comprising: Masking representation of the receiving protein, The masked representation of the protein includes: (i) a representation of the protein's amino acid sequence, comprising multiple embeddings, each embedding corresponding to a specific position in the protein's amino acid sequence; and (ii) a representation of the protein's structure, comprising multiple embeddings, each embedding corresponding to a specific structural feature of the protein. Wherein, at least one of the embeddings included in the masked representation of the protein is masked; and A protein reconstruction neural network is used to process the masked representation of a protein to generate a corresponding predicted embedding that corresponds to one or more masked embeddings included in the masked representation of the protein. Among them, the predicted embedding of the masking embedding in the representation of the amino acid sequence of the protein defines the prediction of the identity of amino acids at corresponding positions in the amino acid sequence. Among them, the prediction embedding, corresponding to the masking embedding in the representation of protein structure, defines the prediction of the corresponding structural features of the protein. The representation of the amino acid sequence of the protein includes multiple single embeddings, each of which corresponds to a specific position in the amino acid sequence of the protein. The representation of the protein structure includes multiple pairs of embeddings, each pair of embeddings corresponding to a specific position pair in the protein's amino acid sequence; The protein-reconstructed neural network includes updating block sequences; Each update block has a corresponding set of update block parameters and performs the following operations: Receive the current paired embeddings and the current single embedding; Based on the current pairwise embeddings, update the current single embedding according to the value of the update block parameter of the update block; and Based on the updated single embedding, update the current pairwise embedding according to the value of the update block parameter of the update block; and The final update block in the update block sequence generates the final paired embedding and the final single embedding.

2. The method according to claim 1, further comprising: The masking representation of a protein is updated by replacing an appropriate subset of the masking embeddings in the masking representation with the corresponding predictive embeddings. The protein reconstruction neural network is used to process the updated masked representation of the protein to generate a corresponding predicted embedding that corresponds to one or more remaining masked embeddings included in the masked representation of the protein.

3. The method according to claim 1 or 2, wherein, The representation of the amino acid sequence of the protein includes one or more masking embeddings, and further includes: A protein folding neural network is used to process the predicted amino acid sequence of a protein to generate data defining the predicted protein structure, wherein the predicted amino acid sequence of the protein is defined by replacing each masking embedding in the representation of the amino acid sequence with the corresponding predicted embedding; and The protein reconstruction neural network processes both (i) the masked representation of the protein and (ii) the predicted protein structure based on the predicted amino acid sequence to generate new predicted embeddings corresponding to one or more masked embeddings included in the masked representation of the protein.

4. The method according to claim 1 or 2, wherein, Each masking embedding included in the masking representation of a protein is a default embedding.

5. The method according to claim 4, wherein, The default embedding includes the zero vector.

6. The method according to claim 1 or 2, wherein, Each predicted embedding, corresponding to the masking embedding in the representation of the protein structure, defines a prediction of the spatial distance between corresponding amino acid pairs in the protein structure.

7. The method according to claim 1 or 2, wherein, At least one of the embedded representations of the amino acid sequence of the protein is masked.

8. The method according to claim 1 or 2, wherein, At least one of the embeddings representing the structure of the protein is masked.

9. The method according to claim 1 or 2, wherein, The protein reconstruction neural network performs further operations, which include masking each of one or more single embeddings in the representation of the protein's amino acid sequence: Based on the corresponding final single embedding generated from the final update block, a predicted embedding is generated for the masked single embedding.

10. The method according to claim 1 or 2, wherein, The protein reconstruction neural network performs further operations, which include masking each of one or more pairs of embeddings in the representation of the protein's amino acid sequence: Based on the corresponding final pairwise embeddings generated from the final update block, predictive embeddings for the masked pairwise embeddings are generated.

11. The method according to claim 1 or 2, wherein, Updating the current single embedding based on the current pairwise embeddings includes: The current single embedding is updated using attention to the current single embedding, wherein the attention is conditioned on the current pairwise embedding.

12. The method according to claim 11, wherein, Updating the current single embedding using attention to the current single embedding includes: Generate multiple attention weights based on the current single embedding; Based on the current pairwise embeddings, generate the corresponding attention bias for each attention weight; Based on attention weights and attention biases, multiple biased attention weights are generated; and Based on biased attention weights, the current single embedding is updated using attention to the current single embedding.

13. The method according to claim 1 or 2, wherein, Updating the current pairwise embedding based on the updated single embedding includes: Apply the transformation operation to the updated single embedding; and The current pairwise embedding is updated by adding the result of the transformation operation to the current pairwise embedding.

14. The method according to claim 13, wherein, The transformation operation includes the outer product operation.

15. The method according to claim 13, wherein, Updating the current pairwise embedding based on the updated single embedding also includes adding the result of the transformation operation to the current pairwise embedding: The current pairwise embedding is updated using attention to the current pairwise embedding, wherein the attention is conditioned on the current pairwise embedding.

16. A method for obtaining a ligand, wherein, The ligand is a drug or an industrial enzyme ligand, and the method includes: Perform the method according to any one of claims 1-15 to determine the predicted structure of the target protein by generating a predicted embedding that defines a complete protein structure representation of the target protein, wherein the masked representation of the protein includes a complete representation of the amino acid sequence of the target protein, and wherein the representation of the structure of the protein includes a completely masked representation of the structure of the target protein. Evaluate the interaction between one or more candidate ligands and the predicted structure of the target protein; and Based on the evaluation results, one or more candidate ligands are selected as ligands.

17. A method for obtaining a ligand, wherein, The ligand is a drug or an industrial enzyme ligand, and the method includes: Perform the method according to any one of claims 1-15 to determine the predicted structure of each of a plurality of target proteins by generating a predicted embedding that defines a complete protein structure representation of each target protein, wherein, for each target protein, the masked representation of the protein includes a complete representation of the amino acid sequence of the target protein, and wherein the representation of the structure of the protein includes a completely masked representation of the structure of the target protein. Evaluate the interaction between one or more candidate ligands and the predicted structure of each target protein; and Select one or more candidate ligands as ligands to i) obtain ligands that interact with each target protein, or ii) obtain ligands that interact with only one target protein.

18. The method according to claim 16 or 17, wherein, The target protein includes a receptor or an enzyme, and wherein the ligand is an agonist or antagonist of the receptor or enzyme.

19. A method for obtaining polypeptide ligands, wherein, The ligand is a drug or an industrial enzyme ligand, and the method includes: For each of one or more candidate polypeptide ligands, the method according to any one of claims 1-15 is performed to determine the predicted structure of the candidate polypeptide ligand by generating a predicted embedding that defines a complete protein structure representation of the candidate polypeptide ligand, wherein, for each of the one or more candidate polypeptide ligands, the masked representation of the protein includes a complete representation of the amino acid sequence of the candidate polypeptide ligand, and wherein, the representation of the structure of the protein includes a completely masked representation of the structure of the candidate polypeptide ligand. Obtain the target protein structure; Evaluate the interaction between the predicted structure of each of one or more candidate polypeptide ligands and the target protein structure; and Based on the evaluation results, one of one or more candidate peptide ligands is selected as the peptide ligand.

20. The method according to claim 19, wherein, The target protein includes a receptor or an enzyme, and wherein the ligand is an agonist or antagonist of the receptor or enzyme, or wherein the polypeptide ligand includes an antibody, and the target protein includes an antigen, and wherein the antibody binds to the antigen to provide a therapeutic effect.

21. A method for obtaining an antibody against an antigen, the method comprising: Perform the method according to any one of claims 1-15 to determine the predicted structure and amino acid sequence of the antibody by generating a predicted embedding, said predicted embedding defining i) a complete amino acid sequence representation of the antibody, and ii) a complete protein structure representation of the antibody. The masked representation of the protein includes a representation of the complement of the antibody that binds to the antigen, and includes i) a partially masked representation of the amino acid sequence of the antibody, and ii) a partially masked representation of the structure of the antibody.

22. The method according to claim 20 or 21, wherein, The antigens include viral proteins or cancer cell proteins.

23. A method for obtaining a diagnostic antibody marker for a disease, the method comprising: For each of the one or more candidate antibodies, the method of any one of claims 1-15 is performed to determine the predicted structure of the candidate antibody by generating a predicted embedding that defines a complete protein structure representation of the candidate antibody, wherein, for each of the one or more candidate antibodies, the masked representation of the protein includes a complete representation of the amino acid sequence of the candidate antibody, and wherein, the representation of the structure of the protein includes a completely masked representation of the structure of the candidate antibody. Obtain the target protein structure; Evaluate the interaction between the predicted structure of each of one or more candidate antibodies and the target protein structure; and Based on the evaluation results, one of one or more candidate antibodies is selected as the diagnostic antibody marker.

24. A method for designing mutant proteins with optimized properties, comprising: To obtain i) a complete representation of the amino acid sequence of a known protein, and ii) a complete protein structure representation of a known protein; as well as For each of one or more candidate mutant proteins, the method according to any one of claims 1-15 is performed to determine the predicted amino acid sequence of the candidate mutant protein by generating a predicted embedding that defines the complete amino acid sequence of the candidate mutant protein, wherein generating the predicted embedding comprises: A partially masked representation of a candidate mutant protein is generated by one or more embeddings in the representation of the amino acid sequence of the candidate mutant protein. For each masked amino acid embedding, a corresponding score distribution is generated that defines the score for each amino acid type in a set of possible amino acid types; Predictive embeddings are generated by sampling the corresponding type of each masked amino acid based on the score distribution of the amino acids; and One of the candidate mutant proteins is selected as the mutant protein by identifying the predicted amino acid sequence that predicts the optimal properties of the candidate mutant protein from among the candidate mutant proteins.

25. The method of claim 24, further comprising synthesizing a mutant protein.

26. A method for identifying the presence of a protein misfolding disease, comprising: Perform the method according to any one of claims 1-15 to determine the predicted structure of a protein by generating a predicted embedding that defines a complete protein structure representation of the protein, wherein the masked representation of the protein includes a complete representation of the amino acid sequence of the protein, and wherein the representation of the structure of the protein includes a completely masked representation of the structure of the protein. Obtain the structure of a version of a protein obtained from a human or animal body; Compare the predicted structure of the protein with the structure of a version of the protein obtained from a human or animal; and Based on the comparison results, the presence of protein misfolding diseases can be identified.

27. A method for obtaining the amino acid sequence of a protein, comprising: The structure of the receiving protein has been obtained experimentally. Determine the complete protein structure representation of the protein from the structure described; and Perform the method according to any one of claims 1-15 to determine the predicted amino acid sequence of a protein by generating a predicted embedding that defines a complete amino acid sequence representation of the protein, wherein the masked representation of the protein includes a complete representation of the protein's structure, wherein the representation of the protein's amino acid sequence includes a completely masked representation of the protein's amino acid sequence, and wherein the predicted amino acid sequence of the protein is the obtained amino acid sequence of the protein.

28. A system comprising: One or more computers; as well as One or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of the corresponding method according to any one of claims 1-27.

29. A non-transitory computer storage medium storing instructions, which, when executed by one or more computers, cause the one or more computers to perform the operation of a corresponding method according to any one of claims 1-27.

Images