Predicting three-dimensional (3D) structures of molecule complexes using embedding neural networks and generative models

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for predicting a 3D structure of a molecule complex In one aspect, there is provided a method comprising: obtaining a network input that characterizes a molecule complex; processing the network input characterizing the molecule complex using an embedding neural network to generate molecule embedding data; and generating, using a generative model and while the generative model is conditioned on the molecule embedding data, a predicted three-dimensional (3D) structure of the molecule complex that defines a respective predicted 3D spatial location of each atom in the molecule complex.

Description

DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT ApplicationPREDICTING THREE-DIMENSIONAL (3D) STRUCTURES OF MOLECULE COMPLEXES USING EMBEDDING NEURAL NETWORKS AND GENERATIVE MODELS CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to U.S. Provisional Application No.63 / 611,674, filed on December 18, 2023. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application. BACKGROUND

[0002] This specification relates to predicting a three-dimensional (3D) structure of a complex of one or more molecules using machine learning.

[0003] Predictions can be made using machine learning models. Machine learning models receive an input and generate an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate the output based on the received input and on values of the parameters of the model. Some machine learning models are deep models that employ multiple layers of models to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a non-linear transformation to a received input to generate an output. SUMMARY

[0004] This specification describes a system implemented as computer programs on one or more computers in one or more locations that can predict a 3D structure of a molecule.

[0005] A “molecule” can be any group of atoms held together by chemical bonds. Examples of molecules include protein molecules (including protein molecules with post-translational modifications), deoxyribonucleic acid (DNA) molecules (including DNA molecules with modified nucleotides), ribonucleic acid (RNA) molecules (including RNA molecules with modified nucleotides), ligand molecules, and so forth.

[0006] A “molecule complex” can refer to a single molecule, or to multiple molecules (e.g., 2 or 3 or 5 or 10 molecules) that associate or interact (e.g., through chemical bonds, or hydrogenDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationbonds, or Van der Waals forces, or hydrophobic interaction, or electrostatic interaction, and so forth) to form a joint structure.

[0007] A “protein” can be understood to refer to any biological molecule that is specified by one or more sequences (or “chains”) of amino acids. For example, the term protein can refer to a protein domain, e.g., a portion of an amino acid chain of a protein that can undergo protein folding nearly independently of the rest of the protein. As another example, the term protein can refer to a protein complex, i.e., that includes multiple amino acid chains that jointly fold into a protein structure.

[0008] A “post-translational modification” (PTM) of a protein refers to a chemical change that the protein undergoes after having been synthesized (translated) by ribosomes in cells. Examples of post-translational modifications include phosphorylation, glycosylation, acetylation, and so forth.

[0009] A “DNA molecule” refers to a molecule that is specified by one or more sequences of nucleotides, each including a nitrogenous base (adenine, thymine, cytosine, or guanine), a sugar molecule (deoxyribose), and a phosphate group. Nucleotides in a DNA molecule can be modified by processes such as methylation, hydroxymethylation, acetylation, phosphorylation, and so forth.

[0010] An “RNA molecule” refers to a molecule that is specified by one or more sequences of nucleotides, each including a nitrogenous base (adenine, uracil, cytosine, or guanine), a sugar molecule (ribose), and a phosphate group. Nucleotides in an RNA molecule can be modified by processes such as methylation, pseudouridylation, capping, polyadenylation, and so forth.

[0011] A “ligand molecule” can refer to a molecule or compound that is capable of binding to and forming a complex with a target molecule, e.g., a protein. Ligands can include, e.g., small organic compounds, macromolecules, and so forth.

[0012] A “multiple sequence alignment” (MSA) for an amino acid chain in a protein specifies a sequence alignment of the amino acid chain with multiple additional amino acid chains, e.g., from other proteins, e.g., homologous proteins. More specifically, the MSA can define a correspondence between the positions in the amino acid chain and corresponding positions in multiple additional amino acid chains. An MSA for an amino acid chain can be generated, e.g., by processing a database of amino acid chains using any appropriate computational sequence alignment technique, e.g., progressive alignment construction. The amino acid chains in theDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT ApplicationMSA can be understood as having an evolutionary relationship, e.g., where each amino acid chain in the MSA may share a common ancestor. The correlations between the amino acids in the amino acid chains in a MSA for an amino acid chain can encode information that is relevant to predicting the structure of the amino acid chain.

[0013] A “multiple sequence alignment” (MSA) for a nucleotide chain in a DNA or RNA molecule specifies a sequence alignment of the nucleotide chain with multiple additional nucleotide chains, e.g., from other DNA or RNA molecules. More specifically, the MSA can define a correspondence between the positions in the nucleotide chain and corresponding positions in multiple additional nucleotide chains. An MSA for a nucleotide chain can be generated, e.g., by processing a database of nucleotide chains using any appropriate computational sequence alignment technique, e.g., progressive alignment construction. The nucleotide chains in the MSA can be understood as having an evolutionary relationship, e.g., where each nucleotide chain in the MSA may share a common ancestor. The correlations between the nucleotides in the nucleotide chains in an MSA for a target nucleotide chain can encode information that is relevant to predicting the structure of the target nucleotide chain.

[0014] A “deletion matrix” corresponding to a MSA (e.g., an MSA for a protein, DNA, or RNA molecule) can refer to a two-dimensional (2D) array with one dimension representing the sequences in the alignment and the other dimension representing the positions in the alignment. Each entry in the deletion matrix can hold a value indicating whether a deletion (or gap) is present at the corresponding position in the corresponding sequence.

[0015] For a given “target” molecule (e.g., for which the molecular structure is to be predicted), a “template” molecule can refer to a molecule having a known molecular structure and that satisfies a similarity criterion in relation to the target molecule. For instance, the similarity criterion can be that a similarity measure between the template molecule and the target molecule satisfies a threshold. A similarity between a first protein molecule and a second protein molecule can be evaluated using, e.g., a sequence identity, or a percent identity similar measure, or any other appropriate similarity measure, between the respective amino acid sequence(s) of the first protein molecule and the second protein molecule. A similarity between a first DNA molecule and a second DNA molecule can be evaluated using, e.g., a sequence identity, or a percent identity similar measure, or any other appropriate similarity measure, between the respective nucleotide sequence(s) of the first DNA molecule and the second DNA molecule. The similarityDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationbetween a first RNA molecule and a second RNA molecule can be evaluated using, e.g., a sequence identity, or a percent identity similar measure, or any other appropriate similarity measure, between the respective nucleotide sequence(s) of the first RNA molecule and the second RNA molecule.

[0016] A “block” in a neural network can refer to a group of one or more neural network layers in the neural network.

[0017] An “embedding” of an entity (e.g., an atom or a nucleotide or an amino acid) can refer to a representation of the entity as an ordered collection of numerical values, e.g., a vector, matrix, or other tensor of numerical values.

[0018] “Conditioning” a model (e.g., a generative model) or a neural network (e.g., a denoising neural network) or an operation (e.g., a self-attention operation) on conditioning data (e.g., an embedding representing a molecule) can refer to providing the conditioning data as an input (e.g., a side input) to the model, neural network, or operation, such that outputs generated by the model, neural network, or operation are influenced by (depend on) the conditioning data.

[0019] A 3D spatial position of an atom can be represented by a set of coordinates in an appropriate coordinate system, e.g., a 3D Cartesian coordinate system or a spherical coordinate system.

[0020] The term “Linear()” denotes a function that represents operations performed by one or more linear (e.g., fully connected) neural network layers.

[0021] The term “LinearNoBias()” denotes a function that represents operations performed by one or more linear (e.g., fully connected) neural network layers without a bias component.

[0022] The term “concat()” denotes a concatenation operation.

[0023] The term “LayerNorm()” denotes layer normalization operations.

[0024] The term “sigmoid()” denotes the operations of a sigmoid function.

[0025] The term “softmax()” denotes the operations of a softmax function.

[0026] The term “swish()” denotes a swish activation function.

[0027] The symbol “⊙” denotes element-wise multiplication.

[0028] According to a first aspect there is provided a method performed by one or more computers, the method comprising: obtaining a network input that characterizes a molecule or molecule complex; processing the network input characterizing the molecule or molecule complex using an embedding neural network to generate molecule embedding data; andDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationgenerating, using a generative model and while the generative model is conditioned on the molecule embedding data, a predicted three-dimensional (3D) structure of the molecule or molecule complex that defines a respective predicted 3D spatial location of each atom in the molecule or molecule complex.

[0029] In some implementations, the generative model is a generative diffusion model that comprises a denoising neural network.

[0030] In some implementations, generating, using the generative model and while the generative model is conditioned on the molecule embedding data, the predicted 3D structure of the molecule or molecule complex comprises: generating positional data defining a respective initial position of each atom in the molecule or molecule complex; denoising the positional data over a sequence of time steps using the denoising neural network and while the denoising neural network is conditioned on the molecule embedding data; wherein the predicted 3D structure of the molecule or molecule complex is defined by the positional data after a final time step in the sequence of time steps.

[0031] In some implementations, generating positional data defining the respective initial position of each atom in the molecule or molecule complex comprises sampling the respective initial position of each atom in the molecule or molecule complex from a probability distribution over 3D space.

[0032] In some implementations, denoising the positional data over the sequence of time steps using the denoising neural network and while the denoising neural network is conditioned on the molecule embedding data comprises, at each of one or more time steps in the sequence of time steps: receiving current positional data that defines a respective current position of each atom in the molecule or molecule complex at the time step; generating a denoising output using the denoising neural network and while the denoising neural network is conditioned on the molecule embedding data, wherein the denoising neural network processes an input comprising the current positional data; and generating positional data that defines a respective position of each atom in the molecule or molecule complex at a next time step using the denoising output.

[0033] In some implementations, the denoising output comprises a respective predicted error in the current position of each atom in the molecule or molecule complex at the time step.

[0034] In some implementations, generating the denoising output using the denoising neural network and while the denoising neural network is conditioned on the molecule embedding dataDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationcomprises: generating a set of atom embeddings using an encoder block of the denoising neural network, wherein each atom embedding represents an atom in the molecule or molecule complex and is based at least in part on the respective current spatial position of the atom at the time step; processing the set of atom embeddings using an update block of the denoising neural network to generate a set of updated atom embeddings, wherein the update block is conditioned on the molecule embedding data; and generating the denoising output based at least in part on the set of updated atom embeddings.

[0035] In some implementations, the update block of the denoising neural network comprises a sequence of self-attention blocks; wherein each of the self-attention blocks applies one or more self-attention operations to a set of current atom embeddings to update the set of current atom embeddings; wherein each of the one or more self-attention operations are conditioned on the molecule embedding data.

[0036] In some implementations, applying a self-attention operation to the set of current atom embeddings to update the set of current atom embeddings comprises: updating each current atom embedding using attention over only a proper subset of the set of current atom embeddings.

[0037] In some implementations, applying a self-attention operation to the set of current atom embeddings to update the set of current atom embeddings comprises: generating, based on the current set of atom embeddings, a respective intermediate attention score for each of a plurality of pairs of current atom embeddings from the set of current atom embeddings; generating, based on the molecule embedding data, a respective attention score bias for each of the plurality of pairs of current atom embeddings; generating a respective final attention score for each pair of current atom embeddings from the set of current atom embeddings based on the intermediate attention scores and the attention score biases; and updating the set of current atom embeddings using the final attention scores.

[0038] In some implementations, the molecule embedding data comprises a set of token- resolution pair embeddings that includes a respective pair embedding for each pair of tokens from a set of tokens associated with the molecule or molecule complex; wherein applying the self-attention operation to the set of current atom embeddings to update the set of current atom embeddings further comprises: generating, from the set of token-resolution pair embeddings, a set of atom-resolution pair embeddings that includes a respective pair embedding for each pair of atoms in the molecule or molecule complex; and wherein generating, based on the moleculeDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationembedding data, the respective attention score bias for each pair of current atom embeddings from the set of current atom embeddings comprises, for each pair of current atom embeddings: generating the attention score bias for the pair of current atom embeddings using a respective atom-resolution pair embedding corresponding to a same pair of atoms as the pair of current atom embeddings.

[0039] In some implementations, for each pair of current atom embeddings from the set of current atom embeddings, generating the final attention score for the pair of current atom embeddings comprises: summing the intermediate attention score for the pair of current atom embeddings and the attention score bias for the pair of current atom embeddings.

[0040] In some implementations, the molecule embedding data comprises a set of token- resolution single embeddings that includes a respective token-resolution single embedding for each token from a set of tokens associated with the molecule or molecule complex; wherein applying the self-attention operation to the set of current atom embeddings to update the set of current atom embeddings further comprises: generating, from the set of token-resolution single embeddings, a set of atom-resolution single embeddings that includes a respective atom- resolution single embedding for each atom in the molecule or molecule complex; and wherein updating the set of current atom embeddings using the final attention scores comprises: generating, for each atom in the molecule or molecule complex, a respective conditioning embedding using the atom-resolution single embedding for the atom; and generating, for each atom in the molecule or molecule complex, a respective updated atom embedding using the final attention scores and the conditioning embedding for the atom.

[0041] In some implementations, for each atom in the molecule or molecule complex, generating the updated atom embedding for the atom using the final attention scores and the conditioning embedding for the atom comprises: updating the atom embedding for the atom using the final attentions scores; and updating the atom embedding for the atom by combining the atom embedding for the atom with the conditioning embedding for the atom.

[0042] In some implementations, for each atom in the molecule or molecule complex, updating the atom embedding for the atom using the final attentions scores comprises: generating, for each of a plurality of atoms in the molecule or molecule complex, a respective value embedding for the atom based on the atom embedding for the atom; and combining the value embeddings forDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationthe plurality of atoms in accordance with final attention scores associated with the atom; and updating the atom embedding for the atom using the combination of the value embeddings.

[0043] In some implementations, for each atom in the molecule or molecule complex, updating the atom embedding for the atom by combining the atom embedding for the atom with the conditioning embedding for the atom comprises: element-wise multiplying the atom embedding and the conditioning embedding for the atom.

[0044] In some implementations, generating the denoising output based at least in part on the set of updated atom embeddings comprises: generating, for each of a plurality of tokens associated with the molecule or molecule complex, a respective token embedding by aggregating one or more corresponding updated atom embeddings; processing the set of token embeddings using an update block of the denoising neural network to generate a set of updated token embeddings, wherein the update block is conditioned on the molecule embedding data; and generating the denoising output based at least in part on the set of updated token embeddings.

[0045] In some implementations, the plurality of tokens associated with the molecule or molecule complex comprises a token representing an entity that includes a plurality of atoms; and wherein generating the token embedding for the token comprises: aggregating the updated atom embeddings for each atom included in the entity represented by the token.

[0046] In some implementations, the entity represented by the token is an amino acid or a DNA nucleotide or an RNA nucleotide.

[0047] In some implementations, the update block of the denoising neural network comprises a sequence of self-attention blocks; wherein each of the self-attention blocks applies one or more self-attention operations to a set of current token embeddings to update the set of current token embeddings; wherein each of the one or more self-attention operations are conditioned on the molecule embedding data.

[0048] In some implementations, applying a self-attention operation to the set of current token embeddings to update the set of current token embeddings comprises: generating, based on the set of current token embeddings, a respective intermediate attention score for each of a plurality of pairs of current token embeddings from the set of current token embeddings; generating, based on the molecule embedding data, a respective attention score bias for each of the plurality of pairs of current token embeddings; generating a respective final attention score for each pair of current token embeddings from the set of current token embeddings based on the intermediateDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationattention scores and the attention score biases; and updating the set of current token embeddings using the final attention scores.

[0049] In some implementations, the molecule embedding data comprises a set of token- resolution pair embeddings that includes a respective token-resolution pair embedding for each pair of tokens from a set of tokens associated with the molecule or molecule complex; wherein generating, based on the molecule embedding data, the respective attention score bias for each of the plurality of pairs of current token embeddings from the set of current token embeddings comprises, for each pair of current token embeddings: generating the attention score bias for the pair of current token embeddings using a respective token-resolution pair embedding corresponding to a same pair of tokens as the pair of current token embeddings.

[0050] In some implementations, the molecule embedding data comprises a set of token- resolution single embeddings that includes a respective single embedding for each token from a set of tokens associated with the molecule or molecule complex; wherein updating the set of current token embeddings using the final attention scores comprises: generating, for each token associated with the molecule or molecule complex, a respective conditioning embedding using the token-resolution single embedding for the token; and generating, for each token associated with the molecule or molecule complex, a respective updated token embedding using the final attention scores and the conditioning embedding for the token.

[0051] In some implementations, for each token associated with the molecule or molecule complex, generating the updated token embedding for the token using the final attention scores and the conditioning embedding for the token comprises: updating the token embedding for the token using the final attention scores; and updating the token embedding for the token by combining the token embedding for the token with the conditioning embedding for the token.

[0052] In some implementations, for each token associated with the molecule or molecule complex, updating the token embedding for the token using the final attention scores comprises: generating, for each of a plurality of tokens associated with the molecule or molecule complex, a respective value embedding for the token based on the token embedding for the token; and combining the value embeddings for the plurality of tokens in accordance with final attention scores associated with the token; and updating the token embedding for the token using the combination of the value embeddings.DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Application

[0053] In some implementations, for each token associated with the molecule or molecule complex, updating the token embedding for the token by combining the token embedding for the token with the conditioning embedding for the token comprises: element-wise multiplying the token embedding and the conditioning embedding for the token.

[0054] In some implementations, generating the denoising output using the denoising neural network and while the denoising neural network is conditioned on the molecule embedding data comprises: generating a set of atom embeddings, wherein each atom embedding represents an atom in the molecule or molecule complex and is based at least in part on the respective current spatial position of the atom at the time step; applying one or more self-attention operations to the set of atom embeddings to update the set of atom embeddings, wherein each of the one or more self-attention operations are conditioned on the molecule embedding data; generating a set of token embeddings, wherein each token embedding represents a respective token associated with the molecule or molecule complex and is generated by aggregating one or more corresponding atom embeddings; and applying one or more self-attention operations to the set of token embeddings to update the set of token embeddings, wherein each of the one or more self- attention operations are conditioned on the molecule embedding data; and generating the denoising output using the set of token embeddings.

[0055] In some implementations, generating the denoising output using the set of token embeddings comprises: updating the set of atom embeddings using the set of token embeddings; applying one or more self-attention operations to the set of atom embeddings to update the set of atom embeddings, wherein each of the one or more self-attention operations are conditioned on the molecule embedding data; and generating the denoising output using the set of atom embeddings.

[0056] In some implementations, processing the network input characterizing the molecule or molecule complex using the embedding neural network to generate the molecule embedding data comprises: generating a set of single embeddings, wherein each single embedding represents a respective token from a set of tokens associated with the molecule or molecule complex; and generating a set of pair embeddings, wherein each pair embedding represents a respective pair of tokens from the set of tokens associated with the molecule or molecule complex.

[0057] In some implementations, the set of tokens comprises one or more tokens that each represent a respective amino acid in the molecule or molecule complex.DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Application

[0058] In some implementations, the set of tokens comprises one or more tokens that each represent a respective nucleotide in the molecule or molecule complex.

[0059] In some implementations, the set of tokens comprises one or more tokens that each represent a respective DNA nucleotide in the molecule or molecule complex.

[0060] In some implementations, the set of tokens comprises one or more tokens that each represent a respective RNA nucleotide in the molecule or molecule complex.

[0061] In some implementations, the set of tokens comprises one or more tokens that each represent a respective atom in the molecule complex.

[0062] In some implementations, the molecule or molecule complex is a molecule complex that comprises one or more protein molecules and one or more DNA molecules; or the molecule or molecule complex is a molecule complex that comprises one or more protein molecules and one or more RNA molecules; or the molecule or molecule complex is a molecule complex that comprises one or more protein molecule and one or more ligand molecules; or the molecule or molecule complex is a molecule complex that comprises a plurality of protein molecules; or the molecule or molecule complex is a protein molecule; or the molecule or molecule complex is a DNA molecule; or the molecule or molecule complex is a RNA molecule.

[0063] In some implementations, the molecule or molecule complex comprises one or more DNA molecules, and one or more of nucleotides in one or more of the DNA molecules are modified DNA nucleotides.

[0064] In some implementations, the molecule or molecule complex comprises one or more RNA molecules, and or more nucleotides in one or more of the RNA molecules are modified RNA nucleotides.

[0065] In some implementations, generating the set of single embeddings and the set of pair embeddings comprises: initializing a set of current single embeddings and a set of current pair embeddings; and updating the set of current single embeddings and the set of current pair embeddings over a sequence of one or more recycling iterations, comprising, for each recycling iteration: updating the set of current single embeddings by applying one or more self-attention operations to the set of current single embeddings, wherein each of the one or more self-attention operations are conditioned on the set of current pair embeddings.

[0066] In some implementations, applying a self-attention operation to the set of current single embeddings comprises: generating, based on the set of current single embeddings, a respectiveDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationintermediate attention score for each pair of current single embeddings from the set of current single embeddings; generating, based on the set of current pair embeddings, a respective attention score bias for each pair of current single embeddings from the set of current single embeddings; generating a respective final attention score for each pair of current single embeddings from the set of single embeddings based on the intermediate attention scores and the attention score biases; and updating the set of current single embeddings using the final attention scores.

[0067] In some implementations, for each pair of current single embeddings that comprises a first single embedding representing a first token in the molecule or molecule complex and a second single embedding representing a second token in the molecule or molecule complex, generating the attention score bias for the pair of single embeddings comprises: processing a current pair embedding, from the set of current pair embeddings, that represents a pair of tokens comprising the first token and the second token using one or more neural network layers to generate the attention score bias for the pair of single embeddings.

[0068] In some implementations, each recycling iteration further comprises updating the set of current pair embeddings using one or more template molecules.

[0069] In some implementations, each recycling iteration further comprises updating the set of current pair embeddings using a multiple sequence alignment (MSA).

[0070] In some implementations, the embedding neural network and the generative model are jointly trained on a set of training data, wherein each training examples corresponds to a respective molecule or molecule complex and includes a prediction target that defines an actual 3D structure of the molecule or molecule complex.

[0071] In some implementations, the set of training data includes: a plurality of training examples corresponding to molecule complexes that each comprise one or more respective protein molecules and one or more respective DNA molecules; and / or a plurality of training examples corresponding to molecule complexes that each include one or more respective protein molecules and one or more respective RNA molecules; and / or a plurality of training examples corresponding to molecule complexes that each include one or more respective protein molecule and one or more respective ligand molecules; and / or a plurality of training examples corresponding to molecule complexes that each comprise a respective plurality of protein molecules; a plurality of training examples corresponding to respective protein molecules; and / orDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationa plurality of training examples corresponding to respective DNA molecules; and / or a plurality of training examples corresponding to respective RNA molecules.

[0072] According to another aspect there is provided a system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of the methods described herein.

[0073] According to another aspect there are provided one or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations of the methods described herein.

[0074] According to another aspect there is provided a method of obtaining a ligand, wherein the ligand is a drug or a ligand of an industrial enzyme, the method comprising: for each of one or more candidate ligands: (a) performing the methods described above to determine a predicted structure of a target molecule or a complex comprising a target molecule and the candidate ligand; and (b) evaluating an interaction of the candidate ligand with the target molecule dependent on the predicted structure; and selecting one or more of the candidate ligands as the ligand dependent on a result of the evaluating.

[0075] In some implementations, the target molecule comprises a receptor or enzyme, and wherein the ligand is an agonist or antagonist of the receptor or enzyme.

[0076] In some implementations, the ligand is a drug, and the method comprises: performing steps (a) and (b) for each of a plurality of target molecules; and selecting one or more of the candidate ligands as the ligand to either i) obtain a ligand that interacts with each of the target molecules, or ii) obtain a ligand that interacts with only one of the target molecules.

[0077] According to anther aspect, there is provided a method of obtaining a ligand, wherein the ligand is a drug or a ligand of an industrial enzyme, the method comprising: selecting a target molecule; for each of one or more candidate ligands: performing the methods described above to determine a predicted structure of the candidate ligand or a complex comprising the candidate ligand and a target molecule; and evaluating an interaction between the candidate ligand and the target molecule dependent on the predicted structure; and selecting one of the one or more of the candidate ligands as the ligand dependent on a result of the evaluating.DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Application

[0078] In some implementations, the target molecule comprises a receptor or enzyme, and the ligand is an agonist or antagonist of the receptor or enzyme; or the ligand comprises an antibody or aptamer and the target molecule comprises an antibody or aptamer target, in particular a virus or cancer cell protein, and wherein the antibody or aptamer binds to the antibody or aptamer target to provide a therapeutic effect.

[0079] In some implementations, the ligand is a polypeptide ligand, a polynucleotide ligand, or a polynucleotide ligand.

[0080] According to another aspect there is provided a method of obtaining a diagnostic antibody or aptamer marker of a disease, the method comprising: selecting a target molecule; for each of one or more candidate antibodies or aptamers: performing the methods described above to determine a predicted structure of the candidate antibody or aptamer or a complex comprising the candidate antibody or aptamer and the target molecule; and evaluating an interaction between the candidate antibody or aptamer and the target molecule; and selecting one of the one or more of the candidate antibodies or aptamers as the diagnostic antibody or aptamer marker dependent on a result of the evaluating.

[0081] In some implementations, evaluating the interaction of one of the candidate ligands comprises determining an interaction score for the candidate ligand, wherein the interaction score comprises a measure of an interaction between the candidate ligand and the target molecule.

[0082] In some implementations, the method further comprises synthesizing the ligand or diagnostic antibody or aptamer marker.

[0083] In some implementations, the method further comprises testing biological activity of the ligand or diagnostic antibody or aptamer marker in vitro and in vivo.

[0084] According to another aspect, there is provided a method of identifying the presence of a protein or nucleic acid mis-folding disease, comprising: performing the methods described above to determine a predicted structure of a protein or nucleic acid; obtaining a structure of a version of the protein or nucleic acid obtained from a human or animal body; comparing the predicted structure of the protein with the structure of a version of the protein or nucleic acid obtained from a human or animal body; and identifying the presence of a protein or nucleic mis-folding disease dependent upon a result of the comparison.DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Application

[0085] According to another aspect there is provided a method of determining the structure of a molecule or molecule complex, comprising: applying an experimental technique to a physical sample comprising a molecule or molecule complex to measure experiment signals dependent on a structure of the molecule complex; performing the methods described above to determine a predicted structure of the molecule or molecule complex; and using the experiment signals and the predicted structure of the molecule complex to determine the structure of the molecule or molecule complex.

[0086] In some implementations, the experimental technique comprises one or more of: x-ray crystallography, nuclear magnetic resonance, and electron microscopy.

[0087] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0088] The folding system described in this specification can include a generative diffusion model that includes a denoising neural network. The denoising neural network can iteratively denoise the 3D spatial positions of the atoms included in a molecule complex in order to predict the 3D structure of the molecule complex. As part of denoising the 3D spatial positions of the atoms in a molecule complex, the denoising neural network can apply self-attention operations over collections of embeddings representing the molecule complex at each of multiple levels of resolution. In particular, the denoising neural network can apply self-attention operations to a collection of atom-resolution embeddings that each represent a respective atom in the molecule complex, and the denoising neural network can apply self-attention operations to a collection of token-resolution embeddings that each represent a respective “token” in the molecule complex. (A “token” can be a unit of the molecule complex that includes multiple atoms, e.g., an amino acid or a nucleotide).

[0089] Applying self-attention operations to the atom-resolution embeddings can enable fine- grained information sharing between individual atoms but can be computationally intensive, e.g., because the number of atoms included in a molecule complex can be very large. Applying self- attention operations to token-resolution embeddings can enable higher-level information sharing between embeddings representing collections of atoms and can be less computationally intensive than atom-resolution self-attention operations. By performing self-attention operations at both the atom-level and the token-level, the folding system can leverage the complementary aspects ofDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationboth types of attention operations to achieve higher prediction accuracy while reducing consumption of computational resources, e.g., memory and computing power.

[0090] As part of applying self-attention operations to a set of atom embeddings, as described above, the folding system can “mask” the self-attention such that each atom embedding is updated using attention over only a proper subset of the full set of atom embeddings. Masking the self-attention operation can dramatically reduce consumption of computational resources, e.g., memory and computing power, and can enable atom-level self-attention operations to be applied even to molecule complexes with large numbers of atoms, e.g., thousands of atoms.

[0091] The folding system described in this specification can predict the structure of a molecule complex by a single forward pass through a collection of jointly trained models (e.g., neural networks), which may take less than one second. In contrast, some conventional systems predict the structure of a molecule complex by an extended search process through the space of possible molecule complex structures to optimize a scalar score function, e.g., using simulated annealing or gradient descent techniques. Such a search process may require millions of search iterations and consume hundreds of central processing unit (CPU) hours. Predicting molecule complex structures by a single forward pass through a collection of neural networks may enable the folding system to consume fewer computational resources (e.g., memory and computing power) than conventional systems, e.g., that predict molecule complex structures by an iterative search process.

[0092] Accurately predicting the structures of molecule complexes can facilitate understanding of the physical and chemical properties of potential drug compounds. This understanding is essential for designing molecules that are not only effective in interacting with specific biological targets but also possess desirable pharmacokinetic properties, such as stability and solubility. Computational methods for predicting molecule complex structures can streamline the process of drug design and reduce time and resources spent on synthesizing and testing less promising compounds.

[0093] The folding system described in this specification can be configured (e.g., through training) to have the capability of generating predicted 3D structures of multiple different types of molecules and molecule complexes, i.e., rather than being limited to generating predicted 3D structures of only one type of molecule or molecule complex. For instance, the folding system can have the capability of generating predicted 3D structures for a complex that includes anyDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationcombination of one or more of: protein molecules, or DNA molecules, or DNA molecules with post-translational modifications, or RNA molecules, or RNA molecules with modifications, or ligand molecules. In contrast, some conventional systems may be limited to generating predicted 3D structures for only one type of molecule, e.g., only for protein molecules. Accurately predicting molecule complex structures requires the models (e.g., neural network models) included in the folding system to learn to implicitly reason about underlying biological and chemical principles that govern the formation of molecule complex structures. Configuring the folding system to predict 3D structures of multiple types of molecule complex structures can enable the folding system to learn to exploit synergies that exist between predicting 3D structures of different types of molecule structures, and thus can enable the folding system to achieve higher overall accuracy in predicting molecule complex structures.

[0094] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] FIG. 1 shows an example folding system.

[0096] FIG. 2 is a flow diagram of an example process for processing molecule complex data characterizing a molecule complex using an embedding neural network to generate molecule embedding data.

[0097] FIG. 3 is a flow diagram of an example process for performing a recycling iteration to update a respective current single embedding for each token in the molecule complex and a respective current pair embedding for each pair of tokens in the molecule complex.

[0098] FIG. 4 is a flow diagram of an example process by which an update block of a neural network (e.g., an embedding neural network) updates a set of single embeddings and a set of pair embeddings.

[0099] FIG. 5 is a flow diagram of an example process for updating a set of single embeddings using a self-attention operation that is conditioned on a set of pair embeddings.

[0100] FIG. 6 is a flow diagram of an example process for generating a predicted 3D structure of a molecule complex using a generative diffusion model that includes a denoising neural network.DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Application

[0101] FIG. 7 is a flow diagram of an example process for generating a denoising output using a denoising neural network conditioned on molecule embedding data.

[0102] FIG.8 is a flow diagram of an example process for updating a set of atoms embeddings for the atoms in the molecule complex using an update block of the denoising neural network.

[0103] FIG.9 is a flow diagram of an example process for applying a self-attention operation to a set of input embeddings (e.g., a set of atom embeddings or a set of token embeddings), where the self-attention operation is conditioned on molecule embedding data (e.g., atom-resolution molecule embedding data or token-resolution molecule embedding data).

[0104] FIG. 10 is a flow diagram of an example process for generating one or more confidence estimates associated with a predicted 3D structure of a molecule complex using a confidence prediction neural network.

[0105] FIG.11 is a flow diagram of an example process for jointly training the embedding neural network and the generative model of the folding system.

[0106] FIG. 12 is a flow diagram of an example process for jointly training an embedding neural network and a generative diffusion model on a training example.

[0107] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0108] FIG. 1 shows an example folding system 100. The folding system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0109] The folding system 100 is configured process molecule complex data 104 characterizing a molecule complex 102 to generate a predicted 3D structure 114 of the molecule complex 102. Throughout this specification, the term “molecule complex” can refer to: (i) a single molecule, or (ii) multiple molecules that associate or interact to form a joint structure. However, in some cases, for clarity, the specification may refer to a “molecule or molecule complex” instead of simply a “molecule complex”.

[0110] The molecule complex 102 can include, e.g., one or more protein molecules, one or more DNA molecules, one or more RNA molecules, one or more ligand molecules, or any combination thereof. The predicted 3D structure 114 of the molecule complex 102 defines a respective predicted 3D spatial location of each atom in the molecule complex 102.DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Application

[0111] In some implementations, the molecule complex may include one or more biological molecules, e.g., the molecule complex may include a plurality of amino acids, such as a polypeptide, and / or a plurality of nucleotides and / or nucleosides, such as a polynucleotide or polynucleoside. For example, the molecule complex may comprise a protein and / or a nucleic acid, such as a deoxyribonucleic acid (DNA) or a ribonucleic acid (RNA). In some implementations, the DNA or RNA may be an aptamer, ribozyme, deoxyribozyme (DNA enzyme), or riboswitch. The protein may comprise canonical and / or non-canonical amino acids, such as amino acids that include post-translational modifications, such as: phosphorylation, acetylation, glycosylation, amidation, hydroxylation, methylation and so on. The DNA or RNA may comprise canonical nucleotides (adenine, cytosine, guanine, thymine, and uracil) and / or non-canonical or modified nucleotides, such as methylated nucleotides. As used herein, the term polypeptide encompasses peptides and oligopeptides, and the term polynucleotide encompasses oligonucleotides. In some cases, the biological molecule complex may occur naturally in one or more living organisms, in which case it may be referred to as a biomolecule.

[0112] In some implementations, the molecule complex includes a plurality of protein molecules.

[0113] In some implementations, the molecule complex includes one or more protein molecules and one or more ligand molecules.

[0114] In some implementations, the molecule complex includes one or more protein molecules and one or more DNA molecules.

[0115] In some implementations, the molecule complex includes one or more protein molecules and one or more RNA molecules.

[0116] In some implementations, the molecule complex comprises one or more antibody molecule and one or more antigen molecules.

[0117] The molecule complex data 104 can be represented with reference to a “tokenization” scheme, e.g., where the input molecule complex is represented as a collection of tokens, and the molecule complex data 104 is expressed with reference to the tokenization scheme. For instance, the tokenization scheme can represent a standard amino acid as a single token. As another example, the tokenization scheme can represent a standard nucleotide residue as a single token. As another example, the tokenization scheme can represent a modified amino acid or modified DNA nucleotide or modified RNA nucleotide by an atom-level tokenization, e.g., where eachDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationatom in the modified amino acid or modified DNA nucleotide or modified RNA nucleotide is represented by a respective token. As another example, the tokenization scheme can represent a ligand by an atom-level tokenization, e.g., where each atom in the ligand is represented by a respective token. Throughout this specification, for convenience, a token representing a unit (e.g., atom, nucleotide, or amino acid residue) in a molecule complex may be referred to as a token “in” or “from” or “associated with” the molecule complex.

[0118] The molecule complex data 104 can be any appropriate type of data characterizing the molecule complex 102. A few examples of molecule complex data 104 are described next.

[0119] In some cases, the molecule complex 102 includes a protein molecule and the molecule complex data 104 includes data defining one or more amino acid sequences of the protein. More specifically, the molecule complex data 104 can identify which amino acid, from a set of possible amino acids, occupies each position in each of the one or more amino acid sequences of the protein. The set of possible amino acids can, in some cases, include modified amino acids associated with post-translational modifications (as described above).

[0120] In some cases, the molecule complex 102 includes a DNA molecule and the molecule complex data 104 includes data defining one or more nucleotide sequences of the DNA molecule. More specifically, the molecule complex data 104 can identify which nucleotide, from a set of possible DNA nucleotides, occupies each position in each of the one or more nucleotide sequences of the DNA molecule. The set of possible DNA nucleotides can, in some cases, include modified DNA nucleotides (as described above).

[0121] In some cases, the molecule complex 102 includes an RNA molecule and the molecule complex data 104 includes data defining one or more nucleotide sequences of the RNA molecule. More specifically, the molecule complex data 104 can identify which nucleotide, from a set of possible RNA nucleotides, occupies each position in each of the one or more nucleotide sequences of the RNA molecule. The set of possible RNA nucleotides can, in some cases, include modified RNA nucleotides (as described above).

[0122] Optionally, for a molecule complex 102 that includes a protein, DNA, or RNA molecule, the molecule complex data 104 for the molecule complex can include MSA features derived from data defining an MSA for the protein, DNA, or RNA molecule. The MSA features can include, e.g., data defining the MSA itself, or data defining a deletion matrix for the MSA, or both.DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Application

[0123] In some cases, the molecule complex 102 includes a ligand and the molecule complex data 104 includes data enumerating the atoms in the ligand, identifying the elemental type of each atom, and identifying which atoms in the ligand connected by bonds.

[0124] Optionally, the molecule complex data 104 can include data characterizing a respective structure of each of one or more template molecules for a molecule included in the molecule complex. The structure of a template molecule can be represented in any appropriate manner, e.g., by a contact map, or by data defining a respective 3D spatial position of each atom in the template molecule.

[0125] Additional examples of molecule complex data 104 that can be received and processed by the folding system 100 are described in more detail below with reference to FIG. 2.

[0126] The folding system 100 processes the molecule complex data 104 using an embedding neural network 106 and a generative model 110 to generate the predicted 3D structure 114 of the molecule. The embedding neural network 106 and the generative model 110 are each described in more detail next (and throughout this specification).

[0127] The embedding neural network 106 is configured to process the molecule complex data 104 to generate molecule embedding data 108 that represents the molecule complex. The molecule embedding data 108 can include data defining a set of “single” embeddings or a set of “pair” embeddings or both. Each single embedding in the set of single embeddings can represent a respective token in the molecule complex 102. Each pair embedding representing in the set of pair embeddings can represent a respective pair of tokens in the molecule complex. As described above, a “token” represents a unit of the molecule complex, e.g., each token can represent a respective atom, or DNA nucleotide, or RNA nucleotide, or amino acid, and so forth.

[0128] The embedding neural network 106 can have any appropriate neural network architecture that enables the embedding neural network 106 to perform its described functions. In particular, the embedding neural network 106 can include any appropriate types of neural network layers (e.g., fully connected layers, convolutional layers, attention layers, etc.) in any appropriate number (e.g., 5 layers, 10 layers, or 20 layers) and connected in any appropriate configuration (e.g., as a directed graph of layers). An example process by which the embedding neural network 106 can generate molecule embedding data 108 representing a molecule complex 102 is described in more detail below with reference to FIG. 2 – FIG.5.DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Application

[0129] The generative model 110, when conditioned on the molecule embedding data 108, is configured to generate one or more predicted 3D structures 114 of the molecule complex. The generative model 110 can be any appropriate conditional generative model. More specifically, the generative model 110 can be any appropriate model that, when conditioned on the molecule embedding data 108, can generate samples from a distribution over a space of possible 3D structures of the molecule complex 102. For instance, the generative model 110 can be implemented as a generative diffusion model, or a generative adversarial neural network (GAN) model, or a flow-based neural network model (normalizing flow model), and so forth. An example process for generating predicted 3D structures using a generative diffusion model is described detail with reference to FIG. 6 – FIG.10.

[0130] Optionally, the folding system 100 can use the generative model 110 to generate multiple distinct predicted 3D structures of the molecule complex. In particular, the generative model 110 can generate multiple samples from the distribution over the space of possible 3D structures of the molecule complex. Differences between the predicted joint 3D structures generated by the generative model 110 can reflect both uncertainty in the predicted structure and also various structural modes of the molecule complex.

[0131] In some implementations, the folding system 100 includes a confidence prediction neural network 112 that is configured to process the molecule embedding data 108 and the predicted 3D structure 114 to generate one or more confidence estimates 116 that each characterize an aspect of the confidence of the folding system 100 in the predicted 3D structure 114. An example process for generating confidence estimates 116 is described below with reference to FIG.10.

[0132] The folding system 100 can jointly train the embedding neural network 106, the generative model 110, and the confidence prediction neural network 112 on a set of training data using an appropriate machine learning training technique. The training data can include a set of training examples, where each training example corresponds to a respective molecule complex. The training data can include training examples corresponding to multiple different types of molecule complexes, e.g., molecule complexes that include various combinations of different types of molecules, e.g., protein molecules, DNA molecules, RNA molecules, ligand molecules, and so forth. Training the embedding neural network 106 and the generative model 110 on a set of training data that includes training examples corresponding to multiple types of molecule complexes can enable the trained embedding neural network 106 and generative model 110 toDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationaccurately generate predicted 3D structures for multiple types of molecule complexes. An example process for jointly training the embedding neural network 106, the generative model 110, and the confidence prediction neural network 112 is described in more detail with reference to FIG.11.

[0133] The folding system 100 can receive the molecule complex data 104 from any appropriate source, e.g., from a user or from another system, by way of an appropriate interface, e.g., an application programming interface (API) or a user interface (e.g., a graphical user interface). After generating the predicted 3D structure 114, the folding system 100 can, e.g., store data defining the predicted 3D structure 114 in a memory, or transmit data defining the predicted 3D structure 114 over a data communication network, or provide data defining the predicted 3D structure 114 directly to a system that performs downstream processing based on the predicted 3D structure 114.

[0134] In some implementations, a user can input molecule complex data characterizing a molecule complex into a user interface made available on a user device of the user by the folding system 100. The user device of the user can be, e.g., a personal computer or a tablet or a smartphone. The user interface by which the user inputs the molecule complex data can be presented to the user, on the user device, as part of an application running on the user device. The molecule complex data input into the user interface of the user device can be transmitted to the folding system, e.g., by transmission over an appropriate data communications network, e.g., the internet. The folding system can process the molecule complex data to generate a predicted 3D structure of the molecule complex, and then output data defining the predicted 3D structure of the molecule complex. For instance, the folding system can transmit the data defining the predicted 3D structure of the molecule complex back to the user device, e.g., over a data communications network, and a user can access the predicted 3D structure by way of the user device, e.g., by viewing data characterizing the predicted 3D structure using an application running on the user device.

[0135] Predicted 3D structures 114 generated by the folding system can be used in any of a variety of possible downstream applications. A few examples of downstream applications that process predicted 3D structures generated by the folding system are described next.

[0136] The folding system and methods described herein can be used to obtain a ligand (i.e. a ligand molecule or ligand molecule complex) such as a drug or a ligand of an industrial enzyme.DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT ApplicationIn general, the drug or industrial enzyme may be a molecule that inhibits or catalyzes a chemical or biochemical process. The molecule complex may include, for example, a protein, a ribozyme (ribonucleic acid enzyme), or a deoxyribozyme (deoxyribonucleic acid enzyme). For example, a method of obtaining a ligand may include obtaining a target amino acid or nucleotide sequence, in particular the amino acid or nucleotide sequence of a target molecule (or target molecule complex), e.g. a drug target, and processing an input based on the target amino acid or nucleotide sequence using the folding system to determine a (tertiary) structure of the target molecule, i.e., the predicted molecule structure, or a predicted structure of a complex comprising the target molecule complex and the candidate ligand. The method may then include evaluating an interaction of one or more candidate ligands with the target molecule. The method may further include selecting one or more of the candidate ligands as the ligand dependent on a result of the evaluating of the interaction. Predicting the structure of a complex comprising the target molecule and a candidate ligand may preferably account for changes in the structure of molecule caused by binding of the candidate ligand and / or changes in the structure of the candidate ligand. Evaluating the interaction of the one or more candidate ligands with the target molecule may, for example, comprise determining a binding energy or an equilibrium constant for the formation of the complex. Alternatively, the structure of the target molecule may be predicted in the absence of the candidate ligand and the interaction may then be evaluated according to an interaction between the candidate ligand and the predicted structure of the target molecule.

[0137] In some implementations, evaluating the interaction may include evaluating binding of the candidate ligand with the structure of the target molecule (such as a biological molecule). For example, evaluating the interaction may include identifying a ligand that binds with sufficient affinity for a biological effect. In some other implementations, evaluating the interaction may include evaluating an association of the candidate ligand with the target molecule which has an effect on a function of the target molecule, e.g., an enzyme. The evaluating may include evaluating an affinity between the candidate ligand and the target molecule or complex, or evaluating a selectivity of the interaction. The candidate ligand(s) may be selected according to which have the highest affinity. Evaluating the interaction may additionally comprise simulating a dynamical behavior of the ligand and target molecule, such as through molecular dynamics simulations, which may allow kinetic aspects of the interaction to be taken into account.DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Application

[0138] The candidate ligand(s) may be derived from a database of candidate ligands, and / or may be derived by modifying ligands in a database of candidate ligands, e.g., by modifying a structure or amino acid sequence or nucleotide sequence of a candidate ligand, and / or may be derived by stepwise or iterative assembly / optimization of a candidate ligand. The candidate ligand(s) may alternately or additionally include one or more candidate ligands generated using a generative model conditioned on (the structure of) the target molecule or part of the target molecule, e.g. a structure of a binding site or other part of the target molecule.

[0139] The evaluation of the interaction of a candidate ligand with the target molecule may be performed using a computer-aided approach in which graphical models of the candidate ligand and target molecule structure are displayed for user-manipulation, and / or the evaluation may be performed partially or completely automatically, for example using standard molecular (e.g. protein-ligand) docking software. In some implementations the evaluation may include determining an interaction score for the candidate ligand, where the interaction score includes a measure of an interaction between the candidate ligand and the target molecule. The interaction score may be dependent upon a strength and / or specificity of the interaction, e.g., a score dependent on binding free energy. A candidate ligand may be selected dependent upon its score.

[0140] In some implementations the target molecule includes a receptor or enzyme and the ligand is an agonist or antagonist of the receptor or enzyme. In some implementations the method may be used to identify the structure of a cell surface marker. This may then be used to identify a ligand, e.g., an antibody or aptamer or a label such as a fluorescent label, which binds to the cell surface marker. This may be used to identify and / or treat cancerous cells.

[0141] In some implementations the ligand is a drug and the interaction of each of a plurality of target molecules (such as a biological molecule) with each of the candidate ligands is evaluated. Then one or more of the candidate ligands may be selected either to obtain a ligand that (functionally) interacts with each of the target molecules, or to obtain a ligand that (functionally) interacts with only one of the target molecules. For example in some implementations it may be desirable to obtain a drug that is effective against multiple drug targets. Also or instead it may be desirable to screen a drug for off-target effects. For example in agriculture it can be useful to determine that a drug designed for use with one plant species does not interact with another, different plant species and / or an animal species.DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Application

[0142] In some implementations the ligand is a drug and the predicted structure of a target protein that is a protein complex, e.g. a dimer or multimer, is determined. Evaluating the interaction of the one or more candidate ligands with the target protein may then comprise identifying a candidate ligand that interacts with the protein complex, and that might therefore be expected to affect the formation or stability of the complex. This could afterwards be confirmed by experimental screening. Thus such a process may be used to identify a drug which is able to disrupt a protein complex or inhibit formation of the complex. Some diseases, e.g. neuro degenerative diseases such as dementia, are caused by protein aggregation. The method may thus be used to identify a ligand that is a drug to treat such a disease.

[0143] In some implementations the candidate ligand(s) may include small molecule complex ligands, e.g., organic compounds with a molecular weight of <900 daltons. In some other implementations the candidate ligand(s) may include polypeptide ligands, i.e., defined by an amino acid sequence.

[0144] In some cases, the folding system can be used to determine the structure of a candidate ligand, e.g., a drug or a ligand of an industrial enzyme, or a complex comprising the candidate ligand and a target molecule. The interaction of this ligand with a target molecule structure may then be evaluated. In some examples, the target molecule structure may have been determined using a structure prediction neural network or using conventional physical investigation techniques such as x-ray crystallography and / or magnetic resonance techniques or cryogenic electron microscopy. The interaction between the predicted structure of the ligand and the structure of the target molecule may then be evaluated.

[0145] In another aspect there is provided a method of using a folding system to obtain a ligand, which may be a biological molecule, such as a polypeptide, polynucleotide, or polynucleoside ligand (e.g., the molecule or its amino acid or nucleotide sequence). For example, the method may include obtaining data defining one or more candidate ligands, e.g. an amino acid sequence or nucleotide sequence of one or more candidate polypeptide or polynucleotide ligands. The method may include selecting a target molecule to which the ligand is to bind. The method may further include, for each of the candidate ligands, using the folding system to determine (tertiary) structure of the candidate ligand or a complex comprising the candidate ligand and the target molecule. The method may further include obtaining a target protein structure of a target molecule, in silico and / or by physical investigation. The method may comprise evaluating an interaction between each ofDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationthe one or more candidate ligands and the target molecule, e.g. by evaluating an interaction between the predicted structure of the candidate ligand and the structure of the target molecule, or using the predicted structure of the complex comprising the candidate ligand and the target molecule. The method may further include selecting one or more of the candidate ligands as the ligand dependent on a result of the evaluation.

[0146] As before, evaluating the interaction may include evaluating binding of the candidate ligand with the structure of the target molecule, e.g., identifying a ligand that binds with sufficient affinity for a biological effect, and / or evaluating an association of the candidate ligand with the structure of the target molecule which has an effect on a function of the target molecule, e.g., an enzyme, and / or evaluating an affinity between the candidate ligand and the structure of the target molecule, or evaluating a selectivity of the interaction. In some implementations the ligand may be an aptamer. Again the candidate ligand(s) may be selected according to which have the highest affinity.

[0147] As before, the selected ligand (e.g. selected polypeptide or polynucleotide ligand) may comprise a receptor or enzyme and the ligand may be an agonist or antagonist of the receptor or enzyme. In some implementations the ligand may comprises an antibody or aptamer and the target molecule (e.g. protein) comprises an antibody or aptamer target, for example a virus, in particular a virus coat protein, or a protein expressed on a cancer cell. In these implementations the antibody or aptamer binds to the antibody or aptamer target to provide a therapeutic effect. For example, the antibody or aptamer may bind to the target and act as an agonist for a particular receptor; alternatively, the antibody or aptamer may prevent binding of another ligand to the target, and hence prevent activation of a relevant biological pathway.

[0148] Implementations of the method may further include synthesizing the ligand, i.e., making, the small molecule, polynucleotide or polypeptide ligand. The ligand may be synthesized by any conventional chemical techniques and / or may already be available, e.g., may be from a compound library or may have been synthesized using combinatorial chemistry.

[0149] The method may further include testing the ligand for biological activity in vitro and / or in vivo. For example the ligand may be tested for ADME (absorption, distribution, metabolism, excretion) and / or toxicological properties, to screen out unsuitable ligands. The testing may include, e.g., bringing the candidate small molecule, polypeptide or polynucleotide ligand intoDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationcontact with the target molecule (e.g. protein) and measuring a change in expression or activity of the target molecule .

[0150] In some implementations a candidate (e.g. polypeptide or polynucleotide) ligand may include: an isolated antibody or aptamer, a fragment of an isolated antibody or aptamer, a single variable domain antibody, a bi- or multi-specific antibody, a multivalent antibody, a dual variable domain antibody, an immuno-conjugate, a fibronectin molecule, an adnectin, an DARPin, an avimer, an affibody, an anticalin, an affilin, a protein epitope mimetic or combinations thereof. A candidate (polypeptide) ligand may include an antibody with a mutated or chemically modified amino acid Fc region, e.g., which prevents or decreases ADCC (antibody-dependent cellular cytotoxicity) activity and / or increases half-life when compared with a wild type Fc region. Candidate (polypeptide or polynucleotide) ligands may include antibodies with different CDRs (Complementarity-Determining Regions).

[0151] As another example, the target molecule may be an enzyme comprising a CRISPR associated protein and the ligand may comprise a guide RNA molecule. The method may be performed to identify a combination of guide RNA molecule and CRISPR associated protein, in particular one that operates efficiently to edit genes. Such a method can involve determining a predicted structure of the enzyme, e.g. as described above, in particular to check that the enzyme shape and the guide RNA shape fit and work together effectively. The guide RNA may have a part with a defined 3D structure, e.g. it may be a single guide RNA (sgRNA), incorporating a guide sequence and a tracrRNA sequence.

[0152] The folding system described herein can also be used to obtain a diagnostic antibody or aptamer marker of a disease. There is also provided a method that comprises selecting a target molecule that is to be recognized by the antibody or aptamer marker, and for each of one or more candidate antibodies or aptamers e.g. as described above, uses the folding system to determine a predicted structure of the candidate antibody or aptamer or a complex comprising the target molecule and the candidate antibody or aptamer. The method may also involve, evaluating an interaction between each of the one or more candidate antibodies or aptamers and the target molecule, and selecting one of the one or more of the candidate antibodies or aptamers as the diagnostic antibody or aptamer marker dependent on a result of the evaluating, e.g. selecting one or more candidate antibodies or aptamers that have the highest affinity to the target protein. The method may include making the diagnostic antibody or aptamer marker. The diagnostic antibodyDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationor aptamer marker may be used to diagnose a disease by detecting whether it binds to the target molecule (e.g. protein) in a sample obtained from a patient, e.g. a sample of bodily fluid. As described above, a corresponding technique can be used to obtain a therapeutic antibody or aptamer (e.g. polypeptide or polynucleotide ligand).

[0153] Misfolded proteins or nucleic acids (e.g. RNA or DNA) are associated with a number of diseases. Thus in a further aspect there is provided a method of using the folding system to identify the presence of a protein or nucleic acid mis-folding disease. The method may include obtaining an amino acid sequence of a protein or nucleic acid sequence and using the folding system to determine a structure of the protein or nucleic acid. The method may further include obtaining a structure of a version of the protein obtained from a human or animal body, e.g., by conventional (physical) methods. The method may then include comparing the structure of the protein or nucleic acid with the structure of the version obtained from the body and identifying the presence of a protein or nucleic acid mis-folding disease dependent upon a result of the comparison. That is, mis-folding of the version of the protein or nucleic acid from the body may be determined by comparison with the in silico determined structure.

[0154] In general identifying the presence of a protein mis-folding disease may involve obtaining an amino acid sequence of a protein, using an amino acid sequence of the protein to determine a structure of the protein, as described herein, and comparing the structure of the protein with the structure of a baseline version of the protein, identifying the presence of a protein mis-folding disease dependent upon a result of the comparison. For example the compared structures may be those of a mutant and wild-type protein. In implementations the wild-type protein may be used as the baseline version but in principle either may be used as the baseline version. A nucleic acid mis- folding disease may be a chromatin mis-folding disease, for example.

[0155] In some other aspects a computer-implemented method as described above or herein may be used to identify active / binding / blocking sites on a target protein from its amino acid sequence.

[0156] The folding systems and methods described herein can also be used to determine the structure of a molecule or molecule complex. For example, an experimental technique may be applied to a physical sample comprising a molecule or molecule complex (i.e. the physical sample of the molecule complex is “interrogated” using the experimental technique) to measure experiment signals dependent on a structure of the molecule or molecule complex. In some implementations, the experimental technique may be a scattering technique or a spectroscopicDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationtechnique. The experimental technique may, for example, comprise one or more of: x-ray crystallography, nuclear magnetic resonance (NMR), and electron microscopy (e.g. cryogenic electron microscopy, cryo-EM). A folding system may be used to determine a predicted structure of the molecule complex. The experiment signals may then be compared with corresponding simulated signals generated using the predicted structure of the molecule complex. For example, the predicted structure of the molecule complex can be used to generate predicted x-ray diffraction patterns (e.g. from an electron density distribution determined using the predicted structure) that can be compared with experimentally measured x-ray diffraction patterns. For example, the experiment signals may comprise NMR signals, electron microscope images, or x-ray diffraction patterns; or signals derived therefrom.

[0157] The predicted structure of the molecule or molecule complex may be determined by adjusting the predicted structure of the molecule or molecule complex dependent upon a result of the comparison. The method may be performed iteratively, wherein for each of one or more iterations, after the predicted structure has been adjusted, the predicted signals may be generated for the adjusted structure, and the comparing and adjusting performed again to refine the predicted structure. Alternatively (or additionally), a plurality of different possible structures of the molecule or molecule complex can be predicted and the expected experiment signals from each compared with the actual experiment signals to determine a match, e.g. a best or most likely match, that can be taken as the determined structure of the molecule or molecule complex.

[0158] In general an aptamer as described above may comprise DNA or RNA. An enzyme as described above may comprise a protein or a DNA enzyme, e.g. a deoxyribozyme or “DNAzyme”, or an RNA enzyme, e.g. a ribozyme. As well as the applications described above such enzymes can also be used for biosensors of many types, e.g. DNAzymes and aptamers can be useful for detecting metal ions, and in general aptamer targets can include small molecule complexes, proteins, and cells. Aptamers have many uses including, e.g. as probes in assays, as biosensors (e.g. they can detectably change shape when binding to a target), to modulate the activity of biomolecule complexes, and to provide a controlled release mechanism. The techniques described herein can be used to design, and then make, such aptamers, sometimes referred to as chemical antibodies.

[0159] FIG. 2 is a flow diagram of an example process 200 for processing molecule complex data characterizing a molecule complex using an embedding neural network to generateDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationmolecule embedding data. For convenience, the process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a folding system, e.g., the folding system 100 of FIG.1, appropriately programmed in accordance with this specification, can perform the process 200.

[0160] The system receives molecule complex data characterizing a molecule complex (202). The molecule complex can include, e.g., one or more protein molecules, one or more DNA molecules, one or more DNA molecules with post-translational modifications, one or more RNA molecules, one or more RNA molecules with modifications, one or more ligand molecules, or any combination thereof. The molecule complex data can be expressed with reference to a tokenization scheme, where the input molecule complex is represented as a collection of tokens and the molecule complex data is expressed with reference to the tokenization scheme, as described with reference to FIG.1.

[0161] In some implementations, the molecule complex data includes at least some of the features described in the table shown below. In this table (and throughout this specification), ^^௧^^^^is the number of tokens in the molecule complex, ^^^^^is the number of MSA sequences in an MSA for the molecule complex, ^^௧^^^^is the number of templates in a set of template molecules, ^^^௧^^is the number of atoms in the molecule complex, ^^^^^^^^is the number of chains in a molecule, ^^^^^ௗ^is the number of bonds in a molecule, and ^^^^^^is the number of atom permutations. In the context of AtomCrossAttention operations (described in more detail below, e.g., with reference to FIG.9), ^^^௨^^^௧^is the size of the atom subsets, ^^^^௬^is the keys subset size, and ^^^௨^^^^^is the queries subset size. Feature & Shape Description Residue_index Residue number in the token’s original input chain ^^^௧^^^^^ Token_index Token number. Increase monotonically; does not restart at 1 for new ^^^௧^^^^^ chains. Asym_id Unique integer for each distinct chain. ^^^௧^^^^^ Entity_id Unique integer for each distinct sequence. ^^^௧^^^^^ Sym_id Unique integer within chains of this sequence. E.g. if chains A, B, and C ^^^௧^^^^^ share a sequence but D does not, their sym_ids would be [0, 1, 2, 0].DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT ApplicationRestype One-hot encoding of the sequence.31 possible values: 20 amino^^^௧^^^^, 31^acids+unknown, 8 nucleotides+unknown, and gap. Ligands represented as “unknown amino acid”. Seq_mask Mask indicating which positions contain a token and which are padding. ^^^௧^^^^^ Is_protein / rna / dna / ligand 4 masks indicating the molecule type of a particular token. ^^^௧^^^^^ Token_to_representative_index Index of the token’s “representative” atom, equivalent to an amino ^^^௧^^^^^ acid’s ^^ఉ. Token_to_center_index Index of the token’s “center” atom, equivalent to an amino acid’s ^^ఈ. ^^^௧^^^^^ Ref_pos Atom positions in the reference conformer, with a random rotation and^^^௧^^^^,^^^௧^^, 3^translation applied. During training, PDB-derived examples zero out this feature with rate 0.25. Ref_mask Mask indicating which atom slots are used in the reference conformer. ^^^௧^^^^,^^^௧^^^ Ref_element One-hot encoding of the element atomic number for each atom in the^^^௧^^^^,^^^௧^^, 128^reference conformer, up to atomic number 128. Ref_charge Charge for each atom in the reference conformer. ^^^௧^^^^,^^^௧^^^ Ref_atom_name_chars One-hot encoding of the unique atom names in the reference conformer.^^^௧^^^^,^^^௧^^, 4,64^Each character is encoded as ord(c) – 32, and names are padded to length 4. Feature & Shape Description Ref_space_uid Numerical encoding of the chain id and residue index associated with ^^^௧^^^^,^^^௧^^^ this reference conformer. Each (chain id, residue index) tuple is assigned an integer on first appearance. Token_atoms_to_res_permutations Stores symmetric permutations of the atoms in each token, calculated ^^^௧^^^^,^^^^^^,^^^௧^^൧ with RDKit’s GetSubstructMatches. Stored as indices into the full atom array. msa One-hot encoding of the processed MSA, using the same classes as^^^^^^,^^௧^^^^, 31^restype. Has_deletion Binary feature indicating if there is a deletion to the left of each position ^^^^^^,^^௧^^^^^ in the MSA Deletion_value Raw deletion counts (the number of deletions to the left of each MSA ^^^^^^,^^௧^^^^^ ଶ ௗ position) are transformed to [0,1] using గ arctan ଷ. Msa_mask Mask indicating which positions contain MSA data and which are ^^^^^^,^^௧^^^^^ padding.DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT ApplicationProfile Distribution across restypes in the main MSA. Computed before MSA^^^௧^^^^, 31^processing (subsection 2.3). Deletion_mean Mean number of deletions at each position in the main MSA. Computed ^^^௧^^^^^ before MSA processing (subsection 2.3) Template_restype One-hot encoding of the template sequence, see restype. ^^^௧^^^^,^^௧^^^^^ Template_atom_mask Mask indicating where a template atom exists and has coordinates. ^^^௧^^^^,^^௧^^^^,^^^௧^^^ Template_pseudo_beta_mask Mask indicating if the ^^ఉ(^^ఈfor glycine) has coordinates for the ^^^௧^^^^^,^^௧^^^^^ template at this residue. Template_backbone_frame_mask Mask indicating if coordinates exist for all atoms required to compute ^^^௧^^^^,^^௧^^^^^ the backbone frame (used in the template_unit_vector feature). Template_distogram A one-hot pairwise feature indicating the distance between ^^ఉatoms^^^௧^^^^, ^^௧^^^^, ^^௧^^^^, 39^(^^ఈfor glycine). Pairwise distances are discretized into 38 bins of equal width between 3.25 ^^°and 50.75; ^^°one more bin contains any larger distances. Template_unit_vector The unit vector of the displacement of the ^^ఈatom of all residues^^^௧^^^^, ^^௧^^^^, ^^௧^^^^, 3^within the local frame of each residue. Local frames are computed as in [1] Token_bonds A 2D matrix indicating if there is a bond between any atom in token ^^ ^^^௧^^^^,^^௧^^^^^ and token ^^, restricted to just polymer-ligand and ligand-ligand bonds and bonds less than 2.4 ^^°during training. Token_atoms_to_queries Indices for mapping from the array of atoms to the array of queries in ^^^^௨^^^௧^,^^^௨^^^^^^ AtomCrossAttention. Feature & Shape Description Tokens_to_queries Indices for mapping from the array of tokens to the array of queries in ^^^^௨^^^௧^,^^^௨^^^^^^ AtomCrossAttention Tokens_to_keys Indices for mapping from the array of tokens to the array of keys in ^^^^௨^^^௧^,^^^^௬^^ AtomCrossAttention. Queries_to_keys Indices for mapping from the array of queries to the array of keys in ^^^^௨^^^௧^,^^^^௬^^ AtomCrossAttention. Queries_to_token_atoms: Indices for mapping from the array of queries to the array of atoms in ^^^௧^^^^,^^^௧^^^ AtomCrossAttention. All_chains_entity_ids Entity_id for all chains in the input, not just those included in the crop. ^^^^^^^^^^ Dense_all_crops_all_chains_pos Records atom positions for the cropped-in regions across all chains.^^^^^^^^^, ^^௧^^^^, ^^^௧^^, 3^DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT ApplicationAll_crops_all_chains_residue_ids For each chain, records the residue ids in the cropped-in region across ^^^^^^^^^,^^^^^^^^,^^௧^^^^^ all other chains Dense_all_crops_all_chains_mask For each chain, records the atoms present in the cropped-in region ^^^^^^^^^,^^^^^^^^,^^௧^^^^,^^^௧^^^ across all other chains. Pred_dense_atom_mask Mask indicating which atoms should be present in the output prediction. ^^^௧^^^^,^^^௧^^^ Does not include information about which atoms were experimentally resolved.

[0162] The system generates a respective initial single embedding for each token in the molecule complex and a respective initial pair embedding for each pair of tokens in the molecule complex (204).

[0163] In particular, for each token, the system can process data characterizing the unit (e.g., amino acid or nucleotide or atom) represented by the token by one or more neural network layers of the embedding neural network (e.g., fully connected layers, or convolutional layers, or attention layers, and so forth) to generate the initial single embedding for the token. Data characterizing a token can include token-specific data (that is specific to the token) and global data (that characterizes the molecule complex as a whole). The token-specific data for a token can include, e.g., one or more of: data characterizing a position of the token (e.g., as an index that identifies a position of the token in one or more sequence of tokens included in the molecule); data identifying which chain of the molecule complex includes the token; data characterizing a type of the token (e.g., the identity of the amino acid or nucleotide represented by the token, or the element number of an atom represented by the token); and so forth. The global data can include, e.g., data indicating whether the molecule complex includes protein molecules, DNA molecules, RNA molecules, ligand molecules, and so forth.

[0164] In some implementations, the system generates the initial single embeddingsby operations including: ^^^௧^^^^௧^ ^ = TargetFeatureEmbedder({f*}) ^^^^^^௧^ = Li௧^^^^௧^ near ^^^^^ where ^^^∗^ denotes the molecule complex data and the operations of TargetFeatureEmbedder() are described in more detail below.

[0165] In some implementations, the operations of TargetFeatureEmbedder() include: def TargetFeatureEmbedder({f*}) :DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Application# Embed per-atom features. 1: ^^^^^ AtomCrossAttentionEncoder^^^^∗^,∅,∅,∅^3: return ^^^^^ where ^^^∗^ denotes the molecule complex data and the operations of AtomCrossAttentionEncoder are described in more detail below.

[0166] In some implementations, the operations of AtomCrossAttentionEncoder include: def AtomCrossAttentionEncoder ({f*}, ^^⃗^^^}, ^^^^௧^௨^^}, {Z^^^^^, c=32) :DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Application16: ^^^ ൌ ^^^^^^^^ ^^^^^^^^^൫^^^^^^^^^^^^^^^^^൯^ Codewhere the operations of CrossAttTransformer() are described in more detail below.

[0167] In some implementations, the operations of CrossAttTransformer() include:def CrossAttTransformer ^^^^^^, ^^^^^, ^^^^^^^:^^ ^^^ , ^^^ , ^^^^ , ^^^^3: return ^^^^^ where the operations of DiffusionTransformer() are described in more detail below.

[0168] In some implementations, the operations of DiffusionTransformer() include:def DiffusionTransformer^^^^^^, ^^^^^, ^^^^^^,^^^^^^^ ൌ 24^1: for all ^^ ∈ ^1, … ,^^^^^^^^ do2: ^^^^^^ൌ AttentionPairBias^^^^^^, ^^^^^, ^^^^^^^3: ^^^^^^ൌ ConditionedTransitionBlock^^^^^^, ^^^^^^4: end for 5: return ^a^^ where the operations of AttentionPairBias() and ConditionedTransitionBlock() are described in more detail below.

[0169] In some implementations, the operations of AttentionPairBias() include:def AttentionPairBias ^^^^^^, ^^^^^, ^^^^^^, ^^^^^^^# Input projections1: ^^^ ← AdaLNInput^^^^ , ^^^^DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Application5: ^^^ ← ^^^^^^^^^^^^^^ ^^^ୃ^ ^^^ ^ √^^^^^^^ ^ ^^^^ ^ ^^^^^6: ^^^^ ^ ^^← ^^^^^^^^^^^^^^# Output projection7: ^^^ ← ^^^^^^^^^^^^^^^^^^^^^^^^^ ^ ^^൫^^^ ൯^ ^^^ ∈ ℝ ^8: return ^AdaLNOutput^^^^ , ^^^^^where the operations of AdaLNInput() and AdaLNOutput are described in more detail below.

[0170] In some implementations, the operations of AdaLNInput include: def AdaLNInput (a,s) 1: v ← LayerNorm^s, scale ൌ False, offset ൌ False^2: m ← Linear^s^3: k ← LinearNoBias^s^4: return sigmoid^m^ ∗ a ^ k

[0171] In some implementations, the operations of AdaLNOutput include: def AdaLNOutput (a,s) 1: s ← LayerNorm^s, scale ൌ False, offset ൌ False^2: m ← Linear^s, biasinit ൌ െ2.0^3: k ← LinearNoBias^s^4: return sigmoid^m^ ∗ a ^ k

[0172] In some implementations, the operations of ConditionedTransitionBlock() include: def ConditionedTransitionBlock(a,s) 1: b ← AdaLNInput^a, s^2: ^^ ← ^^^^^^^^^^^^^^^^3: d ← Linear^b^4: e ← swish^c^ ∗ d5: f ← Linear^e^6: return AdaLNOutput^f, s^

[0173] The system can generate the initial pair embeddings by operations including:DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT ApplicationinitWhere ^^௧^^^^௧^ denotes output of a TargetFeatureEmbedder() operation (as described above) for token ^^, and the operations of RelativePositionEncoding() are described in more detail below.

[0174] In some implementations, the operations of RelativePositionEncoding() include:def RelativePositionEncoding ^^^^∗^ , ,10: ^^^^ ൌ Linear^concat^^^^ ୰^୪_^୭୩^୬ ୰^୪_ୡ୦ୟ୧୬ ே^^, ^^^^ , ^^^^,^^^^ ^^^ ^^^^ ∊ ℝ ^^^^^^^౩11: return ^^^^^^

[0175] The system initializes a respective “current” single embedding for each token in the molecule complex and a respective “current” pair embedding for each pair of tokens in the molecule complex (206). The current single and current pair embeddings will aggregate the results of a sequence of one or more “recycling” iterations that are described below with reference to FIG.3. The system can initialize the current single and current pair embeddings in any appropriate manner, e.g., to default values, e.g., zero. For instance, the system can initialize the current single and current pair embeddings as:^^̂^^^^, ^^̂^^^ ൌ 0,0where ^^̂^^,^^ are the current pair embeddings and ^^̂^^^ are the current single embeddings.DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Application

[0176] The system iteratively updates the current single embeddings and the current pair embeddings at each iteration in a sequence of one or more iterations, which may be referred to for convenience as “recycling” iterations (208). The sequence of recycling iterations can include any appropriate number of recycling iterations, e.g., 1, 5, 10, or 100 recycling iterations. Each recycling iteration can update the current single embeddings and current pair embeddings based at least in part on the initial single embeddings and the initial pair embeddings (e.g., as generated at step 204). More specifically, the current single embeddings and current pair embeddings can be initialized, e.g., to default values (as described at step 206), and then iteratively updated over a sequence of recycling iterations. Each recycling iteration after the first recycling iteration can operate on the current single embeddings and current pair embeddings generated at the conclusion of the preceding recycling iteration. FIG. 3 describes an example process for performing a recycling iteration to update the current single embeddings and the current pair embeddings.

[0177] The system outputs the current single embeddings and the current pair embeddings, as of the last recycling iteration in the sequence of one or more recycling iterations, as the molecule embedding data (210).

[0178] FIG. 3 is a flow diagram of an example process 300 for performing a recycling iteration to update a respective current single embedding for each token in the molecule complex and a respective current pair embedding for each pair of tokens in the molecule complex. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a folding system, e.g., the folding system 100 of FIG.1, appropriately programmed in accordance with this specification, can perform the process 300.

[0179] The system obtains: (i) initial single embeddings and initial pair embeddings, and (ii) current single embeddings and current pair embeddings (302). An example process for generating initial single embeddings and initial pair embeddings is described with reference to step 204 of FIG.2. If the current recycling iteration is the first recycling iteration, then the current single embeddings and the current pair embeddings can have default values, e.g., as described with reference to step 206 of FIG. 2. If the current recycling iteration is after the first recycling iteration, then the current single embeddings and the current pair embeddings can have values generated at the conclusion of the preceding recycling iteration.DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Application

[0180] The system updates the current pair embeddings using the initial pair embeddings (304). For instance, the system can update the current pair embeddings by operations including:^^ ^^^^^^^^^^ ൌ ^^ ^^^^^^ ^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^൫^^^^^^^൯^where ^^^^^^^௧are the initial pair embeddings, ^̂^^^are the current pair embeddings received at the start of the recycling iteration, and ^^^^denote the values of the current pair embeddings after being updated using the initial pair embeddings.

[0181] Optionally, the system updates the current pair embeddings using one or more template molecules (306). For instance, the system can update the current pair embeddings by operations including:^^^^^^^ൌ TemplateEmbedder^^^^∗^, ^^^^^^^where the ^^^∗^ is the molecule complex data, ^^^^are the current pair embeddings, and the operations of TemplateEmbedder() can include:7: ^^^^^ൌ ^^^^8: end forwhere PairFormerStack() refers to a process for updating the “temporary” pair embeddings ^^^^^^ over a sequence of update blocks. An example process for updating a set of pair embeddings by an update block is described with reference to FIG. 4.

[0182] Optionally, in implementations where the molecule complex includes a protein molecule or a DNA molecule or an RNA molecule, the system updates the current pair embeddings usingDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT ApplicationMSA features for the molecule (308). For instance, the system can update the current pair embeddings by operations including:^^^ ^^௧^^^^௧^^^^ൌ MSAEmbedder^^^^ ^^^ ^, ^^^^^^, ^^^^ ^^where ^^^^^are MSA features, ^^^ ^ are the current௧^^^^௧ௌ^ ^^pair embeddings, ^^^can be generated by the TargetFeatureEmbedder() operations described above, and the operations of MSAEmbedder() can include: def MSAEmbedder൫^^^^^^^^ ^, ^^^ ௧^^^^, ^^^ ^^^௧^ ^,^^^^^^^ ൌ 4൯:1: ^^^^ ൌ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ℎ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^3: ^^^^ ൌ Linear^^^^^^^^^4: ^^ ^ൌ L௧^^^^௧^^ inear൫^^^^^൯ 5: for all ^^ ∈ ^1, … ^^^^^^^^ do# Communication 6: ^^^^^^^ൌ OuterProductMean^^m^୧^^# MSA Stack 7: ^^^ ^^ൌ DropoutRowwise^.^ହ8: ^^^ ^^ൌ Transition^# Pair Stack 9: ^^^^^^^ൌ DropoutRowwise^.ଶହ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^൫^^^^^^൯^10: ^^^^^^^ൌ DropoutRowwise^.ଶହ^TriangleMultiplicationIncoming^^z୧୨^^^11: ^^^^^^^ൌ DropoutRowwise^.ଶହ^TriangleAttentionStartingNode൫^^^^^^൯^12:13: Transition൫^^^^^14: end15: return ^^^^^^ where SampleRandomWithoutReplacement() can include operations that randomly sub-sample a set of sequence from the MSA without replacement, the operations of OuterProductMean() and MSARowAttentionOnlyPairBias() are described in more detail below, and the operations of Transition(), TriangleMultiplicationOutgoing(), TriangleMultiplicationIncoming(), TriangleAttentionStartingNode(), TriangleAttentionEndingNode(), and Transition() are described in more detail with reference to FIG.4.DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Application

[0183] The operations of OuterProductMean() can include:def OuterProductMean^^^^^^^, ^^ ൌ 32^:1: ^^^^ ← LayerNorm^^^^^^2: ^^ , ^^ ൌ Linear^^^ ^ ^^ ^^^ ^^ ^^ ^^ , ^^^^ ∈ ℝ3: ^^^^ ൌ4: ^^^^ ൌ Linear^^^ ^^^^ ^^^^ ∈ ℝ ^5: return ^^^^^^

[0184] The operations of MSARowAttentionOnlyPairBias() can include: def MSARowAttentionWithPairBias൫^^^^^^,^^ ൌ 32,^^^^^ௗ ൌ 8൯:# Input projections8: return ^^^^^^^

[0185] The system updates the current single embeddings using the initial single embeddings (310). For instance, the system can update the current single embeddings by operations including:s ୧୬୧^୧ ൌ s୧ ^where ^^^^^^௧are the initial single embeddings, ^̂^^are the current single embeddings at the start of the recycling iteration, and ^^^denote the values of the current single embeddings after being updated using the initial single embeddings.

[0186] The system updates the current single embeddings and the current pair embeddings by processing the current single embeddings and the current pair embeddings using a sequence of update blocks of the embedding neural network (312). Each update block is configured toDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationreceive as input a set of current single embeddings and a set of current pair embeddings, and to process the input in accordance with values of a set of update block parameters to generate a set of updated single embeddings and a set of updated pair embeddings. The first update block is configured to receive the current single embeddings obtained at step 310 and the current pair embeddings at step 304, 306, or 308. Each update block after the first update block is configured to receive the current single embeddings and the current pair embeddings produced by the preceding update block in the sequence of update blocks.

[0187] The sequence of update blocks can include any appropriate number of update blocks, e.g., 1, 10, 50, or 100 update blocks. Each update block can have any appropriate architecture that enables the block to perform its described functions. In particular, each block can include any appropriate types of neural network layers (e.g., fully connected layers, convolutional layers, attention layers, and so forth) in any appropriate number (e.g., 5 layers, 10 layers, or 20 layers) and connected in any appropriate configuration (e.g., as a directed graph of layers). An example process that can be performed by an update block to update a set of single embeddings and a set of pair embeddings is described in more detail with reference to FIG.4.

[0188] The system outputs the current single embeddings and the current pair embeddings (314). If the current recycling iteration is before a final recycling iteration in the sequence of recycling iterations, then the system can provide the current single embeddings and the current pair embeddings for processing at the next recycling iteration. If the current recycling iteration is the final recycling iteration in the sequence of recycling iteration, the system can output the current single embeddings and the current pair embeddings as the molecule embedding data, as described at step 210 of FIG. 2.

[0189] FIG. 4 is a flow diagram of an example process 400 by which an update block of a neural network (e.g., an embedding neural network) updates a set of single embeddings and a set of pair embeddings. For convenience, the process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a folding system, e.g., the folding system 100 of FIG.1, appropriately programmed in accordance with this specification, can perform the process 400.

[0190] The system receives a set of single embeddings and a set of pair embeddings (402). Each single embedding represents a respective token in the molecule complex and each pair embedding represents a respective pair of tokens in the molecule complex.DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Application

[0191] The system processes the set of pair embeddings using a set of neural network layers arranged in a directed graph to update the set of pair embeddings (404). The set of neural network layers can perform operations such as, e.g., self-attention operations, e.g., triangle self- attention operations. For instance, the system can update the set of pair embeddings by performing operations including: DropoutRowwise^.ଶହ^TriangleMultiplicationOutgoing൫^^^^^^൯^^^^^^^^ൌ DropoutRowwise^.ଶହ^TriangleAttentionStartingNode൫^z୧୨^൯^^^^^^^^ൌ DropoutColumnwise^.ଶହ^TriangleAttentionEndingNode൫^z୧୨^൯^^^^^^^^ൌ Transition^^z୧୨^^where ^^^^^^ are the pair embeddings (with ^^ and ^^ indexing the tokens in the molecule complex), DropoutRowwise() is a row-wise dropout operation, and DropoutColumnwise() is a column-wise dropout operation. The TriangleMultiplicationOutoing(), TriangleMultiplicationIncoming(), TriangleAttentionStartingNode(), TriangleAttentionEndingNode(), and Transition() operations are described in more detail below.

[0192] In some implementations, the TriangleMultiplicationOutgoing() operations can include:1: ^^ ←2: ^^ , ^^ ൌ sigmoid^Linear z ^ ⊙ Linear^ ^^^ ^^ ൫ ୧୨൯ ^^^^^ ^^^^ , ^^^^ ∈ ℝ3:^^^^ ൌ sigmoid^Linear൫^^^^൯^ ^^ ^^^ ∈ ℝ ^4: ^̃^^^ ൌ ^^^^ ⊙ Linear^LayerNorm൫^^^^^^^ ⊙ ^^^^൯^ ^̃^^^ ∈ ℝ^^

[0193] In some implementations, the TriangleMultiplicationIncoming() operations can include:1: ^^^^ ← LayerNorm^^^^^^2: ^^ , ^^ ൌ sigmoid ^Linear൫^^ ൯^ ⊙ Lin ^^^ ^^ ^^ ear^^^^^^^ ^^^^ , ^^^^ ∈ ℝ3:^^^^ ൌ sigmoid^Linear൫^^^^൯^ ^^ ^^^ ∈ ℝ ^4: ^̃^^^ ൌ ^^^^ ⊙ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^൫Σ ^^^^^^ ⊙ ^^^^൯^ ^̃^^^ ∈ ℝ ^

[0194] In some implementations, the TriangleAttentionStartingNode() operations can include: # Input projections1: ^^^^ ← LayerNorm൫^^^^൯DeepMind Technologies Limited F&R Ref.: 45288-0393WO1 PCT Application2: ^^^^^ ,^^^^^ , v ^^^ ൌ LinearNoBias^^^^^^ ^^^^^ ^ ^^,^^^^ , v^^ ∈ ℝ ,ℎ ∈ ^1, … ,^^^^^ௗ^3: ^^ ^^^ ൌ LinearNoBias^^^^^^

[0195] In some implementations, the TriangleAttentionEndingNode() operations can include:

[0196] In some implementations, the Transition() operations can include:def Transition^x,^^ ൌ 4^: ^^ ∈ ℝ^1: x ← LayerNorm^x^2: a ൌ Linear^x^ a ∈ ℝ^∙^3: b ൌ Linear^x^ b ∈ ℝ^∙^4: x ← Linear^swish^a^ ⊙ b^ x ∈ ℝ^5: return x

[0197] The system updates the single embeddings using one or more self-attention operations that are conditioned on the set of pair embeddings (406). An example process for applying a self- attention operation to a set of single embeddings that is conditioned on a set of pair embeddings is described with reference to FIG. 5.DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Application

[0198] The system outputs the updated single embeddings and the updated pair embeddings (408).

[0199] FIG. 5 is a flow diagram of an example process 500 for updating a set of single embeddings using a self-attention operation that is conditioned on a set of pair embeddings. For convenience, the process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a folding system, e.g., the folding system 100 of FIG.1, appropriately programmed in accordance with this specification, can perform the process 500.

[0200] The system receives: (i) a set of single embeddings, and (ii) a set of pair embeddings (502). The set of single embeddings includes a respective single embedding for each token in the molecule complex. The set of pair embeddings includes a respective pair embedding for each pair of tokens in the molecule complex.

[0201] The system generates a set of intermediate attention scores based on the set of single embeddings (504). The set of intermediate attention scores includes a respective attention score for each pair of single embeddings from the set of single embeddings.

[0202] The system can generate the intermediate attention scores in any of a variety of possible ways. For instance, the system can generate a respective query embedding for each single embedding by processing the single embedding using a query neural network, e.g., as: ^^ ൌ ^^ொ ⋅ ^^ ^1^where ^^ is a matrix where each column (or row) defines a respective query embedding, ^^ொis a matrix of parameter values (defining the query neural network in this example), and ^^ is a matrix where each column (or row) defines a respective single embedding. Further, the system can generate a respective key embedding for each single embedding by processing the single embedding using a key neural network, e.g., as: ^^ ൌ ^^^ ⋅ ^^ ^2^where ^^ is a matrix where each column (or row) defines a respective key embedding, ^^^is a matrix of parameter values (defining the key neural network in this example), and ^^ is a matrix where each column (or row) defines a respective single embedding. The system can generate the intermediate attention scores based on the query embeddings and the key embeddings, e.g., as:DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationwhere ^^ is a matrix of intermediate attention scores, ^^ is the matrix of query embeddings, and ^^ is the matrix of key embeddings.

[0203] The system generates a set of attention score biases based on the set of pair embeddings (506). The set of attention score biases includes a respective attention score bias for each pair of single embeddings from the set of single embeddings.

[0204] The system can generate the set of attention score biases in any of a variety of possible ways. For instance, for each pair of single embeddings, the system can generate the attention score bias for the pair of single embeddings by processing a corresponding pair embedding from the set of pair embeddings using a projection neural network. In particular, for a pair of single embeddings that includes: (i) a first single embedding representing a first token, and (ii) a second single embedding representing a second token, the system can generate the attention score bias for the pair of single embeddings by processing the pair embedding that jointly represents the first token and the second token.

[0205] The projection neural network can have any appropriate neural network architecture that enables the projection neural network to perform its described functions, e.g., processing a pair embedding to generate an attention score bias. In particular, the projection neural network can include any appropriate number of neural network layers (e.g., 1 layer, or 5 layers, or 10 layers) in any appropriate number (e.g., 1 layer, 3, layers or 5 layers) and connected in any appropriate configuration (e.g., as a directed graph of layers).

[0206] The system generates a set of final attention scores by combining: (i) the intermediate attention scores, and (ii) the attention score biases (508). The set of final attention scores includes a respective final attention score for each pair of single embeddings in the set of single embeddings. The system can generate the final attention score for a pair of single embeddings by combining (e.g., summing): (i) the intermediate attention score for the pair of single embeddings, and (ii) the attention score bias for the pair of single embeddings. Optionally, the system can apply further processing operations to the set of final attention scores, e.g., by applying a soft- max operation to some or all of the final attention scores.

[0207] The system generates a set of updated single embeddings using: (i) the set of single embeddings, and (ii) the set of final attention scores (510). For instance, to generate the set of updated single embeddings, the system can generate a respective value embedding for each single embedding in the set of single embeddings by processing the single embedding using aDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationvalue neural network, e.g., as: ^^ ൌ ^^^ ⋅ ^^ ^4^where ^^ is a matrix where each column (or row) defines a respective value embedding, ^^^is a matrix of parameter values (defining the value neural network in this example), and ^^ is a matrix where each column (or row) defines a respective single embedding. The system can then generate the set of updated single embeddings, e.g., as: ^^ᇱ ൌ ^^ ⋅ ^^ ^5^where each column (or row) of ^^′ defines a respective updated single embedding, each column (or row) of ^^ defines a respective value embedding, and ^^ denotes the set of final attention scores arranged into a matrix.

[0208] In implementations where the self-attention block implements a multi-head attention operation, each head of the attention operation can individually perform the steps of the process 500, and the updated single embeddings generated by each attention head can be combined (e.g., concatenated) to define the overall output of the multi-head attention operation. Each attention head can have a respective set of neural network parameters, having values that are specific to each attention head, that are used for generating the intermediate attention scores and the attention score biases.

[0209] FIG. 6 is a flow diagram of an example process 600 for generating a predicted 3D structure of a molecule complex using a generative diffusion model that includes a denoising neural network. For convenience, the process 600 will be described as being performed by a system of one or more computers located in one or more locations. For example, a folding system, e.g., the folding system 100 of FIG.1, appropriately programmed in accordance with this specification, can perform the process 600.

[0210] The system generates a respective initial 3D spatial position for each atom in the molecule complex (602). For instance, for each atom in the molecule complex, the system can sample a 3D spatial position of the atom from a probability distribution over 3D space, e.g., a standard Normal distribution over 3D space.

[0211] The system performs steps 604, 606, 608, 612, which are described next, over a sequence of iterations that may be referred to as “time steps”. The description of steps 604, 606, 608, 612 which follows will reference a “current” time step for convenience; the current time step can be any time step in the sequence of time steps. The system can perform the steps 604, 606, 608, 612DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationover any appropriate number of time steps, e.g., 3 time steps, 10 time steps, or 100 time steps. The number of time steps can be a predetermined number of time steps.

[0212] The system generates a denoising output using a denoising neural network that is conditioned on the molecule embedding data (604). The denoising output can be any appropriate data that enables estimation of the “final” 3D spatial position of each atom in the molecule. For instance, the denoising output can define, for each atom in the molecule complex, a predicted error in the 3D spatial position of the atom at the current time step. As another example, the denoising output can directly define, for each atom in the molecule complex, a predicted 3D spatial position of the atom. As another example, the denoising output can define, for each atom in the molecule complex, both: (i) a predicted error in the 3D spatial position of the atom at the current time step, and (ii) a predicted 3D spatial position of the atom. As another example, the denoising output can define, for each atom in the molecule complex, a prediction for a value that is a linear combination of: (i) an actual 3D spatial position of the atom, and (ii) an error between the 3D spatial position of the atom at the current time step and the actual 3D spatial position of the atom, e.g., as implemented by the v-parametrization described in: Tim Salimans, Jonathan Ho, “Progressive distillation for fast sampling of diffusion models,” ICLR 2022, arXiv:2202.00512v2. An example process for generating a denoising output using the denoising neural network is described in more detail with reference to FIG.7.

[0213] The system generates an initial estimate of the 3D spatial position for each atom in the molecule complex using at least the denoising output generated by the denoising neural network (606). The system can generate the initial estimate of the 3D spatial position for each atom in the molecule complex in any appropriate way, depending on the form of the denoising output. A few example techniques for generating the initial estimate of the 3D spatial position for each atom in the molecule complex using the denoising output are described next.

[0214] In one example, the denoising output defines, for each atom, a respective prediction for the 3D spatial position of the atom. In this example, the respective predicted 3D spatial position for each atom defines the initial estimate of the 3D spatial position for the atom.

[0215] In another example, the denoising output defines, for each atom, a predicted error in the 3D spatial position of the atom at the current time step. In this example, the system can generate the initial estimate for the 3D spatial position for each atom as a linear combination of: (i) the current 3D spatial position of the atom, and (ii) the predicted error in the 3D spatial position ofDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationthe atom. Each term in the linear combination can be scaled by a respective constant value that is dependent on the time step. For instance, the system can generate the initial estimate for the 3D spatial position ^^௧ି^for an atom in the molecule complex as:where ^^ indexes the current time step, ^^௧, ^ത^௧, and ^^௧are constants specific to time step ^^, and^^ఏ^^^௧ , ^^^ is the predicted error in the 3D spatial position of the atom (e.g., as generated by thedenoising neural network at the time step). (In the notation of equation (6), the time stepsdecrement, such that time step ^^ െ 1 is the “next” time step after time step ^^). The constants inequation (6) (^^௧, ^ത^௧, and ^^௧) can be selected in accordance with a predefined noise schedule.

[0216] In another example, the denoising output defines, for each atom, both: (i) a predicted 3D spatial position of the atom, and (ii) a predicted error in the 3D spatial position of the atom at the current time step. In this example, the system can generate the initial estimate for the 3D spatial position of the atom as a combination (e.g., an average) of: (i) the predicted 3D spatial position of the atom as specified by the denoising output, and (ii) a predicted 3D spatial position of the atom that is derived from the predicted error in the 3D spatial position of the atom at the current time step.

[0217] In another example, the denoising output is expressed using a v-parametrization, and the system generates a respective initial estimate for the 3D spatial position for each atom using the techniques described in Tim Salimans, Jonathan Ho, “Progressive distillation for fast sampling of diffusion models,” ICLR 2022, arXiv:2202.00512v2.

[0218] If the current time step is the final time step (i.e., in the sequence of denoising time steps), the system outputs the initial estimate of the 3D spatial position of each atom in the molecule complex as the predicted 3D structure of the molecule complex (610).

[0219] If the current time step is not the final time step, the system generates a respective 3D spatial position for each atom for the next time step based on the initial estimates of the 3D spatial positions of the atoms (as generated at step 606) using an appropriate diffusion sampling technique (612). A few examples of possible diffusion sampling techniques are described next.

[0220] In one example, the system can generate the 3D spatial position of each atom in the molecule complex at the next time step by combining random noise with the initial estimate of the 3D spatial position of the atom. For instance, for each atom, the system can add respective random noise to the initial estimate of the 3D spatial position of the atom. The random noise canDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationbe sampled from a probability distribution over 3D space. The probability distribution over 3D space can vary based on the time step, e.g., such that the variance of the noise combined with the updated 3D spatial positions of the atoms decreases over the sequence of time steps.

[0221] As another example, the system can generate the 3D spatial position of each atom in the molecule complex at the next time step using a deterministic diffusion sampling technique, i.e., that does not rely on random noise. An example of a deterministic diffusion sampling technique is the denoising diffusion implicit model (DDIM), e.g., as described in: Jiaming Song, Chenlin Meng, Stefano Ermon, “Denoising diffusion implicit models,” ICLR 2021, arXiv:2010.02502v4.

[0222] Optionally, the system can perform the process 600 multiple times to generate multiple predictions for the 3D structure of the molecule complex. Each execution of the steps of the process 600 can result in the generation of a different predicted 3D structure of the molecule complex, e.g., as a result of stochasticity in the random sampling performed to generate the initial positions of the atoms (at step 602), and in some cases, as a result of stochasticity in the diffusion sampler (at step 612).

[0223] FIG. 7 is a flow diagram of an example process 700 for generating a denoising output using a denoising neural network conditioned on molecule embedding data. For convenience, the process 700 will be described as being performed by a system of one or more computers located in one or more locations. For example, a folding system, e.g., the folding system 100 of FIG.1, appropriately programmed in accordance with this specification, can perform the process 700.

[0224] The system receives: (i) data defining a respective current 3D spatial position of each atom in the molecule complex, and (ii) the molecule embedding data (702). The molecule embedding data can be token-resolution molecule embedding data that includes a respective single embedding for each token in the molecule complex and a respective pair embedding for each pair of tokens in the molecule complex. The system can optionally receive additional inputs, e.g., an input defining a current time step in a diffusion process being implemented by a generative diffusion model, as described above with reference to FIG.6.

[0225] Optionally, the system can update the molecule embedding data, e.g., using a DiffusionConditioning() function: ^s ∗ ௧^^^^௧ ௧^௨^^^^, ^z^^^ ൌ DiffusionConditioning^^f ^, ^s௧^௨^^^^, ^z^^ ^, ^^^^^,^^^ where ^^^^^ are the updated single embeddings, ^^^^^^ are the updated pair embeddings, ^^∗is the molecule complex data, ^^௧^^^^௧^ are embeddings generated using the TargetFeatureEmbedder()DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationfunction described with reference to FIG. 2, ^^^^^^୰^୬୩^ denotes the original (non-updated) pair embeddings, ^^^^௧^௨^^^ denotes the original (non-updated) single embeddings, and ^^ denotes a scalar parameter.

[0226] In some implementations, the operations of DiffusionConditioning() can include:def DiffusionConditioning൫^^^∗^, ^^^ ^ୟ୰^^^^ ^, ^^^^^^, ^^^^^,^^൯:# Pair Conditioning 1: z୧୨ ൌ concat൫^z୧୨, RelativePositionEncoding^^f ∗^^൧൯2: z୧୨ ൌ3: ^^^^^^ ^^^^^^ ^^ ∈ ^1,2^ do5: end for # Single Conditioning 6: s ൌ concat^^^ୟ୰^^^୧s୧, s୧൧^ 7: s୧ ൌ Linear^LayerNorm^s୧^^8:^^ ൌ FourierEmbedding^^ ସ log ^^^ / ^^ௗ^௧^^) 9: s୧^ൌ Linear^LayerNorm^n^^10: ^^^^^^ ^^^^^^ ^^ ∈ ^1,2^ ^^^^11: ^^^^ൌ Transition^^^^ ,^^ ൌ 2^12: ^^^^^^ ^^^^^^13: return ^^^^^, ^^^^^^where the operations of RelativePositionEncoding() are described with reference to FIG.2, the operations of Transition() are described with reference to FIG. 4, and the operations of FourierEmbedding() can include: def FourierEmbedding^^^^ # Randomly generate weight / bias once at compile time 1: w, b~^^^0.1^# Compute embeddings 2: return cos ^2πσ^w ^ b^^DeepMind Technologies Limited F&R Ref.: 45288-0393WO1 PCT Application

[0227] The system generates a respective atom embedding for each atom in the molecule complex, using an encoder block of the denoising neural network, based at least in part on the current 3D spatial position of the atom (704). Optionally, the system can generate the atom embeddings for the atoms based at least in part on the molecule embedding data (and, optionally, the current time step in the diffusion process), e.g., in addition to the current 3D spatial positions of the atoms. Example operations for generating atom embeddings for the atoms in the molecule complex are described in more detail below in relation to the description of step 706.

[0228] The system generates atom-resolution molecule embedding data (706). In particular, the molecule embedding data may initially be token-resolution molecule embedding data, that includes: (i) a respective single embedding for each token in the molecule complex, and (ii) a respective pair embedding for each pair of tokens in the molecule complex. The system can process the token-resolution molecule embedding data to generate atom-resolution molecule embedding data that includes: (i) a respective single embedding for each atom in the molecule complex, and (ii) a respective pair embedding for each pair of atoms in the molecule complex.

[0229] Example operations for generating atom embeddings for the atoms in the molecule complex and atom-resolution molecule embedding data is shown next:8: ^^^^ ^x^⃗ୟ୧ ^ ് ∅ ^^^^^^^^# Broadcast the single and pair embedding from the trunk.9: c^^ൌ Linear^LayerNorm൫^^^^^୰^^^୬୩൯^10: ^^^^^ൌ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^൫^^^^^^^^^^൯^DeepMind Technologies Limited F&R Ref.: 45288-0393WO1 PCT Applicationwhere ^^∗is the molecule complex data, ^^^^^^ are the current 3D spatial positions of the atoms, ^^^^௧^௨^^^ are the token-resolution single embeddings, ^^^^^^ are the token-resolution pair embeddings, ^^^^^ are the atom-resolution single embeddings, ^^^^^^ are the atom-resolution pair embeddings, and ^^^^^ are the atom embeddings.

[0230] The system processes the set of atom embeddings using an update block of the denoising neural network to generate a set of updated atom embeddings (708). The update block applies one or more self-attention operations to the set of atom embeddings to update the set of atom embeddings, where the self-attention operations are conditioned on the molecule embedding data (e.g., the atom-resolution molecule embedding data described above). An example process for generating a set of updated atom embeddings using an update block is described in more detail with reference to FIG. 8.

[0231] The system processes the updated atom embeddings (i.e., as generated by the update block of the denoising neural network) using a decoder block of the denoising neural network to generate the denoising output (710). Examples of denoising outputs are described above with reference to FIG.6. The decoder block of the denoising neural network can include any appropriate types of neural network layers (e.g., fully connected layers, convolutional layers, and so forth) in any appropriate number (e.g., 1 layer, 3, layers or 5 layers) and connected in any appropriate configuration (e.g., as a directed graph of layers).

[0232] In a particular example, the decoder block can include a sequence of fully connected neural network layers that are configured to operate separately on each updated atom embedding to generate a predicted error in the 3D spatial position of the corresponding atom at the current time step, or to generate a predicted 3D spatial position of the corresponding atom, or both.

[0233] FIG. 8 is a flow diagram of an example process 800 for updating a set of atoms embeddings for the atoms in the molecule complex using an update block of the denoising neuralDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationnetwork. For convenience, the process 800 will be described as being performed by a system of one or more computers located in one or more locations. For example, a folding system, e.g., the folding system 100 of FIG.1, appropriately programmed in accordance with this specification, can perform the process 800.

[0234] The system receives: (i) a set of atom embeddings, and (ii) molecule embedding data (802). The set of atom embeddings includes a respective atom embedding for each atom in the molecule complex, e.g., as described with reference to step 704 of FIG. 7. The molecule embedding data can include token-resolution molecule embedding data and atom-resolution molecule embedding data. The token-resolution molecule embedding data includes a respective single embedding for each token in the molecule complex and a respective pair embedding for each pair of tokens in the molecule complex. The atom-resolution molecule embedding data includes a respective single embedding for each atom in the molecule complex and a respective pair embedding for each pair of atoms in the molecule complex, e.g., as described with reference to step 706 of FIG.7.

[0235] The system applies a sequence of one or more self-attention operations to the set of atom embeddings to update the set of atom embeddings (804). Each of the one or more self-attention operations are conditioned on the atom-resolution molecule embedding data. An example process for applying a self-attention operation to a set of embeddings (in this case: atom embeddings), where the self-attention operation is conditioned on molecule embedding data (in this case: atom-resolution molecule embedding data), is described with reference to FIG.9.

[0236] The system generates a set of token embeddings that includes a respective token embedding for each token in the molecule complex based on the set of atom embeddings (806). In particular, for each token in the molecule complex, the system generates the token embedding for the token by aggregating the respective atom embedding of each atom included in the entity (e.g., amino acid, DNA nucleotide, RNA nucleotide, or atom) represented by the token. In some implementations, the system can generate the respective token embedding for each token ^^^by operations including:a^ ൌ mean ^relu൫Linear^q^^൯^^^ ∈ ^1, … ,^^^௧^^^^i^^^^ ൌ ^^DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationwhere ^^^^^^ is an indicator function for whether atom ^^ is included in token ^^, and ^^^^^ are theatom embeddings.

[0237] The system applies a sequence of one or more self-attention operations to the set of token embeddings to update the set of token embeddings (808). Each of the one or more self-attention operations are conditioned on the token-resolution molecule embedding data. An example process for applying a self-attention operation to a set of embeddings (in this case: token embeddings), where the self-attention operation is conditioned on molecule embedding data (in this case: token-resolution molecule embedding data), is described with reference to FIG. 9.

[0238] The system updates the set of atom embeddings (e.g., as generated at the output of step 804) using the set of token embeddings (e.g., as generated at the output of step 808) (810). For instance, the system can update the set of atoms embeddings by operations including: Linear൫a^୩୧୮^^^^൯ ^ ^^^where ^^^denotes the updated atom embedding for atom ^^, ^^^^^^denotes the token embedding for the token that includes atom ^^, and ^^^^^^^ denotes the atom embedding from atom ^^ prior to the atom embedding being updated.

[0239] The system applies a sequence of one or more self-attention operations to the set of atom embeddings to update the set of atom embeddings (812). Each of the one or more self-attention operations are conditioned on the atom-resolution molecule embedding data. An example process for applying a self-attention operation to a set of embeddings (in this case: atom embeddings), where the self-attention operation is conditioned on molecule embedding data (in this case: atom-resolution molecule embedding data), is described with reference to FIG. 9.

[0240] The system can provide the set of atom embeddings generated at step 812 to be processed by the denoising neural network to generate a denoising output, as described in more detail with reference to step 710 of FIG.7.

[0241] FIG. 9 is a flow diagram of an example process 900 for applying a self-attention operation to a set of input embeddings (e.g., a set of atom embeddings or a set of token embeddings), where the self-attention operation is conditioned on molecule embedding data (e.g., atom-resolution molecule embedding data or token-resolution molecule embedding data). For convenience, the process 900 will be described as being performed by a system of one or more computers located in one or more locations. For example, a folding system, e.g., theDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationfolding system 100 of FIG.1, appropriately programmed in accordance with this specification, can perform the process 900.

[0242] The system receives: (i) a set of input embeddings, and (ii) molecule embedding data (902). The molecule embedding data can include a set of single embeddings and a set of pair embeddings. The set of single embeddings includes a respective single embedding corresponding to each input embedding in the set of input embeddings. The set of pair embeddings includes a respective pair embedding corresponding to each pair of input embeddings from the set of input embeddings.

[0243] In some implementations, the set of input embeddings is a set of atom embeddings and the molecule embedding data is atom-resolution molecule embedding data. In the atom- resolution molecule embedding data, each single embedding represents a respective atom in the molecule complex and corresponds to the input embedding that represents the same atom. Each pair embedding represents a respective pair of atoms in the molecule complex and corresponds to the pair of input embeddings representing the same pair of atoms.

[0244] In some implementations, the set of input embeddings is a set of token embeddings and the molecule embedding data is token-resolution molecule embedding data. In the token- resolution molecule embedding data, each single embedding represents a respective token in the molecule complex and corresponds to the input embedding that represents the same token. Each pair embedding represents a respective pair of tokens in the molecule complex and corresponds to the pair of input embeddings representing the same pair of tokens.

[0245] The system generates a set of intermediate attention scores based on the set of current atom embeddings (904). The set of intermediate attention scores includes a respective attention score for each of multiple pairs of current atom embeddings from the set of current atom embeddings.

[0246] The system can generate the intermediate attention scores in any of a variety of possible ways. For instance, the system can generate a respective query embedding for each input embedding by processing the input embedding using a query neural network, e.g., as: ^^ ൌ ^^ொ ⋅ ^^ ^7^where ^^ is a matrix where each column (or row) defines a respective query embedding, ^^ொis a matrix of parameter values (defining the query neural network in this example), and ^^ is a matrix where each column (or row) defines a respective input embedding. Further, the system canDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationgenerate a respective key embedding for each input embedding by processing the input embedding using a key neural network, e.g., as: ^^ ൌ ^^^ ⋅ ^^ ^8^where ^^ is a matrix where each column (or row) defines a respective key embedding, ^^^is a matrix of parameter values (defining the key neural network in this example), and ^^ is a matrix where each column (or row) defines a respective input embedding. The system can generate the intermediate attention scores based on the query embeddings and the key embeddings, e.g., as: ^^ ൌ ^^ ⋅ ^^^ ^9^where ^^ is a matrix of intermediate attention scores, ^^ is the matrix of query embeddings, and ^^ is the matrix of key embeddings.

[0247] The system generates a set of attention score biases based on molecule embedding data (906). The set of attention score biases includes a respective attention score bias for each of multiple pair of input embeddings from the set of input embeddings.

[0248] The system can generate the set of attention score biases in any of a variety of possible ways. For instance, for each pair of input embeddings, the system can generate the attention score bias for the pair of input embeddings by processing the corresponding pair embedding (i.e., from the set of pair embeddings included in the molecule embedding data) using a projection neural network.

[0249] The system generates a set of final attention scores by combining: (i) the intermediate attention scores, and (ii) the attention score biases (908). The set of final attention scores includes a respective final attention score for each pair of input embeddings in the set of input embeddings. The system can generate the final attention score for a pair of input embeddings by combining (e.g., summing): (i) the intermediate attention score for the pair of input embeddings, and (ii) the attention score bias for the pair of input embeddings. Optionally, the system can apply further processing operations to the set of final attention scores, e.g., by applying a soft- max operation to some or all of the final attention scores.

[0250] The system updates the set of input embeddings using the set of final attention scores (910). For instance, to update the set of input embeddings, the system can generate a respective value embedding for each input embedding in the set of input embeddings by processing the input embedding using a value neural network, e.g., as: ^^ ൌ ^^^ ⋅ ^^ ^10^DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationwhere ^^ is a matrix where each column (or row) defines a respective value embedding, ^^^is a matrix of parameter values (defining the value neural network in this example), and ^^ is a matrix where each column (or row) defines a respective input embedding. The system can then update the set of input embeddings by performing operations including, e.g.: ^^ᇱ ൌ ^^ ⋅ ^^ ^11^where each column (or row) of ^^′ defines a respective updated input embedding, each column (or row) of ^^ defines a respective value embedding, and ^^ denotes the set of final attention scores arranged into a matrix.

[0251] In some implementations, the system additionally generates a respective “conditioning” embedding for each input embedding, e.g., by processing the single embedding corresponding to the input embedding using one or more neural network layers, e.g., fully connected neural network layers. After updating the input embeddings as described above, the system can further modify each input embedding by combining (e.g., element-wise multiplying, summing, or averaging) the input embedding with the corresponding conditioning embedding.

[0252] In implementations where the self-attention block implements a multi-head attention operation, each head of the attention operation can individually perform the steps of the process 900, and the updated embeddings generated by each attention head can be combined (e.g., concatenated) to define the overall output of the multi-head attention operation. Each attention head can have a respective set of neural network parameters, having values that are specific to each attention head, that are used for generating the intermediate attention scores and the attention score biases.

[0253] Optionally, the system can implement the self-attention operation as a masked self- attention operation. In particular, rather than updating each input embedding using attention over the entire set of input embeddings, the system can update each input embedding using attention over only a proper subset of the set of input embeddings. For instance, the system can implement a masked-self attention operation that updates each given input embedding using attention over only inputs embeddings within a threshold number of positions of the given input embedding in an ordering of the input embeddings as a list.

[0254] An example of a specific implementation of the process 900 is provided next: ^^^,^^^, ^^^ ൌ ^^^^^^^^^^^^^^^^^^^^^^^^^^ ^ ^^ ^ ^ ^a^^ ^^^,^^^ , ^^^ ∈ ℝ ,ℎ ∈ ^1, … , ^^^^^ௗ^^^^^^ ← LinearNoBias^LayerNorm൫^^^^൯^DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationa୧ ← Linear^concat^൫^^^ ^^ ൯^ ^^^ ∈ ℝ ^where ^^^^^ is the set of input embeddings, ℎ indexes each head of a multi-head attention operation, ^^^^^^ are query embeddings (as described at step 904), ^^^^^^ are key embeddings (as described at step 904), ^^^^^^ are value embeddings (as described at step 910), ^^^^^^ is the set of pair embeddings (as described at step 902), ^^^^^^ ^ are the attention score biases (as described at step 906), ^^^^^ is the set of single embeddings (as described at step 902), ^^^^^^ are the conditioning embeddings (as described at step 910), ^ are the intermediate attentionscores (as described at step 904), and ^^^^^^ are masking values that cause the attention operation to be masked (as described at step 910). The masking values can be generated by operations including:

[0255] FIG. 10 is a flow diagram of an example process 1000 for generating one or more confidence estimates associated with a predicted 3D structure of a molecule complex using a confidence prediction neural network. For convenience, the process 1000 will be described as being performed by a system of one or more computers located in one or more locations. For example, a folding system, e.g., the folding system 100 of FIG.1, appropriately programmed in accordance with this specification, can perform the process 1000.

[0256] The system receives: (i) molecule embedding data for the molecule, and (ii) data defining a predicted 3D structure of the molecule complex (1002). The molecule embedding data can include a set of single embeddings (including a respective single embedding for each token in the molecule complex) and a set of pair embeddings (including a respective pair embedding for each pair of tokens in the molecule complex). An example process for generating molecule embedding data using an embedding neural network is described with reference to FIG.2. The data defining the predicted 3D structure of the molecule complex can define, for each atom in the molecule, a predicted 3D spatial position of the atom. An example process for generating aDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationpredicted 3D structure of a molecule complex using a generative diffusion model is described with reference to FIG. 6.

[0257] The system processes the molecule embedding data and the predicted 3D structure of the molecule complex, by an encoder block of the confidence prediction neural network, to update the molecule embedding data (1004).

[0258] For instance, the system can process the predicted 3D structure to generate a set of distance embeddings by, for each of multiple pairs of tokens in the molecule complex, generating an embedding representing a distance between: (i) a representative (designated) atom from the entity (e.g., amino acid or nucleotide) represented by the first token in the pair, and (ii) a representative (designated) atom from the entity (e.g., amino acid or nucleotide) represented by the second token in the pair. (A representative atom in an entity such as an amino acid or nucleotide can be any atom that is designated for that role; e.g., a representative atom for an amino acid can be the alpha carbon atom in the amino acid). The system can then combine the distance embeddings with the set of pair embeddings of the molecule embedding data. In a particular example, the system can perform operations including:^^^^^ൌ Linear^one_hot^^^^^ , ^^^^^^ ൌ ^33ൗ1 3 8Å, 5 ൗ 8 Å, … ,21 ൗ 8 Å൧^^where ^^^^is the distance between representative atoms from tokens ^^ and ^^ in the predicted 3D structure of the molecule complex, one_hot() refers to a one-hot embedding operation, and ^^^^is the pair embedding for tokens ^^ and ^^.

[0259] After incorporating data characterizing the predicted 3D structure into the set of pair embeddings, the system can process the set of single embeddings and the set of pair embeddings using a sequence of update blocks to update the set of single embeddings and the set of pair embeddings. An example of operations that can be performed by an update block to update a set of single embeddings and a set of pair embeddings is described with reference to FIG.4.

[0260] The system can process the set of single embeddings from the molecule embedding data using an atom confidence estimate head of the confidence prediction neural network to generate a per-atom confidence estimate for each atom in the molecule complex (1006). A per-atomDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationconfidence estimate for an atom characterizes a confidence of the folding system in the accuracy of the predicted 3D spatial location of the atom.

[0261] In some implementations, the per-atom confidence head is trained to predict a per-atom confidence estimate that is defined for atom ^^ as:where ^^^^is the distance between atom ^^ and atom ^^ in the predicted 3D structure of the moleculecomplex, ^^ encompasses all atoms in the molecule complex and the set of atoms ^^ ∈ ^^ is definedas including atoms that satisfy one or more (in some case: all) of the following criteria: (i) the distance in the ground truth structure between atom ^^ and atom ^^ is less than a first threshold (e.g., 15A) if ^^ is a protein atom or less than a second threshold (e.g., 30A) if ^^ is a nucleic acid atom; (ii) the atom is included in a polymer chain; and (iii) the atom is of a designated type, e.g., a ^^ఈ-atom type for proteins or a C1’-atom type for nucleic acids.

[0262] The atom confidence estimate head can generate the per-atom confidence estimates by, for each token in the molecule complex, projecting the single embedding for the token into^^்^^^^^௧^^^ ൈ ^^^^^^ values (denoted ^^^^^^), where ^^்^^^^^௧^^^is the maximum number of atoms per token and ^^^^^^is a number of bins that partition the interval between 0 and 1. Each value ^^^^defines a probability that the per-atom confidence estimate for atom ^^ is included in bin ^^. The system can obtain a single value for the per-atom confidence estimate for each atom by taking an expectation across the binned probabilities generated for the atom.

[0263] The system can process the set of pair embeddings from the molecule embedding data using a token pair confidence estimate head of the confidence prediction neural network to generate token pair confidence estimates (1008). A token pair confidence estimate for a pair of tokens characterizes a predicted error in the relative 3D displacement (as defined by the predicted 3D molecule complex structure) between a representative atom in a first token of the pair and a representative atom in the second token of the pair. For instance, for a token pair that includes a first token and a second token in the molecule, the system can process a pair embedding for the token pair using the token pair confidence estimate head to generate a prediction for an error in the relative 3D displacement (as defined by the predicted 3D molecule complex structure) between a representative atom in the first token and a representative atom in the second token.DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Application

[0264] The system can process the set of single embeddings from the molecule embedding data using an experimentally resolved estimate head of the confidence prediction neural network to generate, for each atom in the molecule complex, a prediction for whether the spatial position of the atom is resolved in a ground-truth structure of the molecule complex (1010). The spatial position of an atom in a ground truth structure can be referred to as being “resolved” if the position of the atom has been determined by experimental techniques. Ordered regions of a molecule complex are sometimes more likely to be resolved than disordered regions. Training the confidence prediction neural network to predict which atoms are experimentally resolved can cause the confidence prediction neural network (and in some cases other parts of the folding system) to generate richer features and intermediate representations.

[0265] FIG. 11 is a flow diagram of an example process 1100 for jointly training the embedding neural network and the generative model of the folding system. In the example process 1100, the generative model is a model that implements a differentiable generative process. For instance, the generative model can be a generative diffusion model implemented using a denoising neural network, as described above with reference to FIG.6. For convenience, the process 1100 will be described as being performed by a system of one or more computers located in one or more locations. For example, a folding system, e.g., the folding system 100 of FIG.1, appropriately programmed in accordance with this specification, can perform the process 1100.

[0266] The system receives data characterizing a set of molecule complexes (1102).

[0267] The system generates a set of training examples (1104). Each training example corresponds to a respective molecule complex and includes data defining: (i) a training input to the folding system, and (ii) a target output of the folding system. The training input to the folding system includes molecule complex data characterizing the molecule complex, e.g., as described with reference to FIG.2. The target output of the folding system can be based on the 3D structure of the molecule complex. The system can generate training examples corresponding to multiple different types of molecule complexes, e.g., molecule complexes that each include: one or more protein molecules, one or more DNA molecules, one or more RNA molecules, one or more ligand molecules, and so forth. Training the embedding neural network and the generative model on training examples corresponding to multiple types of molecule complexes can enable the trained embedding neural network and generative model to accurately generate predicted 3D structures for multiple types of molecule complexes.DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Application

[0268] In some implementations, the set of training examples includes: a plurality of training examples corresponding to protein – protein complexes; a plurality of training examples corresponding to protein – DNA complexes; and a plurality of training examples corresponding to protein – RNA complexes. In some implementations, the set of training examples further includes a plurality of training examples corresponding to protein – ligand complexes.

[0269] In some implementations, the set of training examples includes: a plurality of training examples corresponding to molecule complexes with single molecules, e.g., singe proteins, single DNA molecules, single RNA molecules, and so forth.

[0270] In some cases, generative models can “hallucinate,” e.g., by generating predicted 3D structures that include plausible structures even in unstructured regions. An "unstructured region" in a molecule complex refers to a part of the molecule complex that does not adopt a well-defined, stable three-dimensional structure. To counteract this effect, the system can augment the set of training examples to include training examples having target outputs that are based predicted 3D structures of complex which are generated by non-generative structure prediction systems. A “non-generative” structure prediction system refers to a system that can predict the structures of molecules and molecule complexes but that does not include a generative model. Examples of generative models include, e.g., diffusion models, generative adversarial networks, and so forth. An example of a non-generative structure prediction system is: Richard Evans et al., “Protein complex prediction with AlphaFold-Multimer,” March 10, 2022, bioRxiv, https: / / doi.org / 10.1101 / 2021.10.04.463034. The non-generative structure prediction system can generate predicted structures that represent unstructured regions by long, extended loops instead of compact structures. Training the folding system on predicted structures generated by non-generative systems can mitigate the risk of any hallucination behavior by the generative model included in the folding system, e.g., by “teaching” the folding system to mimic the non-hallucinatory behavior of the non-generative structure prediction system.

[0271] The system jointly trains the embedding neural network and the generative model on the set of training examples by a machine learning training technique (1106). More specifically, for each training example, the system can process the training input of the training example using the embedding neural network and the generative model to generate a predicted output of the generative model. The system can evaluate an objective function that measures an error (e.g., a root mean square deviation (RMSD), or a mean absolute error (MAE), or a mean squared errorDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Application(MSE)) between: (i) the predicted output of the generative model, and (ii) the target output of the generative model. The system can determine gradients of the objective function with respect to the parameters of the embedding neural network and the generative model, e.g., using backpropagation. (The parameters of the generative model can include, e.g., a set of neural network parameters of a neural network implemented by the generative model). The system can then update the current values of the parameters of the embedding neural network and the generative model using the gradients, e.g., by the update rule of an appropriate gradient descent optimization algorithm, e.g., RMSprop or Adam.

[0272] Specific aspects of the training (e.g., the operations of the generative model during the training and the objective function) may depend on the implementation of the generative model. An example process for training a generative diffusion model that includes a denoising neural network on a training example is described in more detail next with reference to FIG.12. In the example process of FIG.12, the operations of the generative diffusion model are modified during training (e.g., as compared to the operations of the generative diffusion model during inference, e.g., as described with reference to FIG.6), as will be described in more detail below.

[0273] In some implementations, the system includes a confidence prediction neural network that can process molecule embedding data and a predicted 3D molecule complex structure to generate per-atom confidence estimates, token pair confidence estimates, and estimates for whether atom spatial locations are experimentally resolved (as described above with reference to FIG.10). The system can jointly train the confidence prediction neural network along with the embedding neural network and the generative model to optimize an objective function that measures an accuracy of outputs generated by the confidence prediction neural network. In particular, the system can backpropagate gradients of the objective function through the confidence prediction neural network, and optionally, into the generative model and / or the embedding neural network.

[0274] FIG. 12 is a flow diagram of an example process 1200 for jointly training an embedding neural network and a generative diffusion model on a training example. For convenience, the process 1200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a folding system, e.g., the folding system 100 of FIG.1, appropriately programmed in accordance with this specification, can perform the process 1200.DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Application

[0275] The system generates molecule embedding data for the molecule complex corresponding to the training example using the embedding neural network (1202).

[0276] The system samples a time step from the sequence of denoising time steps (1204). More specifically, during inference, the generative diffusion model can be configured to perform a sequence of denoising time steps, e.g., as described with reference to steps 604, 606, 608, and 612 of FIG.6. During training, the system can randomly sample a single denoising time step from the sequence of denoising time steps, e.g., in accordance with a uniform distribution over the sequence of denoising time steps.

[0277] The system generates a respective noisy spatial position for each atom in the molecule complex by combining random noise with the target spatial position of the atom (i.e., in the target 3D structure of the molecule) (1206). For instance, for each atom in the molecule complex, the system can generate the noisy spatial position for the atom by adding random noise to the target spatial position of the atom. The system can scale the random noise combined with the target spatial positions of the atoms by a constant that depends on the sampled time step, e.g., where the values of the constants corresponding to the denoising time steps are defined by a noise schedule.

[0278] The system generates a denoising output using the denoising neural network while the denoising neural network is conditioned on the molecule embedding data (1208). An example process for generating a denoising output is described in detail with reference to FIG.7. At step 702 of FIG.7, the current position for each atom in the molecule complex can be defined as the noisy spatial position for each atom in the molecule complex.

[0279] The system determines gradients of an objective function that depends on the denoising output, and uses the gradients to update the parameter values of the denoising neural network and the embedding neural network (1210). The objective function can measure an error between: (i) the denoising output of the denoising neural network, and (ii) a target output of the denoising neural network. The target output of the denoising neural network can define an output of the denoising neural network that, if used to generate an initial estimate of the 3D spatial positions of the atoms in the molecule complex (as described in step 606 of FIG. 6), would cause the initial estimate of the 3D spatial positions of the atoms to match the target (actual) 3D spatial positions of the atoms in the molecule complex of the training example.

[0280] Aspects of the disclosure in this specification are further described in Appendix A.DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Application

[0281] This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0282] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0283] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0284] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarativeDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationor procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0285] In this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0286] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0287] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, aDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT ApplicationGlobal Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0288] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0289] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0290] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute- intensive parts of machine learning training or production, i.e., inference, workloads.

[0291] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework, or a Jax framework.

[0292] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. TheDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationcomponents of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0293] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0294] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0295] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Application

[0296] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT ApplicationCLAIMS 1. A method performed by one or more computers, the method comprising: obtaining a network input that characterizes a molecule or molecule complex; processing the network input characterizing the molecule or molecule complex using an embedding neural network to generate molecule embedding data; and generating, using a generative model and while the generative model is conditioned on the molecule embedding data, a predicted three-dimensional (3D) structure of the molecule or molecule complex that defines a respective predicted 3D spatial location of each atom in the molecule or molecule complex.

2. The method of claim 1, wherein the generative model is a generative diffusion model that comprises a denoising neural network.

3. The method of claim 2, wherein generating, using the generative model and while the generative model is conditioned on the molecule embedding data, the predicted 3D structure of the molecule or molecule complex comprises: generating positional data defining a respective initial position of each atom in the molecule or molecule complex; denoising the positional data over a sequence of time steps using the denoising neural network and while the denoising neural network is conditioned on the molecule embedding data; wherein the predicted 3D structure of the molecule or molecule complex is defined by the positional data after a final time step in the sequence of time steps.

4. The method of claim 3, wherein generating positional data defining the respective initial position of each atom in the molecule or molecule complex comprises: sampling the respective initial position of each atom in the molecule or molecule complex from a probability distribution over 3D space.

5. The method of any one of claims 3-4, wherein denoising the positional data over the sequence of time steps using the denoising neural network and while the denoising neural network is conditioned on the molecule embedding data comprises, at each of one or more timeDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationsteps in the sequence of time steps: receiving current positional data that defines a respective current position of each atom in the molecule or molecule complex at the time step; generating a denoising output using the denoising neural network and while the denoising neural network is conditioned on the molecule embedding data, wherein the denoising neural network processes an input comprising the current positional data; and generating positional data that defines a respective position of each atom in the molecule or molecule complex at a next time step using the denoising output.

6. The method of claim 5, wherein the denoising output comprises a respective predicted error in the current position of each atom in the molecule or molecule complex at the time step.

7. The method of any one of claims 5-6, wherein generating the denoising output using the denoising neural network and while the denoising neural network is conditioned on the molecule embedding data comprises: generating a set of atom embeddings using an encoder block of the denoising neural network, wherein each atom embedding represents an atom in the molecule or molecule complex and is based at least in part on the respective current spatial position of the atom at the time step; processing the set of atom embeddings using an update block of the denoising neural network to generate a set of updated atom embeddings, wherein the update block is conditioned on the molecule embedding data; and generating the denoising output based at least in part on the set of updated atom embeddings.

8. The method of claim 7, wherein the update block of the denoising neural network comprises a sequence of self-attention blocks; wherein each of the self-attention blocks applies one or more self-attention operations to a set of current atom embeddings to update the set of current atom embeddings; wherein each of the one or more self-attention operations are conditioned on the molecule embedding data.DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Application9. The method of claim 8, wherein applying a self-attention operation to the set of current atom embeddings to update the set of current atom embeddings comprises: updating each current atom embedding using attention over only a proper subset of the set of current atom embeddings.

10. The method of any one of claims 8-9, wherein applying a self-attention operation to the set of current atom embeddings to update the set of current atom embeddings comprises: generating, based on the current set of atom embeddings, a respective intermediate attention score for each of a plurality of pairs of current atom embeddings from the set of current atom embeddings; generating, based on the molecule embedding data, a respective attention score bias for each of the plurality of pairs of current atom embeddings; generating a respective final attention score for each pair of current atom embeddings from the set of current atom embeddings based on the intermediate attention scores and the attention score biases; and updating the set of current atom embeddings using the final attention scores.

11. The method of claim 10, wherein the molecule embedding data comprises a set of token- resolution pair embeddings that includes a respective pair embedding for each pair of tokens from a set of tokens associated with the molecule or molecule complex; wherein applying the self-attention operation to the set of current atom embeddings to update the set of current atom embeddings further comprises: generating, from the set of token-resolution pair embeddings, a set of atom- resolution pair embeddings that includes a respective pair embedding for each pair of atoms in the molecule or molecule complex; and wherein generating, based on the molecule embedding data, the respective attention score bias for each pair of current atom embeddings from the set of current atom embeddings comprises, for each pair of current atom embeddings: generating the attention score bias for the pair of current atom embeddings using a respective atom-resolution pair embedding corresponding to a same pair of atoms as the pair ofDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationcurrent atom embeddings.

12. The method of any one of claims 10-11, wherein for each pair of current atom embeddings from the set of current atom embeddings, generating the final attention score for the pair of current atom embeddings comprises: summing the intermediate attention score for the pair of current atom embeddings and the attention score bias for the pair of current atom embeddings.

13. The method of any one of claims 9-12, wherein the molecule embedding data comprises a set of token-resolution single embeddings that includes a respective token-resolution single embedding for each token from a set of tokens associated with the molecule or molecule complex; wherein applying the self-attention operation to the set of current atom embeddings to update the set of current atom embeddings further comprises: generating, from the set of token-resolution single embeddings, a set of atom- resolution single embeddings that includes a respective atom-resolution single embedding for each atom in the molecule or molecule complex; and wherein updating the set of current atom embeddings using the final attention scores comprises: generating, for each atom in the molecule or molecule complex, a respective conditioning embedding using the atom-resolution single embedding for the atom; and generating, for each atom in the molecule or molecule complex, a respective updated atom embedding using the final attention scores and the conditioning embedding for the atom.

14. The method of claim 13, wherein for each atom in the molecule or molecule complex, generating the updated atom embedding for the atom using the final attention scores and the conditioning embedding for the atom comprises: updating the atom embedding for the atom using the final attentions scores; and updating the atom embedding for the atom by combining the atom embedding for theDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationatom with the conditioning embedding for the atom.

15. The method of claim 14, wherein for each atom in the molecule or molecule complex, updating the atom embedding for the atom using the final attentions scores comprises: generating, for each of a plurality of atoms in the molecule or molecule complex, a respective value embedding for the atom based on the atom embedding for the atom; and combining the value embeddings for the plurality of atoms in accordance with final attention scores associated with the atom; and updating the atom embedding for the atom using the combination of the value embeddings.

16. The method of any one of claims 14-15, wherein for each atom in the molecule or molecule complex, updating the atom embedding for the atom by combining the atom embedding for the atom with the conditioning embedding for the atom comprises: element-wise multiplying the atom embedding and the conditioning embedding for the atom.

17. The method of any one of claims 7-16, wherein generating the denoising output based at least in part on the set of updated atom embeddings comprises: generating, for each of a plurality of tokens associated with the molecule or molecule complex, a respective token embedding by aggregating one or more corresponding updated atom embeddings; processing the set of token embeddings using an update block of the denoising neural network to generate a set of updated token embeddings, wherein the update block is conditioned on the molecule embedding data; and generating the denoising output based at least in part on the set of updated token embeddings.

18. The method of claim 17, wherein the plurality of tokens associated with the molecule or molecule complex comprises a token representing an entity that includes a plurality of atoms; andDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationwherein generating the token embedding for the token comprises: aggregating the updated atom embeddings for each atom included in the entity represented by the token.

19. The method of claim 18, wherein the entity represented by the token is an amino acid or a DNA nucleotide or an RNA nucleotide.

20. The method of any one of claims 17-19, wherein the update block of the denoising neural network comprises a sequence of self-attention blocks; wherein each of the self-attention blocks applies one or more self-attention operations to a set of current token embeddings to update the set of current token embeddings; wherein each of the one or more self-attention operations are conditioned on the molecule embedding data.

21. The method of claim 20, wherein applying a self-attention operation to the set of current token embeddings to update the set of current token embeddings comprises: generating, based on the set of current token embeddings, a respective intermediate attention score for each of a plurality of pairs of current token embeddings from the set of current token embeddings; generating, based on the molecule embedding data, a respective attention score bias for each of the plurality of pairs of current token embeddings; generating a respective final attention score for each pair of current token embeddings from the set of current token embeddings based on the intermediate attention scores and the attention score biases; and updating the set of current token embeddings using the final attention scores.

22. The method of claim 21, wherein the molecule embedding data comprises a set of token- resolution pair embeddings that includes a respective token-resolution pair embedding for each pair of tokens from a set of tokens associated with the molecule or molecule complex; wherein generating, based on the molecule embedding data, the respective attention score bias for each of the plurality of pairs of current token embeddings from the set of current tokenDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationembeddings comprises, for each pair of current token embeddings: generating the attention score bias for the pair of current token embeddings using a respective token-resolution pair embedding corresponding to a same pair of tokens as the pair of current token embeddings.

23. The method of any one of claims 21-22, wherein the molecule embedding data comprises a set of token-resolution single embeddings that includes a respective single embedding for each token from a set of tokens associated with the molecule or molecule complex; wherein updating the set of current token embeddings using the final attention scores comprises: generating, for each token associated with the molecule or molecule complex, a respective conditioning embedding using the token-resolution single embedding for the token; and generating, for each token associated with the molecule or molecule complex, a respective updated token embedding using the final attention scores and the conditioning embedding for the token.

24. The method of claim 23, wherein for each token associated with the molecule or molecule complex, generating the updated token embedding for the token using the final attention scores and the conditioning embedding for the token comprises: updating the token embedding for the token using the final attention scores; and updating the token embedding for the token by combining the token embedding for the token with the conditioning embedding for the token.

25. The method of claim 24, wherein for each token associated with the molecule or molecule complex, updating the token embedding for the token using the final attention scores comprises: generating, for each of a plurality of tokens associated with the molecule or molecule complex, a respective value embedding for the token based on the token embedding for the token; and combining the value embeddings for the plurality of tokens in accordance with finalDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationattention scores associated with the token; and updating the token embedding for the token using the combination of the value embeddings.

26. The method of claim 25, wherein for each token associated with the molecule or molecule complex, updating the token embedding for the token by combining the token embedding for the token with the conditioning embedding for the token comprises: element-wise multiplying the token embedding and the conditioning embedding for the token.

27. The method of any one of claims 5-26, wherein generating the denoising output using the denoising neural network and while the denoising neural network is conditioned on the molecule embedding data comprises: generating a set of atom embeddings, wherein each atom embedding represents an atom in the molecule or molecule complex and is based at least in part on the respective current spatial position of the atom at the time step; applying one or more self-attention operations to the set of atom embeddings to update the set of atom embeddings, wherein each of the one or more self-attention operations are conditioned on the molecule embedding data; generating a set of token embeddings, wherein each token embedding represents a respective token associated with the molecule or molecule complex and is generated by aggregating one or more corresponding atom embeddings; and applying one or more self-attention operations to the set of token embeddings to update the set of token embeddings, wherein each of the one or more self-attention operations are conditioned on the molecule embedding data; and generating the denoising output using the set of token embeddings.

28. The method of claim 27, wherein generating the denoising output using the set of token embeddings comprises: updating the set of atom embeddings using the set of token embeddings; applying one or more self-attention operations to the set of atom embeddings to updateDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationthe set of atom embeddings, wherein each of the one or more self-attention operations are conditioned on the molecule embedding data; and generating the denoising output using the set of atom embeddings.

29. The method of any preceding claim, wherein processing the network input characterizing the molecule or molecule complex using the embedding neural network to generate the molecule embedding data comprises: generating a set of single embeddings, wherein each single embedding represents a respective token from a set of tokens associated with the molecule or molecule complex; and generating a set of pair embeddings, wherein each pair embedding represents a respective pair of tokens from the set of tokens associated with the molecule or molecule complex.

30. The method of claim 29, wherein the set of tokens comprises one or more tokens that each represent a respective amino acid in the molecule or molecule complex.

31. The method of claim 29, wherein the set of tokens comprises one or more tokens that each represent a respective nucleotide in the molecule or molecule complex.

32. The method of claim 31, wherein the set of tokens comprises one or more tokens that each represent a respective DNA nucleotide in the molecule or molecule complex.

33. The method of claim 31, wherein the set of tokens comprises one or more tokens that each represent a respective RNA nucleotide in the molecule or molecule complex.

34. The method of any one of claims 29-33, wherein the set of tokens comprises one or more tokens that each represent a respective atom in the molecule complex.

35. The method of any one of claims 29-34, wherein: the molecule or molecule complex is a molecule complex that comprises one or more protein molecules and one or more DNA molecules; or the molecule or molecule complex is a molecule complex that comprises one or moreDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationprotein molecules and one or more RNA molecules; or the molecule or molecule complex is a molecule complex that comprises one or more protein molecule and one or more ligand molecules; or the molecule or molecule complex is a molecule complex that comprises a plurality of protein molecules; or the molecule or molecule complex is a protein molecule; or the molecule or molecule complex is a DNA molecule; or the molecule or molecule complex is a RNA molecule.

36. The method of claim 35, wherein the molecule or molecule complex comprises one or more DNA molecules, and one or more of nucleotides in one or more of the DNA molecules are modified DNA nucleotides.

37. The method of claim 35, wherein the molecule or molecule complex comprises one or more RNA molecules, and or more nucleotides in one or more of the RNA molecules are modified RNA nucleotides.

38. The method of any one of claims 29-37, wherein generating the set of single embeddings and the set of pair embeddings comprises: initializing a set of current single embeddings and a set of current pair embeddings; and updating the set of current single embeddings and the set of current pair embeddings over a sequence of one or more recycling iterations, comprising, for each recycling iteration: updating the set of current single embeddings by applying one or more self- attention operations to the set of current single embeddings, wherein each of the one or more self-attention operations are conditioned on the set of current pair embeddings.

39. The method of claim 38, wherein applying a self-attention operation to the set of current single embeddings comprises: generating, based on the set of current single embeddings, a respective intermediate attention score for each pair of current single embeddings from the set of current single embeddings;DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationgenerating, based on the set of current pair embeddings, a respective attention score bias for each pair of current single embeddings from the set of current single embeddings; generating a respective final attention score for each pair of current single embeddings from the set of single embeddings based on the intermediate attention scores and the attention score biases; and updating the set of current single embeddings using the final attention scores.

40. The method of claim 39, wherein for each pair of current single embeddings that comprises a first single embedding representing a first token in the molecule or molecule complex and a second single embedding representing a second token in the molecule or molecule complex, generating the attention score bias for the pair of single embeddings comprises: processing a current pair embedding, from the set of current pair embeddings, that represents a pair of tokens comprising the first token and the second token using one or more neural network layers to generate the attention score bias for the pair of single embeddings.

41. The method of any one of claims 38-40, wherein each recycling iteration further comprises: updating the set of current pair embeddings using one or more template molecules.

42. The method of any one of claims 38-41, wherein each recycling iteration further comprises: updating the set of current pair embeddings using a multiple sequence alignment (MSA).

43. The method of any preceding claim, wherein the embedding neural network and the generative model are jointly trained on a set of training data, wherein each training examples corresponds to a respective molecule or molecule complex and includes a prediction target that defines an actual 3D structure of the molecule or molecule complex.

44. The method of claim 43, wherein the set of training data includes: a plurality of training examples corresponding to molecule complexes that each comprise one or more respective protein molecules and one or more respective DNA molecules; and / orDeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationa plurality of training examples corresponding to molecule complexes that each include one or more respective protein molecules and one or more respective RNA molecules; and / or a plurality of training examples corresponding to molecule complexes that each include one or more respective protein molecule and one or more respective ligand molecules; and / or a plurality of training examples corresponding to molecule complexes that each comprise a respective plurality of protein molecules; a plurality of training examples corresponding to respective protein molecules; and / or a plurality of training examples corresponding to respective DNA molecules; and / or a plurality of training examples corresponding to respective RNA molecules.

45. A system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of the respective method of any one of claims 1-44.

46. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations of the respective method of any one of claims 1-44.

47. A method of obtaining a ligand, wherein the ligand is a drug or a ligand of an industrial enzyme, the method comprising: for each of one or more candidate ligands: (a) performing the method of any one of claims 1-44 to determine a predicted structure of a target molecule or a complex comprising a target molecule and the candidate ligand; and (b) evaluating an interaction of the candidate ligand with the target molecule dependent on the predicted structure; and selecting one or more of the candidate ligands as the ligand dependent on a result of the evaluating.DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Application48. A method as claimed in claim 47, wherein the target molecule comprises a receptor or enzyme, and wherein the ligand is an agonist or antagonist of the receptor or enzyme.

49. A method as claimed in claim 47 or 48 wherein the ligand is a drug, and the method comprises: performing steps (a) and (b) for each of a plurality of target molecules; and selecting one or more of the candidate ligands as the ligand to either i) obtain a ligand that interacts with each of the target molecules, or ii) obtain a ligand that interacts with only one of the target molecules.

50. A method of obtaining a ligand, wherein the ligand is a drug or a ligand of an industrial enzyme, the method comprising: selecting a target molecule; for each of one or more candidate ligands: performing the method of any one of claims 1-44 to determine a predicted structure of the candidate ligand or a complex comprising the candidate ligand and a target molecule; and evaluating an interaction between the candidate ligand and the target molecule dependent on the predicted structure; and selecting one of the one or more of the candidate ligands as the ligand dependent on a result of the evaluating.

51. A method as claimed in claim 50 wherein the target molecule comprises a receptor or enzyme, and wherein the ligand is an agonist or antagonist of the receptor or enzyme; or wherein the ligand comprises an antibody or aptamer and the target molecule comprises an antibody or aptamer target, in particular a virus or cancer cell protein, and wherein the antibody or aptamer binds to the antibody or aptamer target to provide a therapeutic effect.

52. A method as claimed in claim 50 or 51, wherein the ligand is a polypeptide ligand, a polynucleoside ligand, or a polynucleotide ligand.DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Application53. A method of obtaining a diagnostic antibody or aptamer marker of a disease, the method comprising: selecting a target molecule; for each of one or more candidate antibodies or aptamers: performing the method of any one of claims 1-44 to determine a predicted structure of the candidate antibody or aptamer or a complex comprising the candidate antibody or aptamer and the target molecule; and evaluating an interaction between the candidate antibody or aptamer and the target molecule; and selecting one of the one or more of the candidate antibodies or aptamers as the diagnostic antibody or aptamer marker dependent on a result of the evaluating.

54. A method as claimed in any one of claims 47-53 wherein the evaluating the interaction of one of the candidate ligands comprises determining an interaction score for the candidate ligand, wherein the interaction score comprises a measure of an interaction between the candidate ligand and the target molecule.

55. A method as claimed in any one of claims 47-54 further comprising synthesizing the ligand or diagnostic antibody or aptamer marker.

56. A method as claimed in claim 55 further comprising testing biological activity of the ligand or diagnostic antibody or aptamer marker in vitro and in vivo.

57. A method of identifying the presence of a protein or nucleic acid mis-folding disease, comprising: performing the method of any one of claims 1-44 to determine a predicted structure of a protein or nucleic acid; obtaining a structure of a version of the protein or nucleic acid obtained from a human or animal body;DeepMind Technologies LimitedF&R Ref.: 45288-0393WO1 PCT Applicationcomparing the predicted structure of the protein with the structure of a version of the protein or nucleic acid obtained from a human or animal body; and identifying the presence of a protein or nucleic mis-folding disease dependent upon a result of the comparison.

58. A method of determining the structure of a molecule or molecule complex, comprising: applying an experimental technique to a physical sample comprising a molecule or molecule complex to measure experiment signals dependent on a structure of the molecule complex; performing the method of any one of claims 1-44 to determine a predicted structure of the molecule or molecule complex; using the experiment signals and the predicted structure of the molecule complex to determine the structure of the molecule or molecule complex.

59. A method as claimed in claim 58, wherein the experimental technique comprises one or more of: x-ray crystallography, nuclear magnetic resonance, and electron microscopy.

Images