Training structure prediction neural networks using active learning

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a structure prediction neural network for predicting three-dimensional structures of molecules. The system identifies a set of candidate batches of training molecules, determines a batch score for each candidate batch of training molecules, and selects a current batch of training molecules based on the batch scores for training the structure prediction neural network.

Description

TRAINING STRUCTURE PREDICTION NEURAL NETWORKS USING ACTIVE LEARNING CROSS-REFERENCE TO RELATED APPLICATION[00011 This application claims priority to EP Patent Application No. EP24306803.8, filed on October 25, 2024, the disclosure of which is hereby incorporated by reference in its entirety.TECHNICAL FIELD

[0002] This specification relates to using active learning for training structure prediction neural networks.BACKGROUND

[0003] Machine learning models receive an input and generate an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate the output based on the received input and on values of the parameters of the model.

[0004] Some machine learning models are deep neural network models that employ multiple layers of models to generate an output for a received input. For example, a deep neural network includes an output layer and one or more hidden layers that each apply a non-linear transformation to a received input to generate an output.SUMMARY

[0005] This specification generally describes a system implemented as computer programs on one or more computers in one or more locations that can train a structure prediction neural network over a sequence of one or more labeling iterations using an active learning batch selection policy.

[0006] Throughout this specification, a “batch” of data elements (e.g., training molecules for training a structure prediction neural network) can refer to a set of data elements, e.g., a set of 5, or 10, or 100, or 1000 data elements.

[0007] Throughout this specification, the “uncertainty” of a machine learning model in generating a predicted label for a model input can refer to a confidence of the model in the predicted label for the model input.

[0008] Throughout this specification, the “diversity” of model inputs in a batch of model inputs can characterize a level of correlation between predicted outputs generated by a machine learning model for model inputs included in the batch of model inputs. More specifically, lower levels of correlation between the predicted outputs for the model inputs in the batch can be indicative of higher batch diversity, and conversely, higher levels of correlation between the predicted outputs for the model inputs in the batch can be indicative of a lower batch diversity.

[0009] According to a first aspect, there is provided a method performed by one or more computers, the method comprising: training a structure prediction neural network to perform a molecular structure prediction task that comprises processing a network input that characterizes an input molecule to generate a network output that defines a predicted three-dimensional (3D) structure of the input molecule; wherein the training comprises, at each of one or more labeling iterations in a sequence of one or more labeling iterations: selecting a current batch of training molecules for training the structure prediction neural network at the labeling iteration, wherein selecting the current batch of training molecules comprises: identifying a set of candidate batches of training molecules; generating, for each candidate batch of training molecules, a respective batch score for the candidate batch of training molecules that characterizes a predicted utility of the candidate batch of training molecules in improving a performance of the structure prediction neural network; and selecting the current batch of training molecules from the set of candidate batches of training molecules based on the batch scores; obtaining a respective target structure for each training molecule in the current batch of training molecules; and training the structure prediction neural network on at least the current batch of training molecules using the target structures for the current batch of training molecules; and outputting the trained structure prediction neural network.

[0010] In some implementations, for each candidate batch of training molecules, generating the batch score for the candidate batch of training molecules comprises: determining, for each pair of training molecules in the candidate batch of training molecules, a respective joint entropyscore for the pair of training molecules, comprising: determining a plurality of predicted 3D structures of a first training molecule in the pair of training molecules using the structure prediction neural network; determining a plurality of predicted 3D structures of the second training molecule in the pair of training molecules using the structure prediction neural network; and determining the joint entropy score for the pair of training molecules based on the plurality of predicted 3D structures of the first training molecule and the plurality of predicted 3D structures of the second training molecule; and determining the batch score for the candidate batch of training molecules based on the joint entropy scores for the pairs of training molecules included in the candidate batch of training molecules.

[0011] In some implementations, the plurality of predicted 3D structures of the first training molecule and the plurality of predicted 3D structures of the second training molecule are generated by an ensemble of modified versions of the structure prediction neural network.

[0012] In some implementations, each modified version of the structure prediction neural network is determined by dropping a respective subset of neurons from the structure prediction neural network.

[0013] In some implementations, determining the joint entropy score for the pair of training molecules based on the plurality of predicted 3D structures of the first training molecule and the plurality of predicted 3D structures of the second training molecule comprises: processing the plurality of predicted 3D structures to determine, for each of a plurality of pairs of structural units, a covariance between: (i) predicted 3D spatial positions of a first structural unit in the pair of structural units across the plurality of predicted 3D structures, and (ii) predicted 3D spatial positions of a second structural unit in the pair of structural units across the plurality of predicted 3D structures; wherein each of the plurality of pairs of structural units comprises: (i) a first structural unit from the first training molecule or the second training molecule, and (ii) a second structural unit from the first training molecule or the second training molecule; and determining the joint entropy score for the pair of training molecules based on the covariances between the predicted 3D spatial positions of the plurality of pairs of structural units.

[0014] In some implementations, each structural unit is an atom, or an amino acid, or a nucleotide base.

[0015] In some implementations, determining the joint entropy score for the pair of training molecules based on the covariances between the predicted 3D spatial positions of the plurality of pairs of structural units comprises determining the joint entropy score for the pair of training molecules based on a determinant of an array that includes the covariances between the predicted 3D spatial positions of the plurality of pairs of structural units.

[0016] In some implementations, determining the batch score for the candidate batch based on the joint entropy scores for the pairs of training molecules included in the candidate batch comprises computing the batch score based on a combination of a set of joint entropy scores of pairs of different training molecules included in the candidate batch.

[0017] In some implementations, determining the batch score for the candidate batch based on the joint entropy scores for the pairs of training molecules included in the candidate batch comprises computing the batch score based on a summation of the set of joint entropy scores of pairs of different training molecules included in the candidate batch.

[0018] In some implementations, selecting the current batch of training molecules from the set of candidate batches of training molecules based on the batch scores comprises selecting, as the current batch of training molecules, a candidate batch that is associated with the highest batch score from among the set of candidate batches of training molecules.

[0019] In some implementations, selecting the current batch of training molecules from the set of candidate batches of training molecules based on the batch scores comprises: selecting a subset of candidate batches having top batch scores from among the set of candidate batches of training molecules; and perform an optimization of the candidate batches in the subset, the optimization comprises, for each of one or more candidate batches in the subset, replacing one or more training molecules in the candidate batch with different training molecules to maximize the batch score of the candidate batch; and selecting the current batch of training molecules from the optimized subset of candidate batches.

[0020] In some implementations, identifying a set of candidate batches of training molecules comprises: identifying a pool of training molecules; determining a respective uncertainty score for each training molecule in the pool of training molecules, wherein the uncertainty score for a training molecule characterizes an uncertainty of the prediction neural network in generating apredicted 3D structure of the training molecule; determining a probability distribution over the pool of training molecules using the uncertainty scores for the training molecules; and identifying each candidate batch in the set of candidate batches of training molecules by sampling each training molecule included in the candidate batch from the pool of training molecules in accordance with the probability distribution over the pool of training molecules.

[0021] In some implementations, determining the uncertainty score for a training molecule comprises determining the uncertainty score based on a self-entropy score of the training molecule.

[0022] In some implementations, the structure prediction neural network comprises a sequence of multiple subnetworks, wherein a first subnetwork in the sequence processes an initial structure representation of the input molecule to generate an updated structure representation of the input molecule, and each subsequent subnetwork in the sequence processes the structure representation generated by the preceding subnetwork to generate a further updated structure representation of the input molecule.

[0023] In some implementations, each subnetwork in the sequence comprises a respective equivariant subnetwork configured to generate an output structure representation that is equivariant to translations, rotations, and reflections of an input structure representation.

[0024] In some implementations, the sequence of subnetworks comprises a sub-sequence of subnetworks that is consecutively repeated for two or more repetitions, and the repeated subnetworks share the same weights across the repetitions.

[0025] In some implementations, prior to training the structure prediction neural network over each of the one or more labeling iterations, the structure prediction neural network has been pretrained on one or more training examples, and wherein during each of the one or more labeling iterations, parameters of one or more subnetworks in the sequence of multiple subnetworks are held fixed at their pre-trained values.

[0026] In some implementations, outputting the trained structure prediction neural network comprises storing the trained structure prediction neural network in a memory device.

[0027] In some implementations, the input molecule is a protein molecule, and the network input specifies an amino acid sequence of the protein molecule.

[0028] In some implementations, the input molecule is an antibody.

[0029] In some implementations, the input molecule is a heavy chain single variable domain (VHH) antibody molecule.

[0030] In some implementations, prior to training the structure prediction neural network at each of the one or more labeling iterations, the structure prediction neural network has been pretrained on general antibody molecules, and wherein during each of the one or more labeling iterations, the structure prediction neural network is trained on VHH antibody molecules.

[0031] In some implementations, the input molecule is a monomer of a capsid protein.

[0032] In some implementations, the input molecule is a nucleic acid molecule, and the network input specifies a nucleotide sequence of the nucleic acid molecule.

[0033] In some implementations, the input molecule is a messenger ribonucleic acid (mRNA) molecule.

[0034] In some implementations, outputting the trained structure prediction neural network comprises: obtaining a plurality of network inputs, wherein each network input characterizes a respective input molecule; and processing each of the plurality of network inputs using the trained structure prediction neural network to generate a respective predicted 3D structure of the respective input molecule.

[0035] In some implementations, the method further comprises: selecting one or more molecules using the structure prediction neural network; and physically synthesizing the one or more molecules.

[0036] According to another aspect there is provided a system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of the methods described herein.

[0037] According to another aspect there are provided one or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations of the methods described herein.

[0038] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages.

[0039] The function and properties of a molecule, such as a protein or a nucleic acid, is often determined or influenced by its three-dimensional (3D) structure. Predicting the 3D structures of molecules is important in fields such as drug discovery, disease understanding, and bioengineering. Neural networks, referred to in this specification as structure prediction neural networks, can be used to predict 3D structures of molecules. However, training such structure prediction neural networks faces challenges, particularly because the prediction task requires performing high-dimensional regression. For example, predicting a 3D structure of a protein with 100 amino acids may involve predicting 3003D spatial coordinates, e.g., if the prediction is for the position of one atom (e.g., the alpha carbon atom) in each amino acid, or of approximately 7500 spatial coordinates, e.g., if the prediction is for the position of each atom in each amino acid. Performing such a complex prediction task can require the structure prediction neural network to have a complex architecture with a large number of parameters, and further, may require training on a large number of training examples and over a large number of training iterations.

[0040] Obtaining 3D structures for training molecules is time-consuming and expensive, because doing so can require physically synthesizing the molecule and experimentally measuring its 3D structure, e.g., using x-ray crystallography or cryo-electron microscopy (cyro-EM). As a result, the requirement for a large number of labeled training examples in conventional techniques for training structure prediction neural networks presents a significant technical challenge.

[0041] Further, training a structure prediction neural network on a large dataset of molecules demands requires enormous computational resources, both in terms of memory and processing power. This is due to (i) the complexity of the model, which must accommodate the high-dimensional nature of the output, and (ii) the size of the training dataset, which requires a large number of forward passes and backpropagation steps to update the network parameters.

[0042] The techniques described herein address these challenges by using active learning, a computational approach that selects the most informative training data for labeling. Active learning reduces the need for large amounts of labeled data by focusing on samples that are expected to provide the most significant improvement to the model’s performance. This reduces the burden of data labeling and minimizes the computational resources required to train the model. By training on only the most informative samples, the system demands less computational power, memory, and energy compared to conventional approaches that use a fixed or random selection of training examples for labeling.

[0043] In particular, the batch selection process is adapted to the 3D structure prediction task, where the model output includes a high-dimensional regression prediction. For example, in some implementations, the batch score is computed based on the joint entropies of pairs of training molecules in the batch, and in particular, computed based on the covariances between predicted 3D spatial positions of the structural units (e.g., atoms or amino acids) of the pair of molecules. This approach allows the active learning strategy to effectively capture the uncertainty and diversity of the model output in the context of high-dimensional regression, leading to the selection of more informative training samples. Scoring batches based on covariances between predicted 3D spatial positions of individual structural units in pairs of training molecules allows the system to select batches of training molecules in a manner that captures prediction uncertainty and molecular diversity at the granular level of individual structural units, which can enhance the effectiveness of the active learning approach.

[0044] The techniques described herein provide a particular use of artificial intelligence (Al) to solve the problem of effectively and efficiently predicting the 3D structure of a molecule.Furthermore, by selecting training molecule batches using the described techniques, the amount of training data and the number of training iterations required for the training are reduced. This results in faster training times and lower computational resource consumption. Such improvements enhance the scalability and practicality of using Al for complex, high-dimensional tasks like 3D structure prediction.

[0045] Put another way, the system described in this specification provides a technical solution to several technical problems that arise when training a structure prediction neural network to perform a 3D molecule structure prediction task. More specifically, the system implements a batch active learning policy for selecting batches of molecules for labeling for use in training the structure prediction neural network in order to increase the prediction accuracy of the structure prediction neural network, reduce the number of training iterations required for training the structure prediction neural network, and reduce the overall number of molecules for which a label (3D structure) is needed.

[0046] Reducing the number of training iterations required for training the machine learning model can enable more efficient use of resources. For instance, training the machine learning model at each training iteration requires computational resources (e.g., memory and computing power), and therefore reducing the number of training iterations can reduce consumption of computational resources during training. Further, obtaining target 3D structures for molecules can be time consuming and expensive, e.g., because determining a 3D structure for a molecule can require performing physical experiments. Therefore reducing the number of molecules that require labeling can reduce consumption of resources required for obtaining target 3D structures for training molecules.

[0047] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0048] FIG. 1 shows an example machine learning system.

[0049] FIG. 2 shows an example training system.

[0050] FIG. 3 A illustrates an example of a structure prediction neural network.

[0051] FIG. 3B illustrates another example of a structure prediction neural network.

[0052] FIG. 4 is a flow diagram illustrating an example process for training a structure prediction neural network.

[0053] FIG. 5 is a flow diagram illustration of an example process for identifying a set of candidate batches of training molecules.

[0054] FIG. 6 is a flow diagram illustrating an example process for determining a batch score for a current batch of training molecules.

[0055] FIG. 7 demonstrates a comparison of the performances of the described technique with other techniques.

[0056] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0057] Predicting the 3D structures of molecules is important in fields such as drug discovery, disease understanding, and bioengineering. For example, the 3D structure of a protein molecule can determine or significantly influence its properties, such as binding affinity, specificity, stability, and solubility. Accurately predicting the 3D structure of molecules, such as proteins, is crucial for optimizing molecule design in therapeutic applications, such as drug development, antibody engineering, and vaccine design.

[0058] FIG. 1 shows an example machine learning system 100. The machine learning system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0059] The machine learning system 100 includes a training system 200 to train a structure prediction neural network 255 that performs a molecular structure prediction task. In particular, the structure prediction neural network 255 is configured to process an input that characterizes an input molecule to generate an output that defines the input molecule’s predicted three-dimensional (3D) structure. Example architectures of the structure prediction neural network 255 will be described below with references to FIG. 3 A and FIG. 3B.

[0060] In some cases, the input molecule can be a protein molecule. The input to the neural network 255 can specify an amino acid sequence of the protein molecule, and the output specifies a predicted 3D protein structure. For example, the output can specify the 3Dcoordinates of the atoms in the protein molecule. In another example, the output can specify the 3D positions of the amino acid residues, such as the coordinates of the Ca (alpha carbon) atom of each amino acid residue in the protein molecule.

[0061] The input molecule can be any type of protein molecule, such as an antibody, an enzyme, a receptor, a structural protein, a transport protein, a signaling protein (e.g., a hormone or cytokine), a motor protein, a transcription factor, or a chaperone protein.

[0062] In a particular example, the input molecule may be a Nanobody® immunoglobulin single variable domain (ISVD), for example a Nanobody® VHH. Nanobody® VHHs are important because they are small, stable antibody fragments that can bind to specific antigens with high affinity. They have numerous applications in biotechnology and medicine, including drug development, diagnostics, and research. The 3D structure of a Nanobody® VHH can determine its binding affinity and specificity to target antigens, which is crucial in optimizing its efficacy for therapeutic uses, such as neutralizing pathogens, targeting cancer cells, or delivering drugs to specific tissues.

[0063] In another example, the input molecule may be a monomer of a capsid protein. Capsid proteins are essential components of the viral capsid, the protein shell that encases the genetic material of a virus. The 3D structure of a capsid protein influences the virus’s stability, assembly, and infectivity. Predicting the 3D structure of capsid proteins is important in applications such as vaccine development, drug delivery, and antiviral agents.

[0064] In some cases, the input molecule may be a nucleic acid molecule. The network input specifies a nucleotide sequence of the nucleic acid, and the network output specifies a predicted 3D structure of the nucleic acid. For example, the output can specify the 3D coordinates of the nucleotide bases in the nucleic acid.

[0065] In a particular example, the input molecule may be a messenger ribonucleic acid (mRNA) molecule. mRNA-based technologies have become increasingly important in fields such as vaccine development (e.g., mRNA vaccines for COVID-19), gene therapy, and therapeutic protein production. The 3D structure of mRNA can influence its stability, translation efficiency, and interaction with other molecules, such as proteins and microRNAs. Accurately predicting mRNA structure can aid in the design of more stable and effective mRNA-basedtherapies. The input molecule can be a non-coding RNA molecule, such as an aptamer or miRNA, or a regulatory DNA element, such as a promoter.

[0066] Further, in some cases, the structure prediction neural network can process a network input that identifies a plurality of molecules included in a molecular system. The plurality of molecules included in the molecular system can include one or more of small molecule ligands, proteins, DNA molecules, or RNA molecules. The network output generated by the structure prediction neural network can define a predicted 3D structure of the molecular system. In these cases, during training of the structure prediction neural network, at each labeling iteration, the system can select molecular systems for labeling (e.g., as an alternative to or in combination with selecting single molecules for labeling).

[0067] The training system 200 is configured to determine batches of training molecules 225 and use the training batches 225 to train the structure prediction neural network 255. Each training batch 225 includes multiple training molecules used as training inputs to train the structure prediction neural network 255. In some cases, each training molecule in a training batch 225 is a protein molecule, and the training batch 225 specifies an amino acid sequence of the protein molecule. In some cases, each training molecule in the training batch 225 is a nucleic acid molecule, and the training batch 225 specifies a nucleotide sequence of the nucleic acid molecule.

[0068] The operations of the training system 200 will be described in further detail below with reference to FIG. 2. In general, the training system 200 performs a labeling process, where at each of one or more labeling iterations, the system selects a batch of training molecules for labeling, determines a respective 3D structure for each training molecule as the “label” for the molecule, and uses the batch to update the parameter values of the structure prediction neural network 255. At each labeling iteration, the system 200 generates a set of candidate batches of training molecules, determines a respective batch score for each candidate batch based on the predicted utility of the batch in improving the network, selects the batch based on the batch scores, labels the training molecules in the selected batch with the corresponding target structures, and trains the structure prediction neural network 255 on at least the current batch of training molecules.

[0069] Once the structure prediction neural network 255 has been trained, a molecule design / selection system 250 can use the trained structure prediction neural network 255 to design or select target molecule sequences that are optimized for their 3D structure.

[0070] In some cases, the system 250 can screen a library that defines a set of candidate molecule sequences to identify molecule sequences having desirable 3D structures. The system 250 can process each candidate molecule sequence using the trained structure prediction neural network 255 to generate a predicted 3D structure for the candidate molecule sequence, and select one or more candidate molecules as target molecules 260 based on the predicted structures for further processing or development.

[0071] The candidate molecule sequences can be obtained in a number of different ways. In some cases, the candidate molecule sequences can include known molecules, such as those obtained from public databases, or other repositories that house experimentally determined or computationally predicted sequences and structures. In some cases, the candidate molecule sequences can include molecules that have been generated by a generative machine learning model.

[0072] The system 250 can select, based on the predicted 3D structures of the candidate molecules, molecules that exhibit desirable structural characteristics as the target molecules 260. For example, the desirable structural characteristics can include specific motifs or binding sites for a particular biological function. In another example, the system 250 can select the target molecules based on structural properties such as the overall shapes, structural compactness, or flexibility of certain domains, depending on the intended use of the molecules.

[0073] The designed or selected target molecules 260 can be manufactured, i.e., physically synthesized, using any suitable techniques 270 and applied in a variety of applications. For example, the synthesized mRNAs or antibodies may be used as therapeutics 280 administered to a subject 290 to achieve a therapeutic effect, such as mRNA vaccines for infectious diseases or cancer, gene delivery agents for treating genetic disorders, or antibody therapies for targeting specific diseases.

[0074] In one example, the molecule design / selection system 250 can use the structure prediction neural network 255 to determine a respective predicted 3D structure of each moleculein a set of candidate molecules. The system 250 can then determine, for each candidate molecule, a predicted binding affinity between: (i) the candidate molecule, and (ii) a binding target molecule, based at least in part on the predicted 3D structure of the candidate molecule. For instance, the system can determine the binding affinity between the candidate molecule and the binding target molecule, e.g., using molecular docking, or a free energy perturbation calculation, or a thermodynamics integration calculation. The system can then select one or more of the candidate molecules based on their predicted binding affinities for the binding target molecule. In one example, the candidate molecule can be an antibody (e.g., a Nanobody® ISVD such as a Nanobody® VHH) and the binding target molecule can be an antigen of a pathogen (e.g., an influenza pathogen, or a SARS-CoV-2 pathogen, or a human papillomavirus (HPV) pathogen, or a hepatitis B pathogen, or a malaria pathogen, or a dengue pathogen, or a tuberculosis pathogen, or an HIV pathogen, or a Zika virus pathogen, and so forth). The selected candidate molecules can optionally be physically synthesized and included in therapeutics, e.g., drugs, that are administered to subjects. The system can train the structure prediction neural network using the described active learning approach in a manner that causes the structure prediction neural network to predict the structures of candidate molecules more accurately. These improved predictions of the 3D structures of the candidate molecules can allow a candidate molecule with a higher binding affinity for the binding target molecule to be selected and then synthesized for use as a therapeutic.

[0075] FIG. 2 shows an example training system 200. The training system 200 is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0076] The training system 200 trains a structure prediction neural network 255 over one or more labeling iterations to perform a molecular structure prediction task. In particular, the structure prediction neural network 255 is configured to process an input that characterizes an input molecule to generate an output that defines the input molecule’s predicted three-dimensional (3D) structure.

[0077] The model input to the structure prediction neural network 255 can be represented in any appropriate way. In some cases, the input molecule is a nucleic acid (e.g., a DNA or anRNA), and the model input can include a vector that numerically encodes the nucleotide sequence of the input nucleic acid, or the codon sequence of the input DNA or RNA, e.g., by mapping each nucleotide of a list of nucleotides or each codon of a list of codons (e.g., a list of all 64 codons) to a respective numerical value and concatenating the mapped numerical values.

[0078] In some cases, the input molecule is a protein, and the model input can include a numerical representation of the amino acid sequence of the input protein. For example, each amino acid in the sequence can be mapped to a numerical value based on its type (e.g., by using a one-hot encoding scheme to represent the 20 standard amino acids).

[0079] The structure prediction neural network 255 can have any appropriate architecture for predicting molecule structures. The neural network 255 can include any appropriate types of neural network layers (e.g., fully connected layers, message passing layers, convolutional layers, attention layers, and so forth) in any appropriate number (e.g., 5 layers, or 10 layers, or 100 layers) and connected in any appropriate configuration (e.g., as a directed graph of layers).

[0080] FIG. 3 A illustrates an example implementation of structure prediction neural network 300a. The network 300a processes the network input that characterizes the input molecule 305 to generate the network output that defines the predicted 3D structure 340 of the input molecule 305. The network input can include a sequence representation of the input molecule. The network output can specify the spatial positions of the structural units of the input molecule 305. Depending on the type of input molecule and the application, the structural units can be amino acids (for a protein), nucleotides (for a nucleic acid), or atoms (for any molecule). For example, the network output 340 can specify the spatial position of each amino acid of an input protein molecule 305, the spatial position of each nucleotide of an input nucleic acid molecule 305, or the spatial position of each atom of an input molecule 305. The network output 340 can also specify the spatial positions of backbone atoms of an input protein molecule 305 or an input nucleic acid molecule 305.

[0081] The network 300a includes a sequence of subnetworks 310-n, n=l, 2, 3, etc. The first subnetwork 310-1 processes an initial structure representation 320-1 of the input molecule 305 to generate an updated structure representation 330-1 of the input molecule 305. In some cases, the initial structure representation 320-1 can be a randomly generated 3D structure of the inputmolecule 305. Each subsequent subnetwork (e.g., 310-2, 310-3, and 310-4) processes the structure representation generated by the preceding subnetwork to generate a further updated structure representation (e.g., 330-2 and 330-3) of the input molecule. The last subnetwork (e.g., 310-4) is configured to generate a final structure representation for the network output 340.

[0082] In some cases, each subnetwork is an equivariant subnetwork. That is, the subnetwork is configured to generate an output structure representation that is equivariant to spatial transformations, including translations, rotations, and reflections of the input structure representation. Equivariance refers to the property of a model where the output changes in a predictable way in response to transformations of the input. For example, if the input structure is rotated, translated, or reflected in 3D space, the output structure will undergo the same transformation, ensuring that the relative geometry of the molecule remains consistent. This property is important for handling 3D molecular structures because the absolute position and orientation of a molecule in 3D space are often arbitrary. By using equivariant subnetworks, the structure prediction neural network 255 can learn to predict the relative positions of structural units in a molecule without being affected by the absolute position and orientation of the molecule.

[0083] The equivariant subnetworks can be implemented with any appropriate architectures. Implementation examples of the structure prediction neural network including the sequence of subnetworks are described in Lee, at el., “Equifold: Protein structure prediction with a novel coarse-grained structure representation,” bioRxiv, 2022.

[0084] FIG. 3B illustrates another example implementation of structure prediction neural network 300b. The neural network 300b is a modified version of the neural network 300a of FIG.3 A. In particular, the neural network 300b includes a subsequence 312 of subnetworks that is consecutively repeated for two or more repetitions. The repeated subnetworks share the same weights across the repetitions. For example, the sub-sequence 312 can include a sequence of subnetworks 310-3, 310-4, followed by the same subnetworks 310-3, 310-4, sequentially repeated one or more times. This modification increases the depth of the model architecture without increasing the number of parameters, allowing the network to capture more complex relationships and features in the input while maintaining computational efficiency. By reusing thesame weights across repeated subnetworks, the model can perform more iterations of structure refinement without the need for additional parameters, which would otherwise increase memory requirements and computational costs.

[0085] Referring back to FIG. 2, the training system 200 includes a batch generation engine 220, a labeling engine 230, and a training engine 240, which are each described in more detail next. The training system 200 trains the structure prediction neural network 255 over one or more labeling iterations. In each labeling iteration, the training system 200 determines a batch of training examples and updates the parameters of the neural network 255 using the batch of training examples.

[0086] The batch generation engine 220 is configured to select, at each labeling iteration, a respective current batch of training molecules 225 for labeling at the labeling iteration. In general, in order to optimize the sample efficiencies (i.e., to select training examples for labeling that are expected to provide the most significant improvement to the model’s performance), the batch generation engine 220 implements a policy for selecting the current batch of training molecules that accounts for both the prediction uncertainty associated with individual training molecules and for the diversity of the current batch of training molecules as a whole. Training the neural network on molecules associated with high prediction uncertainty can improve the performance of the trained model because these are the examples where the neural network is likely to improve the most by adjusting its parameters. Training the neural network on a batch of training molecules with high diversity can improve generalization because it exposes the neural network to a wider range of molecular structures and prevents it from overfitting to a particular subset of the data.

[0087] More specifically, as part of selecting the current batch of training molecules, the batch generation engine 220 evaluates a set of “candidate” batches of training molecules. For each candidate batch of training molecules, the system determines a batch score for the candidate batch of training molecules that characterizes both: (i) an uncertainty of the structure prediction neural network 225 in generating predicted 3D structures for the training molecules in the candidate batch of training molecules, and (ii) a diversity of the training molecules in the candidate batch of training molecules. The batch generation engine 220 can then select thecurrent batch of training molecules from the set of candidate batches of training molecules based on the scores, e.g., by selecting a candidate batch of training molecules associated with the highest score as the current batch of training molecules.

[0088] The batch generation engine 220 can determine the batch scores and select the batches of training molecules in any of a variety of possible ways. Example techniques for selecting the batches of training molecules are described in more detail below with reference to FIGs. 4-6.

[0089] The labeling engine 230 is configured to obtain, at each labeling iteration, a respective target molecule structure label 235 for each training molecule in the current batch of training molecules. The target molecule structure label 235 for a training molecule defines output data that should be generated by the structure prediction neural network 255 by processing the input characterizing the training molecule.

[0090] The labeling engine 230 can obtain the target molecule structure label 235 for the training molecules in the current batch of training molecules 106 in any of a variety of possible ways. A few example techniques for obtaining target molecule structure label 235 for training molecules are described next.

[0091] For instance, the labeling engine 230 can provide instructions, e.g., by way of a user interface or an application programming interface (API) made available by the system 200, that one or more physical experiments should be performed to obtain the target molecule structure label 235. In some cases, determining a target molecule structure label for a training molecule can require physically synthesizing the training molecule and experimentally determining its 3D structure. Examples of experimental techniques for determining the 3D structure of a molecule can include X-ray crystallography, nuclear magnetic resonance (NMR) spectroscopy, or cryoelectron microscopy (cryo-EM). The labeling engine 230 can receive the results of the experiments, e.g., by way of the user interface or the API, and can associate each training molecule in the current batch with its respective target structure label as determined by the experiments.

[0092] In some cases, the 3D structure of a training molecule can be determined using computational techniques. For example, the 3D structure of certain proteins can be predicted using homology modeling, threading, or ab initio prediction methods. The 3D structure of certainnucleic acids can be predicted using molecular dynamics simulations or other computational methods. The labeling engine 230 can use the output of these computational techniques to associate each training molecule in the current batch with its respective target structure label.

[0093] In some cases, the labeling engine 230 can access target molecule structure labels 235 for the training molecule batch 225 by retrieving the target molecule structure labels 235 from a database of molecule structure labels.

[0094] Training the structure prediction neural network 255 on a training molecule can include training the structure prediction neural network 255 to reduce a discrepancy between: (i) a predicted 3D structure generated by the structure prediction neural network 255 for the training molecule, and (ii) the target structure label 235 for the training molecule. More specifically, the labeling engine 240 can train the structure prediction neural network 255 to optimize (e.g., minimize) an objective function that measures an error between: (i) the predicted 3D structure for the training molecule, and (ii) the target structure label for the training molecule.

[0095] The objective function can measure the error between a predicted structure and a target structure in any appropriate way. The objective function can quantify the error between a predicted structure and a target structure using any appropriate method. In one example, the objective function measures the frame-aligned point error (FAPE) loss, which calculates the alignment error between the predicted 3D coordinates of the structural units and the target (ground truth) coordinates of the same units in the training molecule. FAPE accounts for both the rotation and translation of the predicted structure, ensuring that the predicted 3D coordinates are properly aligned with the target structure, regardless of the molecule’s orientation in 3D space.

[0096] The training engine 240 can train the structure prediction neural network 255 using any appropriate machine learning training technique. For instance, the training engine 240 can train the neural network 255 on a training molecule by processing the training molecule to generate a predicted 3D structure, evaluating the objective function on the predicted 3D structure, determining gradients of the objective function with respect to a set of neural network parameters of the neural network, and adjusting the current values of the set of neural network parameters using the gradients. The labeling engine 240 can determine the gradients of the objective function, e.g., using backpropagation, and can adjust the current values of the neural networkparameters based on the gradients using the update rule of an appropriate gradient descent optimization technique, e.g., RMSprop or Adam.

[0097] In some implementations, prior to training the structure prediction neural network 255 using the selected batches of training molecules at the one or more labeling iterations, the structure prediction neural network 255 has been pre-trained on one or more additional training examples. For example, the pre-training can involve training the structure prediction neural network 255 on a dataset where the structural labels of the training molecules are available, such as in a published dataset. Pre-training on such a dataset helps the network learn general patterns and relationships between molecular sequences and their corresponding structures, which provides an initial foundation for subsequent fine-tuning on more specific molecules.

[0098] In one particular example, the structure prediction neural network 255 has been pretrained on a dataset of general antibody molecules, whereas during each of the plurality of training iterations, the structure prediction neural network 255 is fine-tuned on selected batches of Nanobody® VHH molecules. Pre-training on a larger, more general antibody dataset allows the model to learn fundamental features and patterns common to all antibodies. Fine-tuning then specializes the model to the specific characteristics of Nanobody® VHHs, which are smaller and have unique structural features compared to conventional antibodies. This approach leverages the knowledge gained from the broader dataset while optimizing the model for the target task of predicting the 3D structures of Nanobody® VHH molecules.

[0099] In some cases, a subset of parameters of the structure prediction neural network 255 is held fixed at their pre-trained values during fine-tuning over the one or more labeling iterations. For example, for the neural network 300a or 300b described with references to FIG. 3 A and FIG.3B, the parameters of one or more of the subnetworks 310-n can be held fixed at their pre-trained values during fine-tuning.

[0100] In one particular example, the parameters of one or more earlier subnetworks (those closer to the first layer of the network, e.g., 310-1 and 310-2) are held fixed during fine-tuning. Holding certain parameters fixed during fine-tuning can allow the network to better preserve the general knowledge learned during pre-training, particularly in earlier layers that capture basic structural features. This reduces the risk of overfitting to the smaller fine-tuning dataset andensures that the model retains robust, generalizable features while focusing the fine-tuning process on the parameters of later subnetworks that are more directly responsible for capturing specialized and task-specific details.

[0101] After training the structure prediction neural network 255, the training system 200 can output the trained structure prediction neural network 255, e.g., by storing data defining the trained structure prediction neural network 255 in a memory, or by transmitting data defining the trained structure prediction neural network 255 over a data communications network. The trained structure prediction neural network 255 can be defined by data specifying the architecture of the structure prediction neural network 255 and the trained values of the set of model parameters of the structure prediction neural network.

[0102] A downstream system, e.g., the design / selection system 250 of FIG. 1, can use the trained structure prediction neural network 255 to design or select target molecule sequences with desirable 3D structures. In some cases, the target sequences can be physically synthesized and applied in a clinical application, as described with reference to FIG. 1.

[0103] FIG. 4 is a flow diagram of an example process 400 for training a structure prediction neural network. For convenience, the process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the training system 200 of FIG. 2, appropriately programmed in accordance with this specification, can perform the process 400.

[0104] The system performs 410 - 450 at each labeling iteration in a sequence of one or more labeling iterations. For convenience, the description which follows will reference a “current” labeling iteration in the sequence of labeling iterations.

[0105] At 410, the system identifies a set of candidate batches of training molecules to the structure prediction neural network. The system can identify any appropriate number of candidate batches, e.g., 10 candidate batches, 1000 candidate batches, or 1,000,000 candidate batches. Each candidate batch can include any appropriate number of training molecules, e.g., 10 training molecules, or 50 training molecules, or 100 training molecules. An example process for identifying a set of candidate batches of training molecules is described in detail with reference to FIG. 5.

[0106] At 420, the system generates a respective batch score for each candidate batch of training molecules. The batch score for a candidate batch of training molecules characterizes a predicted utility of the candidate batch of training molecules in improving the performance of the structure prediction neural network. Example processes for generating the batch score characterizing the predicted utility of the candidate batch are described in detail with reference to FIG. 6.

[0107] At 430, the system selects a current batch of training molecules for labeling at the current labeling iteration based on the batch scores of the candidate batches of training molecules. For instance, the system can select the candidate batch of training molecules that is associated with the highest score from among the set of candidate batches of training molecules.

[0108] Optionally, as part of selecting the current batch of training molecules, the system can select a subset of the candidate batches of training molecules (e.g., the candidate batches that are associated with the highest batch scores) and then optimize each selected candidate batch element-wise. The element-wise optimization can include, for each of the selected candidate batches, replacing one or more training molecules in the candidate batch with different training molecules (e.g., selected from a pool of training molecules) to maximize the batch score of the candidate batch. The system can perform the optimization process until convergence is reached, e.g., the batch scores of the selected candidate batches are no longer increasing. The system can then select the batch of training molecules associated with the highest score as the current batch of training molecules for training the structure prediction neural network at the current training iteration.

[0109] At 440, the system obtains a respective target molecule structure for each training molecule in the current batch of training molecules. The target molecule structure for a training molecule defines a network output that should be generated by the structure prediction neural network by processing the model input characterizing the training molecule. A target molecule structure for a training molecule can be generated, e.g., by physically synthesizing the training molecule, and experimentally determining the target molecule structure for the training molecule. As another example, the system can determine a target molecule structure for a training molecule using numerical techniques, as described with reference to FIG. 2.

[0110] At 450, the system trains the structure prediction neural network on at least the current batch of training molecules using the target molecule structures for the current batch of training molecules. Optionally, the system can train the structure prediction neural network on: (i) the current batch of training molecules, and (ii) batches of training molecules selected at any of the preceding labeling iterations. The system can train the structure prediction neural network using any appropriate machine learning training technique, e.g., a stochastic gradient descent training technique that is implemented over a plurality of training iterations.

[0111] At 460, the system determines whether a termination criterion for terminating the labeling of training molecules for training the structure prediction neural network has been satisfied. The system can determine that a termination criterion has been satisfied, e.g., if a predefined number of labeling iterations have been performed, or if a performance (e.g., prediction accuracy) of the structure prediction neural network exceeds a threshold when evaluated on a set of validation data. The system can perform any appropriate number of labeling iterations, e.g., 1, or 5, or 10 labeling iterations.

[0112] In response to determining that the termination criterion has been satisfied, the system outputs the trained structure prediction neural network (470).

[0113] In response to determining that the termination criterion has not been satisfied, the system returns to step (410) and proceeds to the next labeling iteration.

[0114] FIG. 5 is a flow diagram of an example process 500 for identifying a set of candidate batches of training molecules. For convenience, the process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the training system 200 of FIG. 2, appropriately programmed in accordance with this specification, can perform the process 500.

[0115] At 510, the system identifies a pool of training molecules. In some cases, the system obtains a predefined pool of training molecules, e.g., in the form of a preexisting database of molecules. In other cases, the system generates data identifying the pool of training molecules using a generative model, e.g., a machine learning model that can be queried to generate samples from a distribution over a space of possible training molecules. The generative model can be,e.g., a diffusion-based neural network model, a generative adversarial neural network, a flowbased model, and so forth.

[0116] At 520, the system determines a respective uncertainty score for each training molecule in the pool of training molecules. An uncertainty score for a training molecule characterizes an uncertainty of the structure prediction neural network in generating a predicted 3D structure for the training molecule. In one example, the uncertainty score for a training model can be determined based on a self-entropy score computed for the training molecule. The self-entropy score can be computed using a process that is similar to the process of computing a joint entropy score for a pair of training molecules as described with reference to FIG. 6.

[0117] A self-entropy score for a single training molecule can quantify the model’s uncertainty about the training molecule’s predicted 3D structure. To calculate this score, the system can first generate a plurality of different predicted 3D structures for that single molecule. This can be achieved by processing the molecule’s input multiple times through an ensemble of modified versions of the structure prediction neural network. For instance, using a technique like Monte Carlo dropout, different neurons are randomly dropped from the network for each forward pass, resulting in a varied set of structural predictions. This collection of predicted structures forms a distribution, where a wider and more varied distribution corresponds to higher model uncertainty for that molecule.

[0118] This distribution of predicted structures can then be used to compute the final score. In particular, the system can compute a covariance matrix based on the predicted 3D spatial positions of the structural units (e.g., atoms or amino acids) across the entire ensemble of generated structures. The diagonal elements of this matrix represent the variance in the predicted position for each structural unit, while the off-diagonal elements represent the covariance between the positions of different pairs of structural units. The final self-entropy score can then be determined from this covariance matrix, for example, by calculating its determinant. A larger determinant signifies a greater volume of uncertainty in the predicted conformational space, resulting in a higher self-entropy score and marking the molecule as a valuable, high-uncertainty candidate for training.

[0119] At 530, the system determines a probability distribution over the pool of training molecules using the uncertainty scores for the training molecules. To generate the probability distribution, the system can generate a quantile distribution over the pool of training molecules that assigns a respective quantile value to each training molecule that defines a quantile of the uncertainty score of the training molecule. The system can then process the quantile distribution, e.g., by a soft-max function, to generate the probability distribution over the set of training molecules.

[0120] At 540, the system identifies the set of candidate batches of training molecules using the probability distribution over the pool of training molecules. For instance, for each candidate batch of training molecules, the system can sample each training molecule included in the candidate batch of training molecules in accordance with the probability distribution over the pool of training molecules.

[0121] FIG. 6 is a flow diagram of an example process 600 for determining a batch score for a current batch of training molecules. For convenience, the process 600 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the training system 200 of FIG. 2, appropriately programmed in accordance with this specification, can perform the process 600.

[0122] For each pair of training molecules in the candidate batch of training molecules, the system performs 610-630 to determine an entropy score for the pair of training molecules. Each pair of training molecules includes a first training molecule and a second training molecule. The joint entropy score characterizes the uncertainty associated with the joint distribution of the two molecules’ predicted 3D structures. Entropy scores computed for pairs of training molecules in the candidate batch further characterize the diversity of the training molecules in the candidate batch.

[0123] At 610, the system determines a plurality of predicted 3D structures of the first training molecule using the structure prediction neural network according to the current values of the parameters of the structure prediction neural network.

[0124] At 620, the system determines a plurality of predicted 3D structures of the second training molecule using the structure prediction neural network according to the current values of the parameters of the structure prediction neural network.

[0125] In some implementations, the plurality of predicted 3D structures of the first training molecule and the plurality of predicted 3D structures of the second training molecule are generated by an ensemble of modified versions of the structure prediction neural network.

[0126] The ensemble of modified structure prediction neural networks can include any appropriate number of neural networks, e.g., 5 neural networks, or 10 neural networks, or 100 neural networks. Each modified structure prediction neural network in the ensemble differs from each other structure prediction neural network in the ensemble, e.g., as a result of having one or more different parameter values, or having a different architecture, or both.

[0127] In one example, each modified structure prediction neural network in the ensemble is determined by dropping a respective set of parameters from the original structure prediction neural network. “Dropping” a parameter from the neural network can refer to setting the value of the parameter to a default value, e.g., zero, or to a random value, e.g., a value sampled from a Gaussian distribution. The system can randomly sample the set of parameters to be dropped from the neural network in order to generate each modified structure prediction neural network in the ensemble. For instance, for each parameter in the original structure prediction neural network and for each modified structure prediction neural network in the ensemble, the system can determine whether to drop the parameter based on sampling from a probability distribution, e.g., a Bernoulli distribution.

[0128] Given the ensemble of modified structure prediction neural networks, the system generates a respective predicted 3D structure of the first training molecule using each modified structure prediction neural network in the ensemble (610) and generates a respective predicted 3D structure of the second training molecule using each modified structure prediction neural network in the ensemble (620).

[0129] At 630, the system determines the joint entropy score for the pair of training molecules based on the plurality of predicted 3D structures of the first training molecule and the plurality of predicted 3D structures of the second training molecule.

[0130] In some implementations, to determine the joint entropy score, the system computes the covariances between the predicted 3D spatial positions of pairs of structural units in the plurality of predicted 3D structures (at 632) and determines the joint entropy score for the pair of training molecules based on the covariances between the predicted 3D spatial positions of the pairs of structural units (at 634).

[0131] In particular, at 632, the system processes the plurality of predicted 3D structures to determine, for each of a plurality of pairs of structural units, a covariance between: (i) predicted 3D spatial positions of a first structural unit in the pair of structural units in the plurality of predicted 3D structures, and (ii) predicted 3D spatial positions of a second structural unit in the pair of structural units across the plurality of predicted 3D structures. Each of the plurality of pairs of structural units includes: (i) a first structural unit from the first training molecule or the second training molecule, and (ii) a second structural unit from the first training molecule or the second training molecule. An example of the computation process of the joint entropy scores is described below.

[0132] The joint entropy of a pair of structure predictions xt, Xj can be defined asHij = ~ MXu)l°gPu(Xo)'where Xij = (Xi’X7'}JXi7

[0133] In the above equation, each of xtand Xj is a vector representing the predicted 3D spatial positions of the structural units of the corresponding training molecule i and training molecule j, respectively, and x(-, Xj E IRN“x3, where Nuis the number of structural units in the corresponding training molecule. Pi7(Xi7) is the joint probability that the predicted 3D structures (e.g., the 3D coordinates of the structural units) for molecule z and molecule j align with xtand Xj, respective. Pi7(Xi7) characterizes how correlated the predicted 3D structures of the two molecules are, and can be calculated using the predictions generated by the ensemble of modified structure prediction neural network.

[0134] Assuming that (1) Pi7(Xi7) follows a normal distribution characterized by a mean vector Hij and a covariant matrix Sj7and (2) the positions of all groups of structural units are independent, the joint entropycan be determined to follow:1Hij o -log (detS0)

[0135] In the above formula, detS^- represents the determinant of theThe covariant matrix Stj can be estimated using the predictions generated by the ensemble of modified structure prediction neural network as = IE [(Xj —

[0136] The covariant matrix 2^- can be represented as an array that includes the covariances between the predicted 3D spatial positions of the plurality of pairs of structural units. Thus, the entropy score for the pair of training molecules can be determined based on a determinant of an array that includes the covariances between the predicted 3D spatial positions of the plurality of pairs of structural units.

[0137] Once the entropy scores for the pairs of training molecules included in the candidate batch have been determined, the system can determine the batch score for the candidate batch based on the entropy scores computed for the pairs of training molecules included in the candidate batch (at 640).

[0138] In some cases, the batch score for the candidate batch can be computed based on a combination (e.g., summation) of a set of joint entropy scores of pairs of different training molecules included in the candidate batch. In a particular example, the batch score a for a batch of training molecules can be computed asa — Hij.

[0139] FIG. 7 demonstrates a performance of the training technique described above compared with other techniques. In particular, FIG. 7 shows the validation FAPE losses computed at different labeling (selection) iterations (rounds). As shown by FIG. 7, the FAPE loss of the technique described above (shown as “DEWDROP”) decreases faster compared to the other techniques including random sample selection (“RANDOM”), K-Means sample selection (“KMEANS”), and Batch Active Learning by Diverse Gradient Embeddings (“BADGE”). This demonstrates that the described technique requires fewer labeling iterations (and, further, less labeled training data and fewer training iterations) to achieve a predefined training target compared to conventional techniques.

[0140] In this Specification, the term “immunoglobulin single variable domain” (ISV), interchangeably used with “single variable domain”, defines immunoglobulin molecules wherein the antigen binding site is present on, and formed by, a single immunoglobulin domain. This sets immunoglobulin single variable domains apart from “conventional” immunoglobulins (e.g. monoclonal antibodies) or their fragments (such as Fab, Fab’, F(ab’)2, scFv, di-scFv), wherein two immunoglobulin domains, in particular two variable domains, interact to form an antigen binding site. Typically, in conventional immunoglobulins, a heavy chain variable domain (VH) and a light chain variable domain (VL) interact to form an antigen binding site. In this case, the complementarity determining regions (CDRs) of both VH and VL will contribute to the antigen binding site, i.e. a total of 6 CDRs will be involved in antigen binding site formation.

[0141] In contrast, immunoglobulin single variable domains are capable of specifically binding to an epitope of the antigen without pairing with an additional immunoglobulin variable domain. The binding site of an immunoglobulin single variable domain is formed by a single VH, a single VHH or single VL domain. Hence, the antigen binding site of an immunoglobulin single variable domain is formed by no more than three CDRs.

[0142] As such, the single variable domain may be a light chain variable domain sequence (e.g., a Vr-sequence) or a suitable fragment thereof; or a heavy chain variable domain sequence (e.g., a Vn-sequence or VHH sequence) or a suitable fragment thereof; as long as it is capable of forming a single antigen binding unit (i.e., a functional antigen binding unit that essentially consists of the single variable domain, such that the single antigen binding domain does not need to interact with another variable domain to form a functional antigen binding unit).

[0143] An immunoglobulin single variable domain (ISV) can for example be a heavy chain ISV, such as a VH, VHH, including a camelized VH or humanized VHH. In one embodiment, it is a VHH, including a camelized VH or humanized VHH. Heavy chain ISVs can be derived from a conventional four-chain antibody or from a heavy chain antibody.

[0144] For example, the immunoglobulin single variable domain may be a (single) domain antibody (or an amino acid sequence that is suitable for use as a single domain antibody), a "dAb" or dAb (or an amino acid sequence that is suitable for use as a dAb) or a Nanobody® ISV(as defined herein and including but not limited to a VHH); other single variable domains, or any suitable fragment of any one thereof.

[0145] In particular, the immunoglobulin single variable domain may be a Nanobody® ISV (such as a VHH, including a humanized VHH or camelized VH) or a suitable fragment thereof. [Note: Nanobody® is a registered trademark of Ablynx N. V]

[0146] “VHH domains”, also known as VHHS, VHH antibody fragments and VHH immunoglobulins, have originally been described as the antigen binding immunoglobulin variable domain of “heavy chain antibodies” (i.e., of “antibodies devoid of light chains”;Hamers-Casterman et al. 1993 (Nature 363: 446-448). The term “VHH domain” has been chosen in order to distinguish these variable domains from the heavy chain variable domains that are present in conventional 4-chain antibodies (which are referred to herein as “VH domains”) and from the light chain variable domains that are present in conventional 4-chain antibodies (which are referred to herein as “VL domains”). For a further description of VHH’S, reference is made to the review article by Muyldermans 2001 (Reviews in Molecular Biotechnology 74: 277-302).

[0147] For the term “dAb’s” and “domain antibody”, reference is for example made to Ward et al. 1989 (Nature 341: 544), to Holt et al. 2003 (Trends Biotechnol. 21: 484); as well as to for example WO 2004 / 068820, WO 2006 / 030220, WO 2006 / 003388 and other published patent applications of Domantis Ltd. It should also be noted that, although less preferred in the context of the present invention because they are not of mammalian origin, single variable domains can be derived from certain species of shark (for example, the so-called “IgNAR domains”, see for example WO 2005 / 18629).

[0148] Immunoglobulin sequences of different origin, comprising mouse, rat, rabbit, donkey, human and camelid immunoglobulin sequences can be used herein. Also, fully human, humanized or chimeric sequences can be used in the method described herein. For example, camelid immunoglobulin sequences and humanized camelid immunoglobulin sequences, or camelized domain antibodies, e.g. camelized dAb as described by Ward et al. 1989 (Nature 341: 544), WO 1994 / 04678, and Davis and Riechmann (1994, Febs Lett., 339:285-290; and 1996, Prot. Eng., 9:531-537) can be used herein. Moreover, the ISVs are fused forming a multivalent and / or multispecific construct (for multivalent and multispecific polypeptides containing one ormore VHH domains and their preparation, reference is also made to Conrath et al. 2001 (J. Biol. Chem., Vol. 276, 10. 7346-7350) as well as to for example WO 1996 / 34103 and WO 1999 / 23221).

[0149] A “humanized VHH” comprises an amino acid sequence that corresponds to the amino acid sequence of a naturally occurring VHH domain, but that has been “humanized”, i.e. by replacing one or more amino acid residues in the amino acid sequence of said naturally occurring VHH sequence (and in particular in the framework sequences) by one or more of the amino acid residues that occur at the corresponding position(s) in a VH domain from a conventional 4-chain antibody from a human being (e.g. indicated above). This can be performed in a manner known per se, which will be clear to the skilled person, for example on the basis of the prior art (e.g. WO 2008 / 020079). Again, it should be noted that such humanized VHHS can be obtained in any suitable manner known per se and thus are not strictly limited to polypeptides that have been obtained using a polypeptide that comprises a naturally occurring VHH domain as a starting material.

[0150] A “camelized VH” comprises an amino acid sequence that corresponds to the amino acid sequence of a naturally occurring VH domain, but that has been “camelized”, i.e. by replacing one or more amino acid residues in the amino acid sequence of a naturally occurring VH domain from a conventional 4-chain antibody by one or more of the amino acid residues that occur at the corresponding position(s) in a VHH domain of a (camelid) heavy chain antibody. This can be performed in a manner known per se, which will be clear to the skilled person, for example on the basis of the description in the prior art (e.g. Davies and Riechman 1994, FEBS 339: 285; 1995, Biotechnol. 13: 475; 1996, Prot. Eng. 9: 531; and Riechman 1999, J. Immunol. Methods 231: 25). Such “camelizing” substitutions are inserted at amino acid positions that form and / or are present at the VH-VL interface, and / or at the so-called Camelidae hallmark residues, as defined herein (see for example WO 1994 / 04678 and Davies and Riechmann (1994 and 1996, supra). In one embodiment, the VH sequence that is used as a starting material or starting point for generating or designing the camelized VH is a VH sequence from a mammal, such as the VH sequence of a human being, such as a VH3 sequence. However, it should be noted that such camelized VH can be obtained in any suitable manner known per se and thus are not strictlylimited to polypeptides that have been obtained using a polypeptide that comprises a naturally occurring VH domain as a starting material.

[0151] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0152] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0153] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. Aprogram can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0154] In this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0155] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0156] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, aGlobal Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0157] Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

[0158] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0159] Data processing apparatus for implementing structure prediction neural networks can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.

[0160] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework, or a Jax framework, or a Pytorch framework.

[0161] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g.,a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0162] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0163] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment.Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0164] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing maybe advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0165] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0166] What is claimed is:

Claims

CLAIMS1. A method performed by one or more computers, the method comprising:training a structure prediction neural network to perform a molecular structure prediction task that comprises processing a network input that characterizes an input molecule to generate a network output that defines a predicted three-dimensional (3D) structure of the input molecule;wherein the training comprises, at each of one or more labeling iterations in a sequence of one or more labeling iterations:selecting a current batch of training molecules for training the structure prediction neural network at the labeling iteration, wherein selecting the current batch of training molecules comprises:identifying a set of candidate batches of training molecules; generating, for each candidate batch of training molecules, a respective batch score for the candidate batch of training molecules that characterizes a predicted utility of the candidate batch of training molecules in improving a performance of the structure prediction neural network; andselecting the current batch of training molecules from the set of candidate batches of training molecules based on the batch scores;obtaining a respective target structure for each training molecule in the current batch of training molecules; andtraining the structure prediction neural network on at least the current batch of training molecules using the target structures for the current batch of training molecules; and outputting the trained structure prediction neural network.

2. The method of claim 1, wherein for each candidate batch of training molecules, generating the batch score for the candidate batch of training molecules comprises:determining, for each pair of training molecules in the candidate batch of training molecules, a respective joint entropy score for the pair of training molecules, comprising:determining a plurality of predicted 3D structures of a first training molecule in the pair of training molecules using the structure prediction neural network;determining a plurality of predicted 3D structures of the second training molecule in the pair of training molecules using the structure prediction neural network; and determining the joint entropy score for the pair of training molecules based on the plurality of predicted 3D structures of the first training molecule and the plurality of predicted 3D structures of the second training molecule; anddetermining the batch score for the candidate batch of training molecules based on the joint entropy scores for the pairs of training molecules included in the candidate batch of training molecules.

3. The method of claim 2, wherein the plurality of predicted 3D structures of the first training molecule and the plurality of predicted 3D structures of the second training molecule are generated by an ensemble of modified versions of the structure prediction neural network.

4. The method of claim 3, wherein each modified version of the structure prediction neural network is determined by dropping a respective subset of neurons from the structure prediction neural network.

5. The method of any one of claims 2-4, wherein determining the joint entropy score for the pair of training molecules based on the plurality of predicted 3D structures of the first training molecule and the plurality of predicted 3D structures of the second training molecule comprises:processing the plurality of predicted 3D structures to determine, for each of a plurality of pairs of structural units, a covariance between: (i) predicted 3D spatial positions of a first structural unit in the pair of structural units across the plurality of predicted 3D structures, and (ii) predicted 3D spatial positions of a second structural unit in the pair of structural units across the plurality of predicted 3D structures;wherein each of the plurality of pairs of structural units comprises: (i) a first structural unit from the first training molecule or the second training molecule, and (ii) a second structural unit from the first training molecule or the second training molecule; anddetermining the joint entropy score for the pair of training molecules based on the covariances between the predicted 3D spatial positions of the plurality of pairs of structural units.

6. The method of claim 5, wherein each structural unit is an atom, or an amino acid, or a nucleotide base.

7. The method of claim 5 or claim 6, wherein determining the joint entropy score for the pair of training molecules based on the covariances between the predicted 3D spatial positions of the plurality of pairs of structural units comprises:determining the joint entropy score for the pair of training molecules based on a determinant of an array that includes the covariances between the predicted 3D spatial positions of the plurality of pairs of structural units.

8. The method of any one of claims 2-7, wherein determining the batch score for the candidate batch based on the joint entropy scores for the pairs of training molecules included in the candidate batch comprises:computing the batch score based on a combination of a set of joint entropy scores of pairs of different training molecules included in the candidate batch.

9. The method of claim 8, wherein determining the batch score for the candidate batch based on the joint entropy scores for the pairs of training molecules included in the candidate batch comprises:computing the batch score based on a summation of the set of joint entropy scores of pairs of different training molecules included in the candidate batch.

10. The method of any preceding claim, wherein selecting the current batch of training molecules from the set of candidate batches of training molecules based on the batch scores comprises:selecting, as the current batch of training molecules, a candidate batch that is associated with the highest batch score from among the set of candidate batches of training molecules.

11. The method of any of claims 1-9, wherein selecting the current batch of training molecules from the set of candidate batches of training molecules based on the batch scores comprises:selecting a subset of candidate batches having top batch scores from among the set of candidate batches of training molecules; andperform an optimization of the candidate batches in the subset, the optimization comprises, for each of one or more candidate batches in the subset, replacing one or more training molecules in the candidate batch with different training molecules to maximize the batch score of the candidate batch; andselecting the current batch of training molecules from the optimized subset of candidate batches.

12. The method of any preceding claim, wherein identifying a set of candidate batches of training molecules comprises:identifying a pool of training molecules;determining a respective uncertainty score for each training molecule in the pool of training molecules, wherein the uncertainty score for a training molecule characterizes an uncertainty of the prediction neural network in generating a predicted 3D structure of the training molecule;determining a probability distribution over the pool of training molecules using the uncertainty scores for the training molecules; andidentifying each candidate batch in the set of candidate batches of training molecules by sampling each training molecule included in the candidate batch from the pool of training molecules in accordance with the probability distribution over the pool of training molecules.

13. The method of claim 12, wherein determining the uncertainty score for a training molecule comprises:determining the uncertainty score based on a self-entropy score of the training molecule.

14. The method of any preceding claim, wherein the structure prediction neural network comprises a sequence of multiple subnetworks, wherein a first subnetwork in the sequence processes an initial structure representation of the input molecule to generate an updated structure representation of the input molecule, and each subsequent subnetwork in the sequence processes the structure representation generated by the preceding subnetwork to generate a further updated structure representation of the input molecule.

15. The method of claim 14, wherein each subnetwork in the sequence comprises a respective equivariant subnetwork configured to generate an output structure representation that is equivariant to translations, rotations, and reflections of an input structure representation.

16. The method of claim 14 or claim 15, wherein the sequence of subnetworks comprises a sub-sequence of subnetworks that is consecutively repeated for two or more repetitions, and the repeated subnetworks share the same weights across the repetitions.

17. The method of any of claims 14-16, wherein prior to training the structure prediction neural network over each of the one or more labeling iterations, the structure prediction neural network has been pre-trained on one or more training examples, and wherein during each of the one or more labeling iterations, parameters of one or more subnetworks in the sequence of multiple subnetworks are held fixed at their pre-trained values.

18. The method of any preceding claim, wherein outputting the trained structure prediction neural network comprises:storing the trained structure prediction neural network in a memory device.

19. The method of any preceding claim, wherein the input molecule is a protein molecule, and the network input specifies an amino acid sequence of the protein molecule.

20. The method of claim 19, wherein the input molecule is an antibody.

21. The method of claim 20, wherein the input molecule is a heavy chain single variable domain (VHH) antibody molecule.

22. The method of claim 21, wherein prior to training the structure prediction neural network at each of the one or more labeling iterations, the structure prediction neural network has been pre-trained on general antibody molecules, and wherein during each of the one or more labeling iterations, the structure prediction neural network is trained on VHH antibody molecules.

23. The method of any of claims 19-22, wherein the input molecule is a monomer of a capsid protein.

24. The method of any of claims 1-18, wherein the input molecule is a nucleic acid molecule, and the network input specifies a nucleotide sequence of the nucleic acid molecule.

25. The method of claim 24, wherein the input molecule is a messenger ribonucleic acid (mRNA) molecule.

26. The method of any preceding claim, wherein outputting the trained structure prediction neural network comprises:obtaining a plurality of network inputs, wherein each network input characterizes a respective input molecule; andprocessing each of the plurality of network inputs using the trained structure prediction neural network to generate a respective predicted 3D structure of the respective input molecule.

27. The method of 26, further comprising:selecting one or more molecules using the structure prediction neural network; and physically synthesizing the one or more molecules.

28. A system comprising:one or more computers; andone or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of the respective method of any one of claims 1-27.

29. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations of the respective method of any one of claims 1-27.

Images