Predicting the properties of a single variable domain using machine learning models

Abstract

A method, computer system, and apparatus for predicting ISVD molecular properties, comprising a computer program encoded on a computer storage medium. The system generates an input vector by obtaining a token sequence representing an ISVD molecule and numerically encoding the amino acid sequence; generates an embedded feature vector by processing the input vector using an embedded machine learning model having a first set of model parameters; and generates an output predicting one or more properties of the ISVD molecule by processing the embedded feature vector using a property prediction machine learning model.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims priority to European Patent Application No. 23305893, filed on June 5, 2023, the disclosure of which is incorporated herein by reference in its entirety.

[0002] This specification generally relates to predicting a subject's motor function based on biomarkers.

Background Art

[0003] Nanobody (registered trademark) molecules, also known as heavy - chain single - variable domain (VHH) antibodies, are important tools in medicine and biotechnology. Similar to all mammals, camelids (e.g., llamas) can produce conventional antibodies consisting of two heavy chains and two light chains linked to each other by disulfide bonds in a Y - shape (e.g., IgG1). However, they also produce two unique subclasses of IgG, IgG2 and IgG3, which are also known as heavy - chain IgGs. These antibodies are made from only two heavy chains, which lack the CH1 region but still have an antigen - binding domain called VHH (or Nanobody (registered trademark) molecule) at their N - terminus.

[0004] Conventional Ig requires the association of variable domains from both the heavy and light chains to enhance the diversity of antigen-antibody interactions. Isolated heavy and light chains still exhibit this ability, but they show significantly lower affinity compared to paired heavy and light chains. A unique feature of heavy chain IgG is the ability of their monomeric antigen-binding region to bind to antigens with comparable specificity, affinity, and especially diversity to conventional antibodies, without requiring pairing with other regions. This feature is primarily due to two major mutations in the amino acid sequences of the variable regions of the two heavy chains, which induce deep conformational changes compared to conventional Ig. The major substitutions in the variable region prevent the light chain from binding to the heavy chain, but also prevent the unbound heavy chain from being repurposed by immunoglobulin-binding proteins.

[0005] The single variable domain of these antibodies (called VHH, sdAb, or nanobody® molecule) is the smallest antigen-binding domain produced by the adaptive immune system. The third complementarity-determining region (CDR3) of the variable region of these antibodies has been found to be twice as long as that of conventional antibodies. This results in an increased interaction surface with the antigen and greater diversity of antigen-antibody interactions, thereby compensating for light chain deficiencies. Having a longer complementarity-determining region 3 (CDR3) allows VHHs to extend into gaps on proteins that conventional antibodies cannot access, including functionally interesting sites such as enzyme active sites or receptor-binding canyons on the viral surface. Furthermore, the additional cysteine ​​residues make the structure more stable and thus increase the strength of the interaction.

[0006] VHH offers many advantages over conventional antibodies with variable domains (VH and VL), including higher stability, solubility, expression yield, and refolding ability, as well as superior in vivo tissue penetration. Furthermore, in contrast to the VH domain of conventional antibody VHH, VHH does not exhibit an intrinsic tendency to bind to light chains. This facilitates the induction of heavy chain antibodies in the presence of functional light chain loci. In addition, because VHH does not bind to the VL domain, reformatting VHH into bispecific antibody constructs is far easier than constructs containing a single domain based on conventional VH-VL pairs or VH domains.

[0007] The term "immunoglobulin single variable domain" (ISVD), used synonymously with "single variable domain," defines an immunoglobulin molecule in which the antigen-binding site resides on a single immunoglobulin domain, thereby forming the molecule. This term distinguishes an immunoglobulin single variable domain from "conventional" immunoglobulins (e.g., monoclonal antibodies) or their fragments (e.g., Fab, Fab', F(ab')2, scFv, di-scFv) which have two immunoglobulin domains, particularly two variable domains that interact to form an antigen-binding site.

[0008] Typically, in conventional immunoglobulins, the heavy chain variable domain (VH) and the light chain variable domain (VL) interact to form an antigen-binding site. In this case, the complementarity-determining regions (CDRs) of both the VH and VL contribute to the antigen-binding site; that is, a total of six CDRs are involved in the formation of the antigen-binding site.

[0009] In contrast, a single immunoglobulin variable domain can specifically bind to an antigen epitope without pairing with additional immunoglobulin variable domains. The binding site of a single immunoglobulin variable domain is formed by a single VH, a single VHH, or a single VL domain. Therefore, the antigen-binding site of a single immunoglobulin variable domain is formed by three or fewer CDRs.

[0010] Therefore, a single variable domain can be a light chain variable domain sequence (e.g., a VL- sequence) or a suitable fragment thereof, or a heavy chain variable domain sequence (e.g., a VH sequence or a VHH sequence) or a suitable fragment thereof, insofar as it can form a single antigen-binding unit (i.e., a functional antigen-binding unit that is essentially composed of a single variable domain, such that a single antigen-binding domain does not need to interact with another variable domain to form a functional antigen-binding unit).

[0011] A machine learning model is a computational model that learns patterns and relationships in data and uses that knowledge to make predictions or decisions about new data. A neural network is a machine learning model that uses one or more nonlinear units of layers to predict the output of an incoming input. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or output layer. Each layer of the network generates an output from an incoming input according to the current values ​​of its respective set of parameters. [Overview of the project] [Means for solving the problem]

[0012] This disclosure describes a method, a computer system, and an apparatus including a computer program encoded on a computer storage medium for predicting the properties of nanobody (registered trademark) molecules.

[0013] In this specification, the terms "VHH," "nanobody® molecule," and "nanobody" are used synonymously.

[0014] In one embodiment, the Disclosure provides a prediction method for predicting one or more properties of a nanobody® molecule. The method can be performed by a system comprising one or more computers. The system acquires data representing the amino acid sequence of a nanobody® molecule, generates an input vector by numerically encoding the amino acid sequence, and generates an embedded feature vector by processing the input vector using an embedded machine learning model having a first set of model parameters. The first set of model parameters is updated using self-supervised learning of the first machine learning model, which includes the embedded machine learning model and is configured to perform a sequence reconstruction task. The system uses a feature prediction machine learning model to further process the embedded feature vector to generate an output that predicts one or more properties of a nanobody® molecule. The feature prediction machine learning model has a second set of model parameters, which is updated using supervised learning based on multiple training examples of the second machine learning model, which includes the feature prediction machine learning model. Each of the respective training examples includes (i) a respective training input specifying a representation of each nanobody® molecule, and (ii) a respective label specifying one or more properties of each nanobody® molecule.

[0015] In some implementations of the prediction method, the properties of the nanobody (registered trademark) molecule may include binding affinity to one or more target molecules, binding specificity to one or more target molecules, production yield, stability under one or more environmental conditions, cross-reactivity to one or more target molecules, melting temperature, and / or immunogenicity.

[0016] In some implementations of the prediction method, to generate an input vector, the system can generate a token vector by mapping each amino acid in an amino acid sequence to its respective numerical value and then combining (e.g., concatenating) these values.

[0017] In some implementations of the prediction method, the first machine learning model can include a Large-Scale Language Model (LLM). For example, the first machine learning model can include a Variational Autoencoder (VAE), an Autoregressive Transformer, and / or a Bidirectional Transformer.

[0018] In several implementations of prediction methods, characteristic prediction machine learning models can include neural networks, K-nearest neighbor models, support vector machines, decision tree models, random forest models, and / or ridge regression models.

[0019] In some implementations of the prediction method, the first model parameter set can be fixed after the self-supervised learning process and during the supervised learning process. In some other implementations of the prediction method, the first model parameter set can be further updated during the supervised learning process, and the embedded machine learning model and the characteristic prediction machine learning model are co-trained end-to-end.

[0020] In some implementations of the prediction method, the system further maintains a first set of candidate machine learning models trained using self-supervised learning and selects an embedding machine learning model for a specific prediction task from this first set of candidate models. To select an embedding machine learning model from the first set of candidate models, the system can evaluate the performance of each model in the first set of candidate models for a specific prediction task on a labeled dataset and select an embedding machine learning model from the first set of candidate models based on performance. The first set of candidate models may include two or more of the following: variational autoencoders (VAEs), autoregressive transformers, or bidirectional transformers.

[0021] In several implementations of the prediction method, based on the predicted properties of a nanobody® molecule, the system can select a nanobody® molecule from a set of candidate nanobody® molecules to perform a downstream task. The system can predict the properties of each candidate nanobody® molecule using the process described above and select that nanobody® molecule from the set of candidate nanobody® molecules based on the predicted properties. Downstream tasks may include binding to a target protein, agonist or antagonist function, achieving thermal stability, and / or achieving oral stability.

[0022] In some implementations of the prediction method, the system further generates prediction results by using an embedded machine learning model to generate an embedded feature vector for each of the multiple nanobody® molecules and then performing cluster analysis on the embedded feature vectors of the multiple nanobody® molecules.

[0023] In some implementations of the prediction method, the system further maintains a dataset containing data specifying (i) the amino acid sequence of each known nanobody® molecule and (ii) the respective embedding feature vectors generated for each known nanobody® molecule, for each of several known nanobody® molecules. The system receives an input specifying the amino acid sequence of a particular nanobody® molecule, uses an embedding machine learning model to generate a specific embedding feature vector for that particular nanobody® molecule, searches the dataset to identify a set of one or more embedding feature vectors within a predetermined distance from a specific embedding feature vector in the feature space, identifies the amino acid sequences corresponding to the identified set of embedding feature vectors in the dataset, and outputs data specifying the identified amino acid sequences.

[0024] In some implementations of prediction methods, the characteristic prediction machine learning model is configured to perform regression and / or classification tasks.

[0025] In another aspect, the present disclosure provides a design method for determining an optimal amino acid sequence of a Nanobody® molecule for performing a particular task. This design method can be executed by a system including one or more computers. The system maintains data representing a set of candidate sequences for the Nanobody® molecule and uses a reinforcement learning (RL) model to process one or more of the candidate sequences to generate one or more new sequences. The RL model is trained using a reward signal that includes Nanobody® molecule properties predicted using the prediction method described above. The system can select an optimal sequence from the new sequences.

[0026] In another aspect, the present disclosure provides a reinforcement learning (RL) method for training an RL model for determining an optimal amino acid sequence of a Nanobody® molecule for performing a particular task. To perform the training, a computer-implemented system can use the RL model to process an input sequence representing a Nanobody® molecule to generate one or more sets of actions, which modify the input sequence, identify a new sequence based on the input sequence and its set of actions, and calculate one or more reward values indicating how well a particular task is performed by the Nanobody® molecule represented by the new sequence. The reward values are calculated using one or more Nanobody® molecule properties predicted using the prediction method described. The computer-implemented system can adjust one or more parameters of the RL model based at least on the reward values.

[0027] In another aspect, the present disclosure provides a training method for training a prediction model for predicting characteristics of Nanobody® molecules. The training method can be executed by a system including one or more computers. The prediction model includes (i) an embedding machine learning model configured to generate an embedding feature vector for model input representing the amino acid sequence of a Nanobody® molecule, and (ii) a characteristic prediction machine learning model configured to process the embedding feature vector to generate an output specifying one or more characteristics of the Nanobody® molecule. The system obtains a first dataset including a set of sequence representations of Nanobody® molecules, and uses the first dataset to perform self-supervised learning of a first machine learning model including the embedding machine learning model for a reconstruction task, and obtains a second dataset including a plurality of training examples. Each of the training examples includes (i) each training input specifying the representation of each Nanobody® molecule, and (ii) each label specifying one or more characteristics of each Nanobody® molecule. The system performs supervised learning of a second machine learning model including characteristic prediction machine learning on the second dataset.

[0028] In some implementations of the training method, the characteristics of the Nanobody® molecule include binding affinity, specificity, yield, cross-reactivity, melting temperature, stability, and / or immunogenicity for a target molecule.

[0029] In some implementations of the training method, the first machine learning model includes a large language model (LLM).

[0030] In some implementations of the training method, the first machine learning model includes a variational autoencoder (VAE), an autoregressive transformer, and / or a bidirectional transformer.

[0031] In some implementations of the training method, the characteristic prediction machine learning model includes one or more of a neural network, a K-nearest neighbor model, a support vector machine, a decision tree model, a random forest model, and / or a ridge regression model.

[0032] In some implementations of the training method, the first model parameter set is fixed both after the self-supervised learning process and during the supervised learning process.

[0033] In several other implementations of the training method, the primary model parameter set is further updated during the supervised learning process, and the embedded machine learning model and the feature prediction machine learning model are co-trained end-to-end.

[0034] The disclosure also provides a system including one or more computers and one or more storage devices that, when executed by one or more computers, store instructions causing one or more computers to perform the aforementioned prediction method, training method, design method, or RL method.

[0035] The disclosure also provides one or more computer storage media that, when executed by one or more computers, store instructions causing one or more computers to perform the aforementioned prediction method, training method, design method, or RL method.

[0036] In another aspect, the disclosure provides a second prediction method for training a prediction model to predict the properties of heavy chain antibody single variable domain (ISVD) molecules. The prediction method can be performed by a system comprising one or more computers. The system obtains token sequences representing ISVD molecules, generates an input vector by numerically encoding the token sequences representing ISVD molecules, and generates an embedded feature vectoral by processing the input vector using an embedded machine learning model having a first model parameter set. The first model parameter set is updated using self-supervised learning of the first machine learning model, which includes an embedded machine learning model and is configured to perform a sequence reconstruction task. In some cases, the ISVD molecule is a VHH molecule.

[0037] The system uses a feature prediction machine learning model to process embedded feature vectors and generate outputs that predict one or more properties of an ISVD molecule. The feature prediction machine learning model has a second set of model parameters updated using supervised learning based on multiple training examples of a second machine learning model that contains the feature prediction machine learning model. Each of the respective training examples includes (i) a respective training input specifying a representation of each ISVD molecule, and (ii) a respective label specifying one or more properties of each ISVD molecule.

[0038] In some implementations of the second prediction method, the ISVD molecule is a single variable domain of immunoglobulin G that (i) contains two heavy chains and (ii) lacks one of the CH1 domains.

[0039] In some implementations of the second prediction method, to obtain a token sequence representing the ISVD molecule, the system obtains an initial token sequence representing the amino acid sequence of the ISVD molecule and generates a token sequence of a predetermined length by adding padding tokens at one or more positions to the initial token sequence. The predetermined length is longer than the length of the initial token sequence.

[0040] In some implementations of the second prediction method, to obtain a token sequence representing the ISVD molecule, the system obtains an initial token sequence representing the amino acid sequence of the ISVD molecule and generates the token sequence by performing an alignment of the initial token sequence. Alignment involves inserting gap tokens at one or more positions in the initial token sequence. In some cases, the alignment is performed according to annotation information of the initial token sequence. In some cases, the annotation information of the initial token sequence is generated using an annotation scheme selected from the IMGT annotation scheme, Kabat annotation scheme, Chothia annotation scheme, Martin annotation scheme, Wolfguy annotation scheme, or AHo annotation scheme. In a specific example, the annotation information of the initial token sequence is generated using the AHo annotation scheme.

[0041] In some implementations of the second prediction method, one or more properties of the ISVD molecule include one or more of the following: binding affinity to one or more target molecules, binding specificity to one or more target molecules, production yield, stability under one or more environmental conditions, cross-reactivity to one or more target molecules, melting temperature, or immunogenicity.

[0042] In some implementations of the second prediction method, to generate an input vector, the system generates the input vector by mapping each token in the token array to its respective numerical value and concatenating the numerical values.

[0043] In some implementations of the second prediction method, the first machine learning model includes a Large-Scale Language Model (LLM).

[0044] In some implementations of the second prediction method, the first machine learning method includes variational autoencoders (VAEs), autoregressive transformers, or bidirectional transformers.

[0045] In some implementations of the second prediction method, the characteristic prediction machine learning model includes one or more of the following: neural networks, K-nearest neighbor models, support vector machines, decision tree models, random forest models, or ridge regression models.

[0046] In some implementations of the second prediction method, the first model parameter set is fixed both after the self-supervised learning process and during the supervised learning process.

[0047] In some implementations of the second prediction method, the first model parameter set is further updated during the supervised learning process, and the embedded machine learning model and the characteristic prediction machine learning model are co-trained end-to-end.

[0048] In some implementations of the second prediction method, the system further maintains a first set of candidate machine learning models trained using self-supervised learning, and selects an embedded machine learning model for a specific prediction task from the first set of candidate machine learning models.

[0049] In some implementations of the second prediction method, in order to select an embedding machine learning model from the first candidate machine learning model set, the system evaluates the performance of each of the first candidate machine learning model sets for a specific prediction task on a labeled dataset and selects an embedding machine learning model from the first candidate machine learning model set based on its performance.

[0050] In some implementations of the second prediction method, the first candidate machine learning model set includes two or more of the following: variational autoencoders (VAEs), autoregressive transformers, or bidirectional transformers.

[0051] In some implementations of the second prediction method, the system further generates prediction results by using an embedded machine learning model to generate an embedded feature vector for each of the multiple ISVD molecules and then performing cluster analysis on the embedded feature vectors of the multiple ISVD molecules.

[0052] In some implementations of the second prediction method, the system further maintains a dataset containing, for each of several known ISVD molecules, (i) the amino acid sequence of each known ISVD molecule and (ii) data specifying the respective embedding feature vectors generated for each known ISVD molecule. The system receives an input specifying the amino acid sequence of a particular ISVD molecule, uses an embedding machine learning model to generate a specific embedding feature vector for that particular ISVD molecule, searches the dataset to identify a set of one or more embedding feature vectors within a predetermined distance from that specific embedding feature vector in the feature space, identifies the amino acid sequences corresponding to the identified set of embedding feature vectors in the dataset, and outputs data specifying the identified amino acid sequences.

[0053] In some implementations of the second prediction method, the characteristic prediction machine learning model is configured to perform one or more regression or classification tasks.

[0054] In another embodiment, the disclosure provides a second selection method for selecting an ISVD molecule from a set of candidate ISVD molecules to perform a downstream task. This method can be performed by a system including one or more computers. The system predicts the properties of each candidate ISVD molecule using one of the methods described herein and selects the ISVD molecule from the set of candidate ISVD molecules based on the predicted properties.

[0055] In some implementations of the second selection method, the downstream task includes one or more of the following: binding to a target protein, agonist or antagonist function, achieving thermal stability, or achieving oral stability.

[0056] In another aspect, the disclosure provides a second design method for determining the optimal amino acid sequence of an ISVD molecule for performing a particular task. The design method can be performed by a system comprising one or more computers. The system maintains data representing a set of candidate sequences of the ISVD molecule and processes one or more of the candidate sequences using a reinforcement learning (RL) model to generate one or more new sequences. The RL model is trained with a reward signal containing ISVD molecular properties predicted using the method of any one of the preceding claims. The system selects the optimal sequence from the new sequences.

[0057] In another aspect, the disclosure provides a second reinforcement learning (RL) method for training an RL model to identify the optimal amino acid sequence of an ISVD molecule for performing a particular task. The method can be performed by a system comprising one or more computers. The system uses the RL model to process an input sequence representing an ISVD molecule to generate one or more sets of actions that modify the input sequence, to identify a new sequence based on the input sequence and the sets of actions, and to calculate one or more reward values ​​indicating how well the ISVD molecule represented by the new sequence performs a particular task. The reward values ​​are calculated using one or more ISVD molecule properties predicted using the method of any one of the preceding claims. The system adjusts one or more parameters of the RL model based on at least the reward values.

[0058] In another aspect, the disclosure provides a second training method for training a predictive model for predicting the properties of ISVD molecules. The method can be performed by a system comprising one or more computers. The predictive model comprises (i) an embedding machine learning model configured to generate an embedding feature vector for model inputs representing ISVD molecules, and (ii) a property predictive machine learning model configured to process the embedding feature vector to generate an output specifying one or more properties of the ISVD molecules. The system obtains a first dataset comprising a set of sequence representations of ISVD molecules, performs self-supervised training of a first machine learning model comprising the embedding machine learning model on the reconstruction task using the first dataset, obtains a second dataset comprising multiple training examples, each training example comprising (i) a training input specifying a representation of each ISVD molecule, and (ii) a label specifying one or more properties of each ISVD molecule, and performs supervised training of a second machine learning model comprising property predictive machine learning on the second dataset.

[0059] In some implementations of the second training method, one or more properties of the ISVD molecule include one or more of the following: binding affinity to the target molecule, specificity, yield, cross-reactivity, melting temperature, stability, or immunogenicity.

[0060] In some implementations of the second training method, the first machine learning model includes a Large-Scale Language Model (LLM).

[0061] In some implementations of the second training method, the first machine learning model includes a variational autoencoder (VAE), an autoregressive transformer, or a bidirectional transformer.

[0062] In some implementations of the second training method, the feature prediction machine learning model includes one or more of the following: neural networks, K-nearest neighbor models, support vector machines, decision tree models, random forest models, or ridge regression models.

[0063] In some implementations of the second training method, the first model parameter set is fixed both after the self-supervised learning process and during the supervised learning process.

[0064] In some implementations of the second training method, the first model parameter set is further updated during the supervised learning process, and the embedded machine learning model and the feature prediction machine learning model are co-trained end-to-end.

[0065] The disclosure also provides a system including one or more computers and one or more storage devices that, when executed by one or more computers, store instructions causing one or more computers to perform the aforementioned second prediction method, second training method, second design method, or second RL method.

[0066] The disclosure also provides one or more computer storage media that, when executed by one or more computers, store instructions causing one or more computers to perform the aforementioned second prediction method, second training method, second design method, or second RL method.

[0067] The subject matter described herein can be implemented in specific embodiments to achieve one or more advantages.

[0068] The goal of nanobody® molecular engineering is to design, create, and / or select nanobody® molecules with properties optimized for specific applications. One approach to nanobody® molecular engineering involves generating a large number of candidate nanobody® molecules, measuring the suitability of each nanobody® molecule for a specific application, and selecting the most suitable nanobody® molecule. This process can be carried out iteratively by diversifying the selected nanobody® molecules to generate candidate nanobody® molecules for subsequent iterations.

[0069] Existing nanobody (registered trademark) molecular engineering processes are associated with many challenges, including multi-objective optimization (i.e., the need to optimize multiple properties for a specific application) and large search spaces. For example, in the case of 14 amino acid CDRH3, the sequence space is 1.6 × 10⁻⁶. 18 It is of such a large size that experimentally benchmarking such a search space becomes impossible.

[0070] The described technology uses deep learning to computationally predict the properties of nanobody® molecules based on their amino acid sequences. Specifically, the described technology uses self-supervised learning to pre-train a language model to generate effective embeddings of nanobody® molecular sequences, and then performs transfer learning using labeled data for downstream tasks. The self-supervised pre-training process enables the generation of high-performance embeddings when the labeled data is limited to downstream tasks.

[0071] Based on predicted nanobody® molecular properties, the described system or other systems can identify the optimal nanobody® molecular sequence for performing a particular application. For example, the system can generate an output indicating whether a particular nanobody® molecule is suitable for a particular application, an output specifying the optimal nanobody® molecular sequence for a particular application, or an output indicating where mutations can be made in the sequence. The system can send the output to a manufacturing apparatus that can operate to execute instructions to produce the nanobody® molecule. Overall, by training a high-performance predictive model based on limited experimental data and using the trained model to predict nanobody® molecular properties, the described techniques can significantly improve the effectiveness and efficiency of nanobody® molecular engineering.

[0072] Details of one or more embodiments of the subject matter of this specification will be described in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawing]

[0073] [Figure 1] An example of a nanobody (registered trademark) molecular property prediction system is shown. [Figure 2] This flowchart illustrates an example process for predicting the properties of nanobody (registered trademark) molecules. [Figure 3] This flowchart illustrates an example process for training a predictive model to predict the properties of nanobody (registered trademark) molecules. [Figure 4] This is a block diagram of an exemplary computer system. [Modes for carrying out the invention]

[0074] Similar reference numbers and designations in different drawings refer to the same element.

[0075] Figure 1 shows an example of a nanobody® molecular property prediction system 100. System 100 is an example of a system that can be implemented as a computer program on one or more computers in one or more locations where the systems, components, and technologies described below can be implemented.

[0076] The Nanobody® molecular property prediction system 100 uses a machine learning model to predict one or more properties 140 of a Nanobody® molecule based on input data 110 specifying the amino acid sequence of the Nanobody® molecule. In some embodiments, the entire Nanobody® molecular sequence is used. In some embodiments, three CDR sequences (CDR1, CDR2, and CDR3) are used. In some embodiments, only the CDR3 sequence is used.

[0077] Generally, a machine learning model includes an embedded machine learning model 120 having a first set of model parameters 122, and a feature prediction machine learning model 130 having a second set of model parameters 132.

[0078] System 100 includes a sequence tokenizer 125 configured to generate an input vector as input to an embedded machine learning model 120. The sequence tokenizer 125 generates the input vector by numerically encoding a token sequence representing nanobody® molecules. For example, the system can generate the input vector by concatenating numerical values ​​assigned to different tokens within the token sequence.

[0079] In some implementations, the token sequence for generating the input vector corresponds to the amino acid sequence of the nanobody® molecule. That is, the input vector is generated by numerically encoding the raw amino acid sequence of the nanobody® molecule. In some cases, to ensure a fixed length for the input vector, the amino acid sequence of the nanobody® molecule may be extended to a specified length, e.g., 200 tokens, by adding padding tokens at the end positions.

[0080] In some implementations, the amino acid sequences of nanobody® molecules are aligned to generate token sequences that represent the nanobody® molecules. For example, the sequence tokenizer 125 can align each amino acid sequence by inserting gap tokens at specific positions in each amino acid sequence to match the positions of identical or similar regions, such as framework regions and CDRs, across multiple sequences. In some cases, alignment can be performed based on structural and / or functional annotations of the amino acid sequences of nanobody® molecules. Any suitable annotation scheme can be used to generate the annotations. Examples of annotation schemes include IMGT, Kabat, Chothia, Martin, Wolfguy, and AHo annotation schemes. In one particular example, alignment can be performed based on annotations obtained using the AHo annotation scheme, which provides hydrophobic information for amino acids. The length of the aligned token sequence may depend on the annotation scheme. In the exemplary example, the aligned token sequence has a length of 151. The array tokenizer 125 can generate an input vector by numerically encoding an ordered token array.

[0081] Several advantages can be obtained by generating input vectors using aligned token sequences of nanobody® molecules. Mapping amino acids to their functional and / or structural locations within nanobody® molecules makes it easier for machine learning models to establish clearer relationships between sequence elements. Pre-aligned sequences can help models focus on meaningful patterns, such as aligned framework regions and CDRs, rather than being distracted by uncertainty in the location of regions. In addition, shorter token sequences (e.g., 151 vs. 200) may improve the training speed of the model.

[0082] The embedded machine learning model 120 is configured to process an input vector to generate an embedded feature vector 125. The embedded feature vector 125 is a numerical representation of the input data that captures essential information required for one or more tasks. In particular, the embedded feature vector 125 can be a high-dimensional vector of real numbers that captures features of the model input that specify nanobody® molecules. These can be amino acid-level embeddings or protein-level embeddings (global embeddings).

[0083] In some implementations, the embedded machine learning model 120 is a neural network. The embedded neural network can be used in any suitable architecture. In particular, the embedded neural network 120 can include at least a portion (e.g., the embedding portion) of a state-of-the-art language model.

[0084] For example, in some implementations, the embedded neural network 120 can include a variational autoencoder (VAE) encoder network. An example of a VAE implementation is described in "Auto-encoding Variational Bayes," Kingma et al., arXiv:1312.6114, 2013.

[0085] In some implementations, the embedded neural network 120 can include an embedding layer of an autoregressive transformer, such as a generative pre-trained transformer (GPT). An example of a GPT implementation is described in "Language Models Are Few-Shot Learners," Brown et al., Advances in Neural Information Processing Systems 33:1877-1901, 2020.

[0086] In some implementations, the embedded neural network 120 can include a bidirectional transformer, such as a bidirectional encoder representation from a BERT model. An example of a BERT implementation is described in "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," Devlin et al., arXiv:1810.04805, 2018.

[0087] Using language models to generate embeddings of nanobody® molecular sequences offers several advantages for predicting the function and properties of nanobody® molecules in downstream tasks. Due to evolutionary pressures, nanobody® molecular sequences are not random. For example, nanobody® molecules can contain evolutionarily conserved regions, and parts of the nanobody® molecular sequence can be reused. Furthermore, correlations and interactions may exist between positional pairs within the nanobody® molecule. Language models can learn complex patterns in nanobody® molecular sequences and can be used to identify previously unknown patterns and correlations within them. As will be described in more detail below, language models for generating nanobody® molecular embeddings are specifically trained using nanobody® molecular sequence data. This is important to obtain a high-performance model for predicting nanobody® molecular properties, i.e., a model with high predictive accuracy and that avoids model bias and overfitting.

[0088] System 100 further includes a property prediction machine learning model 130 configured to process an embedded feature vector 125 to generate an output 140 predicting one or more properties of a nanobody® molecule. The predicted properties may be nanobody® molecule properties related to one or more applications and may include one or more of the following: binding affinity to one or more target molecules, binding specificity to one or more target molecules, production yield, stability under one or more environmental conditions, cross-reactivity to one or more target molecules, melting temperature, and / or immunogenicity of the nanobody® molecule. In an exemplary example, the intended application is to screen for suitability of nanobody® molecules for use as therapeutic agents via oral delivery, and the properties of interest would include oral stability of the nanobody® molecule.

[0089] The feature prediction machine learning model 130 can employ any suitable machine learning technique, which may include one or more of the following: neural networks, K-nearest neighbor models, support vector machines, decision tree models, random forest models, or ridge regression models. The feature prediction machine learning model 130 can be used to perform regression tasks, classification tasks, or both.

[0090] In some implementations, System 100 or other systems include a self-supervised learning engine 150 configured to update model parameters 122 of an embedded machine learning model using self-supervised learning based on a set of nanobody® molecular sequence representations 155. The goal of self-supervised learning is to learn meaningful embeddings of nanobody® molecular sequences without requiring the use of labeled data. Instead, the self-supervised learning engine 150 learns embeddings using unlabeled nanobody® molecular sequence data, i.e., data that specifies or represents each of the nanobody® molecular sequence sets that do not contain nanobody® molecular property labels. Thus, the self-supervised learning engine 150 can learn embeddings using a large number of known nanobody® molecular sequences, without the need to acquire a large amount of experimental benchmark data about the properties of known nanobody® molecules.

[0091] To effectively learn embeddings from unlabeled data, the self-supervised learning engine 150 can be configured to train a first machine learning model to perform a reconstruction task, namely, the task of generating embeddings for input nanobody® molecular arrangement representations and reconstructing the input nanobody® molecular arrangement representations from the embeddings. The first machine learning model includes an embedding machine learning model 120 as a subnetwork for generating embeddings.

[0092] The self-supervised learning engine 150 can update the parameters of the first machine learning model (including the model parameters 122 of the embedded machine learning model 120) by minimizing the reconstruction error between the input nanobody® molecular arrangement and the reconstructed nanobody® molecular arrangement. The self-supervised learning engine 150 can use any suitable backpropagation-based machine learning technique to update the model parameters, for example, the Adam or AdaGrad optimizer can be used.

[0093] In some implementations, after self-supervised learning, the embedded machine learning model 120 can be used for different prediction tasks, namely, tasks that predict different combinations of nanobody® molecular properties. That is, the embedded machine learning model 120 may not need to be retrained to predict different nanobody® molecular properties.

[0094] System 100 or other systems may further include a supervised learning engine 160 configured to update model parameters 132 of a characteristic prediction model 130 based on a labeled dataset 165. The labeled dataset 165 includes a plurality of labeled training examples. Each training example includes (i) a training input specifying a representation of each nanobody® molecule, and (ii) a label specifying one or more characteristics of each nanobody® molecule. The nanobody® molecule labels may be based on experimental measurements of the characteristics of the corresponding nanobody® molecules.

[0095] The supervised learning engine 160 is configured to perform supervised learning of a second machine learning model, which includes the feature prediction machine learning model 130, on a labeled dataset 165. That is, the supervised learning engine 160 is configured to update the parameters of the second machine learning model (including the model parameters 132 of the feature prediction machine learning model 130) based on the labeled dataset 165.

[0096] In some implementations, the second machine learning model further includes an embedded machine learning model 120. That is, the model parameters 122 of the embedded machine learning model 120 are further fine-tuned end-to-end with the feature prediction machine learning model 130 through supervised learning based on a labeled dataset 165.

[0097] In some other implementations, the model parameters 122 of the embedded machine learning model 120 are fixed during supervised learning while the model parameters 132 of the feature prediction machine learning model 130 are being updated.

[0098] The supervised learning engine 160 can update the parameters of a second machine learning model (including the model parameters 132 of the characteristic prediction machine learning model 130, and optionally the model parameters 122 of the embedding machine learning model 120) by minimizing the prediction error between the predicted nanobody (registered trademark) molecular properties and the properties specified by the label. The supervised learning engine 160 can update the model parameters using any suitable backpropagation-based machine learning technique, for example, using the Adam or AdaGrad optimizer.

[0099] In some implementations, system 100 or another system can maintain a set of candidate embedding machine learning models trained using self-supervised learning. For example, system 100 can maintain embedding machine learning models having different network architectures, including, for example, variational autoencoders (VAEs), autoregressive transformers, and bidirectional transformers. System 100 can select an embedding machine learning model 120 from a set of pre-trained candidate machine learning models for a particular prediction task by evaluating the performance of each candidate embedding machine learning model for a specific prediction task on a labeled dataset, and by selecting the best-performing embedding machine learning model from the first set of candidate machine learning models. Here, the particular prediction task may be the task of predicting a specific set of nanobody® molecular properties related to a particular application. To evaluate the performance of the pre-trained candidate embedding machine learning models for a particular prediction task, the system can pair the pre-trained candidate embedding machine learning models for the particular prediction task with a property prediction model 130 for the particular prediction task, and evaluate the performance of the paired models for the prediction task, e.g., prediction accuracy.

[0100] In some implementations, system 100 can mix and match different candidate embedding machine learning models with different types of feature prediction models (e.g., neural networks, K-nearest neighbor models, support vector machines, decision tree models, random forest models, or ridge regression models) to select the best-performing combination for a particular prediction task. System 100 can use any appropriate optimization method to identify the best-performing combination of the embedding machine learning model and the feature prediction model.

[0101] Based on the predicted nanobody® molecular properties 140, the system 100 can identify the optimal nanobody® molecular sequence for specific downstream tasks, such as binding to a target protein, agonist, or antagonist function, achieving thermal stability, and / or achieving oral stability. For example, the system can generate an output indicating whether a particular nanobody® molecule is suitable for a particular application, or an output specifying the optimal nanobody® molecular sequence for a particular application. The system can send the output to a manufacturing apparatus capable of executing instructions to produce the nanobody® molecule.

[0102] In some implementations, system 100 or other systems can perform cluster analysis using the embedded feature vectors 125 generated for multiple nanobody® molecules. Cluster analysis involves grouping nanobody® molecules based on the distance between pairs of corresponding embedded feature vectors 125 for the nanobody® molecules in the embedded feature space. From the results of the cluster analysis, common properties between different nanobody® molecular arrangements can be identified. These findings can be used to select and / or design nanobody® molecular arrangements with properties suitable for specific applications.

[0103] In some implementations, System 100 or other systems can perform a search of nanobody® molecular sequences in the embedding feature vector space to identify nanobody® molecules that have specific properties similar to a particular nanobody® molecule. This system can maintain a database of sequences and corresponding embedding feature vectors for a population of nanobody® molecules. If a search query specifies the amino acid sequence of a particular nanobody® molecule, the system can use the embedding machine learning model 120 to generate a specific embedding feature vector for that particular nanobody® molecule. The system can then search the database for sets of embedding feature vectors within a predetermined distance from the specific embedding feature vector in the embedding feature space, and identify nanobody® molecules corresponding to the identified sets of embedding feature vectors in the database. The resulting nanobody® molecular sequences can be used to guide the selection and / or design of nanobody® molecular sequences with properties suitable for a particular application.

[0104] In some implementations, System 100 or other systems can use reinforcement learning based on predicted nanobody® molecular properties 140 to generate an output specifying the optimal nanobody® molecular sequence for a particular application. For example, System 100 can train a reinforcement learning (RL) model to identify the optimal amino acid sequence of a nanobody® molecule for performing a particular task. System 100 can use any appropriate reinforcement learning technique to train the RL model. Generally, the RL model is configured to process an input sequence representing a nanobody® molecule and generate one or more sets of actions that modify the input sequence. System 100 can identify a new sequence based on the output of the RL model. System 100 can calculate one or more reward values ​​indicating how well the nanobody® molecule represented by the new sequence performs a particular task. The reward values ​​are calculated using one or more nanobody® molecular properties predicted using an embedded machine learning model 120 and a property prediction model 130. The system can tune the parameters of the RL model based on at least the reward values.

[0105] After the RL model is trained, system 100 or another system can use the RL model to generate nanobody® molecular sequences for target properties. In a particular example, system 100 can maintain a curriculum of nanobody® molecular sequences, i.e., data representing a set of candidate sequences for nanobody® molecules. The system can process the candidate sequences using the trained RL model to generate new sequences and select one or more optimal sequences from the new sequences. System 100 can iteratively update the curriculum of nanobody® molecular sequences by repeatedly running the process until certain conditions are met, for example, when it is determined that the nanobody® molecular sequences have satisfactory performance, or when a threshold for the number of iterations is reached.

[0106] Figure 2 is a flowchart illustrating an exemplary process 200 for predicting the properties of nanobody® molecules. For convenience, the process 200 is described as being performed by a system of one or more computers located in one or more locations. For example, a nanobody® molecular property prediction system, e.g., the nanobody® molecular property prediction system 100 shown in Figure 1, appropriately programmed according to the disclosure, can perform the process 200.

[0107] At step 210, the system obtains a nanobody® molecular sequence, i.e., a token sequence representing the nanobody® molecule. In some implementations, the token sequence can be an alignment sequence generated by aligning the amino acid sequences of the nanobody® molecule.

[0108] In 220, the system generates an input vector by numerically encoding a token sequence. For example, the system can generate a vector by concatenating numerical values ​​assigned to different amino acids to form the input vector.

[0109] At step 230, the system generates an embedded feature vector by processing the input vector using an embedded machine learning model. The embedded machine learning model has a first set of model parameters. The first set of model parameters is updated using self-supervised learning of the first machine learning model, which includes the embedded machine learning model and is configured to perform an array reconstruction task.

[0110] At 240, the system processes the embedded feature vector using a characteristic prediction machine learning model to generate an output that predicts one or more properties of the nanobody® molecule, including, for example, binding affinity to one or more target molecules, binding specificity to one or more target molecules, production yield, stability under one or more environmental conditions, cross-reactivity to one or more target molecules, melting temperature, and / or immunogenicity of the nanobody® molecule.

[0111] The feature prediction machine learning model has a second set of model parameters. This second set of parameters is updated using supervised learning based on multiple training examples of the second machine learning model, which includes the feature prediction machine learning model.

[0112] Figure 3 is a flowchart illustrating an example process 300 for training a predictive model to predict the properties of nanobody® molecules. For convenience, process 300 is described as being performed by a system of one or more computers located at one or more locations. For example, a nanobody® molecular property prediction system appropriately programmed in accordance with this disclosure, such as the nanobody® molecular property prediction system 100 in Figure 1, can perform process 300.

[0113] Generally, a predictive model includes (i) an embedding machine learning model configured to generate an embedding feature vector for model input representing the amino acid sequence of a nanobody® molecule, and (ii) a feature predictive machine learning model configured to process the embedding feature vector to generate an output specifying one or more properties of the nanobody® molecule.

[0114] In step 310, the system obtains a first dataset containing a set of sequence representations of nanobody® molecules. For example, a sequence representation can be a token vector that numerically encodes a sample sequence of a nanobody® molecule whose amino acid sequence is known.

[0115] In step 320, the system performs self-supervised learning of a first machine learning model, including an embedded machine learning model, for a reconstruction task using the first dataset. The first machine learning model can be a large-scale language model and may include a variational autoencoder (VAE), an autoregressive transformer, or a bidirectional transformer.

[0116] In step 330, the system acquires a second dataset containing multiple training examples. Each training example includes (i) a training input specifying a representation of each nanobody® molecule, and (ii) a label specifying one or more properties of each nanobody® molecule.

[0117] In step 340, the system performs supervised learning of a second machine learning model, which includes a feature prediction machine learning model based on the second dataset. The second machine learning model can include one or more of the following: a neural network, a K-nearest neighbor model, a support vector machine, a decision tree model, a random forest model, or a ridge regression model. The second machine learning model can be used to perform regression tasks, classification tasks, or both.

[0118] Figure 4 is a block diagram of an example computer system 400 that can be used to perform the operations described above. System 400 includes a processor 410, memory 420, storage device 430, and input / output device 440. Each of the components 410, 420, 430, and 440 can be interconnected, for example, using a system bus 450. The processor 410 can process instructions to be executed within system 400. In one implementation, the processor 410 is a single-threaded processor. In other implementations, the processor 410 is a multi-threaded processor. The processor 410 can process instructions stored in memory 420 or storage device 430.

[0119] Memory 420 stores information within the system 400. In one implementation, memory 420 is a computer-readable medium. In another implementation, memory 420 is a volatile memory unit. In yet another implementation, memory 420 is a non-volatile memory unit.

[0120] The storage device 430 can provide high-capacity storage for the system 400. In one implementation, the storage device 430 is a computer-readable medium. In various implementations, the storage device 430 may include, for example, a hard disk device, an optical disk device, a storage device shared over a network by multiple computing devices (e.g., cloud storage), or any other high-capacity storage device.

[0121] The input / output device 440 provides input / output operations for the system 400. In one implementation, the input / output device 440 may include one or more network interface devices, such as an Ethernet card, a serial communication device, such as an RS-232 port, and / or a wireless interface device, such as a 502.11 card. In other implementations, the input / output device may include a driver device configured to receive data and send output data to other input / output devices, such as a keyboard, printer, and display device 460. However, other implementations such as mobile computing devices, mobile communication devices, and set-top box television client devices may also be used.

[0122] An exemplary processing system is illustrated in Figure 4, but the subject matter and functional operations described herein can be implemented in other types of digital electronic circuits or computer software, firmware, or hardware, or a combination of one or more thereof, including the structures disclosed herein and their structural equivalents.

[0123] In this disclosure, the term “configured” is used in relation to systems and computer program components. For a system of one or more computers, being configured to perform a particular operation or action means that software, firmware, hardware, or a combination thereof is installed thereon, causing the system to perform that operation or action during operation. One or more computer programs being configured to perform a particular operation or action means that, when executed by a data processing device, the programs include instructions that cause the device to perform that operation or action. The subject matter and functional operation embodiments described in this disclosure can be implemented in digital electronic circuits, tangibly implemented computer software or firmware, computer hardware including structures disclosed herein and their structural equivalents, or one or more combinations thereof. Embodiments of the subject matter described in this disclosure can be implemented as one or more modules of computer program instructions, i.e., computer program instructions, that are executed by one or more computer programs, i.e., data processing devices, or encoded in tangible non-temporary storage media for controlling their operation. Computer storage media can be machine-readable storage devices, machine-readable storage boards, random or serial access memory devices, or one or more combinations thereof. Alternatively or additionally, program instructions can be encoded with mechanically generated electrical, optical, or electromagnetic signals, such as those generated to encode information for transmission to a receiving device suitable for execution by a data processing device.

[0124] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, machines, and equipment for data processing, including, for example, programmable processors, computers, or multiple processors or computers. A device may be a dedicated logic circuit, such as an FPGA (Field-Programmable Gate Array) or ASIC (Application-Specific Integrated Circuit), or may further include such circuits. Optionally, in addition to hardware, a device may include code that forms the execution environment for computer programs, such as processor firmware, protocol stacks, database management systems, operating systems, or one or more combinations thereof.

[0125] Computer programs, also called or written as programs, software, software applications, apps, modules, software modules, scripts, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including standalone programs or modules, components, subroutines, or other units suitable for use in a computer environment. A program may or may not correspond to a file in a file system. A program may be stored in other programs or data, such as markup language documents, in a single file dedicated to the program, or in part of a file containing one or more scripts stored in a file containing one or more modules, subprograms, or parts of code. A computer program can be deployed to run on one computer, or on multiple computers located in one place or distributed across multiple locations and interconnected by a data communication network.

[0126] In this disclosure, the term “database” is used in a broad sense to refer to any collection of data, that is, data that does not need to be structured in any particular way, or does not need to be structured at all, and can be stored in storage devices in one or more locations. Thus, for example, an index database may contain multiple collections of data, each of which may have a different organization and access.

[0127] Similarly, in this disclosure, the term “engine” is used in a broad sense to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components and installed on one or more computers in one or more locations. One or more computers may be dedicated to a particular engine, or multiple engines may be installed and operate on one or more of the same computers.

[0128] The processes and logic flows described herein can be executed by one or more programmable computers running one or more computer programs to perform functions by acting on input data and producing outputs. The processes and logic flows can also be executed by dedicated logic circuits, such as FPGAs or ASICs, or by a combination of dedicated logic circuits and one or more programmed computers.

[0129] A computer suitable for executing computer programs may be based on a general-purpose or dedicated microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory, random-access memory, or both. Essential elements of a computer are a central processing unit that executes or carries out instructions, and one or more memory devices that store instructions and data. The central processing unit and memory may be reinforced by or incorporated into dedicated logic circuits. Generally, a computer also includes or is operationally coupled to one or more mass storage devices for storing data, such as magnetic, magneto-optical disks, or optical disks, to receive and transmit data to and from such devices, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in other devices, to name a few, such as mobile phones, personal digital assistants (PDAs), mobile audio or video players, game consoles, Global Positioning System (GPS) receivers, or portable storage devices, such as Universal Serial Bus (USB) flash drives.

[0130] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0131] To enable user interaction, embodiments of the subject matter described herein can be implemented on a computer having a display device that displays information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to enable user interaction; for example, the feedback provided to the user may be any form of sensory feedback, such as visual, auditory, or tactile feedback, and the input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, the computer can interact with the user by sending documents to and receiving documents from the device the user is using, for example, by sending a web page to a web browser on a user's device in response to a request received from that web browser. The computer can also interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving response messages from the user in return.

[0132] The data processing unit that implements the machine learning model may also include, for example, a dedicated hardware accelerator unit to handle the general, computationally intensive parts of machine learning training or production, i.e., inference, workload.

[0133] Machine learning models can be implemented and deployed using machine learning frameworks, such as PyTorch or TensorFlow.

[0134] Multiple embodiments of the subject matter described in this disclosure can be implemented in a computing system that includes, for example, a backend component as a data server, or a middleware component, such as an application server, or a frontend component, such as a client computer equipped with an implementation of the subject matter described in this disclosure and a graphical user interface, a web browser, or an application through which a user can interact, or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected by digital data communication in any form or medium, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0135] A computing system may include a client and a server. The client and server are generally located remotely from each other and typically interact via a communication network. The client-server relationship is established by computer programs running on each computer that have a client-server relationship with each other. In some embodiments, the server sends data, such as an HTML page, to a user device for the purpose of displaying data to a user interacting with a device acting as a client and receiving user input from the user. Data generated on the user device, such as the results of the interaction with the user, can be received from the device on the server side.

[0136] This disclosure includes many specific implementation details, which should not be interpreted as limitations on the scope of any invention or the claims, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described in this disclosure in relation to separate embodiments may be implemented in combination within a single embodiment. Conversely, various features described in relation to a single embodiment may be implemented separately in multiple embodiments or in any suitable partial combination. Furthermore, even if features are described as operating in a particular combination, and are initially claimed as such, one or more features from the claimed combination may, in some cases, be excluded from that combination, and the claimed combination may refer to a partial combination or a variation of a partial combination.

[0137] Similarly, although the operations are shown in the drawings and enumerated in the claims in a specific order, this should not be understood as meaning that such operations must be performed in a specific illustrated or sequential order, or that all the exemplified operations must be performed, in order to achieve the desired results. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as meaning that such separation is required in all embodiments, and the described program components and systems can generally be incorporated together in one software product or packaged into multiple software products.

[0138] Furthermore, experiments can be performed to identify predicted properties of the nanobody® molecule, such as binding affinity to a target molecule, specificity, yield, cross-reactivity, melting temperature, stability, or immunogenicity. In some embodiments, the experimental results can be used to refine the methods described herein. In some embodiments, the amino acid sequence of a nanobody® molecule having desired properties can be identified or predicted by the methods described herein. The nanobody® molecule can be expressed using a recombinant vector (e.g., an expression vector) containing an isolated polynucleotide (e.g., a polynucleotide encoding the desired nanobody® molecule sequence). In some embodiments, an expression vector is used. The polynucleotide of interest is positioned to be expressed in the vector by operationally linking it to a regulatory element such as a promoter, enhancer, and / or poly-A tail, either within the vector, at, near, or adjacent to the integration site of the polynucleotide of interest, and thereby the polynucleotide of interest is translated in the host cell into which it is introduced along with the expression vector. Vectors can be introduced into host cells by methods known in the art, such as electroporation, chemical transfection (e.g., DEAE-dextran), transformation, transfection, infection, and / or transduction (e.g., by recombinant viruses). Non-limiting examples of vectors include viral vectors (which can be used to generate recombinant viruses), naked DNA or RNA, plasmids, cosmids, phage vectors, and DNA or RNA expression vectors associated with cationic condensants.

[0139] Specific embodiments of the subject matter have been described. Other embodiments are also included in the scope of the following claims. For example, the actions mentioned in the claims may be performed in a different order, and the desired results may still be achieved. As an example, the process shown in the accompanying drawings does not necessarily require the specific illustrated or sequential order to achieve the desired results. Multitasking and parallel processing may be advantageous in some cases.

Claims

1. A computer implementation method for predicting the properties of an immunoglobulin single variable domain (ISVD) molecule, Obtaining a token sequence representing the ISVD molecule, The input vector is generated by numerically encoding the token sequence representing the ISVD molecule, Generating an embedding feature vector by processing the input vector using an embedding machine learning model having a first set of model parameters, wherein the first set of model parameters is updated using self-supervised learning of a first machine learning model configured to include the embedding machine learning model and perform an array reconstruction task. Processing the embedding feature vector using a feature prediction machine learning model to generate an output that predicts one or more properties of the ISVD molecule, wherein the feature prediction machine learning model has a second set of model parameters, which is updated using supervised learning based on a plurality of training examples of a second machine learning model comprising the feature prediction machine learning model, and each training example includes (i) a training input specifying a representation of each ISVD molecule, and (ii) a label specifying one or more properties of each ISVD molecule. Computer implementation methods, including those mentioned above.

2. The method according to claim 1, wherein the ISVD molecule is a heavy-chain antibody single variable domain (VHH) molecule.

3. The method according to claim 1 or 2, wherein the ISVD molecule is a single variable domain of immunoglobulin G comprising (i) two heavy chains and (ii) lacking either CH1 domain.

4. Obtaining the token sequence representing the ISVD molecule is, Obtaining an initial token sequence representing the amino acid sequence of the ISVD molecule, The method involves generating a token array of a predetermined length by adding padding tokens to the initial token array at one or more positions, wherein the predetermined length is longer than the length of the initial token array. The method according to any one of claims 1 to 3, including

5. Obtaining the token sequence representing the ISVD molecule is, Obtaining an initial token sequence representing the amino acid sequence of the ISVD molecule, Generating the token array by performing an alignment of the initial token array, wherein the alignment includes inserting gap tokens at one or more positions in the initial token array. The method according to any one of claims 1 to 3, including

6. The method according to claim 5, wherein the alignment is performed according to the annotation information of the initial token sequence.

7. The method according to claim 6, wherein the annotation information of the initial token sequence is generated using an annotation scheme selected from the IMGT annotation scheme, Kabat annotation scheme, Chothia annotation scheme, Martin annotation scheme, Wolfguy annotation scheme, or AHo annotation scheme.

8. The method according to claim 6, wherein the annotation information of the initial token array is generated using the AHo annotation scheme.

9. The method according to any one of claims 1 to 8, wherein the one or more properties of the ISVD molecule include one or more of the following: binding affinity to one or more target molecules, binding specificity to one or more target molecules, production yield, stability under one or more environmental conditions, cross-reactivity to one or more target molecules, melting temperature, or immunogenicity.

10. Generating the aforementioned input vector means Mapping each token in the aforementioned token array to its respective numerical value, The above numerical values ​​are concatenated to generate the input vector, The method according to any one of claims 1 to 9, including

11. The first machine learning model includes a large-scale language model (LLM), according to any one of claims 1 to 10.

12. The first machine learning method according to any one of claims 1 to 11, wherein the first machine learning method includes a variational autoencoder (VAE), an autoregressive transformer, or a bidirectional transformer.

13. The method according to any one of claims 1 to 12, wherein the characteristic prediction machine learning model includes one or more of the following: a neural network, a K nearest neighbor model, a support vector machine, a decision tree model, a random forest model, or a ridge regression model.

14. The method according to any one of claims 1 to 13, wherein the first model parameter set is fixed after the self-supervised learning process and during the supervised learning process.

15. The method according to any one of claims 1 to 13, wherein the first model parameter set is further updated during the supervised learning process, and the embedded machine learning model and the feature prediction machine learning model are jointly trained end-to-end.

16. Maintain a first candidate set of machine learning models trained using self-supervised learning, Selecting the embedded machine learning model for a specific prediction task from the aforementioned first set of candidate machine learning models, The method according to any one of claims 1 to 15, further comprising:

17. Selecting the embedded machine learning model from the aforementioned first candidate set of machine learning models is, Evaluating the performance of each of the first candidate machine learning model sets for the specific prediction task on a labeled dataset, Based on the performance, the embedded machine learning model is selected from the first candidate set of machine learning models, The method according to claim 16, including the method described in claim 16.

18. The method according to claim 16 or 17, wherein the first candidate set of machine learning models includes two or more of variational autoencoders (VAEs), autoregressive transformers, or bidirectional transformers.

19. A method for selecting an ISVD molecule from a set of candidate ISVD molecules to perform a downstream task, Using one of the methods described above, predict the characteristics of each of the candidate ISVD molecules, Based on the predicted characteristics, select the ISVD molecule from the set of candidate ISVD molecules, Methods that include...

20. The aforementioned downstream task is, Binding to target protein, agonist, or antagonist function; achieving thermal stability; or achieving oral stability. 19 methods, including one or more of the following.

21. A method for identifying the optimal amino acid sequence of an ISVD molecule for performing a specific task, Maintaining data representing a set of candidate sequences for the aforementioned ISVD molecule, Using a reinforcement learning (RL) model to process one or more of the candidate sequences to generate one or more new sequences, wherein the RL model is trained using a reward signal that includes ISVD molecular properties predicted using the method according to any one of claims 1 to 20. Selecting the optimal sequence from the aforementioned new sequences, Methods that include...

22. A method for training a reinforcement learning (RL) model to identify the optimal amino acid sequence of an ISVD molecule for performing a specific task, Using the aforementioned RL model, the input sequence representing the ISVD molecule is processed to generate one or more sets of actions that modify the input sequence. Identifying a new sequence based on the aforementioned input sequence and the aforementioned action set, The calculation involves determining how well a particular task is performed by the ISVD molecule represented by the new sequence, wherein the reward value is calculated using one or more ISVD molecular properties predicted using the method according to any one of claims 1 to 21. Adjusting at least one parameter of the RL model based on the reward value, Methods that include...

23. A computer implementation method for training a predictive model for predicting the properties of an ISVD molecule, wherein the predictive model includes (i) an embedding machine learning model configured to generate an embedding feature vector for a model input representing the ISVD molecule, and (ii) a property predictive machine learning model configured to process the embedding feature vector to generate an output specifying one or more properties of the ISVD molecule, the method being: Obtaining a first dataset containing a sequence representation set of the ISVD molecule, Using the aforementioned first dataset, perform self-supervised learning of the first machine learning model, including the embedded machine learning model, for the reconstruction task. Obtaining a second dataset containing multiple training examples, wherein each training example includes (i) a training input specifying a representation of each ISVD molecule, and (ii) a label specifying one or more properties of each ISVD molecule. Supervised learning of a second machine learning model, including the characteristic prediction machine learning model, is performed on the second dataset. Computer implementation methods, including those mentioned above.

24. The method according to claim 23, wherein the one or more properties of the ISVD include one or more of binding affinity to a target molecule, specificity, yield, cross-reactivity, melting temperature, stability, or immunogenicity.

25. The method according to claim 23 or 24, wherein the first machine learning model includes a large-scale language model (LLM).

26. The method according to any one of claims 23 to 25, wherein the first machine learning model includes a variational autoencoder (VAE), an autoregressive transformer, or a bidirectional transformer.

27. The method according to any one of claims 23 to 26, wherein the characteristic prediction machine learning model includes one or more of a neural network, a K nearest neighbor model, a support vector machine, a decision tree model, a random forest model, or a ridge regression model.

28. The method according to any one of claims 23 to 27, wherein the first model parameter set is fixed after the self-supervised learning process and during the supervised learning process.

29. The method according to any one of claims 23 to 28, wherein the first model parameter set is further updated during the supervised learning process, and the embedded machine learning model and the feature prediction machine learning model are jointly trained end-to-end.

30. Using the aforementioned embedding machine learning model, the embedding feature vectors for each of the multiple ISVD molecules are generated, By performing cluster analysis on the embedding feature vectors of the plurality of ISVD molecules, prediction results are generated. The method according to any one of claims 1 to 18, further comprising:

31. Maintain a dataset containing data specifying (i) the amino acid sequence of each known ISVD molecule and (ii) the respective embedding feature vectors generated for each known ISVD molecule, Receiving input specifying the amino acid sequence of a particular ISVD molecule, Using the aforementioned embedding machine learning model, a specific embedding feature vector for the specific ISVD molecule is generated, Searching the dataset to identify one or more sets of embedding feature vectors that are within a predetermined distance from a particular embedding feature vector in the feature space, Identifying the amino acid sequences corresponding to the identified set of embedding feature vectors in the dataset, Outputting data specifying the identified amino acid sequence, The method according to any one of claims 1 to 18, further comprising:

32. The method according to any one of claims 1 to 18, wherein the characteristic prediction machine learning model is configured to perform one or more regression tasks or classification tasks.

33. One or more computers, When executed by the one or more computers, one or more storage devices store instructions that cause the one or more computers to perform the operation of each of the methods described in any one of claims 1 to 32, A system that includes this.

34. One or more computer-readable storage media that, when executed by one or more computers, store instructions causing one or more computers to perform the operation of each of the methods described in any one of claims 1 to 32.

Images