Optimizing viral capsids using machine learning

Abstract

In one aspect, the method includes receiving data specifying the amino acid sequences of the capsid monomers, processing the data specifying the amino acid sequences of the capsid monomers with an embedded neural network according to values ​​of a set of embedded neural network parameters to generate embeddings of the capsid monomers, processing the embeddings of the capsid monomers with a fitness predictive machine learning model to generate fitness scores characterizing a predicted quality of the capsid, and outputting the fitness scores characterizing the predicted quality of the capsid.

Description

[Technical Field]

[0001] This specification relates to optimizing viral capsids using machine learning. [Background technology]

[0002] Virus can refer to a microscopic infectious agent capable of replicating within the living cells of an organism, and capsid can refer to the protein shell of a virus that encloses the virus's genetic material.

[0003] A capsid may comprise an assembly of repeating protein structural units referred to as "capsid monomers" or simply "monomers." A capsid may comprise any suitable number of monomers, e.g., 20, 50, 100, etc. A capsid monomer may comprise a sequence (chain) of amino acids, each amino acid selected from a set of possible amino acids, e.g., the standard set of 20 alpha-amino acids, such as glycine, alanine, valine, etc. The amino acid sequence of a capsid monomer may have any suitable length (i.e., the "length" of an amino acid sequence refers to the number of positions occupied by an amino acid in the amino acid sequence), e.g., 10, 20, 30, etc. Summary of the Invention [Problem to be solved by the invention]

[0004] "Adeno-associated viruses" (AAVs) are small viruses that infect humans but produce only a mild immune response and are not currently known to cause disease. AAVs have properties that make them suitable candidates for generating viral vectors for gene therapy, such as apparent nonpathogenicity, the ability to infect non-dividing cells, and the ability to stably integrate into the host cell genome at specific sites. AAV capsids may contain a mixture of capsid monomers VP1, VP2, and VP3 for a total of 60 monomers, which assemble to form a capsid with icosahedral symmetry. [Means for solving the problem]

[0005] This specification describes a fitness prediction system implemented as a computer program on one or more computers located at one or more locations that can process data specifying the amino acid sequences of capsid monomers and generate a fitness score that characterizes the predicted quality of the capsid. Capsid quality may refer to any suitable property of the capsid, such as the manufacturability of the capsid, the ability of a virus containing the capsid to evade neutralization, the immunoreactivity of the capsid, etc., as described in more detail below.

[0006] Throughout this specification, an "embedding" may refer to an ordered collection of numbers, for example, a vector, matrix, or other tensor of numbers.

[0007] Throughout this specification, a "block" in a neural network may refer to a collection of neural network layers within the neural network.

[0008] According to a first aspect, there is provided a method comprising receiving data specifying amino acid sequences of capsid monomers; processing the data specifying the amino acid sequences of the capsid monomers using an embedded neural network according to values ​​of a set of embedded neural network parameters to generate embeddings of the capsid monomers; processing the capsid monomer embeddings using a fitness predictive machine learning model to generate fitness scores characterizing a predicted quality of the capsid; and outputting the fitness scores characterizing the predicted quality of the capsid.

[0009] In some implementations, the method further includes processing the representation of the capsid monomers with an error prediction machine learning model to generate a fitness error, where the fitness error is an estimate of the error in the fitness score generated by the fitness prediction machine learning model.

[0010] In some implementations, the method includes updating the relevance score by combining (i) a relevance score generated by a relevance prediction machine learning model and (ii) a relevance error generated by an error prediction machine learning model.

[0011] In some implementations, updating the relevance score includes summing the relevance score generated by the relevance prediction machine learning model and the relevance error generated by the error prediction machine learning model.

[0012] In some implementations, the relevance predictive machine learning model has a different model architecture than the error predictive machine learning model.

[0013] In some implementations, processing the representation of the capsid monomers with the error-predictive machine learning model includes processing the representation of the amino acid sequence of the capsid monomers with the error-predictive machine learning model.

[0014] In some implementations, the error prediction machine learning model processes model inputs differently than the relevance prediction machine learning model.

[0015] In some implementations, processing the data defining the amino acid sequence of the capsid using an embedded neural network includes instantiating each amino acid embedding corresponding to each position in the amino acid sequence of the capsid monomer, processing the amino acid embeddings through one or more neural network layers of the embedded neural network to update each amino acid embedding, and generating an embedding of the capsid protein monomer based on the one or more updated amino acid embeddings.

[0016] In some embodiments, generating a capsid protein monomer embedding based on one or more updated amino acid embeddings comprises generating a capsid protein monomer embedding by combining the updated amino acid embeddings.

[0017] In some implementations, processing the amino acid embeddings through one or more neural network layers of the embedding neural network to update each amino acid embedding includes processing the amino acid embeddings using one or more self-attention neural network layers of the embedding neural network.

[0018] In some implementations, processing the data defining the amino acid sequence of the capsid monomer using an embedded neural network includes instantiating an array of amino acid embeddings including each amino acid embedding corresponding to each position in the amino acid sequence of the capsid monomer; sequentially processing each amino acid embedding in the array of amino acid embeddings using a recurrent neural network to generate an updated hidden state of the recurrent neural network; and determining an embedding of the capsid monomer based on the updated hidden state of the recurrent neural network.

[0019] In some implementations, an embedded neural network is trained on a set of training proteins to perform the token unmasking task.

[0020] In some implementations, training the embedded neural network to perform the token unmasking task includes: for each training protein, generating a masked amino acid sequence by masking one or more positions in the amino acid sequence of the training protein; processing data defining the masked amino acid sequence with the embedded neural network to determine, for each masked position in the amino acid sequence of the training protein, a respective probability distribution over the set of possible amino acids; and training the embedded neural network to optimize, for each masked position in the amino acid sequence of the training protein, an objective function that measures the error between (i) the probability distribution generated for the masked position and (ii) the attribute of the amino acid identified at the masked position in the training protein.

[0021] In some implementations, an embedded neural network is trained using a set of training proteins to perform a next token prediction task.

[0022] In some implementations, training the embedded neural network to perform a next token prediction task includes, for each training protein, using the embedded neural network to generate, for each of one or more target positions in the amino acid sequence of the training protein, a respective probability distribution over the set of possible amino acids, such that for each target position, the probability distribution generated for the target position depends only on the amino acid at the preceding position in the amino acid sequence of the training protein; and training the embedded neural network to optimize, for each target position in the amino acid sequence of the training protein, an objective function that measures the error between (i) the probability distribution generated for the target position and (ii) the attributes of the amino acid identified at the target position in the training protein.

[0023] In some implementations, the fitness prediction neural network is trained using a set of training examples, each training example corresponding to a respective training monomer of the capsid, and including (i) an embedding of the training monomer of the capsid, and (ii) a target fitness score for the training monomer of the capsid.

[0024] In some implementations, training the fitness-prediction neural network on the set of training examples includes rebalancing the set of training examples based on target fitness scores specified by the training examples.

[0025] In some implementations, the relevance predictive machine learning model is a parametric machine learning model.

[0026] In some implementations, the fitness-predicting machine learning model includes one or more of a neural network model, a random forest model, or a support vector regression model.

[0027] In some implementations, the relevance predictive machine learning model is a non-parametric model.

[0028] In some implementations, the relevance predictive machine learning model includes a k-nearest neighbor model.

[0029] In some implementations, the predicted quality of the capsid characterizes the predicted manufacturability of the capsid.

[0030] In some embodiments, the manufacturability of a capsid is defined based on the number of instances of the capsid that are produced when a plasmid coding for the capsid is transfected into one or more cells.

[0031] In some implementations, the number of instances of capsid produced when a plasmid encoding the capsid is transfected into one or more cells is normalized by some wild-type sequence produced.

[0032] In some implementations, the predicted quality of a capsid characterizes the predicted ability to evade neutralization of a virus containing the capsid.

[0033] In some implementations, the predicted quality of the capsid characterizes the predicted immunoreactivity of the capsid.

[0034] In some implementations, the predicted quality of a capsid characterizes the predicted ability of a virus containing the capsid to penetrate a target tissue.

[0035] In some implementations, the predicted quality of the capsid characterizes the packing capacity of the capsid.

[0036] In some implementations, the predicted quality of a capsid characterizes the predicted ability of the virus that contains the capsid to integrate into the host genome.

[0037] In some implementations, the capsid corresponds to the capsid of an adeno-associated virus (AAV).

[0038] In some implementations, the method further includes selecting a capsid monomer from the set of possible capsid monomers using a fitness predictive machine learning model, and physically producing one or more viruses containing proteins having the selected capsid protein monomer.

[0039] In some implementations, selecting a capsid monomer from the set of possible capsid monomers using a fitness predictive machine learning model includes generating a respective fitness score for each capsid monomer in the set of possible capsid monomers using the fitness predictive machine learning model, and selecting the capsid monomer based on the fitness score.

[0040] In some implementations, selecting a capsid monomer based on the fitness score includes selecting a capsid monomer having the highest fitness score from among the capsid monomers in the set of possible capsid monomers.

[0041] In some implementations, the method further includes administering the produced virus to a subject to achieve a therapeutic effect in the subject.

[0042] According to another aspect, there is provided a system including one or more computers and one or more storage devices communicatively coupled to the one or more computers, the one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the methods described herein.

[0043] According to another aspect, one or more non-transitory computer storage media are provided that store instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the methods described herein.

[0044] Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.

[0045] The fitness prediction system described herein utilizes machine learning models to predict capsid quality.

[0046] In general, the accuracy and robustness of a machine learning model can be limited by the quantity and quality of training data available to train the model. Generating capsid fitness training data for training a fitness prediction system often involves performing laboratory experiments, e.g., by transfecting cells with a plasmid encoding the capsid and measuring the number of capsid instances generated. These laboratory experiments can be expensive, time-consuming, and generate "noisy" (e.g., inaccurate or inconsistent) data.

[0047] Furthermore, because capsid quality (e.g., manufacturability) can result from a complex interplay between biology, chemistry, and mechanics, machine learning models must perform sophisticated implicit inference to predict capsid quality. The inference power of a machine learning model can depend on the number of parameters of the model, and so, for example, increasing the number of parameters of the machine learning model can improve inference power. A particular machine learning model performing a complex prediction task may include a large number of parameters, e.g., millions or billions of parameters. However, increasing the number of parameters of a machine learning model may increase the likelihood that the machine learning model will "overfit" during training, particularly by capturing some residual variation (noise) in the training data as if that variation represents the underlying structure of the training data. The risk of overfitting may be exacerbated when the training data is limited in quantity and subject to noise, as in the case of the capsid fitness training data described above.

[0048] The relevance prediction system described herein implements various innovations to address these issues.

[0049] To generate a fitness score for a capsid, the fitness prediction system can process data specifying the amino acid sequences of the capsid monomers using an embedded neural network to generate embeddings of the capsid monomers. The fitness prediction system can then process the embeddings of the capsid monomers using a fitness prediction machine learning model to generate a fitness score for the capsid.

[0050] The fitness prediction system can use unsupervised learning techniques to train an embedded neural network on large amounts of protein data for which labels (e.g., capsid fitness scores) are not available. For example, the fitness prediction system can train the embedded neural network to perform a token unmasking task or a next token prediction task, as described in more detail below. By training the embedded neural network to perform unsupervised tasks, the embedded neural network learns to generate capsid monomer embeddings that compactly encode the biologic, chemical, and mechanical properties of capsid monomers, which can be used by the fitness prediction machine learning model to generate accurate fitness scores. Furthermore, training the embedded neural network for unsupervised tasks does not rely on capsid fitness training data, and is therefore not affected by the limited quantity and quality of available capsid fitness training data.

[0051] After generating a fitness score for a capsid using the fitness prediction machine learning model, the fitness prediction system can use a separate error prediction machine learning model to generate a fitness error that predicts an error in the fitness score. The fitness prediction system can then use the fitness error to update the fitness score, e.g., to correct the fitness score error, thereby improving the overall accuracy of the fitness prediction system. More specifically, the fitness prediction system can train the error prediction machine learning model to learn to correct systematic errors in the fitness scores generated by the fitness prediction machine learning model. The error prediction machine learning model can detect and avoid certain systematic errors in the fitness prediction machine learning model, for example, by having a different model architecture or by processing inputs in a different format than the fitness prediction machine learning model, as described in more detail below.

[0052] The fitness prediction system can rebalance the capsid fitness training data to improve the accuracy and learning speed of the machine learning model included in the fitness prediction system. For example, without rebalancing the training data, the number of training examples with low capsid fitness scores may increase significantly more than the number of training examples with high capsid fitness scores. Training on unbalanced training data may limit the ability of training examples with high capsid fitness scores to influence the parameter values ​​of the machine learning model, thereby reducing the ability of the machine learning model to learn the properties of capsid monomers that result in high capsid fitness scores. The fitness prediction system can rebalance the training data to increase the influence of training examples with high capsid fitness scores. For example, the fitness prediction system can resample the training data to increase the proportion of training examples with high capsid fitness scores or modify the objective function used during training to increase the penalty for prediction errors of training examples corresponding to high capsid fitness scores.

[0053] After training, a fitness prediction system can be used to screen large libraries of capsid monomers (e.g., containing millions or billions of possible capsid monomers) to identify capsid monomers that result in capsids with high fitness scores, e.g., capsids that are likely to be manufacturable. That is, a fitness prediction system allows for the automatic exploration of the space of possible capsid monomers to identify capsid monomers with desired properties. A fitness prediction system allows for the exploration of orders of magnitude more capsids than can be physically generated and tested in laboratory experiments. Furthermore, a fitness prediction system operates quickly (e.g., taking less than one second to generate a predicted fitness score) while being cheaper and consuming relatively fewer operating resources than laboratory experiments.

[0054] A compatibility prediction system can be used to identify viral vector capsids for gene therapy. Gene therapy has the potential to transform medicine by enabling diseases (e.g., hemophilia, leukemia, melanoma, etc.) to be addressed with a single dose of a therapeutic viral vector (e.g., AAV vector). However, gene therapy treatments are often expensive, in some cases costing over a million dollars to treat a single patient. The high cost is a major barrier to fully realizing the transformative potential of gene therapy. A major factor in the cost of gene therapy is the cost of manufacturing capsids for viral vectors. The compatibility prediction system described herein enables the discovery of viral capsids with a high level of manufacturability, which could significantly reduce the cost of gene therapy.

[0055] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0056] [Figure 1] 1 shows an exemplary environment for screening a capsid monomer library using a fitness prediction system. [Figure 2] 1 illustrates an exemplary relevance prediction system. [Figure 3] FIG. 1 is a flow diagram of an exemplary process for generating a fitness score that characterizes the predicted quality of a capsid. [Figure 4] FIG. 1 is a flow diagram of an exemplary process for generating embeddings of capsid monomers using an embedded neural network with a recurrent neural network architecture. [Figure 5] FIG. 1 is a flow diagram of an exemplary process for generating embeddings of capsid monomers using an embedded neural network having a neural network architecture that includes an array of update blocks. [Figure 6] FIG. 1 is a flow diagram of an exemplary process for training an embedded neural network to perform a next token prediction task. [Figure 7] FIG. 1 is a flow diagram of an exemplary process for training an embedded neural network to perform a token unmasking task. [Figure 8] 1 shows the accuracy achieved by the fitness prediction system described herein on the task of predicting a fitness score that characterizes the manufacturability of a capsid. [Figure 9] 1 shows a comparison of the accuracy achieved by a relevance prediction system with and without an error prediction machine learning model. [Figure 10] 1 shows a comparison of the prediction accuracy achieved by the relevance prediction system for different choices of model architecture for the relevance prediction machine learning model. [Figure 11] 1 shows a comparison of the prediction accuracy achieved by the adaptive prediction system when no rebalancing of the training data is performed and when rebalancing of the training data is performed. [Figure 12] A comparison of the prediction accuracy achieved by the adaptive predictive machine learning model when an embedding neural network generates a 1900-dimensional capsid monomer embedding and when an embedding neural network generates a 256-dimensional capsid monomer embedding is shown. DETAILED DESCRIPTION OF THE INVENTION

[0057] Like reference numbers and designations in the various drawings indicate like elements.

[0058] FIG. 1 illustrates an exemplary environment 100 for screening a capsid monomer library 102 using a fitness prediction system 200.

[0059] Capsid monomer library 102 defines a set of capsid monomers, each capsid monomer represented by a respective sequence of amino acids. Capsid monomer library 102 may include any suitable number of capsid monomers, for example, a thousand, a million, or a billion capsid monomers.

[0060] Capsid monomer library 102 may be generated in any of a variety of possible ways. For example, some or all of the capsid monomers in capsid monomer library 102 may be variants of an "original" capsid monomer, such as the VP1, VP2, or VP3 capsid monomers of AAV. More specifically, each capsid monomer in capsid monomer library 102 may be generated by altering the attributes of each amino acid at one or more positions within the amino acid sequence of the original capsid monomer. Positions within the amino acid sequence of the original capsid monomer may be selected for mutation in any suitable manner, such as random selection or selection according to a predetermined rule. The attributes of the new amino acids substituted into positions within the amino acid sequence of the original capsid monomer may be selected in any suitable manner, such as randomly from a probability distribution over the set of possible amino acids. The amino acid sequence of each capsid monomer in the capsid monomer library may differ from the amino acid sequence of the original capsid monomer at any suitable number of positions, for example, 1 position, 3 positions, or 10 positions.

[0061] The fitness prediction system 200 is configured to process capsid monomers (e.g., from the capsid monomer library 102) and generate a capsid monomer fitness score 104 that characterizes the predicted quality of the corresponding capsid. The capsid monomer fitness score 104 may characterize any suitable quality of the corresponding capsid. Some examples of possible fitness scores 104 are described below.

[0062] In some implementations, the fitness prediction system 200 generates a fitness score 104 for the capsid monomer that characterizes the manufacturability of the corresponding capsid. Capsid manufacturability characterizes the rate at which a capsid can be produced when a plasmid encoding the capsid is transfected into one or more cells. (More specifically, capsid manufacturability may be based, at least in part, on the number of capsid instances produced when the plasmid encoding the capsid is transfected into one or more cells.) That is, capsid manufacturability characterizes how efficiently a capsid can be produced. Capsid manufacturability indirectly characterizes various structural and biochemical properties that affect capsid production, such as the ease with which the capsid monomers of a capsid assemble into the capsid's three-dimensional structure, capsid stability, etc.

[0063] In some implementations, the fitness prediction system 200 can generate a fitness score 104 for a capsid monomer that characterizes the predicted ability of a virus containing the corresponding capsid to evade neutralization in an organism. Virus neutralization refers to the process of reducing a virus's ability to infect cells, for example, by antibody binding to epitopes on the surface of the virus.

[0064] In some implementations, the fitness prediction system 200 can generate a fitness score 104 for a capsid monomer that characterizes the predicted immune reactivity of the corresponding capsid. The immune reactivity of a capsid refers to a measure of the immune response elicited by the capsid.

[0065] In some implementations, the fitness prediction system 200 can generate a fitness score 104 for a capsid monomer that characterizes the predicted ability of the virus containing the capsid to penetrate a target tissue within an organism, which may be, for example, tissue corresponding to a particular organ, such as liver tissue, brain tissue, eye tissue, etc.

[0066] In some implementations, the fitness prediction system 200 can generate a fitness score 104 for a capsid monomer that characterizes the predicted packing capacity of the corresponding capsid. The packing capacity of a capsid refers to a measure of the amount of genetic material (e.g., deoxyribonucleic acid (DNA) or ribonucleic acid (RNA)) that the capsid can contain.

[0067] In some implementations, the fitness prediction system 200 can generate a fitness score 104 for a capsid monomer that characterizes the predicted ability of a virus containing the corresponding capsid to integrate into a host genome.

[0068] The fitness prediction system 200 can screen the capsid monomer library 102 to identify capsid monomers that correspond to capsids with desired properties. More specifically, the fitness prediction system 200 can generate a respective fitness score 104 for each capsid monomer in the capsid monomer library 102. The fitness prediction system 200 can designate an appropriate subset of the capsid monomers in the capsid monomer library 102 as "target" capsid monomers 106 based, at least in part, on the fitness scores 104. For each target capsid monomer 106, a virus containing a capsid corresponding to the target capsid monomer 106 can then be manufactured 108, i.e., physically produced using an appropriate manufacturing technique.

[0069] The generated viruses may be used in any of a variety of applications, such as administering the generated viruses to a subject 112 for treatment 110 to achieve a therapeutic effect in the subject. For example, the generated viruses may be used as viral vectors to perform gene therapy as described above. The generated viruses may be administered to any suitable subject, such as mice, cats, dogs, pigs, or humans, to achieve any suitable therapeutic effect, such as treatment of diseases such as hemophilia, leukemia, melanoma, etc.

[0070] In certain instances, viral vectors (e.g., AAV vectors, e.g., recombinant AAV vectors) containing target capsids selected using a compatibility prediction system can be generated as described above and administered to a subject for gene therapy.

[0071] The fitness prediction system 200 can select an appropriate subset of capsid monomers in the capsid monomer library to be designated as target capsid monomers in any of a variety of possible ways. For example, the fitness prediction system 200 can designate as target capsid monomers any capsid monomers having a fitness score 104 that meets a predetermined threshold. As another example, the fitness prediction system 200 can designate as target capsid monomers a predetermined number of capsid monomers having the highest fitness scores 104.

[0072] The fitness prediction system 200 can designate any suitable number of capsid monomers from the capsid monomer library 102 as target capsid monomers 106, for example, 10 capsid monomers, 100 capsid monomers, or 1000 capsid monomers. In some cases, the fitness prediction system 200 designates only a small fraction of the total number of capsid monomers in the capsid monomer library as target capsid monomers (e.g., less than 1%, less than 0.1%, or less than 0.01% of the total number of capsid monomers in the capsid monomer library).

[0073] 2 illustrates an exemplary relevance prediction system 200. Relevance prediction system 200 is an exemplary system implemented as a computer program on one or more computers at one or more locations where the systems, components, and techniques described below are implemented.

[0074] The fitness prediction system 200 is configured to process data specifying the amino acid sequence of a capsid monomer 202 to generate a fitness score 104 for the capsid monomer 202. The fitness score 104 characterizes the predicted qualities of a capsid corresponding to the capsid monomer 202. For example, the fitness score 104 may characterize the predicted manufacturability of the capsid, the predicted ability of a virus containing the capsid to evade neutralization, the predicted immunoreactivity of the capsid, the predicted ability of a virus containing the capsid to penetrate a target tissue, the predicted packaging ability of the capsid, or the predicted ability of a virus containing the capsid to integrate into a host genome, as described above with reference to FIG. 1. The fitness score 104 generated by the fitness prediction system 200 can be used to screen a library of capsid monomers to identify target capsid monomers with desirable properties, as described above with reference to FIG. 1.

[0075] The relevance prediction system 200 includes an embedded neural network 204, a relevance prediction machine learning model 208, and an error prediction machine learning model 212, each of which is described in more detail below.

[0076] The embedding neural network 204 is configured to process data defining the amino acid sequence of the capsid monomer 202 to generate an embedding 206 of the capsid monomer 202.

[0077] The embedded neural network 204 may be configured to process any suitable representation of the amino acid sequence of the capsid monomer 202. For example, the embedded neural network 204 may process the representation of the amino acid sequence of the capsid monomer 202 as a sequence of amino acid embeddings. The amino acid embeddings at each position in the sequence of amino acid embeddings may represent the attributes of the amino acid at the corresponding position in the amino acid sequence of the capsid monomer. The amino acid embeddings representing the amino acids may be, for example, one-hot embeddings or any other suitable predetermined embeddings.

[0078] The embedded neural network 204 may have any suitable neural network architecture that enables the embedded neural network 204 to perform its described function. In particular, the embedded neural network 204 may include any suitable number (e.g., 5, 10, 50, etc.) of any suitable type of neural network layers (e.g., fully connected layers, convolutional layers, recurrent layers, attention layers, etc.), connected in any suitable configuration (e.g., as a linear sequence of layers).

[0079] In some implementations, the embedded neural network 204 may be a recurrent neural network configured to sequentially process each amino acid embedding in a sequence of amino acid embeddings representing the amino acid sequence of the capsid monomer 202. An exemplary process for generating embeddings 206 of a capsid monomer 202 using an embedded neural network 204 having a recurrent neural network architecture is described in more detail below with reference to FIG.

[0080] In some implementations, the embedding neural network 204 may include an array of update blocks, each configured to process a current set of embeddings representing a capsid monomer to generate an updated set of embeddings representing a capsid monomer. Each update block may include, for example, one or more self-attention neural network layers. An exemplary process for generating embeddings 206 for capsid monomers 202 using an embedding neural network 204 including an array of updated blocks is described in more detail below with reference to FIG. 5.

[0081] The fitness prediction system 200 can train the embedded neural network 204 on a set of training examples, where each training example corresponds to a respective protein and defines the protein's amino acid sequence. The fitness prediction system 200 can train the embedded neural network on the set of training examples using unsupervised learning techniques, i.e., training techniques that do not rely on training examples labeled with fitness scores. Some exemplary techniques for training the embedded neural network 204 on the set of training examples using unsupervised learning techniques are described below.

[0082] In some implementations, the fitness prediction system 200 trains an embedded neural network 204 to perform a token unmasking task. More specifically, for each training example, the fitness prediction system 200 can "mask" the attributes of amino acids at one or more positions within the amino acid sequence of a protein. (Masking an amino acid attribute may refer to replacing an amino acid embedding representing an amino acid with a predefined embedding, as described in more detail below with reference to FIG. 5.) The fitness prediction system 200 can then train the embedded neural network 204 to process the masked amino acid sequence and predict the attributes of the amino acid at the masked position within the amino acid sequence. The token unmasking task requires the embedded neural network 204 to learn to infer the attributes of the masked amino acid from contextual information provided by the unmasked remainder of the amino acid sequence. An exemplary process for training an embedded neural network to perform a token unmasking task is described in more detail below with reference to FIG. 7.

[0083] In some implementations, the fitness prediction system 200 trains an embedded neural network to perform a next-token prediction task. More specifically, for each training example, the fitness prediction system 200 can designate one or more positions within the amino acid sequence of a protein as "target" positions. For each target position, the fitness prediction system 200 can train the embedded neural network 204 to learn to predict the attributes of the amino acid at the target position based solely on the attributes of amino acids at previous positions in the amino acid sequence. The next-token prediction task requires the embedded neural network 204 to learn to infer the attributes of an amino acid at a position within an amino acid sequence based on the attributes of amino acids at previous positions within the amino acid sequence. An exemplary process for training an embedded neural network to perform a next-token prediction task is described in more detail below with reference to FIG. 6.

[0084] For example, by training the embedding neural network 204 using unsupervised learning techniques to perform tasks such as token unmasking, next token prediction, or both, the embedding neural network 204 can be taught to generate embeddings that densely encode the rich information that characterizes a protein, such as a capsid monomer.

[0085] In some cases, in combination with, or as an alternative to, training the embedded neural network 204 using unsupervised learning techniques, the fitness prediction system 200 can train the embedded neural network 204 using supervised learning techniques on an auxiliary set of training examples. Each training example can correspond to a respective protein and can define (i) the amino acid sequence of the protein and (ii) a label associated with the protein. The label can represent any suitable characteristic of the protein, such as the protein's structure, protein stability, etc.

[0086] The fitness prediction system 200 can jointly train an embedding neural network and a "projection" neural network on a set of supervised training examples. The projection neural network may be configured to process the protein embeddings generated by the embedding neural network to generate predictions of the labels associated with the proteins.

[0087] To train the embedding neural network on the supervised training examples, the fitness prediction system 200 can use the embedding neural network 204 to process the amino acid sequences of the proteins corresponding to the training examples to generate embeddings for the proteins. The fitness prediction system 200 can then use the projection neural network to process the protein embeddings to generate predicted labels associated with the proteins. The fitness prediction system 200 can evaluate a supervised objective function that measures the error of the predicted labels generated for the proteins and back-propagate the gradient of the supervised objective function through the projection neural network to the embedding neural network 204. After training is complete, the projection neural network can be discarded, and the fitness prediction system 200 can use the trained embedding neural network 204 to generate embeddings used to generate the fitness scores 104.

[0088] The supervised objective function may measure the error of the predicted labels produced by the embedded neural network, for example, as cross-entropy error, squared error, or in any other suitable manner. Backpropagating the gradient of the objective function through a neural network may refer to determining the gradient of the objective function with respect to the parameters of the neural network and then using the gradient to adjust the values ​​of the parameters of the neural network, for example, using a suitable gradient descent optimization technique, such as Adam or RMSprop.

[0089] The fitness predictive machine learning model 208 is configured to process the embeddings 206 of the capsid monomers 202 to generate a fitness score 104 that characterizes the predicted quality of the capsid corresponding to the capsid monomers 202.

[0090] The relevance prediction machine learning model 208 may have any suitable machine learning model architecture that enables the relevance prediction machine learning model 208 to perform its described functions. For example, the relevance prediction machine learning model may be a parametric machine learning model, such as a neural network model, a random forest model, or a support vector regression model. As another example, the relevance prediction machine learning model may be a non-parametric model, such as a k-nearest neighbor model.

[0091] The fitness prediction system 200 can train a fitness prediction machine learning model on a set of training examples. Each training example corresponds to a respective capsid monomer and specifies (i) an embedding of the capsid monomer's amino acid sequence, generated using an embedding neural network, and (ii) a target fitness score associated with the capsid monomer. The target fitness score of a capsid monomer represents the prediction goal of the fitness prediction system, i.e., the "ground truth" fitness that should be generated by the fitness prediction system for the capsid monomer.

[0092] The fitness prediction system 200 can use any suitable supervised machine learning training technique to train the fitness prediction machine learning model 208 on a set of training examples and their associated target fitness scores. More specifically, for each training example, the fitness prediction system 200 can train the fitness prediction machine learning model to process the capsid monomer embeddings contained in that training example to generate a predicted fitness score that matches the target fitness score specified by that training example. The fitness prediction system 200 can train the fitness prediction machine learning model 208 to optimize, for each training example, an objective function that measures the error between (i) the target fitness score and (ii) the predicted fitness score generated by the fitness prediction machine learning model 208. The objective function can measure the error in any suitable way, for example, as squared error or absolute error.

[0093] During training of the relevance prediction machine learning model 208, the relevance prediction system 200 can rebalance the set of training examples based on target relevance scores associated with the training examples. In particular, the set of training examples may include significantly more training examples associated with low relevance scores than training examples associated with high relevance scores. Training the relevance prediction machine learning model 208 without rebalancing the set of training examples may result in a decrease in the accuracy of the relevance prediction machine learning model 208. The relevance prediction system 200 can rebalance the set of training examples in any of a variety of possible ways. Some exemplary techniques for rebalancing the set of training examples are described below.

[0094] In some implementations, the relevance prediction system 200 can rebalance the set of training examples by replicating training examples associated with high relevance scores (e.g., relevance scores above a threshold) until at least a threshold fraction of the set of training examples is associated with high relevance scores.

[0095] In some implementations, the relevance prediction system 200 can rebalance the training data set by determining a respective weighting factor associated with each training example as a function of the relevance score associated with the training example. For example, the relevance prediction system 200 can generate the training example weighting factors by processing the training example's relevance scores using a sigmoid function. The training example weighting factors can control the influence of the training examples on the parameter values ​​of the relevance prediction machine learning model during training, e.g., such that training examples with higher weighting factors have a greater influence on the relevance prediction machine learning model. For example, if the relevance prediction machine learning model is a neural network model, the relevance prediction system 200 can scale the gradient of the objective function by the weighting factor as part of training the relevance prediction machine learning model on the training examples. The weighting factor can therefore define an individualized learning rate for each training example during training of the neural network implementing the relevance prediction machine learning model.

[0096] In some implementations, the relevance prediction system 200 can rebalance the training data set by determining a respective penalty coefficient associated with each training example as a function of the fitness score associated with the training example. For example, the relevance prediction system 200 can generate the training example penalty coefficients by processing the training example fitness scores with a sigmoid function. The training example penalty coefficients can control the penalty imposed on the relevance prediction machine learning model during training for inaccurately predicting the fitness scores of the training examples. For example, the training example penalty coefficients can multiplicatively scale a term in an objective function that measures the error of the predicted fitness scores generated by the relevance prediction machine learning model 208 for the training examples.

[0097] The target fitness scores for the capsid monomers of the training examples can be generated in any suitable manner, for example, by physical experimentation. Some exemplary techniques for generating target fitness scores for capsid monomers are described below.

[0098] In some implementations, the fitness score of a capsid monomer characterizes the manufacturability of the corresponding capsid. To generate a target fitness score for a capsid monomer, a plasmid encoding the capsid can be transfected into one or more cells. Experimental measurements are then performed to determine (i) the number of instances of the capsid produced by the one or more cells, and (ii) the number of instances of wild-type capsid produced by the one or more cells. The fitness score of a capsid can be defined, for example, as the ratio of (i) the number of produced instances of the capsid to (ii) the number of produced instances of the wild-type capsid.

[0099] In some implementations, the fitness score of a capsid monomer characterizes the ability of a virus containing the corresponding capsid to evade neutralization. To generate a target fitness score for a capsid monomer, a virus containing the corresponding capsid can be introduced into an organism and experimental measurements can be performed to determine the number of viruses that are subsequently neutralized. The fitness score of a capsid can be defined, for example, based on the ratio of (i) the number of neutralized virus (with capsid) instances to (ii) the total number of virus (with capsid) instances.

[0100] In some embodiments, the fitness score of a capsid monomer characterizes the immune reactivity of the capsid. To generate a target fitness score for a capsid monomer, the corresponding capsid can be introduced into an organism and experimental measurements can be performed to determine the resulting immune response of the organism. A target fitness score for the capsid can then be defined based on the measured immune response.

[0101] In some embodiments, the fitness score of a capsid monomer characterizes the ability of a virus containing the corresponding capsid to penetrate a target tissue. To generate a target fitness score for a capsid monomer, a virus containing the corresponding capsid can be introduced into an organism, and experimental measurements can be performed to determine the number of viruses that penetrate the target tissue. The target fitness score for the capsid monomer can then be determined based on the ratio of (i) the number of instances of the virus (having a capsid) that penetrate the target tissue to (ii) the total number of instances of the virus (having a capsid) introduced into the organism.

[0102] In some implementations, the fitness score of a capsid monomer characterizes the packing ability of the corresponding capsid. To generate a target fitness score for a capsid monomer, one or more examples of the corresponding capsid can be generated, and the amount of genetic material contained in the generated capsids can be experimentally determined. A target fitness score for the capsid monomer can then be defined based on the experimentally determined amount of genetic material contained in the corresponding capsid.

[0103] In some implementations, the fitness score of a capsid monomer characterizes the ability of a virus containing the corresponding capsid to integrate into a host genome. To generate a target fitness score for a capsid monomer, one or more instances of the virus containing the corresponding capsid can be introduced into an organism. Experimental measurements can be performed to determine the number of instances of the virus that successfully integrated into the host genome. The target fitness score for the capsid monomer can then be defined based on the ratio of (i) the number of instances of the virus (having a capsid) that successfully integrated into the host genome in the organism to (ii) the total number of instances of the virus (having a capsid) introduced into the organism.

[0104] The error prediction machine learning model 212 is configured to process the representation of the amino acid sequence of the capsid monomer 202 to generate a fitness error 214. The fitness error 214 is an estimate of the error in the fitness score 104 generated by the fitness prediction machine learning model 208.

[0105] The fitness prediction system 200 can update the fitness score 104 generated by the fitness prediction machine learning model 208 by combining (i) the fitness score 104 and (ii) the fitness error 214. The fitness prediction system 200 can combine the fitness score 104 and the fitness error 214, for example, by summing the fitness score 104 and the fitness error 214. The fitness prediction system 200 can then output the updated fitness score 104 as the predicted fitness score for the input capsid monomer 202.

[0106] In some implementations, the error prediction machine learning model 212 may have a different model architecture than the relevance prediction machine learning model 208. For example, the error prediction machine learning model 212 may be a linear regression model, while the relevance prediction machine learning model 208 may be a random forest model.

[0107] In some implementations, the error prediction machine learning model 212 may be configured to process different inputs (e.g., having a different format) than the fitness prediction machine learning model 208. For example, the error prediction machine learning model 212 may process representations of capsid monomers as sequences of amino acid embeddings, while the fitness prediction machine learning model 208 may process capsid monomer embeddings 206 generated using an embedding neural network 204.

[0108] Configuring the error prediction machine learning model 212 to have a different architecture or to process different inputs than the relevance prediction machine learning model 208 can help make the two machine learning models "orthogonal" to one another. More specifically, configuring the models differently allows each model to perform distinct implicit inferences independent of the other models, thus attenuating the mutual information between their outputs and maximizing the information gain achieved by combining their outputs to generate the updated relevance score 104.

[0109] The error prediction machine learning model 212 may have any suitable machine learning model architecture that enables the error prediction machine learning model 212 to perform its described function. For example, the error prediction machine learning model may be a parametric machine learning model such as a neural network model, a random forest model, or a support vector regression model. As another example, the relevance prediction machine learning model may be a non-parametric model such as a k-nearest neighbor model.

[0110] The fitness prediction system 200 can train an error prediction machine learning model on a set of training examples. Each training example corresponds to a respective capsid monomer and specifies (i) the amino acid sequence of the capsid monomer and (ii) a target fitness error associated with the capsid monomer. The target fitness error for a capsid monomer represents the prediction goal of the error prediction machine learning model, i.e., the "ground truth" fitness error to be generated by the capsid monomer error prediction system. The fitness prediction system 200 can generate the target fitness error for a training example by determining the error between (i) the target fitness score for the training example and (ii) the predicted fitness score generated by the (trained) fitness prediction machine learning model 208 for the training example.

[0111] The fitness prediction system 200 can train the error prediction machine learning model 212 based on a set of training examples and their associated target fitness errors using any suitable supervised machine learning training technique. More specifically, for each training example, the fitness prediction system 200 can train the error prediction machine learning model to process data specifying the amino acid sequences of capsid monomers included in the training example and generate a predicted fitness error that matches the target fitness error specified by the training example. For each training example, the fitness prediction system 200 can train the error prediction machine learning model 212 to optimize an objective function that measures the error between (i) the target fitness error and (ii) the predicted fitness error generated by the error prediction machine learning model 212. The objective function can measure the error in any suitable way, for example, as squared error or absolute error.

[0112] 3 is a flow diagram of an exemplary process 300 for generating a fitness score that characterizes the predicted quality of a capsid. For convenience, process 300 will be described as being performed by one or more computer systems located at one or more locations. For example, a fitness prediction system, such as fitness prediction system 200 of FIG. 2, suitably programmed in accordance with this specification, can perform process 300.

[0113] The system receives data defining the amino acid sequence of a monomer of a capsid (302).

[0114] The system processes the data specifying the amino acid sequences of the capsid monomers using the embedded neural network according to the values ​​of the set of embedded neural network parameters to generate an embedding of the capsid monomers (304).

[0115] The system processes the capsid monomer embeddings using a fitness prediction machine learning model to generate a fitness score that characterizes the predicted quality of the capsid (306).

[0116] The system processes the representation of the capsid monomers using an error prediction machine learning model to generate a fitness error (308), which is an estimate of the error in the fitness score generated by the fitness prediction machine learning model.

[0117] The system updates (310) the relevance score by combining (i) the relevance score generated by the relevance prediction machine learning model and (ii) the relevance error generated by the error prediction machine learning model.

[0118] The system outputs the updated relevance score (312).

[0119] 4 is a flow diagram of an exemplary process 400 for generating capsid monomer embeddings using an embedded neural network having a recurrent neural network architecture. For convenience, process 400 will be described as being performed by one or more computer systems located at one or more locations. For example, a fitness prediction system, such as fitness prediction system 200 of FIG. 2, suitably programmed in accordance with the present disclosure, can perform process 400.

[0120] The system instantiates 402 an array of amino acid embeddings representing the amino acid sequence of the capsid monomer. The array of amino acid embeddings includes an amino acid embedding corresponding to each position in the amino acid sequence of the capsid monomer. The amino acid embeddings corresponding to positions in the amino acid sequence of the capsid monomer can represent the attributes of the amino acid at that position, for example, by one-hot embedding.

[0121] The system uses an embedded neural network having a recurrent neural network architecture to sequentially process each amino acid embedding in the sequence of amino acid embeddings to generate updated hidden states of the embedded neural network (404). More specifically, before processing the first amino acid embedding in the sequence of amino acid embeddings, the system initializes the hidden state of the embedded neural network to, for example, a default hidden state. The embedded neural network then sequentially processes each amino acid embedding in the sequence, starting with the first amino acid embedding and following the order of the amino acid embeddings in the sequence of amino acid embeddings. Each time the embedded neural network processes an amino acid embedding, the embedded neural network uses the amino acid embedding to update the hidden state of the embedded neural network.

[0122] The embedded neural network includes one or more recurrent neural network layers, the hidden states of which collectively define the hidden state of the embedded neural network. Each recurrent neural network layer may have any suitable architecture, such as a long short-term memory (LSTM) architecture or a gated recurrent unit (GRU) architecture.

[0123] The system determines a capsid monomer embedding for the capsid based on the updated hidden state of the embedding neural network after processing the last amino acid embedding in the sequence of amino acid embeddings (406). For example, the system can generate the capsid monomer embedding by processing the updated hidden state of the embedding neural network with one or more neural network layers, e.g., fully connected layers.

[0124] 5 is a flow diagram of an exemplary process 500 for generating embeddings of capsid monomers using an embedded neural network having a neural network architecture that includes an array of update blocks. For convenience, process 500 will be described as being performed by one or more computer systems located at one or more locations. For example, a fitness prediction system, such as fitness prediction system 200 of FIG. 2, suitably programmed in accordance with the present disclosure, can perform process 500.

[0125] The system instantiates 502 an array of amino acid embeddings representing the amino acid sequence of the capsid monomer. The array of amino acid embeddings includes each amino acid embedding corresponding to a position in the amino acid sequence of the capsid monomer. The amino acid embeddings corresponding to a position in the amino acid sequence of the capsid monomer can represent the attributes of the amino acid at that position, for example, by one-hot embedding. Optionally, the system can combine each amino acid embedding with a position embedding representing the corresponding position in the amino acid sequence of the capsid monomer.

[0126] The system processes the sequence of amino acid embeddings using an array of one or more blocks, referred to as update blocks, included in an embedded neural network (504). Each update block is configured to process the sequence of amino acid embeddings using one or more neural network layers included in the update block to generate updated sequences of amino acid embeddings. A first update block may be configured to receive an initial sequence of amino acid embeddings, and each subsequent update block may receive the updated sequence of amino acid embeddings generated by the previous update block. Each update block may have any suitable neural network architecture. In some implementations, some or all of the update blocks include one or more self-attention neural network layers.

[0127] The system generates an embedding for the capsid monomer based on one or more updated amino acid embeddings generated by the last update block of the sequence of update blocks (506). For example, the system can specify that an updated amino acid embedding corresponding to the last amino acid of the amino acid sequence of the capsid monomer is an embedding representing the capsid monomer. As another example, the system can combine (e.g., average or sum) the updated amino acid embeddings generated by the last update block to generate an embedding for the capsid monomer.

[0128] 6 is a flow diagram of an exemplary process 600 for training an embedded neural network to perform a next token prediction task. For convenience, process 600 will be described as being performed by one or more computer systems located at one or more locations. For example, a relevance prediction system, such as relevance prediction system 200 of FIG. 2, suitably programmed in accordance with this specification, can perform process 600.

[0129] The system accepts a set of training proteins (602).

[0130] The system performs steps 604-608 for each training protein. For convenience, steps 604-608 are described with reference to a particular training protein from the set of training proteins.

[0131] The system designates one or more positions within the amino acid sequence of the training protein as target positions 604. The system can select target positions, for example, randomly or according to predetermined selection rules.

[0132] The system uses an embedded neural network to generate a respective probability distribution over the set of possible amino acids for each target position in the amino acid sequence of the training protein (606). For each target position, the probability distribution generated for the target position depends only on the amino acids at the preceding positions in the amino acid sequence of the training protein. To generate the probability distributions for the target positions, the system can use the embedded neural network to generate an embedding for the target position and then process the embedding using one or more neural network layers to generate the probability distributions. In some implementations, the embedded neural network is a recurrent neural network, and to generate an embedding for the target position, the embedded neural network sequentially processes the amino acid embeddings corresponding to each position before the target position to generate an updated hidden state. The system can then designate the updated hidden state of the embedded neural network as the embedding for the target position.

[0133] For each target position in the amino acid sequence of the training protein, the system trains the embedded neural network to optimize an objective function that measures the error between (i) the probability distribution generated for the target position and (ii) the attributes of the amino acid identified at the target position in the training protein (608). The objective function can measure the error, for example, as a cross-entropy error.

[0134] 7 is a flow diagram of an exemplary process 700 for training an embedded neural network to perform a token unmasking task. For convenience, the process 700 will be described as being performed by one or more computer systems located at one or more locations. For example, a relevance prediction system, such as the relevance prediction system 200 of FIG. 2, suitably programmed in accordance with this specification, can perform the process 700.

[0135] The system accepts a set of training proteins (702).

[0136] The system performs steps 704-708 for each training protein. For convenience, steps 704-708 will be described with reference to a particular training protein from the set of training proteins.

[0137] The system generates a masked amino acid sequence by masking one or more positions in the amino acid sequence of the training protein (704). Masking a position in the amino acid sequence of the training protein may refer to replacing an amino acid embedding representing an attribute of the amino acid at that position with a predefined embedding. The predefined embedding may be, for example, a predetermined embedding, such as an embedding with each entry having a value of zero. The system may determine the positions to mask in the amino acid sequence, for example, by randomly selecting the positions to mask or by selecting the positions to mask according to a predefined rule.

[0138] The system processes the data defining the masked amino acid sequences using an embedded neural network to determine a respective probability distribution over the set of possible amino acids for each masked position in the amino acid sequence of the training protein (706). To generate the probability distributions for the masked positions, the system can generate an embedding of the masked position using the embedded neural network and then process the embedding using one or more neural network layers to generate the probability distribution. In some implementations, the embedded neural network includes one or more update blocks, each configured to process an input sequence of embeddings to generate an updated sequence of embeddings. In these implementations, to generate the embeddings for the masked positions, the embedded neural network can process the masked sequence of amino acids using a sequence of update blocks of the embedded neural network to generate an updated sequence of amino acid embeddings. Then, for each masked position, the system can designate the updated amino acid embedding of the masked position as the embedding that is processed to generate the probability distribution for the masked position.

[0139] For each masked position in the amino acid sequence of the training protein, the system trains an embedded neural network to optimize an objective function that measures the error between (i) the probability distribution generated for the masked position and (ii) the attributes of the amino acid identified at the masked position in the training protein (708). The objective function can measure the error, for example, as a cross-entropy error.

[0140] Figure 8 shows the accuracy achieved by the fitness prediction system described herein for the task of predicting a fitness score that characterizes the manufacturability of a capsid. The horizontal axis of the plot represents the experimental (ground truth) fitness score, and the vertical axis of the plot represents the predicted fitness score. It can be seen that the experimental and predicted fitness scores are highly correlated.

[0141] Figure 9 shows a comparison of the accuracy achieved by the relevance prediction system with and without the error-prediction machine learning model. More specifically, the left scale represents the prediction accuracy when the relevance prediction system does not use the error-prediction machine learning model, and the right scale represents the prediction accuracy when the relevance prediction system uses the error-prediction machine learning model. Accuracy is measured by the Pearson correlation coefficient between the experimental and predicted relevance scores.

[0142] A comparison of the prediction accuracy achieved by the relevance prediction system for different choices of model architecture for the relevance prediction machine learning model is shown in Figure 10. Accuracy is measured by the Pearson correlation coefficient between the experimental and predicted relevance scores.

[0143] Figure 11 shows a comparison of the prediction accuracy achieved by the fitness prediction system when no training data rebalancing was performed (left scale) and when training data rebalancing was performed (right scale). Accuracy is measured by the Pearson correlation coefficient between experimental and predicted fitness scores.

[0144] Figure 12 shows a comparison of the prediction accuracy achieved by the fitness prediction machine learning model when the embedding neural network generates a capsid monomer embedding with 1900 dimensions (left scale) versus when the embedding neural network generates a capsid monomer embedding with 256 dimensions (right scale). Accuracy is measured by the Pearson correlation coefficient between experimental and predicted fitness scores.

[0145] The term "configured" is used herein in connection with systems and computer program components. In the case of a system of one or more computers, being configured to perform a particular operation or action means that software, firmware, hardware, or a combination thereof is installed on the system that causes the operating system to perform that operation or action. One or more computer programs being configured to perform a particular operation or action means that one or more computer programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0146] Embodiments of the subject matter and functional operations described herein may be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware including the structures disclosed herein and their structural equivalents, or one or more combinations thereof. Embodiments of the subject matter described herein may also be implemented as one or more computer programs, i.e., one or more modules of computer program instructions coded on a tangible, non-transitory storage medium for execution by or control the operation of a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these. Alternatively or additionally, the program instructions may be coded in an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal generated to encode information for transmission to an appropriate receiving apparatus for execution by a data processing apparatus.

[0147] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, apparatus, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. A device may also be or further include special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, a device may optionally include code that forms the execution environment for a computer program, such as code comprising processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0148] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, may be written in any type of programming language, including a compiled or interpreted language, or a declarative or procedural language, and may be deployed in any form, including components, subroutines, or other units, suitable for use in a computing environment, as a standalone program or as a module. A program may, but need not, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data, for example, in one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple cooperating files, for example, files storing one or more modules, subprograms, or portions of code. A computer program may be deployed to be executed on one computer or on multiple computers, which may be located at one site or distributed across multiple sites and interconnected by a data communications network.

[0149] As used herein, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine is implemented as one or more software modules or components and installed on one or more computers located at one or more locations. In some cases, one or more computers are dedicated to a particular engine, while in other cases, multiple engines may be installed and operational on the same computer.

[0150] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, such as an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0151] A computer suitable for executing a computer program may be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a central processing unit, which executes or carries out instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory may be augmented by, or incorporated with, special-purpose logic circuitry. Typically, a computer will also include one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or be operatively coupled to receive data from or transfer data to them, or both. However, a computer need not have such devices. Furthermore, a computer may be incorporated in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name just a few.

[0152] Computer-readable media suitable for storing computer program instructions and data include all types of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0153] To facilitate user interaction, the subject matter described herein may be implemented in a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a pointing device, such as a keyboard and mouse or trackball, for allowing the user to provide input to the computer. Other types of devices may be used to facilitate user interaction as well; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, speech, or tactile input. A computer may also interact with a user by sending and receiving documents to and from a device used by the user, for example, by sending a web page to a web browser on the user device in response to a request received from the web browser. A computer may also interact with a user by sending text messages or other types of messages to a personal device, such as a smartphone running a messaging application, and then receiving a reply message from the user.

[0154] A data processing device implementing a machine learning model may also include, for example, a dedicated hardware accelerator unit to handle the common and computationally intensive part of machine learning training or production, i.e., inference, workload.

[0155] The machine learning model may be implemented and deployed using a machine learning framework, such as the TensorFlow framework.

[0156] Embodiments of the subject matter described herein may be implemented in a computing system that includes back-end elements, such as data servers, or middleware elements, such as application servers, or front-end elements, such as client computers with graphical user interfaces, web browsers, or apps through which users can interact with implementations of the subject matter described herein, or any combination of such back-end, middleware, or front-end elements. The elements of the system may be interconnected by any form or medium of digital data communication, such as a communications network. Examples of communications networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0157] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data, e.g., HTML pages, to a user device, e.g., for the purpose of displaying data to and receiving input from a user interacting with the device acting as a client. Data generated by the user device, e.g., a result of a user interaction, can be received by the server from the device.

[0158] While this specification contains many specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather as descriptions of features specific to particular embodiments of a particular invention. Features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while multiple features may be described as working in combination in a particular way and may be originally claimed as such, one or more features from a claimed combination may in some cases be excluded from the combination, and a claimed combination may refer to a subcombination or a variation of a subcombination.

[0159] Similarly, while operations may be illustrated in the figures and recited in the claims in a particular order, it should not be understood that such operations need to be performed in the particular order illustrated, or in sequential order, or that all of the operations illustrated need to be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program elements and systems described may generally be integrated together in a single software product or packaged in multiple software products.

[0160] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order to achieve desirable results. By way of example, the processes depicted in the accompanying figures need not necessarily be performed in the particular order shown, or in a sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. receiving data specifying the amino acid sequences of the monomers of the capsid; processing data specifying the amino acid sequences of the monomers of the capsid with an embedded neural network according to values ​​of a set of embedded neural network parameters to generate embeddings of the monomers of the capsid; processing the monomer embeddings of the capsid with a fitness predictive machine learning model to generate a fitness score that characterizes the predicted quality of the capsid; and outputting the fitness score characterizing the predicted quality of the capsid.

2. 10. The method of claim 1, further comprising processing the representation of the monomers of the capsid with an error-prediction machine learning model to generate a fitness error, wherein the fitness error is an estimate of an error in the fitness score generated by the fitness-prediction machine learning model.

3. 3. The method of claim 2, further comprising: updating the relevance score by combining (i) the relevance score generated by the relevance prediction machine learning model and (ii) the relevance error generated by the error prediction machine learning model.

4. 4. The method of claim 3, wherein updating the relevance score comprises summing the relevance score generated by the relevance prediction machine learning model and the relevance error generated by the error prediction machine learning model.

5. The method of any one of claims 2 to 4, wherein the relevance-prediction machine learning model has a different model architecture than the error-prediction machine learning model.

6. processing the representations of the monomers of the capsid with the error prediction machine learning model, 6. The method of any one of claims 2 to 5, comprising processing a representation of the amino acid sequences of the monomers of the capsid with an error prediction machine learning model.

7. The method of claim 6 , wherein the error-prediction machine learning model processes different model inputs than the relevance-prediction machine learning model.

8. processing the data defining the amino acid sequence of the capsid using the embedded neural network, instantiating each amino acid embedding corresponding to each position within the amino acid sequence of the monomer of the capsid; processing the amino acid embeddings through one or more neural network layers of the embedded neural network to update each amino acid embedding; generating embeddings of monomers of said capsid protein based on one or more updated amino acid embeddings; The method of claim 1 , comprising:

9. generating embeddings of monomers of the capsid protein based on the one or more updated amino acid embeddings, 9. The method of claim 8, comprising generating an embedding of a monomer of the capsid protein by combining the updated amino acid embeddings.

10. processing the amino acid embeddings through one or more neural network layers of the embedding neural network to update each amino acid embedding; 10. The method of any one of claims 8 to 9, comprising processing amino acid embeddings using one or more self-attention neural network layers of the embedding neural network.

11. processing the data defining the amino acid sequences of the monomers of the capsid with the embedded neural network, instantiating a sequence of amino acid fillers, the sequence including each amino acid filler corresponding to each position within the amino acid sequence of the monomer of the capsid; using a recurrent neural network to sequentially process each amino acid embedding in the sequence of amino acid embeddings to generate updated hidden states of the recurrent neural network; determining the embedding of the monomers of the capsid based on an updated hidden state of the recurrent neural network; The method according to any one of claims 1 to 7, comprising:

12. The method of any one of claims 1 to 11, wherein the embedded neural network is trained to perform a token unmasking task on a set of training proteins.

13. training the embedded neural network to perform the token unmasking task includes, for each training protein: generating a masked amino acid sequence by masking one or more positions within the amino acid sequence of the training protein; processing data defining the masked amino acid sequences with the embedded neural network to determine, for each masked position within the amino acid sequences of the training proteins, a respective probability distribution over a set of possible amino acids; training an embedded neural network to optimize, for each masked position in the amino acid sequences of the training proteins, an objective function that measures the error between (i) the probability distribution generated for the masked position and (ii) the attribute of the amino acid specified at the masked position in the training protein; The method of claim 12, comprising:

14. The method of any one of claims 1 to 11, wherein the embedded neural network is trained on a set of training proteins to perform a next token prediction task.

15. training the embedded neural network to perform the next token prediction task includes, for each training protein: For each of one or more target positions within the amino acid sequence of the training protein, the embedded neural network is used to generate a respective probability distribution over the set of possible amino acids: for each target position, generating the probability distribution for that target position such that the probability distribution depends only on the amino acid at the preceding position in the amino acid sequence of the training protein; For each target position in the amino acid sequence of the training protein, training an embedded neural network to optimize (i) a probability distribution generated for the target position and (ii) an objective function that measures the attributes of the amino acid identified at the target position in the training protein.

15. The method of claim 14, comprising:

16. 16. The method of any one of claims 1 to 15, wherein the fitness-prediction neural network is trained on a set of training examples, each training example corresponding to a respective training monomer of a capsid, and including (i) an embedding of the training monomer of the capsid, and (ii) a target fitness score for the training monomer of the capsid.

17. 17. The method of claim 16, wherein training the fitness-prediction neural network on the set of training examples comprises rebalancing the set of training examples based on the target fitness scores specified by the training examples.

18. The method of any one of claims 1 to 17, wherein the fitness predictive machine learning model is a parametric machine learning model.

19. 20. The method of claim 18, wherein the fitness-predicting machine learning model comprises one or more of a neural network model, a random forest model, or a support vector regression model.

20. The method of any one of claims 1 to 17, wherein the fitness predictive machine learning model is a non-parametric model.

21. The method of claim 20 , wherein the relevance predictive machine learning model comprises a k-nearest neighbor model.

22. 22. The method of any one of claims 1 to 21, wherein the predicted quality of the capsid characterizes the predicted manufacturability of the capsid.

23. 23. The method of claim 22, wherein the manufacturability of the capsid is defined based on the number of instances of the capsid produced when a plasmid coding for the capsid is transfected into one or more cells.

24. 24. The method of claim 23, wherein the number of instances of the capsid produced when a plasmid encoding the capsid is transfected into one or more cells is normalized by a number of generated wild-type sequences.

25. 22. The method of any one of claims 1 to 21, wherein the predicted quality of the capsid characterizes the predicted ability to evade neutralization of a virus containing the capsid.

26. The method of any one of claims 1 to 21, wherein the predicted quality of the capsid characterizes the predicted immunoreactivity of the capsid.

27. 22. The method of any one of claims 1 to 21, wherein the predicted quality of the capsid characterizes the predicted ability of a virus containing the capsid to penetrate a target tissue.

28. 22. The method of any one of claims 1 to 21, wherein the predicted quality of the capsid characterizes the packing capacity of the capsid.

29. 22. The method of any one of claims 1 to 21, wherein the predicted quality of the capsid characterizes the predicted ability of a virus containing the capsid to integrate into a host genome.

30. 30. The method of any one of claims 1 to 29, wherein the capsid corresponds to the capsid of an adeno-associated virus (AAV).

31. selecting a capsid monomer from the set of possible capsid monomers using the fitness predictive machine learning model; 31. The method of any one of claims 1 to 30, further comprising the step of physically producing one or more viruses comprising proteins having said selected capsid protein monomers.

32. selecting a capsid monomer from a set of possible capsid monomers using the fitness predictive machine learning model, generating a respective fitness score for each capsid monomer in the set of possible capsid monomers using the fitness predictive machine learning model; 32. The method of claim 31, comprising selecting capsid monomers based on the fitness score.

33. selecting capsid monomers based on the fitness scores, 33. The method of claim 32, comprising selecting the capsid monomer from the set of possible capsid monomers that has the highest fitness score.

34. 34. The method of any one of claims 31 to 33, further comprising the step of administering the produced virus to a subject to achieve a therapeutic effect in the subject.

35. one or more computers; one or more storage devices communicatively coupled to the one or more computers; A system comprising: A system in which the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of each of the methods described in any one of claims 1 to 34.

36. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform each of the methods of any one of claims 1 to 34.

Images