Designing de novo antibody structures by repurposing esmfold as a probabilistic denoising diffusion model

Abstract

An antigen-aware antibody folding computing system includes a processor and a memory having stored thereon a trained machine-learned model and instructions that, when executed, cause the system to receive input biomolecule digital representations and process the input using the model to generate predicted output biomolecule digital representations. A method includes receiving input biomolecule digital representations and processing the input using a trained machine-learned model to generate predicted output biomolecule digital representations. A non-transitory computer-readable storage medium includes a trained machine-learned model and instructions that, when executed, cause a computer to: receive input biomolecule digital representations and process the input using the machine-learned model to generate predicted output biomolecule digital representations.

Description

DESIGNING DE NOVO ANTIBODY STRUCTURES BY REPURPOSING ESMFOLD AS A PROBABILISTIC DENOISING DIFFUSION MODELFIELD OF THE DISCLOSURE[1] The present disclosure generally relates to predicting their structure and / or sequence biomolecules, including, based on associated biomolecules, using machine learning techniques, and more particularly, predicting a structure and / or sequence of an antibody based on an antigen by using machine learning techniques.BACKGROUND[2] Antibodies are diverse proteins used by the immune system to naturally bind and neutralize foreign objects, such as viruses, fungi, and bacteria. Accordingly, antibodies have the potential to serve as drug candidates for many infectious diseases and cancers. Conventional techniques for automatic design, discovery, and / or improvement of antibodies are lacking, for example, because such techniques design proteins unconditionally, use only open-source data, and do not enable targeted design to a specified binding partner biomolecule, or predicting the structure of the antibody. Such techniques have also not demonstrated the ability to condition design on other criteria / aims.[3] Conventional laboratory-based techniques are time-consuming and expensive. For example, traditional protein structure discovery requires time, and methods such as X-ray crystallography, nuclear magnetic resonance techniques and cryo-electron microscopy are resource intensive. These methods also offer little control over the conditional output structures to inform corresponding sequences, which can result in lead candidates with sub-optimal binding and poor developability attributes. Conventional generative antibody design and prediction techniques have not demonstrated de novo antibody design or structure prediction. Thus, improved techniques for protein design and prediction are needed.[4] Antibodies in particular are a growing class of therapeutic molecules due to their attractive drug-like properties, including high target selectivity and minimal immunogenic effects. Antibody drug development commonly begins with initial lead molecule discovery. Existing approaches for lead discovery typically consist of searching through a massive combinatorial sequence space by screening large libraries of antibodyvariants against a target antigen. Techniques such as phage, yeast display, immunization coupled with hybridoma screening or B-cell sequencing are typically employed for initial discovery, followed by further molecule development. These methods are time and resource intensive, lack control over the properties of the designed antibodies, often produce sub-optimal leads, and require further experimentation for validation methods, such as by antibody structure validation.[5] The application of artificial intelligence (Al) methods to antibody design, antibody structure prediction, and more generally, protein therapeutic design, is compelling given the availability of large protein sequence and structure databases that may fuel model training. Indeed, recent work has shown that models trained on these data may be used for de novo design of certain classes of proteins, by screening dozens to thousands of protein designs.[6] However, no conventional techniques have achieved antibody structure prediction, let alone antigen-aware antibody structure prediction, despite the fact that antibody-based therapeutics are immensely therapeutically relevant, having accounted for nearly a third of FDA-approved biologies in 2022. Antigen-aware antibody structure prediction is of particular interest as the key determinants of antibody function emerge from the complementary determining regions (CDRs) of the sequence. These hypervariable regions interact directly with the antigen and among these, heavy chain CDR3 (HCDR3) is often most critical to binding but also most variable, making it particularly challenging to model. Further, CDRs fold into beta-strands, loose loops with undefined structures existing as disordered regions of the protein, adding to the difficulty of predicting the structure of an antibody.[7] A particular challenge of designing de novo antibodies and / or predicting antibody structures with therapeutic or scientific relevance is predicting how a linear amino acid sequence (primary protein structure) may fold with other chains of amino acid sequences into one quaternary protein structure wherein some regions remain as secondary protein structures (the beta-strands of CDRs) until conforming to an antigen and establishing a defined structure.[8] In general, conventional techniques have difficulty with the multiple chains and predicting accurate structure when loose beta stranded CDRs have undefined structure,conventional techniques. For example, ML models may use multiple sequence alignments (MSA) to predict quaternary protein structures, however antibody MSA databanks are not as commonly available. Additionally, MSAs infer the evolutionary history of a protein by analyzing the conservation and variation of amino acid sequences across related species, whereas antibodies are often developed by a body in response to exposure to an antigen, a process which may vary from individual to individual and require several steps of evolutionary mutations. Thus, antibody MSAs may not be as beneficial to protein prediction as MSAs of other proteins, and if useful, may be prohibitively accessible. Furthermore, MSAs require significantly more processing power compared to single sequence methods. However, conventional techniques of ML models trained for single sequences are not trained to predict structures with multiple chains, or structures with undefined regions unless in a binding state.[9] Another challenge is lack of antibody structure training data. Antibodies evolve in the human body to match specific and unique targets possibly unique to the individual, and this contributes to lack of data and need for design method steps. Further, conventional techniques relying on training from known data / biophysical properties of biomolecules may not capture de novo structures.

[0010] Thus, improved techniques and platforms for training and operating generative Al models to perform protein design and prediction are needed.BRIEF SUMMARY

[0011] In one aspect, a computing system for performing antigen-aware antibody folding includes one or more processors; and one or more memories having stored thereon a machine-learned model. The machine-learned model may be trained to predict an output biomolecule digital representation corresponding to an output biomolecule using one or more training inputs including one or more training biomolecule digital representations each corresponding to a respective one of a plurality of training biomolecules. The memories may store one or more sets of computer-executable instructions that, when executed by the one or more processors, cause the computing system to: (i) receive a plurality of input biomolecule digital representations each corresponding to a respective one of a plurality of input biomolecules; and (ii) processone or more of the plurality of input biomolecule digital representations using the machine-learned model to generate one or more predicted output biomolecule digital representations.

[0012] In another aspect, a computer-implemented method for performing antigen- aware antibody folding may include (i) receiving a plurality of input biomolecule digital representations each corresponding to a respective one of a plurality of input biomolecules; and (ii) processing one or more of the plurality of input biomolecule digital representations using a machine-learned model to generate one or more predicted output biomolecule digital representations, the machine-learned model trained to predict an output biomolecule digital representation corresponding to an output biomolecule using one or more training inputs including one or more training biomolecule digital representations each corresponding to a respective one of a plurality of training biomolecules.

[0013] In yet another aspect, a non-transitory computer-readable storage medium may have stored thereon a machine-learned model trained to predict an output biomolecule digital representation corresponding to an output biomolecule using one or more training inputs including one or more training biomolecule digital representations each corresponding to a respective one of a plurality of training biomolecules. The storage medium may have stored thereon instructions that, when executed by one or more processors, cause a computer to: (i) receive a plurality of input biomolecule digital representations each corresponding to a respective one of a plurality of input biomolecules; and (ii) process one or more of the pluralities of input biomolecule digital representations using the machine-learned model to generate one or more predicted output biomolecule digital representations.BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The figures described below depict various aspects of the applications, methods, and systems disclosed herein. It should be understood that each figure depicts one embodiment of a particular aspect of the disclosed applications, systems and methods, and that each of the figures is intended to accord with a possible embodiment thereof. Furthermore, wherever possible, the following description refers to the referencenumerals included in the following figures, in which features depicted in multiple figures are designated with consistent reference numerals.

[0015] FIG. 1 depicts an exemplary computing environment for training and / or operating machine learning models to perform antigen-aware antibody folding and prediction, according to some aspects.

[0016] FIG. 2 depicts a block flow diagram of an exemplary denoising diffusion probabilistic module, according to some aspects.

[0017] FIG. 2 depicts an exemplary computer-implemented method for performing antigen-aware antibody folding, according to some aspects.

[0018] FIG. 3 depicts an exemplary computer-implemented method for performing antigen-aware antibody folding using a masked autoencoder, according to some aspects.

[0019] Advantages will become more apparent to those skilled in the art from the following description of the preferred embodiments which have been shown and described by way of illustration. As will be realized, the present embodiments may be capable of other and different embodiments, and their details are capable of modification in various respects. Accordingly, the drawings and description are to be regarded as illustrative in nature and not as restrictive.DETAILED DESCRIPTION

[0020] The present disclosure addresses the need for artificial intelligence (Al) and machine learning (ML) models trained to predict biomolecules using known biomolecule / binding partner complexes. In particular, generative Al has the potential to greatly increase the speed, quality and controllability of biomolecule (e.g., antibody) design and structure prediction by augmenting models with awareness of target biomolecule / binding partner complexes (e.g., antigens).

[0021] The present techniques may include predicting the structure of a biomolecule from one or more of: an amino acid sequence of the biomolecule, a sequence of a binding-partner biomolecule which binds to the biomolecule, or a structure of the binding-partner biomolecule. By predicting how the biomolecule sequence may fold into a biomolecule structure, the present techniques may be applied downstream andupstream of drug discovery as biomolecule design and / or biomolecule validation. For example, an optimized model may output sequences by determining how a sequence may fold based on desired specifications of the resultant structure e.g., for the structure to be nonimmunogenic, robust to varying pH, more compact, and / or be a particular size, etc. Accordingly, techniques of iterative biomolecule structure prediction may design a de novo biomolecule.

[0022] Similarly, the sequence or structure of an efficacious and proprietary biomolecule may not be available to the public. Biomolecule sequence folding techniques may allow for diversification and further discovery of drugs (biomolecules) sharing a similar structure but a different and non-proprietary sequence. Additionally, or alternatively, downstream drug discovery may utilize biomolecule structure prediction to perform in- silico validation of an Al-designed biomolecule sequence, for example, by comparing a reference structure for the sequence to a biomolecule structure predicted by the present techniques.

[0023] The present techniques include improvements in computer functionality or in improvements to other technologies at least because the disclosure describes that, e.g., a hosting server (e.g., computing system, external server, etc.), or otherwise computing device (e.g., a user computing device), is improved where the intelligence or predictive ability of the hosting server or computing device is enhanced by a trained machine- learned model. This model, executing on the computing system or computing device, is able to accurately and efficiently predict biomolecules, including how a linear sequence of amino acids may fold and what amino acid sequence(s) may fold into a biomolecule compatible with binding to specific partner-biomolecule(s). That is, the present disclosure describes advantageous improvements in the functioning of the computer itself or any other technology or technical field because a hosting server or user computing device, is enhanced with a trained machine-learned model to accurately interpret input biomolecule digital representations corresponding to biomolecule sequences and binding-partner biomolecules, and output predicted biomolecule digital representations corresponding to biomolecules. This is an improvement over conventional techniques by providing control and / or predictive functionality and accurate prediction of the biomolecule structures (e.g., antibody structures). Additionally, these improvements specifically address the challenges of predicting antibody folding andantibody design, among other proteins / biomolecules with comparable properties (e.g., quaternary structure, light chains and heavy chains, sections of undefined structure before binding to binding-partner, etc.). And further, the predictions of the present techniques are aware of antigen context, which improves upon conventional techniques that do not consider binding partner information, resulting in richer data models with more predictive power and capability for modeling real world processes.

[0024] Specifically, the present techniques may include computer-implemented machine learning training and operation. Thus, in these instances, the techniques of the present disclosure may further include improvements in computer functionality or in improvements to other technologies at least because the present techniques include training models with a plurality of training data (e.g., training biomolecule digital representations corresponding to training biomolecule sequences and structures and binding-partner biomolecule sequences and structures, etc.) to output the predicted biomolecule structure, e.g., an antigen-aware antibody. The present techniques further describe improvements to computer architecture that provide solutions to the challenges of biomolecule (e.g., protein, antibody) structure prediction and design, such as for biomolecules with regions of undefined structure(s) before binding to a binding-partner biomolecule and / or biomolecules composed of more than one chain of amino acids. Further, the present techniques enable parallelization across multiple computational instances, improving computer architecture for the present techniques.

[0025] Moreover, the present disclosure may include effecting a transformation or reduction of a particular article to a different state or thing, e.g., transforming or reducing biomolecule design and / or biomolecule structure prediction from a non-optimal or error state to an optimal state. Applying generative artificial intelligence to predict antigen- aware antibody structures may be implemented in the drug discovery process to drastically reduce the time and resources necessary for therapeutic antibody development, thereby transforming or reducing biomolecule design and / or structure prediction from a non-optimal error state to an optimal state.

[0026] Still further, the present disclosure includes specific features other than what is well-understood, routine, conventional activity in the field, or adding unconventional steps that demonstrate, in various embodiments, particular useful applications, e.g.,initializing backbone frames of an unknown biomolecule structure according to spatial coordinates of binding-partner biomolecule backbone frames to achieve equivariance compared to the conventional invariance achieved by initializing all backbone frames of an unknown biomolecule structure to a single point and incepting the structure solely from the biomolecule sequence.Exemplary Computing Environment

[0027] FIG. 1 depicts an exemplary computing environment 100 for training and / or operating machine learning (ML) models to perform antigen-aware antibody folding and prediction, according to some aspects.

[0028] The environment 100 may include a computing system 120, a data repository 130, an external server 140, a client computing device 160, a sequencing device 170, and an electronic network 150. In some aspects, the environment 100 may include a plurality of computing systems 120, a plurality of data repositories 130, a plurality of external servers 140, a plurality of networks 150, a plurality of client computing devices 160, and / or a plurality of sequencing devices 170. The computing system 120 includes a processor 122, a network interface controller (NIC) 124 and a memory 126.

[0029] The computing system 120 may further include or be communicatively coupled to a data repository 130. The data repository 130 may be a structured query language (SQL) database (e.g., a MySQL database, an Oracle database, etc.) or another type of database (e.g., a non-SQL (NoSQL) database, a flat file database, etc.). The computing system 120 may include a library of client bindings for accessing the data repository130. The library may be stored as a set of computer-executable instructions in the memory 120, for example. The data repository 130 may be located remote from the computing system 120. For example, the data repository 130 may be implemented using a RESTdb.lO database, an Amazon Relational Database Service (RDS), etc. in some aspects. The data repository 130 may include file system (e.g., an EXT file system, Apple file system (APFS), a networked filesystem (NFS), local filesystem, etc.), an object store (e.g., Amazon Web Services S3), a data lake, etc. The data repository 130 may include a plurality of data types, such as pre-training data sourced from public data sources (e.g., SAbDab data, PDB data, DIPS data), pre-training data, and fine- tuning data. The data repository 130 may include machine learning model training data(training inputs) represented in any suitable data format(s), such as protein data bank (PDB) format, JavaScript Object Notation (JSON) format, extensible Markup Language (XML) format, etc. In some aspects, the data repository 130 may store 100 GB or more of training inputs for training, pre-training, and / or fine-tuning.

[0030] In some aspects, the computing system 120 may include client-server platform technology such as Python, PHP, ASP.NET, Java J2EE, Ruby on Rails, Node.js, a web service or online API, responsive for receiving and responding to electronic requests. Further, the computing system 120 may include sets of instructions (e.g., software) for performing machine learning operations, as discussed below, that may be integrated with the client-server platform technology.

[0031] The processor 122 may include any suitable number of processors and / or processor types, such as one or more graphics processing units (GPUs), one or more central processing units (CPUs), etc. Generally, the processor 122 may be configured to execute software instructions stored in the memory 126. The memory 126 may include one or more persistent memories (e.g., a hard drive / solid state memory) and store one or more set of computer executable instructions / modules 128, including an input / output (I / O) module 130, a machine learning operation module 134, and a machine learning training module 134. In some aspects, more or fewer modules may be included, and in some aspects, one or more of the models may be combined or aggregated into a fewer number of modules.

[0032] Each of the modules 128 implements specific functionality related to the present techniques, as will be described further, below. The modules 128 may store machine readable instructions, including one or more application(s), one or more software component(s), and / or one or more APIs, which may be implemented to facilitate or perform the features, functions, or other disclosure described herein, such as any methods, processes, elements or limitations, as illustrated, depicted, or described for the various flowcharts, illustrations, diagrams, figures, and / or other disclosure herein. In some aspects, a plurality of modules 128 may act in concert to implement a particular technique. For example, the machine learning operation module 134 may load information from one or more other models prior to, during and / or after initiating an inference operation. Thus, the modules 128 may exchange data via suitabletechniques, e.g., via inter-process communication (IPC), a Representational State Transfer (REST) API, etc. within a single computing device, such as the computing system 120. In some aspects one or more modules 128 may be implemented in a plurality of computing devices (e.g., a plurality of computing systems 120, client computing device 160). The modules 128 may exchange data among the plurality of computing devices via a network such as the network 150. The modules 128 of FIG. 1 will now be described in greater detail.

[0033] Generally, the I / O module 130 includes instructions that enable a user (e.g., an employee of the entity or company) to access and operate the computing system 120 (e.g., via the client computing device 160). For example, the employee may be a software developer who trains one or more ML models using the ML training module132 in preparation for using the one or more trained ML models to generate outputs used in an antibody prediction project, an antibody structure prediction project, and / or an antibody design project. Once the one or more ML models are trained, the same user (or another) may access the computing system 120 via the I / O module to cause the biomolecule prediction process to be initiated. The I / O module 130 may include instructions for generating one or more graphical user interfaces (GUIs) (not depicted) that collect and store parameters related to biomolecule prediction, such as a user selection of a particular protein database, biomolecule, binding partner, etc. from a list stored in the data repository 130.

[0034] In general, a computer program or computer based product, application, or code (e.g., the model(s), such as machine learning models, or other computing instructions described herein) may be stored on a computer usable storage medium, or tangible, non-transitory computer-readable medium (e.g., standard random access memory (RAM), an optical disc, a universal serial bus (USB) drive, or the like) having such computer-readable program code or computer instructions embodied therein, wherein the computer-readable program code or computer instructions may be installed on or otherwise adapted to be executed by the processor(s) 122 (e.g., working in connection with the respective operating system in memory 126) to facilitate, implement, or perform the machine readable instructions, methods, processes, elements or limitations, as illustrated, depicted, or described for the various flowcharts, illustrations, diagrams, figures, and / or other disclosure herein. In this regard, the program code may beimplemented in any desired program language, and may be implemented as machine code, assembly code, byte code, interpretable source code or the like (e.g., via Golang, Python, C, C++, C#, Objective-C, Java, Scala, ActionScript, JavaScript, HTML, CSS, XML, etc.).

[0035] In some aspects, the ML training module 132 may include a set of computerexecutable instructions implementing machine learning training, configuration, parameterization and / or storage functionality. The ML training module 132 may initialize, train and / or store one or more ML models, as discussed herein. The trained ML models and their weights / parameters may be stored in the data repository 130 and / or in the memory 122, which is accessible or otherwise communicatively coupled to the computing system 120.

[0036] For example, the ML training module 132 may train one or more models (e.g., one or more generative models) to automatically design and / or fold antibodies from an amino acid sequence while considering a specific binding-partner biomolecule of interest (e.g., a target biomolecule, antigen). The ML training module 132 may train the models on biomolecule sequences and / or structures (e.g., antigen structures, antibody sequences, antibody-antigen structures, etc.) and / or on a large collection of extant biomolecule sequences and / or structures (e.g., 50,000 proteins). The ML training module 132 may include instructions for training, pre-training, and / or fine-tuning the models using digital representations of biomolecule sequences and / or structures (e.g., antigen structures, antibody sequences, antibody-antigen structures, etc.) and / or a large collection of extant biomolecule sequences and / or structures, as discussed further herein.

[0037] The ML training module 132 may train and / or pre-train the models using proprietary biomolecule data (e.g., pseudo-CDR datasets). The ML training module 132 may include instructions for validating the models, in some aspects, by designing an antibody that binds to an antigen and comparing the antibody to other antibodies capable of binding the antigen.

[0038] In some aspects, the ML training module 132 may validate the models by comparing an antibody structure the predictions generated by the models (e.g., by using the antibody’s sequence and the structure of a compatible antigen) to the actualstructure of the folded antibody derived using other techniques (e.g., crystallography, NMR, Cryo-EM, etc.).

[0039] The present techniques may enable predicting the structure of and / or designing of the complementarity-determining regions (CDRs) of antibodies, in some aspects, which confer binding specificity to a target (e.g., an antibody or antigen), and in some aspects, predicting how multiple amino acid sequences that compose an antibody may fold to confer binding specificity to the target. In some aspects, the ML training module 132 may train and / or pre-train a model on a collection of biomolecules and / or biomolecule-binding partner biomolecule complexes (e.g., antibody-antigen complexes) mined from public data sources (e.g., the Protein Data Bank) or proprietary data sets. For example, total training inputs in such circumstances may be on the order of 10,000- 50,000 biomolecules or more.

[0040] In some aspects, the ML training module 132 may include computer-executable instructions for performing ML model pre-training and / or ML model fine-tuning. Model pre-training may be known as transfer learning and may enable training of a base model that is universal, in the sense that it can be used as a common grammar for all biomolecule sequences or structures, for example. The ML training module 132 may pre-train one or more models to learn common biological concepts, to capture general features, patterns, and representations from the data, and then further train those models for more specific tasks. It should be appreciated that alternatives to fine-tuning are also envisioned. These may include using ensembles of lighter-weight models, zero-shot learning, few-shot learning, data augmentation, etc.

[0041] In some examples, the ML training module 132 may pre-train multiple models (e.g., a large language model, a masked autoencoder, a denoising diffusion probabilistic model, etc.). In some aspects, the ML training module 132 pre-training may include training a model on training inputs corresponding to many (e.g., 50,000 training biomolecules or more).

[0042] The term “pre-training” may be used to describe scenarios wherein a second training may occur (i.e., when the model may be “fine-tuned”). Transfer learning refers to the ability of the model to leverage the result (weights) of a first pre-training to better initialize the second training, which otherwise requires a random initialization. Thesecond training, i.e., fine-tuning, may be performed using training inputs such as training biomolecule digital representations as discussed herein. In general, fine-tuning adapts a pre-trained model to a specific target task by further training on a smaller taskspecific dataset. During fine-tuning, the ML training module 132 may update only a subset of the model layers, while some layers may remain frozen. This prevents the model from forgetting its general knowledge gained during pre-training

[0043] The technique of combining pre-training and fine-tuning advantageously boosts performance, in that the result of the training-on-training biomolecule digital representations performs better after pre-training (e.g., using natural antibody structures from SAbDab) than when no pre-training is performed. It is also much faster to finetune a pre-trained model to produce a plurality of trained models, as opposed to training each one individually. However, transfer learning is not strictly necessary.

[0044] The ML training module 132 may include instructions for fine-tuning a model using fine tuning inputs that include antibody-antigen pairs, in some aspects. In some aspects, ML model fine-tuning may be performed to endow the model with an understanding of antibody and / or antigen grammar. In some aspects, the ML training module 132 may fine-tune a model on many (e.g., 4,000 or more) antibodies.

[0045] The ML training module 132 may train one or more ML models (e.g., a pretrained generative model, a large language model (LLM), a masked autoencoder model, a denoising diffusion probabilistic model, etc.). The ML training module 132 may access one or more training data sets (e.g., training inputs) for model training, as discussed herein. The ML training module 132 may include instructions for retrieving training data from the data repository 130, for example, via an SQL query.

[0046] In some aspects, the ML training module 132 may train a model to predict a structure of a biomolecule (e.g., an antibody) based on received inputs including (i) an input biomolecule digital representation including information of a binding-partner biomolecule structure, (ii) an input biomolecule digital representation including information of a sequence of the binding-partner biomolecule, and (iii) an input biomolecule digital representation including information of a sequence of a biomolecule.

[0047] As used herein, the term “digital representation” may refer to a machine-readable representation of a corresponding biomolecule (e.g., a binding-partner biomolecule, anantibody, an antigen, etc.), or a fragment or portion thereof. The digital representations herein may include information used to represent biomolecules, or fragment or portions thereof, in a digital environment (e.g., a digital file format common to proprietary or open source-licensed biomolecule modeling software).

[0048] The digital representations may include information that digitally represents a sequence and / or structure of a biomolecule. The digital representations may include information to digitally represent a sequence and / or structure of a fragment or portion of a biomolecule, such as a region (e.g., CDR, epitope, paratope, binding site interface, heavy chain CDR 1 , antigen, etc.). This information may be included in digital representations and / or in any other suitable format now known or later developed. In some aspects, a digital representation of a biomolecule may include a digital representation of fragment or portion(s) of the biomolecule. For example, an input biomolecule digital representation may include a digital representation of a complementary determining region (CDR).

[0049] As used herein, the term “biomolecule” may generally refer to an entirety of a biomolecule, a fragment or portion of a biomolecule, an entirety of a biomolecule complex or a fragment or portion of a biomolecule complex. For example, an antigenantibody complex may be digitally represented as a biomolecule digital representation. This antigen-antibody complex may include a digital representation of an antibody and / or fragment thereof, and a digital representation of an antigen and / or a fragment or portion thereof. In some aspects, a biomolecule digital representation (e.g., a training biomolecule digital representation or input biomolecule digital representation) may include a digital representation of a concatenated antibody and antigen. Digital representations of biomolecules may be generated by a digital representation module of the modules 128 (not depicted). In some aspects, the digital representation module may load existing digital representations into the environment 100 (e.g., via the network 150, the data repository 130, and / or the client computing device 160).

[0050] In some aspects, digital representations may correspond to biomolecule sequence(s) and binding-partner biomolecule structure(s) and / or sequence(s). In some aspects, biomolecules, biomolecule sequences, and / or fragment or portions thereof may be represented digitally according to various file formats such as (i) a protein databank (PDB) data format, (ii) a JSON data format, (iii) an XML data format, or (iv) XYZ format, for example. Digital representations may be opened in programs such as PyMol or NGLView, for example.

[0051] In some aspects, a digital representation may include one or more strings encoding sequence and / or structure of a biomolecule. For example, an input biomolecule digital representation may include a string encoding an amino acid sequence of a corresponding input biomolecule. Additionally, or alternatively, the input biomolecule digital representation may include a string encoding positional coordinates of atoms in a three-dimensional (3D) space, the atoms and respective coordinates forming a digital representation of a structure of the corresponding input biomolecule (e.g., an antigen). In some aspects, a digital representation may include an array or other data structure(s) encoding the sequence and / or structure of a biomolecule. For example, a digital representation may be featurized to enrich data with node features and pair features, as discussed herein. In this example, a sequence of a biomolecule may be digitally represented by one or more node features concatenated into an NxN- channel array or matrix. In some aspects, a digital representation may include additional / different data structures besides those explicitly described herein which may sufficiently represent a biomolecule sequence and / or structure to be utilized by the methods herein.

[0052] In operation, the ML training module 132 may train one or more machine learning models. The ML training module 132 may process edge features and node features during training.

[0053] In general, node features may refer to respective attributes of each node in a graph. For example, each node may correspond to a biomolecule, and the node features may correspond to attributes of the biomolecule. In general, pair features (or edge features) may refer to respective attributes of edges between nodes in the graph. Edge features may encode relationships between nodes. In some aspects, the digital representations may include features (e.g., node feature(s), pair feature(s)) that enable the present techniques to enable the ML training module 132 to process and propagate information throughout one or more graph neural networks (GNNs) during training and operation. For example, digital representations may include graph-structured datadigitally representing a biomolecule as a graph with node feature(s) and pair feature(s) (e.g., edge features).

[0054] For example, the ML training module 132 may aggregate information from neighboring nodes and the respective features of those nodes, and propagate information among the nodes. The ML training module 132 may also process pair features during this process, wherein the processing affects the propagation of information between nodes. The node features and / or pair features may be represented as tensors, in some aspects.

[0055] In some aspects, the ML training module 132 may process a digital representation (e.g., an input biomolecule digital representation, a training biomolecule digital representation, etc.) that includes at least one respective node feature corresponding to a biomolecule and at least one pair feature. Specifically, the node feature(s) may include a residue-level representation of each amino acid of the respective biomolecule being digitally represented, and the pair feature may include a pair-level digital representation of at least two amino acid residues. In some aspects, the respective node feature(s) include a digital representation of one or more of (i) an amino acid sequence, (ii) one or more backbone dihedrals, and (iii) a binding interface flag. In some aspects, backbone dihedrals correspond to the dihedral angles formed by the atoms within a single amino acid residue. In some aspects, backbone dihedrals may correspond to the dihedral angles formed by two amino acid residues. The backbone dihedrals may correspond to dihedral angles formed by atoms within a single amino acid residue. For example, a node feature for an atom (object) may encode for three dihedral angles corresponding to three coordinates of a backbone (e.g., alpha carbon atom, carbon atom, and nitrogen atom of the backbone) in a local coordinate frame of the atom, as discussed in “Iterative Refinement Graph Neural Network for Antibody Sequence-Structure Co-design,” Jin et aL, arXiv:21 10.04624 [q-bio.BM], hereby incorporated by reference in its entirety for all purposes.

[0056] In some aspects, the ML training module 132 may process data including a binding interface flag (e.g., Boolean flag) indicating whether the respective amino acid residue is part of the binding interface. This flag may be included in data sets used for training and / or inference. In some aspects, the binding interface includes an epitope ofan antigen. In some aspects, the binding interface includes a superset of amino acid residues involved in the binding of two biomolecules (e.g., an antigen and antibody, two antibodies). .

[0057] In some aspects, the two amino acid residues of a pair-level digital representation may correspond to the same respective biomolecule that the respective node feature(s) is digitally representing. In some aspects, the two amino acid residues may correspond to different biomolecules, wherein only one amino acid residue corresponds to the biomolecule which the respective node feature(s) is digitally representing. For example, a trained ML model may receive an input biomolecule digital representation corresponding to an antigen and receive an input biomolecule digital representation corresponding to an antibody. The data may be featurized to include node feature(s) including residue-level digital representations of each amino acid residue of the antigen and the antibody. The data may be featurized to include pair feature(s) including pair-level digital representations of (i) each amino acid residue of the antibody paired with each other amino acid residue of the antibody, (ii) each amino acid of the antibody with each amino acid residue of the antigen, (iii) each amino acid residue of the antigen paired with each amino acid residue of the antibody and / or (iv) each amino acid residue of the antigen paired with each other amino acid residue of the antigen.

[0058] In some aspects, the respective pair features may include a digital representation of one or more of: (i) a relative distance, (ii) a relative orientation and (iii) a relative chain. In some aspects, a relative distance may be the relative distance between two amino acid residues in 3D space. In some aspects, the relative distance may be the relative distance between an atom of an amino acid residue backbone and another atom of a different amino acid residue backbone. For example, four atoms are included in the backbone of an amino acid residue and the distance between each backbone atom of the residue and each backbone atom of a different residue may be determined for the relative distance to be encoded in the respective pair feature(s). In some aspects, the relative orientation may be the relative rotation of one amino acid residue of a pair of residues in its local coordinate frame in 3D space to the other amino acid residue of the pair in its local coordinate frame. For example, the relative orientation encoding of a pair feature may be the quaternion representation of a spatial rotationmatrix based on an orientation matrix representing the local coordinate frame of the residue, as discussed in Jin et aL, supra.

[0059] In some aspects, the relative chain may be positional information encoding differences between the two amino acid residues of a pair. For example, the relative chain may encode whether the two amino acid residues are of the same chain (e.g., heavy chain of an antibody, light chain of an antibody) and / or of the same biomolecule (e.g., antigen, antibody). In some aspects each amino acid residue may be indexed. Node feature(s) and / or pair feature(s) may utilize residue indices when encoding information to differentiate residues. The computing system 120 may additionally, or alternatively, utilize residue indices when featurizing unfeaturized biomolecule digital representations.

[0060] In some aspects, the respective pair feature may further include a digital representation of one or more of (iv) a relative contact flag. In some aspects, a relative contact flag may be a flag (e.g., Boolean) indicating whether two residues are in contact with each other. For example, when an amino acid residue chain folds, amino acid residues may become adjacent to other amino acid residues (not including residues in direct contact via peptide bond). Accordingly, the newly adjacent residues may be in relative contact, and the relationship encoded as a relative contact flag.

[0061] In some aspects, digital representations may be unfeaturized. A digital representation may include information to digitally represent a biomolecule (e.g., coordinates of atoms in 3D space) and not include features encoding information which enable model(s) to generate outputs as described in the techniques herein. For example, a digital representation may not include features encoding information (i) about each atom (e.g., a residue frame (local coordinate frame) the atom resides, if the atom is a part of the binding interface, etc.), (ii) about a relationship (e.g., dihedral angles, distance, orientation, relative chain, relative contact) between each atom and another object (e.g., a different atom, backbone, frame), and / or (iii) formatted in a data structure a model is trained to process. Accordingly, in some aspects, digital representations may be featurized by the computing system 120 to include features. In some aspects, featurizing digital representations may include generating machine-readable tensors based on the multilinear relationships of one or more objects in the vector space of the data.

[0062] In some aspects, an object may include one or more atoms, one or more molecules (e.g., one or more amino acids), and / or any other object that may be included, or implied to exist, when digitally representing a biomolecule. For example, featurizing a biomolecule may include generating machine-readable tensors corresponding to the relationship(s) between each atom of a biomolecule with each other atom in the biomolecule.

[0063] In some aspects, a prediction may be performed by iteratively adding noise to a predicted biomolecule structure according to a noising schedule and predicting a new biomolecule structure based on the noised-predicted biomolecule structure and received data as inputs. In such cases, this forward noising process may output multiple predicted biomolecule structures.

[0064] In addition, or alternative, to pre-training on training inputs (e.g., training biomolecule digital representations), the present techniques may include pre-training a language model on a large database of antibody and antigen sequences (e.g., PDB, SabDab, etc.). The language model and predictive model may be conjoined or concatenated (or otherwise combined) to generate an ensemble model to further improve predictions of antibody structure.

[0065] After training, the ensembled model may generate antigen-aware antibody structures. For example, the ML training module 132 may train one or more ML models (e.g., a pre-trained generative model, a large language model (LLM), a masked autoencoder model, a denoising diffusion probabilistic model, etc.) and the ML operation module 134 may operate one or more ML models. One or more training data sets (e.g., training inputs) may be used for language model training in the present techniques, as discussed herein.

[0066] In some aspects, the ML training module 132 may train an LLM to apply natural language processing (NLP) techniques to biological data. Although unconventional, LLMs can be useful subcomponents of a biomolecule structure prediction model because LLMs operate based on the assumption that meaning can be inferred from functional constraints imposed on one thing by another (e.g., the meaning of textimposing constraints on patterns of words used in the text). In practice, this means that, for example, token sequence can be predicted based on the overall frequency of tokens in a corpus. Transformer machine learning architectures have enabled this style of prediction to be parallelized in LLMs, making LLMs more efficient and thus practically workable as compared to past, less efficient approaches (e.g., recurrent neural networks) and other statistical language modeling approaches that struggle with sparsity, such as N-gram language models.

[0067] For example, in the present techniques, the ML training module 132 may use an LLM as a subcomponent to infer a structure of a biomolecule based on an amino acid sequence of the biomolecule.

[0068] ML operation module 134 may operate the trained model and provide amino acid sequences of a de novo biomolecule (a biomolecule that the trained model has not seen previously) as input to the trained model.

[0069] Advantageously, the ability to train the model and infer structure may be possible because the structure constrains the possible sequences of amino acids that may fold to become the structure. An LLM may utilize additional sources of information that impose constraints on the protein’s structure, such as a structure of a binding partner biomolecule, to infer the protein’s structure as well. Accordingly, the present techniques may train an LLM to infer biomolecule structure from biomolecule sequences and binding-partner biomolecule structures and / or sequences, each partially, or wholly, digitally represented. The combination of biomolecule digital representations with binding partner biomolecule digital representations as training data may cause the resulting trained LLM to be a richer and more comprehensive model than would using only one type of the training data.

[0070] The LLM may include aspects of a generative pre-trained transformer (GPT) machine learning architecture, a bidirectional encoder representations from transformers (BERT) machine learning architecture, and / or another machine learning architecture (generative or otherwise) suitable for inferring biomolecule structures from biological data. For example, ESM-2 of ESMFold is a BERT-style encoder that may be used to predict atomic level structure from an individual sequence of a protein, as discussed in “Evolutionary-scale prediction of atomic level protein structure with alanguage model,” Lin, Z. et aL, bioRxiv 2022.07.20.500902, hereby incorporated by reference in its entirety for all purposes.

[0071] In some aspects the ML training module 132 may train, and the ML operation module 134 may operate, one or more models that include aspects (e.g., layers, model architecture, hyperparameters, etc.) of ESMFold. However, as discussed, the present techniques may include additional data, such as a digital representation of the bindingpartner biomolecule structure as input for the model. Advantageously, the models of the present techniques may use this additional data to generate embedding(s) processed by downstream modules and predict biomolecule structure(s) compatible to form biomolecule complexes with the binding-partner biomolecule. Doing so creates an advantageous emergent property of binding-partner biomolecule-aware- biomolecule(s) such as antigen-aware antibodies that represents a significant improvement over known conventional techniques. Training is discussed further with respect to FIG. 2.

[0072] The external server 140 may include a processor 142, a NIC 144, and a memory 146. The external server 140 may be a publicly accessible server such as PDB, SabDab, or DIPs, wherein the memory 146 stores respective information of protein data, structural antibody data, or protein interaction data. In general, the external server 140 may contain any data utilized by the computing system 120 and / or the client computing device 160. The external server 140 may be an individual server, a group (e.g., cluster) of multiple servers communicatively coupled, a group of multiple servers not communicatively coupled, or another suitable type of computing device or system (e.g., a collection of computing resources). Further, the external server 140 may be operated by one or more entities, including and / or not including the operating entity of the computing system 120.

[0073] The network 150 may be a single communication network or may include multiple communication networks of one or more types (e.g., one or more wired and / or wireless local area networks (LANs), and / or one or more wired and / or wireless wide area networks (WANs) such as the Internet). The network 108 may enable bidirectional communication between the client computing device 160 and the computing system 120, for example.

[0074] The client computing device 160 may be an individual server, a group (e.g., cluster) of multiple servers, or another suitable type of computing device or system (e.g., a collection of computing resources). For example, the client computing device 160 may be any suitable computing device (e.g., a server, a mobile computing device, a smart phone, a tablet, a laptop, a wearable device, etc.). In some aspects, one or more components of the client device 160 may be embodied by one or more virtual instances (e.g., a cloud-based virtualization service) and / or may be included in a respective remote data center (e.g., a cloud computing environment, a public cloud, a private cloud, hybrid cloud, etc.). The client computing device 160 includes a processor and a network interface controller (NIC). The processor may include any suitable number of processors and / or processor types, such as CPUs and one or more graphics processing units (GPUs). Generally, the processor is configured to execute software instructions stored in a memory. The memory may include one or more persistent memories (e.g., a hard drive / solid state memory) and stores one or more set of computer executable instructions / modules. For example, the executable instructions may receive and / or display results generated by the server 104.

[0075] The client computing device 160 may include a respective input device and a respective output device. The respective input devices may include any suitable device or devices for receiving input, such as one or more microphones, one or more camera, a hardware keyboard, a hardware mouse, a capacitive touch screen, etc. The respective output devices may include any suitable device for conveying output, such as a hardware speaker, a computer monitor, a touch screen, etc. In some cases, the input device and the output device may be integrated into a single device, such as a touch screen device that accepts user input and displays output. The NIC of the client computing device 160 may include any suitable network interface controller(s), such as wired / wireless controllers (e.g., Ethernet controllers), and facilitate bidirectional / multiplexed networking over the network 150 between the client computing device 160 and other components of the environment 100.

[0076] The sequencing device 170 may be a machine capable of analyzing a biomolecule and generating biomolecule data such as amino acid sequence data and / or protein structure data. The device 170 may be physically connected to either the computing system 120 or the network 150, as depicted. The device 170 may be locatedin a laboratory and may be accessible from one or more computers within the laboratory (not depicted) and / or from the computing system 120. The device 170 may generate data and upload that data directly to the data repository 130 and / or computing system 120, and / or upload that data indirectly via the network 150 to the client computing device 160, in some aspects. The data may include protein analysis from low resolution (e.g., amino acid composition, N and C terminus amino acids) to high resolution (e.g., amino acid sequence, secondary structure, tertiary structure). The device 170 may include instructions for receiving one or more sequences (e.g., designed sequences) and for synthesizing those sequences. The synthesis may sometimes be performed via another technique (e.g., via a different device or via a human). In some aspects, the device 170 may be configured not as a device, but as an alternative assay that can measure protein-protein interactions. For example, the device 170 may instead be configured as a suite of devices / workflows, including plates and liquid handling. In general, the device 170 may be substituted with suitable hardware and / or software optionally including human operators to generate biomolecule data such as amino acid sequence data and / or protein structure data.Exemplary Computer-Implemented Denoising-Diffusion Modeling Aspects

[0077] FIG. 2 depicts a block flow diagram 200 of an exemplary denoising diffusion probabilistic module utilized in a computer implemented method, according to some aspects.

[0078] In general, diffusion models work in two stages, training and inference. Herein, the models may be trained to perform reverse diffusion in training and inference. During training, both a forward and reverse step of diffusion may be performed while at inference only reverse steps are performed. In the first stage, training (which may be performed, for example, by the ML training module 132 of FIG. 1 ), a neural network is trained to predict noise added to a known input according to a schedule defining the forward diffusion process. Randomly selected timesteps in the schedule are sampled during training, and the model is optimized to recover the original input before the noise was added. In the second stage, inference (performed, for example, by the ML operation module 134 of FIG. 1 ), the neural network is provided with context and pure noise as input, and the output of the network (a novel generated output) is repeatedlygenerated, such that noise is progressively removed until an output matching the provided context is generated.

[0079] Context may be provided in a number of different ways to the diffusion model.. For example, FIG. 2 depicts a large language model (LLM) 210 trained / pretrained to output embeddings using a plurality of input biomolecule digital representations 212 and / or a masked language model 214. The LLM 210 may produce one or more embeddings corresponding to the input biomolecule digital representations. These embeddings may be provided as input (context) along with featurized noised structure 214 to a neural network model 216. In the forward direction, the input structure may be provided in featurized form with the embedding and no noise (or very little), at a firsttime step. In subsequent time steps, more and more noise may be added as the model learns on the embeddings and featurized structures to predict structures and, optionally, a confidence value 218. The predicted structure may be processed by featurization block 220 and noised at block 222, in a loop. Generally, featurization means taking a protein and converting it into machine readable tensors that the neural networks can consume. As discussed herein, input proteins may be factorized into node features, which represent residue-level information about each amino acid, as well as pair features, which represent interactions, distances, and orientations between pairs of residues. The pair features can represent information between residues on the same chain or on different chains, including antigen-antibody residue interactions. A loss function may be adjusted during the iterative looping process. The LLM 210 may be trained on SabDab, PDB, DIPS, and / or proprietary data sets.

[0080] Node features that may be used include amino acid sequence: The amino acid type; backbone dihedrals: the dihedral angles formed by the atoms within a single residue; binding interface: a flag specifying whether or not the residue is part of the epitope. Pair (edge) features may include relative distance: the distances between pairs of amino acids; relative orientation: the relative rotation in space of pairs of residue frames; relative sequence separation / relative chain: positional information encoding differences between residue indices, and whether or not they are on the same chain. Optionally, pair features may include contacts: a flag specifying whether or not two residues can be considered to be in contact with each other.

[0081] Once trained, the trained neural network model 216 may be sampled by providing context information (e.g., embeddings) and pure noise to the trained neural network model 216. The neural network model 216 may output predicted noise repeatedly until structure is generated that corresponds to the context. Here, the embeddings may be input biomolecule digital representations, and the structure may be that of a structure. In this way, the neural network model 216 is capable of learning to take different noisy inputs and turn them into structural predictions.

[0082] The context can include the antigen as context, which as mentioned, is critical to folding antibodies, both based on empirical findings as well as biophysical intuition. The hypervariable regions (CDRs) are loop-like in structure and are disordered regions of the protein, meaning they lack well-defined structure unless they are in the bound state with an antigen. Thus, it is not possible to predict these in high fidelity without providing the model with the antigen context. The present generative (diffusion-based) techniques cover the different binding of antibodies to targets, which is important because this binding can occur in different poses. Thus, the distribution of folds is probabilistic and multimodal. The present techniques advantageously capture the inherent variability in these poses whereas existing techniques fail to do so.

[0083] In some aspects, a masked design model may be used instead of the diffusion model.

[0084] The masked design model may be a masked autoencoder that is based upon a graph transformer architecture. An autoencoder is a neural network that has the same input and output which is typically trained for the purpose of data compression. Typically, autoencoders force the data through a “latent bottleneck,” which is a compressed vector representation of the input and learn to reconstruct the output strictly from the compressed representation. These representations are then typically used for dimensionality reduction or as inputs in further downstream models, taking advantage of the fact that the latent representation of the input is much more information dense and may be invariant to irrelevant artifacts (for example, an image autoencoder trained to represent object semantics may want to be invariant to lighting and shadow conditions), which leads to better generalization to a held-out test set. A masked autoencoder is a variant of this kind of neural network that is useful for prediction. Rather than keepingthe input and the output the same, as in a traditional autoencoder, the masked autoencoder replaces parts of its input with “mask tokens” and learns to replace the mask tokens with their true values.

[0085] In the case of the masked design model, the present techniques may take a traditional sequence-structure autoencoder architecture and replace the CDR sequence values with mask tokens. Some or all structural features that would rely on an unknown coordinate (e.g., the antibody coordinates) may also be replaced with mask tokens as well. In some aspects, only the pair features between antigen residues would be unmasked. In some aspects, CDR sequences are not masked. The masked design model architecture may include an encoder-decoder style neural network that alternates updates between residue and pair features. In some aspects, the node and pair blocks may follow the architecture of AlphaFold’s evoformer, as discussed in Jumper, J., Evans, R., Pritzel, A. et al. Highly accurate protein structure prediction with AlphaFold; Nature 596, 583-589 (2021 ), incorporated herein by reference in its entirety for all purposes. The model may be supervised with a FARE loss, as well as several auxiliary losses that encourage the model to predict structures that are physically plausible.

[0086] In the diffusion modeling case, rather than masking unknown features between coordinates, the present techniques initialize the antibody to a random point cloud initialized from Gaussian noise and compute all features unmasked. The present techniques then combine the input featurization with a language model embedding from ESM2, which can be used unmasked or masked, depending on whether the present techniques are being used to solve CDR design or the folding task (as depicted in FIG. 2), and run it through the folding trunk and structure model of ESMFold. This pretrained architecture may reap significant benefits in terms of training efficiency and data coverage.

[0087] The present forward noising process consists of gradually adding Gaussian noise to the native protein according to a predefined schedule. At training time, a random timestep from the schedule may be sampled, and the protein partially noised according to the schedule. The model may then be trained by the ML operation module 132 to predict the original structure before it was noised. At inference time, the ML operation module 134 may start with pure noise, and then interpolate the predicted structure withthe current guess. The ML operation module 134 may then refeaturize the new structure and feed it back into the model via the recycling embedding.Exemplary Computer-Implemented Methods

[0088] FIG 2 depicts an exemplary computer-implemented method 200 for generating a predicted output biomolecule digital representation. The method 200 may be performed by the computing system 120 of FIG. 1 , in some aspects. As discussed herein, biomolecules and binding-partner biomolecules may be represented digitally as biomolecule digital representations.

[0089] The method 200 may include receiving a plurality of input biomolecule representations each corresponding to a respective one of a plurality of input biomolecules (block 202).

[0090] Generally, the input biomolecules may be (i) biomolecules and / or (ii) bindingpartner biomolecules. In some aspects, the input biomolecule digital representations may include one or more digital representations of (i) a binding-partner biomolecule structure and / or sequence, and (ii) a biomolecule sequence. In some aspects, the input biomolecule digital representations may include fragments or portions of (i) a bindingpartner biomolecule structure and / or sequence, and / or (ii) a biomolecule sequence, represented digitally.

[0091] The method 200 may include processing one or more of the plurality of input biomolecule representations using a machine-learned model to generate one or more predicted output biomolecule representations, the machine-learned model trained to predict an output biomolecule digital representation corresponding to an output biomolecule using one or more training inputs including one or more training biomolecule digital representations each corresponding to a respective one of a plurality of training biomolecules (block 204).

[0092] In some aspects, the one or more predicted output biomolecule digital representations includes a digital representation of a structure of a biomolecule, or a fragment or portion thereof.

[0093] In some aspects, at least one of the one or more training biomolecule digital representations includes a digital representation of a complementary determining region(CDR) and (ii) at least one of the plurality of input biomolecule digital representations includes a digital representation of a complementary determining region (CDR).

[0094] In some aspects, method 200 may include transmitting, via an electronic network (e.g., the network 150), the predicted output biomolecule digital representations. For example, the method 200 may include transmitting the predicted biomolecule digital representations from the computing system 120 of FIG. 1 to the external server, the data repository 130, and / or the client device 160.

[0095] In some aspects, the one or more training biomolecule digital representations and / or the plurality of input biomolecule digital representations may include one or more digital representations of a binding-partner biomolecule structure and / or sequence, and / or a biomolecule sequence. The one or more predicted output biomolecule digital representations may include a digital representation of a structure of a biomolecule. In some aspects, at least one of the one or more training biomolecule digital representations includes a digital representation of a complementary determining region (CDR) and / or at least one of the pluralities of input biomolecule digital representations include a digital representation of a complementary determining region (CDR). At least some of the digital representation of the CDR may be masked.

[0096] Further, one or more of the one or more training biomolecule digital representations may include a digital representation of a binding interface; and one or more of the pluralities of input biomolecule digital representations may include a digital representation of a binding interface. The digital representation of the binding interface may include a digital representation of an epitope. Some of the digital representation of the binding interface may be masked. Some of the digital representation of the CDR may be masked and at least some of the digital representation of the binding interface is masked.

[0097] The one or more training biomolecule digital representations may include a digital representation of an antigen and / or an antibody. The plurality of input biomolecule digital representations may include a digital representation of an antigen, and / or an antibody. At least one of the one or more training biomolecule digital representations may include a digital representation of a concatenated antibody and antigen. At least one of the pluralities of input biomolecule digital representations may include a digitalrepresentation of a concatenated antibody and antigen. The output biomolecule digital representation may include a digital representation of an antibody. The one or more training biomolecule digital representations may include digital representations of one or more training biomolecules. The plurality of training biomolecules may include at least one antigen, and / or at least one antibody.

[0098] Each of the plurality of input biomolecule digital representations may include at least one respective node feature including a residue-level digital representation of each amino acid sequence; and / or at least one respective pair feature including a pair-level digital representation of at least two amino acid residues.

[0099] The method 200 may further include receiving the plurality of input biomolecules; and generating the plurality of input biomolecule digital representations by featurizing the plurality of input biomolecules. The method 200 may further include generating the plurality of input biomolecule digital representations by featurizing each of the plurality of input biomolecules with: at least one respective node feature including a residue-level digital representation of each amino acid residue; and / or at least one respective pair feature including a pair-level digital representation of at least two amino acid residues.

[0100] The method 200 may include generating a machine-readable tensor corresponding to at least one of the plurality of input biomolecules. At least one respective node feature may include a digital representation of one or more of an amino acid sequence, one or more backbone dihedrals, and / or a binding interface flag.

[0101] At least one respective pair feature may include a digital representation of one or more of (i) a relative distance, (ii) a relative orientation, and (iii) a relative chain. The one respective pair feature further includes a digital representation of one or more of a relative contact flag. The input biomolecules may include at least one antigen and at least one antibody. The input biomolecules may include one or more antigens. The input biomolecules may include one or more antibodies.

[0102] The machine-learned model of method 200 may be a masked autoencoder model. The machine-learned model may be further trained to predict the output biomolecule digital representation by reconstructing a compressed digital representation of one or more of the pluralities of input biomolecule digital representations by using the training inputs. The one or more training biomoleculedigital representations may be one or more masked training biomolecule digital representations, wherein at least some values of one of the one or more training biomolecule digital representations are masked. The plurality of input biomolecule digital representations may be a plurality of masked input biomolecule digital representations, wherein at least some values of one of the pluralities of input biomolecule digital representations is masked.

[0103] The machine-learned model may be a masked autoencoder model further trained to predict the output biomolecule digital representation by reconstructing a compressed digital representation of one or more of the pluralities of input biomolecule digital representations by using the training inputs. The one or more training biomolecule digital representations may be one or more masked training biomolecule digital representations, wherein at least some values of one of the one or more training biomolecule digital representations of the training inputs are masked. The method 200 may further include generating a plurality of masked input biomolecule digital representations by masking at least some values of one or more of the pluralities of input biomolecule digital representations. The machine-learned model may be trained to reconstruct the compressed digital representation by predicting one or more true values of one or more masked values using the training inputs.

[0104] One or more amino acid residues of a pair feature may correspond to an antigen, and one or more of the two amino acid residues of a pair feature may correspond to an antibody. One or more pair features corresponding to the antigen and the antibody may be masked.

[0105] The machine-learned model may include an encoder-decoder neural network architecture that alternates updates of predicted true values between (i) one or more respective node features of each one of the pluralities of masked input biomolecule digital representations and (ii) one or more respective pair features of each one of the pluralities of masked input biomolecule digital representations. The machine- learned model may be supervised by a Frame Aligned Point Error (FAPE) loss. A runtime for the machine-learned model may be approximately 1 second or less.

[0106] The method 200 may include further training the machine-learned model to initialize one or more backbone frames according to one or more of the one or morebinding-partner biomolecule structures by initializing at least some of backbone frames using a respective spatial position one or more backbone frames of the binding-partner biomolecule structure to achieve equivariance.

[0107] The machine-learned model may be a denoising diffusion probabilistic model, and the method 200 may include computing an embedding by processing at least one of the plurality of input digital representations using a large language model; receiving at least one structure corresponding to the output biomolecule digital representations; and for each time step t in T time steps: generating a featurized structure by featurizing the raw coordinate input; generating a noised featurized structure by adding at least some noise to the featurized structure based upon time step t; generating a folding trunk output by processing (i) the embedding and (ii) the noised featurized structure using a plurality of folding trunk layers; and updating the structure by processing the folding trunk output using a plurality of structure module layers, wherein the update includes generating a new output biomolecule digital representation of the structure.

[0108] The received structure may be blank. The noise may be Gaussian noise. The featurized structure may be generated by featurizing the structure with: at least one respective node feature including a residue-level digital representation of each amino acid residue; and / or at least one respective pair feature including a pair-level digital representation of at least two amino acid residues. The machine-learned model may further include one or more layers of ESMFold. The large language model may be ESM2. The large language model may be pre-trained via a masked language model. The machine-learned model may be executed by one or more processors of an electronic device. Training inputs may include protein representations from at least one of (i) a protein data bank, (ii) structural antibody database, or (iii) a protein interaction database.Additional Considerations

[0109] Before the present disclosure is further described, it is to be understood that this disclosure is not limited to particular embodiments described, as such may, of course, vary. It is also to be understood that the terminology used herein is for thepurpose of describing particular embodiments only, and is not intended to be limiting, since the scope of the present disclosure will be limited only by the appended claims.

[0110] Where a range of values is provided, it is understood that each intervening value, to the tenth of the unit of the lower limit unless the context clearly dictates otherwise, between the upper and lower limit of that range and any other stated or intervening value in that stated range, is encompassed within the disclosure. The upper and lower limits of these smaller ranges may independently be included in the smaller ranges, and are also encompassed within the disclosure, subject to any specifically excluded limit in the stated range. Where the stated range includes one or both of the limits, ranges excluding either or both of those included limits are also included in the disclosure.

[0111] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. Although any methods and materials similar or equivalent to those described herein can also be used in the practice or testing of the present disclosure, the preferred methods and materials are now described. All publications mentioned herein are incorporated herein by reference to disclose and describe the methods and / or materials in connection with which the publications are cited.

[0112] It must be noted that as used herein and in the appended claims, the singular forms "a," "and," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a conformation switching probe" includes a plurality of such conformation switching probes and reference to "the microfluidic device" includes reference to one or more microfluidic devices and equivalents thereof known to those skilled in the art, and so forth. It is further noted that the claims may be drafted to exclude any element, e.g., any optional element. As such, this statement is intended to serve as antecedent basis for use of such exclusive terminology as "solely," "only" and the like in connection with the recitation of claim elements, or use of a "negative" limitation.

[0113] As will be apparent to those of skill in the art upon reading this disclosure, each of the individual embodiments described and illustrated herein has discretecomponents and features which may be readily separated from or combined with the features of any of the other several embodiments without departing from the scope or spirit of the present disclosure. Any recited method can be carried out in the order of events recited or in any other order which is logically possible. This is intended to provide support for all such combinations.

[0114] The various embodiments described above can be combined to provide further embodiments. All U.S. patents, U.S. patent application publications, U.S. patent application, foreign patents, foreign patent application and non-patent publications referred to in this specification and / or listed in the Application Data Sheet are incorporated herein by reference, in their entirety. Aspects of the embodiments can be modified if necessary to employ concepts of the various patents, applications, and publications to provide yet further embodiments.

[0115] These and other changes can be made to the embodiments in light of the above-detailed description. In general, in the following claims, the terms used should not be construed to limit the claims to the specific embodiments disclosed in the specification and the claims but should be construed to include all possible embodiments along with the full scope of equivalents to which such claims are entitled. Accordingly, the claims are not limited by this disclosure.

Claims

WHAT IS CLAIMED:1 . A computing system for performing antigen-aware antibody folding, comprising: one or more processors; and one or more memories having stored thereon: a machine-learned model trained to predict an output biomolecule digital representation corresponding to an output biomolecule using one or more training inputs including one or more training biomolecule digital representations each corresponding to a respective one of a plurality of training biomolecules; and a set of computer-executable instructions that, when executed by the one or more processors, cause the computing system to: receive a plurality of input biomolecule digital representations each corresponding to a respective one of a plurality of input biomolecules; and process one or more of the pluralities of input biomolecule digital representations using the machine-learned model to generate one or more predicted output biomolecule digital representations.

2. The computing system of claim 1 , the set of computer-executable instructions having stored thereon instructions that, when executed, cause the computing system to: transmit, via an electronic network, the predicted output biomolecule digital representations.

3. The computing system of claim 1 , wherein at least one of (i) the one or more training biomolecule digital representations or (ii) the plurality of input biomolecule digital representations includesone or more digital representations of (i) a binding-partner biomolecule structure and / or sequence, and (ii) a biomolecule sequence.

4. The computing system of claim 3, wherein the one or more predicted output biomolecule digital representations includes a digital representation of a structure of a biomolecule.

5. The computing system of claim 1 , wherein one or both of(i) at least one of the one or more training biomolecule digital representations includes a digital representation of a complementary determining region (CDR); and(ii) at least one of the pluralities of input biomolecule digital representations includes a digital representation of a complementary determining region (CDR).

6. The computing system of claim 5, wherein at least some of the digital representation of the CDR is masked.

7. The computing system of claim 1 , wherein one or both of(i) one or more of the one or more training biomolecule digital representations includes a digital representation of a binding interface; and(ii) one or more of the pluralities of input biomolecule digital representations includes a digital representation of a binding interface.

8. The computing system of claim 7, wherein the digital representation of the binding interface includes a digital representation of an epitope.

9. The computing system of claim 7, wherein at least some of the digital representation of the binding interface is masked.

10. The computing system of claim 5 or claim 7, wherein at least some of the digital representation of the CDR is masked and at least some of the digital representation of the binding interface is masked.1 1 . The computing system of claim 1 , wherein one or both of(i) the one or more training biomolecule digital representations include a digital representation of one or both of:(a) an antigen, and(b) an antibody; and(ii) the plurality of input biomolecule digital representations includes a digital representation of one or both of:(c) an antigen, and(d) an antibody.

12. The computing system of claim 1 , wherein one or both of(i) at least one of the one or more training biomolecule digital representations include a digital representation of a concatenated antibody and antigen, and(ii) at least one of the pluralities of input biomolecule digital representations include a digital representation of a concatenated antibody and antigen.

13. The computing system of claim 1 , wherein the output biomolecule digital representation includes a digital representation of an antibody.

14. The computing system of claim 1 , wherein the one or more training biomolecule digital representations include digital representations of one or more training biomolecules.

15. The computing system of claim 14, wherein the plurality of training biomolecules includes at least one antigen, and wherein the one or more training biomolecules include at least one antibody.

16. The computing system of claim 1 , wherein each of the plurality of input biomolecule digital representations includes:(i) at least one respective node feature including a residue-level digital representation of each amino acid sequence; and(ii) at least one respective pair feature including a pair-level digital representation of at least two amino acid residues.

17. The computing system of claim 1 , the set of computer-executable instructions having stored thereon instructions that, when executed, cause the computing system to: receive the plurality of input biomolecules; and generate the plurality of input biomolecule digital representations by featurizing the plurality of input biomolecules.

18. The computing system of claim 17, the set of computer-executable instructions having stored thereon instructions that, when executed, cause the computing system to: generate the plurality of input biomolecule digital representations by featurizing each of the plurality of input biomolecules with:(i) at least one respective node feature including a residue-level digital representation of each amino acid residue; and(ii) at least one respective pair feature including a pair-level digital representation of at least two amino acid residues.

19. The computing system of claim 17, the set of computer-executable instructions having stored thereon instructions that, when executed, cause the computing system to: generate a machine-readable tensor corresponding to at least one of the pluralities of input biomolecules.

20. The computing system of claim 16 or 18, wherein at least one respective node feature includes a digital representation of one or more of:(i) an amino acid sequence, (ii) one or more backbone dihedrals, and (iii) a binding interface flag.21 . The computing system of claim 16 or 18, wherein at least one respective pair feature includes a digital representation of one or more of:(i) a relative distance, (ii) a relative orientation, and (iii) a relative chain.

22. The computing system of claim 21 , wherein the one respective pair feature further includes a digital representation of one or more:(iv) a relative contact flag.

23. The computing system of claim 16 or 18, wherein the input biomolecules include at least one antigen and at least one antibody.

24. The computing system of claim 16 or 18, wherein the input biomolecules include one or more antigens.

25. The computing system of claim 16 or 18, wherein the input biomolecules include one or more antibodies.

26. The computing system of claim 1 , wherein the machine-learned model is a masked autoencoder model; wherein the machine-learned model is further trained to predict the output biomolecule digital representation by reconstructing a compressed digital representation of one or more of the pluralities of input biomolecule digital representations by using the training inputs; wherein the one or more training biomolecule digital representations is one or more masked training biomolecule digital representations, wherein at least some values of one of the one or more training biomolecule digital representations are masked; and wherein the plurality of input biomolecule digital representations is a plurality of masked input biomolecule digital representations,wherein at least some values of one of the pluralities of input biomolecule digital representations are masked.

27. The computing system of claim 1 , wherein the machine-learned model is a masked autoencoder model; wherein the machine-learned model is further trained to predict the output biomolecule digital representation by reconstructing a compressed digital representation of one or more of the pluralities of input biomolecule digital representations by using the training inputs; wherein the one or more training biomolecule digital representations are one or more masked training biomolecule digital representations, wherein at least some values of one of the one or more training biomolecule digital representations of the training inputs are masked; and the set of computer-executable instructions having stored thereon instructions that, when executed, cause the computing system to: generate a plurality of masked input biomolecule digital representations by masking at least some values of one or more of the pluralities of input biomolecule digital representations.

28. The computing system of claim 26 or 27, wherein the machine-learned model is trained to reconstruct the compressed digital representation by predicting one or more true values of one or more masked values using the training inputs.

29. The computing system of claims 16, 18, 26 or 27, wherein one of the two amino acid residues of a pair feature correspond to an antigen, andwherein one of the two amino acid residues of a pair feature corresponds to an antibody.

30. The computing system of claim 29, wherein one or more pair features corresponding to an antigen and an antibody is masked.

30. The computing system of any of claims 26-28, wherein the machine-learned model includes encoder-decoder neural network architecture that alternates updates of predicted true values between (i) one or more respective node features of each one of the pluralities of masked input biomolecule digital representations and (ii) one or more respective pair features of each one of the pluralities of masked input biomolecule digital representations.

32. The computing system of claim 26 or 27, wherein the machine-learned model is supervised by a Frame Aligned Point Error (FAPE) loss.

33. The computing system of claim 26 or 27, wherein runtime for the machine-learned model is approximately 1 second.

34. The computing system of claim 3, wherein the machine-learned model is further trained to initialize one or more backbone frames according to one or more of the one or more binding-partner biomolecule structures by initializing at least some of backbone frames using a respective spatial position one or more backbone frames of the binding-partner biomolecule structure to achieve equivariance.

35. The computing system of claim 1 , wherein the machine-learned model is a denoising diffusion probabilistic model, and wherein the memory has stored thereon instructions that, when executed, cause the computing system to: compute an embedding by processing at least one of the pluralities of input digital representations using a large language model; receive at least one structure corresponding to the output biomolecule digital representations; and for each time step t in T time steps: generate a featurized structure by featurizing the structure; generate a noised featurized structure by adding at least some noise to the featurized structure based upon time step t; generate a folding trunk output by processing (i) the embedding and (ii) the noised featurized structure using a plurality of folding trunk layers; and update the structure by processing the folding trunk output using a plurality of structure module layers, wherein the update includes generating a new output biomolecule digital representation of the structure.

36. The computing system of claim 35, wherein the received structure is blank.

37. The computing system of claim 35, wherein the featurized structure is generated by featurizing the structure with:(i) at least one respective node feature including a residue-level digital representation of each amino acid residue; and(ii) at least one respective pair feature including a pair-level digital representation of at least two amino acid residues.

38. The computing system of claim 35, wherein the machine-learned model further includes ESMFold.

39. The computing system of claim 35, wherein the large language model is ESM2.

40. The computing system of claim 35, wherein the large language model is pre-trained via a masked language model.41 . The computing system of claim 1 , wherein the machine-learned model is executed by one or more processors of an electronic device.

42. The computing system of claim 1 , wherein training inputs include protein representations from at least one of (i) a protein data bank, (ii) structural antibody database, or (iii) a protein interaction database.

43. A computer-implemented method for performing antigen-aware antibody folding, comprising: receiving a plurality of input biomolecule digital representations each corresponding to a respective one of a plurality of input biomolecules; and processing one or more of the plurality of input biomolecule digital representations using a machine-learned model to generate one or more predictedoutput biomolecule digital representations, the machine-learned model trained to predict an output biomolecule digital representation corresponding to an output biomolecule using one or more training inputs including one or more training biomolecule digital representations each corresponding to a respective one of a plurality of training biomolecules.

44. The computer implemented method of claim 43, wherein each of the plurality of input biomolecule digital representations includes:(i) at least one respective node feature including a residue-level digital representation of each amino acid sequence; and(ii) at least one respective pair feature including a pair-level digital representation of at least two amino acid residues.

45. The computer implemented method of claim 43, wherein the machine-learned model is a masked autoencoder model; wherein the machine-learned model is further trained to predict the output biomolecule digital representation by reconstructing a compressed digital representation of one or more of the pluralities of input biomolecule digital representations by using the training inputs; wherein the one or more training biomolecule digital representations is one or more masked training biomolecule digital representations, wherein at least some values of one of the one or more training biomolecule digital representations are masked; and wherein the plurality of input biomolecule digital representations is a plurality of masked input biomolecule digital representations, wherein at least some values of one of the pluralities of input biomolecule digital representations are masked.

46. The computer implemented method of claim 43, wherein the machine-learned model is a masked autoencoder model; wherein the machine-learned model is further trained to predict the output biomolecule digital representation by reconstructing a compressed digital representation of one or more of the pluralities of input biomolecule digital representations by using the training inputs; and wherein the one or more training biomolecule digital representations are one or more masked training biomolecule digital representations, wherein at least some values of one of the one or more training biomolecule digital representations of the training inputs are masked; and further comprising: generating a plurality of masked input biomolecule digital representations by masking at least some values of one or more of the pluralities of input biomolecule digital representations.

47. The computer implemented method of claim 45 or 46, wherein the machine-learned model is trained to reconstruct the compressed digital representation by predicting one or more true values of one or more masked values using the training inputs.

48. The computer implemented method of claim 43, wherein the machine-learned model is a denoising diffusion probabilistic model; and further comprising: computing an embedding by processing at least one of the pluralities of input digital representations using a large language model; receiving at least one structure corresponding to the output biomolecule digital representations; andfor each time step t in T time steps: generating a featurized structure by featurizing the structure; generating a noised featurized structure by adding at least some noise to the featurized structure based upon time step t; generating a folding trunk output by processing (i) the embedding and (ii) the noised featurized structure using a plurality of folding trunk layers; and updating the structure by processing the folding trunk output using a plurality of structure module layers, wherein updating includes generating a new output biomolecule digital representation of the structure.

49. The computer implemented method of claim 48, wherein the received structure is blank.

50. A non-transitory computer-readable storage medium having stored thereon: a machine-learned model trained to predict an output biomolecule digital representation corresponding to an output biomolecule using one or more training inputs including one or more training biomolecule digital representations each corresponding to a respective one of a plurality of training biomolecules; and a set of computer-executable instructions that, when executed on one or more processors, cause a computer to: receive a plurality of input biomolecule digital representations each corresponding to a respective one of a plurality of input biomolecules; and process one or more of the pluralities of input biomolecule digital representations using the machine-learned model to generate one or more predicted output biomolecule digital representations.51 . The computer-readable storage medium of claim 50,wherein each of the plurality of input biomolecule digital representations includes:(i) at least one respective node feature including a residue-level digital representation of each amino acid sequence; and(ii) at least one respective pair feature including a pair-level digital representation of at least two amino acid residues.

52. The computer-readable storage medium of claim 50, wherein the machine-learned model is a masked autoencoder model; wherein the machine-learned model is further trained to predict the output biomolecule digital representation by reconstructing a compressed digital representation of one or more of the pluralities of input biomolecule digital representations by using the training inputs; wherein the one or more training biomolecule digital representations is one or more masked training biomolecule digital representations, wherein at least some values of one of the one or more training biomolecule digital representations are masked; and wherein the plurality of input biomolecule digital representations is a plurality of masked input biomolecule digital representations, wherein at least some values of one of the pluralities of input biomolecule digital representations are masked.

53. The computer-readable storage medium of claim 50, wherein the machine-learned model is a masked autoencoder model; wherein the machine-learned model is further trained to predict the output biomolecule digital representation by reconstructing a compressed digital representation of one or more of the pluralities of input biomolecule digital representations by using the training inputs;wherein the one or more training biomolecule digital representations are one or more masked training biomolecule digital representations, wherein at least some values of one of the one or more training biomolecule digital representations of the training inputs are masked; and wherein the instructions, when executed, cause a computer to: generate a plurality of masked input biomolecule digital representations by masking at least some values of one or more of the pluralities of input biomolecule digital representations.

54. The computer-readable storage medium of claims 52 or 53, wherein the machine-learned model is trained to reconstruct the compressed digital representation by predicting one or more true values of one or more masked values using the training inputs.

55. The computer-readable storage medium of claim 50, wherein the machine-learned model is a denoising diffusion probabilistic model; and wherein the instructions, when executed, cause a computer to: compute an embedding by processing at least one of the pluralities of input digital representations using a large language model; receive at least one structure corresponding to the output biomolecule digital representations; and for each time step t in T time steps: generate a featurized structure by featurizing the structure; generate a noised featurized structure by adding at least some noise to the featurized structure based upon time step t; generate a folding trunk output by processing (i) the embedding and (ii) the noised featurized structure using a plurality of folding trunk layers; andupdate the structure by processing the folding trunk output using a plurality of structure module layers, wherein the update includes generating a new output biomolecule digital representation of the structure.

56. The computer-readable storage medium of claim 55, wherein the received structure is blank.

Images