Methods and systems for efficient sequence-based ligand binding pocket prediction

Abstract

This disclosure describes novel methods and systems for identifying a ligand binding site (such as a cryptic ligand binding pocket) or ligand binding residues of a protein. The disclosed methods and systems employ "Efficient Sequence-based cryptic Pocket prediction" (ESP) leveraging advanced Protein Language Models (PLMs) and demonstrate significant improvement in predictive efficacy compared to existing machine learning (ML)-based cryptic pocket prediction methods. In the disclosed methods and systems, detection of cryptic pockets is achieved via training on readily available, non-cryptic pocket-specific data from a PDBBind dataset, rather than costly simulation and post-processing. Further, while SOTA's predictions often include positive signal broadly distributed over a target structure, the disclosed methods and systems produce more focused predictions and thus significantly increase downstream utility.

Description

[0001] METHODS AND SYSTEMS FOR EFFICIENT SEQUENCE-BASED LIGAND BINDING POCKET PREDICTION

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority under 35 U.S.C. §119(e) to U.S. Provisional Patent Application No. 63 / 584,952, filed September 25, 2024. The foregoing application is incorporated by reference herein in its entirety.

[0004] FIELD

[0005] This invention relates generally to methods and systems for efficient sequence-based ligand binding pocket prediction.

[0006] BACKGROUND

[0007] Proteins are critical cellular components; they participate in various critical activities and processes in the life cycle of organisms, which can achieve or help achieve important biological functions. Proteins often need to bind to other molecules or ions, called ligands, to create specific interactions to achieve corresponding functions. Intermolecular interactions between proteins and ligands, such as small compounds, occur via amino acid residues at specific positions in the protein, usually located in pocket-like regions. These specific key amino acid residues are called ligand binding sites. Ligand binding sites have attracted much attention in the fields of molecular docking, drug-target interactions, compound design, ligand affinity prediction, and even molecular dynamics (MD). Thus, identification of ligand binding sites not only helps to explore the mechanism of intermolecular interactions but also effectively explains the pathogenesis of diseases, which provides insights for drug discovery and design.

[0008] Compared with highly accurate but time-consuming biological experiments, the advantage of computational methods is that ligand binding site predictions can be performed based on sequence and structure information without relying on annotating biological functions of proteinbinding residues. Accurate prediction of protein-ligand binding pockets is a critical task in protein functional analysis and small molecule pharmaceutical design. However, the flexible and dynamic nature of proteins conceal an unknown number of potentially invaluable “cryptic” pockets. Current approaches for cryptic pocket discovery rely on MD, leading to poor scalability and bias. Even recent machine learning (ML)-based cryptic pocket discovery approaches rely on large MD datasets to train their models.

[0009] Therefore, there is a critical need to devise a strategy capable of efficiently and accurately predicting protein-ligand binding pocket.

[0010] SUMMARY

[0011] This disclosure addresses the need mentioned above in a number of aspects. In one aspect, this disclosure provides a method of identifying a ligand binding site or ligand binding residues of a protein. In some embodiments, the method comprises: (a) inputting a sequence of the protein into a machine learning model comprising a trained protein language model and a trained prediction head; (b) generating, by the protein language model, a contextual vector embedding for each of residues contained in the sequence of the protein, wherein the contextual vector embedding encodes functional and structural properties of a corresponding residue; and (c) inputting contextual vector embeddings of the residues contained in the sequence into the prediction head.

[0012] In some embodiments, the method comprises: (a) inputting a sequence of the protein into a machine learning model comprising a trained protein language model and a trained prediction head; (b) generating, by the protein language model, a contextual vector embedding for each of residues contained in the sequence of the protein, wherein the contextual vector embedding encodes functional and structural properties of a corresponding residue; (c) inputting contextual vector embeddings of the residues contained in the sequence into the prediction head; (d) determining a pocket prediction score for each of the residues contained in the sequence based on the contextual vector embeddings of the sequence; and (e) identifying a residue as a ligand binding residue constituting the ligand binding site of the protein if a value of the pocket prediction score of the residue is equal to or greater than a threshold value.

[0013] In some embodiments, the method further comprises mapping the ligand binding residue on a three-dimensional structure model of the protein.

[0014] In some embodiments, the method comprises inputting a classification token into the protein language model, and generating, by the protein language model, a summary or classification embedding. In some embodiments, the method further comprises training the machine learning model by: (a) retrieving a set of sequences from a database; (b) determining a reference pocket prediction score of each of residues contained in each of the sequences from the database; (c) inputting each of the sequences into the machine learning model comprising a protein language model and a prediction head; (d) generating by the protein language model a contextual vector embedding for each of residues contained in each of the sequences, wherein the contextual vector embedding encodes functional and structural properties of a corresponding residue; (e) inputting contextual vector embeddings of the residues contained in each of the sequence into the prediction head; (f) determining a pocket prediction score for each of the residues contained in each of the sequences based on the contextual vector embeddings of the residues contained in each of the sequences; (g) determining a value of a loss function associated with the prediction head based on deviation between the pocket prediction score for each of the residues and a reference pocket prediction score of the corresponding residue; and (h) performing iterative adjustments of parameters of the prediction head until a stopping criterion is reached.

[0015] In yet another aspect, this disclosure provides a method of training a machine learning model for identifying a ligand binding site or ligand binding residues of a protein. In some embodiments, the method comprises: (i) retrieving a set of sequences from a database; (ii) determining a reference pocket prediction score of each of residues contained in each of the sequences from the database; (iii) inputting each of the sequences into a machine learning model comprising a protein language model and a prediction head; (iv) generating by the protein language model a contextual vector embedding for each of residues contained in each of the sequences, wherein the contextual vector embedding encodes functional and structural properties of a corresponding residue; (v) inputting contextual vector embeddings of the residues contained in each of the sequence into the prediction head; (vi) determining a pocket prediction score for each of the residues contained in each of the sequences based on the contextual vector embeddings of the residues contained in each of the sequences; (vii) determining a value of a loss function associated with the prediction head based on deviation between the pocket prediction score for each of the residues and a reference pocket prediction score of the corresponding residue; and (viii) performing iterative adjustments of parameters of the prediction head until a stopping criterion is reached. In some embodiments, the database comprises a protein ligand-binding pocket database. In some embodiments, the protein ligand-binding pocket database is P-2024.

[0016] In some embodiments, the database also comprises a PDBBind dataset, an ESM-2 SSP dataset, or a combination thereof.

[0017] In some embodiments, the step of retrieving the set of sequences from the database comprises removing sequences that have a greater than 30%, 50%, 70%, 90%, 95%, 100%, or other arbitrary sequence identity with another sequence in the set of validation and testing sequences.

[0018] In some embodiments, the prediction head comprises a Multi-Head Attention (MHA) prediction head. In some embodiments, the MHA prediction head utilizes a PLM’s classification token. In some embodiments, the MHA prediction head does not utilize a PLM’s classification token.

[0019] In some embodiments, the method further comprises tokenizing each of the residues contained in each of the sequences with an additional contextual vector embedding.

[0020] In some embodiments, the step of inputting the contextual vector embeddings of each of the sequence into the prediction head further comprises inputting one or more additional features of each of the residues contained in the sequence.

[0021] In some embodiments, the step of determining the pocket prediction score for each of the residues contained in each of the sequences comprising determining the pocket prediction score for each of the residues contained in each of the sequences based on the contextual vector embeddings of each of the sequences and the one or more additional features of each of the residues contained in the sequence.

[0022] In some embodiments, the one or more additional features comprise one or more dynamic features or sequence features. In some embodiments, the one or more dynamic features comprise one or more of b-factor features, square fluctuation features, cross-correlation features, and perturbation response scanning features. In some embodiments, the one or more sequence features comprise one or more of residue identity features, conservation features, co-evolution features, and structure features. In some embodiments, the ligand binding site comprises a cryptic ligand binding pocket. In some embodiments, the ligand binding site comprises an allosteric ligand binding site.

[0023] In some embodiments, the three-dimensional structure model of the protein is experimentally determined or generated by homology modeling or other computational approaches.

[0024] In some embodiments, the sequence comprises an amino acid sequence, a DNA sequence, or a RNA sequence.

[0025] In some embodiments, the machine learning model exhibits a receiver operating characteristic area under the curve (AUC) value or average prediction score (APS) of at least 0.8. In some embodiments, the machine learning model exhibits a receiver operating characteristic area under the curve (AUC) value of at least 0.9.

[0026] In another aspect, this disclosure provides a system of identifying a ligand binding site or ligand binding residues of a protein. In some embodiments, the system comprises one or more processors configured to: (a) input a sequence of the protein into a machine learning model comprising a trained protein language model and a trained prediction head; (b) generate, by the protein language model, a contextual vector embedding for each of residues contained in the sequence of the protein, wherein the contextual vector embedding encodes functional and structural properties of a corresponding residue; and (c) input contextual vector embeddings of the residues contained in the sequence into the prediction head.

[0027] In some embodiments, the system comprises one or more processors configured to: (a) input a sequence of the protein into a machine learning model comprising a trained protein language model and a trained prediction head; (b) generate, by the protein language model, a contextual vector embedding for each of residues contained in the sequence of the protein, wherein the contextual vector embedding encodes functional and structural properties of a corresponding residue; (c) input contextual vector embeddings of the residues contained in the sequence into the prediction head; (d) determine a pocket prediction score for each of the residues contained in the sequence based on the contextual vector embeddings of the sequence; and (e) identify a residue as a ligand binding residue constituting the ligand binding site of the protein if a value of the pocket prediction score of the residue is equal to or greater than a threshold value. In some embodiments, the one or more processors are further configured to map the ligand binding residue on a three-dimensional structure model of the protein.

[0028] In some embodiments, the one or more processors are further configured to input a classification token into the protein language model and generate, by the protein language model, a summary or classification embedding.

[0029] In some embodiments, the one or more processors are further configured to train the machine learning model by: (a) retrieving a set of sequences from a database; (b) determining a reference pocket prediction score of each of residues contained in each of the sequences from the database; (c) inputting each of the sequences into the machine learning model comprising a protein language model and a prediction head; (d) generating by the protein language model a contextual vector embedding for each of residues contained in each of the sequences, wherein the contextual vector embedding encodes functional and structural properties of a corresponding residue; (e) inputting contextual vector embeddings of the residues contained in each of the sequence into the prediction head; (f) determining a pocket prediction score for each of the residues contained in each of the sequences based on the contextual vector embeddings of the residues contained in each of the sequences; (g) determining a value of a loss function associated with the prediction head based on deviation between the pocket prediction score for each of the residues and a reference pocket prediction score of the corresponding residue; and (h) performing iterative adjustments of parameters of the prediction head until a stopping criterion is reached.

[0030] In yet another aspect, this disclosure further provides a system of training a machine learning model for identifying a ligand binding site of a protein. In some embodiments, the system comprises one or more processors configured to: (i) retrieve a set of sequences from a database; (ii) determine a reference pocket prediction score of each of residues contained in each of the sequences from the database; (iii) input each of the sequences into a machine learning model comprising a protein language model and a prediction head; (iv) generate by the protein language model a contextual vector embedding for each of residues contained in each of the sequences, wherein the contextual vector embedding encodes functional and structural properties of a corresponding residue; (v) input contextual vector embeddings of the residues contained in each of the sequence into the prediction head; (vi) determine a pocket prediction score for each of the residues contained in each of the sequences based on the contextual vector embeddings of the residues contained in each of the sequences; (vii) determine a value of a loss function associated with the prediction head based on deviation between the pocket prediction score for each of the residues and a reference pocket prediction score of the corresponding residue; and (viii) perform iterative adjustments of parameters of the prediction head until a stopping criterion is reached.

[0031] In some embodiments, the database comprises a protein ligand-binding pocket database. In some embodiments, the protein ligand-binding pocket database is P-2024.

[0032] In some embodiments, the database also comprises a PDBBind dataset, an ESM-2 SSP dataset, or a combination thereof.

[0033] In some embodiments, the step of retrieving the set of sequences from the database comprises removing sequences that have a greater than 30%, 50%, 70%, 90%, 95%, 100%, or other arbitrary sequence identity with another sequence in the set of validation and testing sequences.

[0034] In some embodiments, the prediction head comprises a Multi-Head Attention (MHA) prediction head. In some embodiments, the MHA prediction head utilizes a PML’s classification token. In some embodiments, the MHA prediction head does not utilize a PML’s classification token.

[0035] In some embodiments, the one or more processors are further configured to tokenize each of the residues contained in each of the sequences with an additional contextual vector embedding.

[0036] In some embodiments, the step of inputting the contextual vector embeddings of each of the sequence into the prediction head further comprises inputting one or more additional features of each of the residues contained in the sequence. In some embodiments, the step of determining the pocket prediction score for each of the residues contained in each of the sequences comprising determining the pocket prediction score for each of the residues contained in each of the sequences based on the contextual vector embeddings of each of the sequences and the one or more additional features of each of the residues contained in the sequence.

[0037] In some embodiments, the one or more additional features comprise one or more dynamic features or sequence features. In some embodiments, the one or more dynamic features comprise one or more of b-factor features, square fluctuation features, cross-correlation features, and perturbation response scanning features. In some embodiments, the one or more sequence features comprise one or more of residue identity features, conservation features, co-evolution features, and structure features.

[0038] In some embodiments, the ligand binding site comprises a cryptic ligand binding pocket. In some embodiments, the ligand binding site comprises an allosteric ligand binding site.

[0039] In some embodiments, the three-dimensional structure model of the protein is experimentally determined or generated by homology modeling.

[0040] In some embodiments, the sequence comprises an amino acid sequence, a DNA sequence, or a RNA sequence.

[0041] In some embodiments, the machine learning model exhibits a receiver operating characteristic area under the curve (AUC) value or average prediction score (APS) of at least 0.8. In some embodiments, the machine learning model exhibits a receiver operating characteristic area under the curve (AUC) value of at least 0.9.

[0042] The foregoing summary is not intended to define every aspect of the disclosure, and additional aspects are described in other sections, such as the following detailed description. The entire document is intended to be related as a unified disclosure, and it should be understood that all combinations of features described herein are contemplated, even if the combinations of features are not found together in the same sentence, or paragraph, or section of this document. Other features and advantages of the invention will become apparent from the following detailed description. It should be understood, however, that the detailed description and the specific examples, while indicating specific embodiments of the disclosure, are given by way of illustration only, because various changes and modifications within the spirit and scope of the disclosure will become apparent to those skilled in the art from this detailed description.

[0043] BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figures la and lb show an example process of training a machine learning (ML) model for efficient sequence-based protein-ligand binding pocket prediction (Figure la) and an example process of efficient sequence-based protein-ligand binding pocket prediction enabled by protein language models (PLMs) (Figure lb).

[0045] Figures 2a, 2b, 2c, and 2d show the best prediction head overall or for a class for each protein language model (PLM) and evaluation metric (average prediction score (APS) and area under curve (AUC)). The text above each bar indicates the configuration for each prediction head. The integers indicate either the width of the single-layer MLP or the number of attention heads of the multi-head attention (MHA). PDB or SSP indicates the single task or multitask setting, respectively.

[0046] Figures 3a, 3b, 3c, and 3d show the best prediction head and task regime for the model with the highest APS and AUC for each sequence identity threshold. Ankh-Large was used for all models in this subsection.

[0047] Figures 4a, 4b, and 4c show inference results and labels for PDB ID: 1KM0. Figure 4a shows ESP inference results, blue being negative prediction and red being positive. Figure 4b shows PM inference results using the same color scale. Figure 4c shows the binary PM test labels, where red indicates residues known to be associated with cryptic pocket formation, and green indicates unknown status and is masked during APS and AUC calculation.

[0048] Figures 5a, 5b, and 5c show inference results and labels for PDB ID:1BTP are visualized by projection onto the structure. Figure 5a shows ESP inference results. Figure 5b shows PM inference results. Figure 5c shows the binary PM test labels, where blue indicates residues known to not be associated with cryptic pocket formation, and green as above.

[0049] Figure 6 shows an example computing system for implementing the disclosed methods.

[0050] DETAILED DESCRIPTION

[0051] This disclosure describes novel methods and systems for identifying a ligand binding site (such as a cryptic ligand binding pocket) or ligand binding residues of a protein. The disclosed methods and systems employ “Efficient Sequence-based cryptic Pocket prediction” (ESP) leveraging advanced Protein Language Models (PLMs), and demonstrate significant improvement in predictive efficacy compared to existing machine learning (ML)-based cryptic pocket prediction methods. In the disclosed methods and systems, detection of cryptic pockets is achieved via training on readily available, non-cryptic pocket-specific data from a PDBBind dataset, rather than costly simulation and post-processing. Further, while predictions of existing methods often include positive signal broadly distributed over a target structure, the disclosed methods and systems produce more focused predictions and thus significantly increases downstream utility. In one aspect, this disclosure provides a method of identifying a ligand binding site of a protein. In some embodiments, the method comprises: (a) inputting a sequence of the protein into a machine learning model comprising a trained protein language model and a trained prediction head; (b) generating, by the protein language model, a contextual vector embedding for each of residues contained in the sequence of the protein, wherein the contextual vector embedding encodes functional and structural properties of a corresponding residue; and (c) inputting contextual vector embeddings of the residues contained in the sequence into the prediction head.

[0052] In some embodiments, the method comprises: (a) inputting a sequence of the protein into a machine learning model comprising a trained protein language model and a trained prediction head; (b) generating, by the protein language model, a contextual vector embedding for each of residues contained in the sequence of the protein, wherein the contextual vector embedding encodes functional and structural properties of a corresponding residue; (c) inputting contextual vector embeddings of the residues contained in the sequence into the prediction head; (d) determining a pocket prediction score for each of the residues contained in the sequence based on the contextual vector embeddings of the sequence; and (e) identifying a residue as a ligand binding residue constituting the ligand binding site of the protein if a value of the pocket prediction score of the residue is equal to or greater than a threshold value.

[0053] An example process for identifying a ligand binding site of a protein or identifying ligand binding residues of a protein is shown in Figures la and lb.

[0054] In some embodiments, the ligand binding site may include a cryptic ligand binding pocket. In some embodiments, the ligand binding site may include an allosteric ligand binding site.

[0055] In some embodiments, the method comprises inputting a classification token into the protein language model, and generating, by the protein language model, a summary or classification embedding.

[0056] In machine learning, an individual constituent of a sequence is often referred to as a “token.” Tokens can represent any semantic unit, such as a word, a part of a word, punctuation, a medical diagnosis code, a medical prescription code, an amino acid, a nucleic acid, etc. Machine learning models must convert each individual token into a group of numbers that represents the token in mathematical terms that the machine learning model can operate on and modify. A group of numbers representing an individual token is called an “embedding,” or a “representation,” or a “latent representation.” A “classification” token is a special token prepended (or appended) to a sequence of tokens that is input into a machine learning model. The machine learning model(s) most often associated with a classification token is the Transformer (and its derivatives), however use of a classification token is not necessarily limited to Transformers (or their derivatives). The embeddings output by Transformer models (and their derivatives) are “dynamic contextual embeddings” which contain both token-specific non-contextual information and token-specific contextual information unique to each sequence. Other types of machine learning models may also be capable of producing dynamic contextual embeddings.

[0057] An example of dynamic context in natural language is as follows. The word “bank” can refer to a financial institution. If it always had that meaning, it would have a static contextual definition. However, “bank” can also refer to the land adjacent a moving waterway. The specific meaning of any particular use of the word “bank” in general is therefore dynamic in its dependence on the context. The classification token, despite its identity being same for all sequences, produces an embedding that is dynamically context dependent in the manner just described. The classification token’s resultant embedding is dependent on the entire sequence, which makes the classification embedding useful for downstream prediction processes such as classifiers. This explains the origin of the nomenclature. Classification embeddings can also be used for regression and generative applications. Since the classification token produces an embedding that is dependent on the entire sequence, it may also be referred to as a “summary” token.

[0058] In some embodiments, the method may further include mapping the ligand binding residue on a three-dimensional structure model of the protein. In some embodiments, the three- dimensional structure model of the protein is experimentally determined (e.g., X-ray crystallography or cryo-EM) or generated by homology modeling or any suitable methods (e.g., AlphaFold).

[0059] In some embodiments, the sequence may include an amino acid sequence, a DNA sequence, or a RNA sequence.

[0060] As used herein, the term “protein” as used throughout this specification generally encompasses macromolecules comprising one or more polypeptide chains, i.e., polymeric chains of amino acid residues linked by peptide bonds. The term may encompass naturally, recombinantly, semi-synthetically or synthetically produced proteins. The term also encompasses proteins that carry one or more co- or post-expression -type modifications of the polypeptide chain(s), such as, without limitation, glycosylation, acetylation, phosphorylation, sulfonation, methylation, ubiquiti nation, signal peptide removal, N-terminal Met removal, conversion of pro-enzymes or pre-hormones into active forms, etc. The term further also includes protein variants or mutants which carry amino acid sequence variations vis-a-vis a corresponding native protein, such as, e.g., amino acid deletions, additions and / or substitutions. The term contemplates both full-length proteins and protein parts or fragments, e.g., naturally occurring protein parts that ensue from processing of such full-length proteins.

[0061] In another aspect, this disclosure provides a system of identifying a ligand binding site or ligand binding residues of a protein. In some embodiments, the system may include one or more processors configured to: (a) input a sequence of the protein into a machine learning model comprising a trained protein language model and a trained prediction head; (b) generate, by the protein language model, a contextual vector embedding for each of residues contained in the sequence of the protein, wherein the contextual vector embedding encodes functional and structural properties of a corresponding residue; (c) input contextual vector embeddings of the residues contained in the sequence into the prediction head; (d) determine a pocket prediction score for each of the residues contained in the sequence based on the contextual vector embeddings of the sequence; and (e) identify a residue as a ligand binding residue constituting the ligand binding site of the protein if a value of the pocket prediction score of the residue is equal to or greater than a threshold value.

[0062] In another aspect, this disclosure provides a system of identifying a ligand binding site or ligand binding residues of a protein. In some embodiments, the system comprises one or more processors configured to: (a) input a sequence of the protein into a machine learning model comprising a trained protein language model and a trained prediction head; (b) generate, by the protein language model, a contextual vector embedding for each of residues contained in the sequence of the protein, wherein the contextual vector embedding encodes functional and structural properties of a corresponding residue; and (c) input contextual vector embeddings of the residues contained in the sequence into the prediction head.

[0063] In some embodiments, the one or more processors are further configured to map the ligand binding residue on a three-dimensional structure model of the protein. In some embodiments, the three-dimensional structure model of the protein is experimentally determined (e.g, X-ray crystallography or cryo-EM) or generated by homology modeling or any suitable methods (e.g., AlphaFold).

[0064] In some embodiments, the one or more processors are further configured to input a classification token into the protein language model and generate, by the protein language model, a summary or classification embedding.

[0065] In some embodiments, the prediction head comprises a Multi-Head Attention (MHA) prediction head. In some embodiments, the MHA prediction head utilizes a PLM’s classification token. In some embodiments, the MHA prediction head does not utilize a PLM’s classification token. Transformers have been widely used for a variety of machine learning tasks. In general, the feature responsible for this uptake is the multi-head attention mechanism. Multi-head attention allows for the neural network to control the mixing of information between pieces of an input sequence, leading to the creation of richer representations, which in turn allows for increased performance on machine learning tasks. The key concept behind self-attention is that it allows the network to learn how best to route information between pieces of an input sequence (known as tokens). Attention applies to any type of data that can be formatted as a sequence. For instance, it can also be used with imaging data, where typically an image is represented as a sequence of patches that are then used as tokens in a sequence (more details on vision transformers in my post here). There are three main inputs to the attention mechanism: queries (Q), keys (K), and values (V). These inputs can all be described in terms of an "information retrieval" system of sorts. The general idea is that a query for each token in the input sequence is submitted. Those queries are then matched against a series of keys that describe values. The similarity of a given query to the keys determines how much information from each value to retrieve for that particular query. The output of the first matrix multiplication, where the similarity of each query is taken to each of the keys, is known as the attention matrix. The attention matrix depicts how much each token in the sequence is paying attention to each of the keys (hence the n x n shape).

[0066] In some embodiments, the one or more processors are further configured to tokenize each of the residues contained in each of the sequences with an additional contextual vector embedding. As used herein, the term “tokenization” refers to the process in which a token is substituted as a proxy for data. Accordingly, a token is a surrogate value that can be used to replace some underlying value in, for example, in a sequence.

[0067] In some embodiments, the step of inputting the contextual vector embeddings of each of the sequence into the prediction head may further include inputting one or more additional features of each of the residues contained in the sequence. In some embodiments, the step of determining the pocket prediction score for each of the residues contained in each of the sequences comprising determining the pocket prediction score for each of the residues contained in each of the sequences based on the contextual vector embeddings of each of the sequences and the one or more additional features of each of the residues contained in the sequence.

[0068] In some embodiments, the one or more additional features comprise one or more dynamic features or sequence features. In some embodiments, the one or more dynamic features comprise one or more of b-factor features, square fluctuation features, cross-correlation features, and perturbation response scanning features. In some embodiments, the one or more sequence features comprise one or more of residue identity features, conservation features, co-evolution features, and structure features.

[0069] In some embodiments, the machine learning model exhibits a receiver operating characteristic area under the curve (AUC) value or average prediction score (APS) of at least 0.8. In some embodiments, the machine learning model exhibits a receiver operating characteristic area under the curve (AUC) value of at least 0.9.

[0070] As used herein, a “machine learning model,” a “model,” or a “classifier” refers to a set of algorithmic routines and parameters that can predict an output(s) for a process input based on a set of input features, with or without being explicitly programmed. A structure of the software routines (e. , number of subroutines and relation between them) and / or the values of the parameters can be determined in a training process, which can use actual results of the process that is being modeled. Such systems or models are understood to be necessarily rooted in computer technology, and in fact, cannot be implemented or even exist in the absence of computing technology. While machine learning systems utilize various types of statistical analyses, machine learning systems are distinguished from statistical analyses by virtue of the ability to learn without explicit programming and being rooted in computer technology. A neural network or an artificial neural network is one set of algorithms used in machine learning for modeling the data using graphs of neurons. Any network structure may be used. Any number of layers, nodes within layers, types of nodes (activations), types of layers, interconnections, learnable parameters, and / or other network architectures may be used. Machine training uses the defined architecture, training data, and optimization to learn values of the learnable parameters of the architecture based on the samples and ground truth of training data.

[0071] A typical machine learning pipeline may include building a machine learning model from a sample dataset (referred to as a “training set”), evaluating the model against one or more additional sample datasets (referred to as a “validation set” and / or a “test set”) to decide whether to keep the model and to benchmark how good the model is, and using the model in “production” to make predictions or decisions against live input data captured by an application service. For training the model to be applied as a machine-learned model, training data is acquired and stored in a database or memory. The training data is acquired by aggregation, mining, loading from a publicly or privately formed collection, transfer, and / or access. Ten, hundreds, or thousands of samples of training data are acquired. The samples are from scans of different patients and / or phantoms. Simulation may be used to form the training data. The training data includes the desired output (ground truth), such as segmentation, and the input, such as protocol data and imaging data.

[0072] In some embodiments, the training set will be used to create a single classifier using any now or hereafter-known methods. In other embodiments, a plurality of training sets will be created to generate a plurality of corresponding classifiers. Each of the plurality of classifiers can be generated based on the same or different learning algorithm that utilizes the same or different features in the corresponding one of the pluralities of training sets.

[0073] Once trained, the machine-learned or trained classifier is stored for later application. The training determines the values of the learnable parameters of the network. The network architecture, values of non-learnable parameters, and values of the learnable parameters are stored as the machine-learned network. Once stored, the machine-learned network may be fixed. The same machine-learned network may be applied to different patients, different scanners, and / or with different imaging protocols for the scanning. The machine-learned network may be updated. As additional training data is acquired, such as through application of the network for patients and corrections by experts to that output, the additional training data may be used to re-train or update the training. For the machine learning model, input data structures of subreads can be used for the training. The input data structure may correspond to a window of in a sequence. Each training sample can include one of the first plurality of first data structures and a pocket prediction score likelihood of a residue being a ligand binding residue. The training is performed by optimizing parameters of the model based on outputs of the model matching or not matching corresponding labels of the first labels and optionally the second labels when the first plurality of first data structures and optionally the second plurality of second data structures are input to the model. An output of the model specifies whether a residue in sequence is likely a ligand binding residue. In some embodiments, the output of the model may include a probability of being in each of a plurality of states. The state with the highest probability can be taken as the state.

[0074] In some embodiments, the method further comprises training the machine learning model by: (a) retrieving a set of sequences from a database; (b) determining a reference pocket prediction score of each of residues contained in each of the sequences from the database; (c) inputting each of the sequences into the machine learning model comprising a protein language model and a prediction head; (d) generating by the protein language model a contextual vector embedding for each of residues contained in each of the sequences, wherein the contextual vector embedding encodes functional and structural properties of a corresponding residue; (e) inputting contextual vector embeddings of the residues contained in each of the sequence into the prediction head; (f) determining a pocket prediction score for each of the residues contained in each of the sequences based on the contextual vector embeddings of the residues contained in each of the sequences; (g) determining a value of a loss function associated with the prediction head based on deviation between the pocket prediction score for each of the residues and a reference pocket prediction score of the corresponding residue; and (h) performing iterative adjustments of parameters of the prediction head until a stopping criterion is reached. In some embodiments, the sopping criterian may include arbitrary number of iterations, validation loss escalation, the value of the loss function is smaller than a threshold loss function score, etc.

[0075] In yet another aspect, this disclosure provides a method of training a machine learning model for identifying a ligand binding site or ligand binding residues of a protein. In some embodiments, the method comprises: (i) retrieving a set of sequences from a database; (ii) determining a reference pocket prediction score of each of residues contained in each of the sequences from the database; (iii) inputting each of the sequences into a machine learning model comprising a protein language model and a prediction head; (iv) generating by the protein language model a contextual vector embedding for each of residues contained in each of the sequences, wherein the contextual vector embedding encodes functional and structural properties of a corresponding residue; (v) inputting contextual vector embeddings of the residues contained in each of the sequence into the prediction head; (vi) determining a pocket prediction score for each of the residues contained in each of the sequences based on the contextual vector embeddings of the residues contained in each of the sequences; (vii) determining a value of a loss function associated with the prediction head based on deviation between the pocket prediction score for each of the residues and a reference pocket prediction score of the corresponding residue; and (viii) performing iterative adjustments of parameters of the prediction head until a stopping criterion is reached.

[0076] In some embodiments, the one or more processors are further configured to train the machine learning model by: (a) retrieving a set of sequences from a database; (b) determining a reference pocket prediction score of each of residues contained in each of the sequences from the database; (c) inputting each of the sequences into the machine learning model comprising a protein language model and a prediction head; (d) generating by the protein language model a contextual vector embedding for each of residues contained in each of the sequences, wherein the contextual vector embedding encodes functional and structural properties of a corresponding residue; (e) inputting contextual vector embeddings of the residues contained in each of the sequence into the prediction head; (f) determining a pocket prediction score for each of the residues contained in each of the sequences based on the contextual vector embeddings of the residues contained in each of the sequences; (g) determining a value of a loss function associated with the prediction head based on deviation between the pocket prediction score for each of the residues and a reference pocket prediction score of the corresponding residue; and (h) performing iterative adjustments of parameters of the prediction head until a stopping criterion is reached.

[0077] In yet another aspect, this disclosure further provides a system of training a machine learning model for identifying a ligand binding site or ligand binding residues of a protein. In some embodiments, the system comprises one or more processors configured to: (i) retrieve a set of sequences from a database; (ii) determine a reference pocket prediction score of each of residues contained in each of the sequences from the database; (iii) input each of the sequences into a machine learning model comprising a protein language model and a prediction head; (iv) generate by the protein language model a contextual vector embedding for each of residues contained in each of the sequences, wherein the contextual vector embedding encodes functional and structural properties of a corresponding residue; (v) input contextual vector embeddings of the residues contained in each of the sequence into the prediction head; (vi) determine a pocket prediction score for each of the residues contained in each of the sequences based on the contextual vector embeddings of the residues contained in each of the sequences; (vii) determine a value of a loss function associated with the prediction head based on deviation between the pocket prediction score for each of the residues and a reference pocket prediction score of the corresponding residue; and (viii) perform iterative adjustments of parameters of the prediction head until a stopping criterion is reached.

[0078] In some embodiments, the database may include a protein ligand-binding pocket database, termed P-2024, comprising 258,000 unique protein structures, which have been curated from the protein database as of January 2024. A distinguishing feature of this database is precisely labeling of all residues engaged in ligand binding within each protein, thereby enabling the training of a pocket prediction algorithm.

[0079] In some embodiments, constructing the protein pocket database, such as P-2024, is delineated in several steps: (a) classification of proteins into families according to the SCOP 2 protocol (Antonina Andreeva, et al. (2020) Nucl. Acid Res., 48 (DI): D376-D382); (b) further subdivision of each protein family into subfamilies based on a sequence similarity threshold exceeding 80%, thereby grouping closely related proteins to enhance the efficiency and accuracy of subsequent pocket labeling; (c) removal of small and irrelevant ligands, alongside the filtering of shallow and non-druggable pockets, utilizing volume and pocket scores derived from the fpocket algorithm (Le Guilloux, V., et al. BMC Bioinformatics 10, 168 (2009)); and (d) assignment of scores to residues within a given protein structure, where a score of 0 indicates a non-pocket residue (i.e., does not bind to ligands in any protein structures within the respective subfamily), a score of 1 indicates a pocket residue (i.e., binds to ligands in the protein structure), and a score of -1 indicates a masked residue (i.e., does not bind to ligand in the current protein but binds to ligands in other structures within the subfamily). These residue labels are integral to the computation of the loss function employed in the training of the pocket prediction algorithm. In some embodiments, the database may include a PDBBind dataset, an ESM-2 SSP dataset, or a combination thereof.

[0080] In some embodiments, the step of retrieving the set of sequences from the database may include removing sequences that have a greater than 20% (e.g., 20%, 25%, 30%, 35%, 40%, 45%, 50%, 55%, 60%, 65%, 70%, 75%, 80%, 85%, 90%, or greater) sequence identity with another sequence in the set of validation and testing sequences.

[0081] In some embodiments, the machine learning model may further include a supervised learning model. Supervised learning models may include different approaches and algorithms including analytical learning, artificial neural network, backpropagation, boosting (metaalgorithm), Bayesian statistics, case-based reasoning, decision tree learning, inductive logic programming, Gaussian process regression, genetic programming, group method of data handling, kernel estimators, learning automata, learning classifier systems, minimum message length (decision trees, decision graphs, etc.), multilinear subspace learning, naive Bayes classifier, maximum entropy classifier, conditional random field, Nearest Neighbor Algorithm, probably approximately correct learning (PAC) learning, ripple down rules, a knowledge acquisition methodology, symbolic machine learning algorithms, subsymbolic machine learning algorithms, support vector machines, Minimum Complexity Machines (MCM) , random forests, ensembles of classifiers, ordinal classification, data pre-processing, handling imbalanced datasets, statistical relational learning, or Proaftn, a multicriteria classification algorithm, linear regression, logistic regression, deep recurrent neural network (e.g., long short term memory, LSTM), Bayes classifier, hidden Markov model (HMM) , linear discriminant analysis (LDA) , k-means clustering, densitybased spatial clustering of applications with noise (DBSCAN) , random forest algorithm, support vector machine (SVM), or any model described herein.

[0082] In some embodiments, the classifier may include a supervised or unsupervised Machine Learning or Deep Learning algorithm, Logistic Regression, Naive Bayes, Support Vector Machine, Decision Tree, Random Forest, Gradient Boosting, Regularizing Gradient Boosting, K-Nearest Neighbors, a continuous regression approach, Ridge Regression, Kernel Ridge Regression, Support Vector Regression, deep learning approach, Neural Networks, Convolutional Neural Network (CNNs), Recurrent Neural Networks (RNNs), Gated Recurrent Units (GRUs), Long Short Term Memory Networks (LSTMs), Generative Models, Generative Adversarial Networks (GANs), Deep Belief Networks (DBNs), Feedforward Neural Networks, Autoencoders, Variational Autoencoders, Normalizing Flow Models, Deniosing Diffusion Probabilistic Models (DDPMs), Score Based Generative Models (SGMs), Radial Basis Function Networks (RBFNs), Multilayer Perceptrons (MLPs), Stochastic Neural Networks, or any combination thereof.

[0083] In some embodiments, the model may include a convolutional neural network (CNN). The CNN may include a set of convolutional filters configured to filter the first plurality of data structures and, optionally, the second plurality of data structures. The filter may be any filter described herein. The number of filters for each layer may be from 10 to 20, 20 to 30, 30 to 40, 40 to 50, 50 to 60, 60 to 70, 70 to 80, 80 to 90, 90 to 100, 100 to 150, 150 to 200, or more. The kernel size for the filters can be 2, 3, 4, 5, 6, 7, 8, 9, 10, 11 , 12, 13, 14, 15, from 15 to 20, from 20 to 30, from 30 to 40, or more. The CNN may include an input layer configured to receive the filtered first plurality of data structures and, optionally, the filtered second plurality of data structures. The CNN may also include a plurality of hidden layers, including a plurality of nodes. The first layer of the plurality of hidden layers is coupled to the input layer. The CNN may further include an output layer coupled to a last layer of the plurality of hidden layers and configured to output an output data structure. The output data structure may include the properties.

[0084] Figure 6 is a functional diagram illustrating a programmed computer system in accordance with some embodiments. As will be apparent, other computer system architectures and configurations can be used to perform the described methods. Computer system 600, which includes various subsystems as described below, includes at least one microprocessor subsystem (also referred to as a processor or a central processing unit (CPU) 606). For example, processor 606 can be implemented by a single-chip processor or by multiple processors. In some embodiments, processor 606 is a general-purpose digital processor that controls the operation of the computer system 600. In some embodiments, processor 606 also includes one or more coprocessors or special purpose processors (e.g., a graphics processor, a network processor, etc.). Using instructions retrieved from memory 607, processor 606 controls the reception and manipulation of input data received on an input device (e.g., image processing device 603, I / O device interface 602), and the output and display of data on output devices e.g., display 601).

[0085] Processor 606 is coupled bi-directionally with memory 607, which can include, for example, one or more random access memories (RAM) and / or one or more read-only memories (ROM). As is well known in the art, memory 607 can be used as a general storage area, a temporary (e.g., scratchpad) memory, and / or a cache memory. Memory 607 can also be used to store input data and processed data, as well as to store programming instructions and data, in the form of data objects and text objects, in addition to other data and instructions for processes operating on processor 606. Also, as is well known in the art, memory 607 typically includes basic operating instructions, program code, data, and objects used by the processor 606 to perform its functions (e.g., programmed instructions). For example, memory 607 can include any suitable computer-readable storage media described below, depending on whether, for example, data access needs to be bi-directional or uni-directional. For example, processor 606 can also directly and very rapidly retrieve and store frequently needed data in a cache memory included in memory 607

[0086] A removable mass storage device 608 provides additional data storage capacity for the computer system 600 and is optionally coupled either bi-directionally (read / write) or unidirectionally (read-only) to processor 606. A fixed mass storage 609 can also, for example, provide additional data storage capacity. For example, storage devices 608 and / or 609 can include computer-readable media such as magnetic tape, flash memory, PC-CARDS, portable mass storage devices such as hard drives (e.g., magnetic, optical, or solid-state drives), holographic storage devices, and other storage devices. Mass storages 608 and / or 609 generally store additional programming instructions, data, and the like that typically are not in active use by the processor 606. It will be appreciated that the information retained within mass storages 608 and 609 can be incorporated, if needed, in a standard fashion as part of memory 607 e.g., RAM) as virtual memory.

[0087] In addition to providing processor 606 access to storage subsystems, bus 610 can be used to provide access to other subsystems and devices as well. As shown, these can include a display 601, a network interface 604, an input / output (I / O) device interface 602, an image processing device 603, as well as other subsystems and devices. For example, image processing device 603 can include a camera, a scanner, etc.; VO device interface 602 can include a device interface for interacting with a touchscreen (e.g., a capacitive touch sensitive screen that supports gesture interpretation), a microphone, a sound card, a speaker, a keyboard, a pointing device (e.g., a mouse, a stylus, a human finger), a global positioning system (GPS) receiver, a differential global positioning system (DGPS) receiver, an accelerometer, and / or any other appropriate device interface for interacting with system 600. Multiple I / O device interfaces can be used in conjunction with computer system 600. The I / O device interface can include general and customized interfaces that allow the processor 606 to send and, more typically, receive data from other devices such as keyboards, pointing devices, microphones, touchscreens, transducer card readers, tape readers, voice or handwriting recognizers, biometrics readers, cameras, portable mass storage devices, and other computers.

[0088] The network interface 604 allows processor 606 to be coupled to another computer, computer network, or telecommunications network using a network connection as shown. For example, through the network interface 604, the processor 606 can receive information (e.g., data objects or program instructions) from another network, or output information to another network in the course of performing method / process steps. Information, often represented as a sequence of instructions to be executed on a processor, can be received from and outputted to another network. An interface card or similar device and appropriate software implemented by (e.g., executed / performed on) processor 606 can be used to connect the computer system 600 to an external network and transfer data according to standard protocols. For example, various process embodiments disclosed herein can be executed on processor 606 or can be performed across a network such as the Internet, intranet networks, or local area networks, in conjunction with a remote processor that shares a portion of the processing. Additional mass storage devices (not shown) can also be connected to processor 606 through network interface 604.

[0089] In addition, various embodiments disclosed herein further relate to computer storage products with a computer-readable medium that includes program code for performing various computer-implemented operations. The computer-readable medium includes any data storage device that can store data that can thereafter be read by a computer system. Examples of computer- readable media include, but are not limited to: magnetic media such as disks and magnetic tape; optical media such as CD-ROM disks; magneto-optical media such as optical disks; and specially configured hardware devices such as application-specific integrated circuits (ASICs), programmable logic devices (PLDs), and ROM and RAM devices. Examples of program code include both machine code as produced, for example, by a compiler, or files containing higher level code (e.g., script) that can be executed using an interpreter. The computer system as shown in Figure 6 is an example of a computer system suitable for use with the various embodiments disclosed herein. Other computer systems suitable for such use can include additional or fewer subsystems. In some computer systems, subsystems can share components (e.g., for touchscreen-based devices such as smartphones, tablets, etc., I / O device interface 602 and display 601 share the touch-sensitive screen component, which both detects user inputs and displays outputs to the user). In addition, bus 610 is illustrative of any interconnection scheme serving to link the subsystems. Other computer architectures having different configurations of subsystems can also be utilized.

[0090] Additional Definitions

[0091] To aid in understanding the detailed description of the compositions and methods according to the disclosure, a few express definitions are provided to facilitate an unambiguous disclosure of the various aspects of the disclosure. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0092] Unless defined otherwise, all technical and scientific terms used herein have the meaning commonly understood by a person skilled in the art to which this invention belongs. The following references provide one of skill with a general definition of many of the terms used in this invention: Singleton et al., Dictionary of Microbiology and Molecular Biology (2nd ed. 1994); The Cambridge Dictionary of Science and Technology (Walker ed., 1988); The Glossary of Genetics, 5th Ed., R. Rieger et al. (eds ), Springer Verlag (1991); and Hale & Marham, The Harper Collins Dictionary of Biology (1991). As used herein, the following terms have the meanings ascribed to them below, unless specified otherwise.

[0093] As used herein, the term “classifiers” refers generally to various types of classifier frameworks, such as neural network classifiers, hierarchical classifiers, ensemble classifiers, etc. In addition, a classifier design can include a multiplicity of classifiers that attempt to partition data into two groups, either organized hierarchically or run in parallel, and then combined to find the best classification. Further, a classifier can include ensemble classifiers wherein a large number of classifiers all attempting to perform the same classification task are learned, but trained with different data / variables / parameters, and then combined to produce a final classification label. The classification methods implemented may be “black boxes” that are unable to explain their prediction to a user (which is the case if classifiers are built using neural networks, for example). The classification methods may be “white boxes” that are in a human-readable form (which is the case if classifiers are built using decision trees, for example). In other embodiments, the classification models may be “gray boxes” that can partially explain how solutions are derived (e.g., a combination of “white box” and “black box” type classifiers).

[0094] As used herein, the term “classification” refers to any number or other characters that are associated with a particular property of a sample. The classification can be binary (e.g., positive or negative) or have more levels of classification (e.g., a scale from 1 to 10 or 0 to 1). The term “cutoff’ or “threshold” refers to a predetermined number used in an operation. For example, a cutoff value can refer to a classification score as used above. A threshold value may be a value above or below which a particular classification applies. Either of these terms can be used in either of these contexts.

[0095] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. In some embodiments, the flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, a segment, or a portion of instructions, which may include one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the Figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustration, and combinations of blocks in the block diagrams and / or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.

[0096] These computer readable program instructions may be provided to a processor of a general- purpose computer, a special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function / act specified in the flowchart and / or block diagram block or blocks.

[0097] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0098] It will be understood that, although the terms “first,” “second,” etc., may be used herein to describe various elements, components, regions, layers and / or sections, these elements, components, regions, layers and / or sections should not be limited by these terms. These terms are only used to distinguish one element, component, region, layer or section from another element, component, region, layer or section. Thus, a first element, component, region, layer or section discussed below could be termed a second element, component, region, layer or section without departing from the teachings of example embodiments.

[0099] Unless specifically stated otherwise, as apparent from the above discussion, it is appreciated that throughout the description, discussions utilizing terms such as “processing,” “performing,” “receiving,” “computing,” “calculating,” “determining,” “identifying,” “displaying,” “providing,” “merging,” “combining,” “running,” “transmitting,” “obtaining,” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (or electronic) quantities within the computer system memories or registers or other such information storage, transmission or display devices.

[0100] As used herein, the term “if may be construed to mean “when” or “upon” or “in response to determining” or “in response to detecting,” depending on the context. Similarly, the phrase “if it is determined” or “if [a stated condition or event] is detected” may be construed to mean “upon determining” or “in response to determining” or “upon detecting [the stated condition or event]” or “in response to detecting [the stated condition or event],” depending on the context.

[0101] The terms or acronyms like “convolutional neural network,” “CNN,” “neural network,” “NN,” “deep neural network,” “DNN,” “recurrent neural network,” “RNN,” and / or the like may be interchangeably referenced throughout this document.

[0102] An “electronic device” or a “computing device” refers to a device that includes a processor and memory. Each device may have its own processor and / or memory, or the processor and / or memory may be shared with other devices as in a virtual machine or container arrangement. The memory will contain or receive programming instructions that, when executed by the processor, cause the electronic device to perform one or more operations according to the programming instructions.

[0103] The terms “memory,” “memory device,” “computer-readable medium,” “data store,” “data storage facility” and the like each refer to a non-transitory device on which computer-readable data, programming instructions or both are stored. Except where specifically stated otherwise, the terms “memory,” “memory device,” “computer-readable medium,” “data store,” “data storage facility” and the like are intended to include single device embodiments, embodiments in which multiple memory devices together or collectively store a set of data or instructions, as well as individual sectors within such devices.

[0104] The terms “processor” and “processing device” refer to a hardware component of an electronic device that is configured to execute programming instructions, such as a microprocessor or other logical circuit. A processor and memory may be elements of a microcontroller, custom configurable integrated circuit, programmable system-on-a-chip, or other electronic device that can be programmed to perform various functions. Except where specifically stated otherwise, the singular term “processor” or “processing device” is intended to include both single-processing device embodiments and embodiments in which multiple processing devices together or collectively perform a process.

[0105] In this document, the terms “communication link” and “communication path” mean a wired or wireless path via which a first device sends communication signals to and / or receives communication signals from one or more other devices. Devices are “communicatively connected” if the devices are able to send and / or receive data via a communication link. “Electronic communication” refers to the transmission of data via one or more signals between two or more electronic devices, whether through a wired or wireless network, and whether directly or indirectly via one or more intermediary devices.

[0106] As used herein, “zzz vitro" refers to events that occur in an artificial environment, e.g., in a test tube or reaction vessel, in cell culture, etc., rather than within a multi-cellular organism.

[0107] As used herein, “in vivo” refers to events that occur within a multi-cellular organism, such as a non-human animal.

[0108] It is noted here that, as used in this specification and the appended claims, the singular forms “a,” “an,” and “the” include plural reference unless the context clearly dictates otherwise.

[0109] As used herein, “plurality” means two or more. As used herein, a “set” of items may include one or more of such items.

[0110] As used herein, “including,” “comprising,” “containing,” or “having” and variations thereof are meant to encompass the items listed thereafter and equivalents thereof as well as additional subject matter unless otherwise noted.

[0111] As used herein, the phrases “in one embodiment,” “in various embodiments,” “in some embodiments,” and the like do not necessarily refer to the same embodiment, but may unless the context dictates otherwise.

[0112] As used herein, the terms “and / or” or “ / ” means any one of the items, any combination of the items, or all of the items with which this term is associated.

[0113] As used herein, the term “substantially” does not exclude “completely,” e.g., a composition which is “substantially free” from Y may be completely free from Y. Where necessary, the word “substantially” may be omitted from the definition of the present disclosure.

[0114] As used herein, the term “approximately” or “about,” as applied to one or more values of interest, refers to a value that is similar to a stated reference value. In some embodiments, the term “approximately” or “about” refers to a range of values that fall within 25%, 20%, 19%, 18%, 17%, 16%, 15%, 14%, 13%, 12%, 11%, 10%, 9%, 8%, 7%, 6%, 5%, 4%, 3%, 2%, 1%, or less in either direction (greater than or less than) of the stated reference value unless otherwise stated or otherwise evident from the context (except where such number would exceed 100% of a possible value). Unless indicated otherwise herein, the term “about” is intended to include values, e.g., weight percents, proximate to the recited range that are equivalent in terms of the functionality of the individual ingredient, the composition, or the embodiment.

[0115] As used herein, the term “each,” when used in reference to a collection of items, is intended to identify an individual item in the collection but does not necessarily refer to every item in the collection. Exceptions can occur if explicit disclosure or context clearly dictates otherwise.

[0116] As disclosed herein, a number of ranges of values are provided. It is understood that each intervening value, to the tenth of the unit of the lower limit, unless the context clearly dictates otherwise, between the upper and lower limits of that range is also specifically disclosed. Each smaller range between any stated value or intervening value in a stated range and any other stated or intervening value in that stated range is encompassed within the present disclosure. The upper and lower limits of these smaller ranges may independently be included or excluded in the range, and each range where either, neither, or both limits are included in the smaller ranges is also encompassed within the present disclosure, subject to any specifically excluded limit in the stated range. Where the stated range includes one or both of the limits, ranges excluding either or both of those included limits are also included in the present disclosure.

[0117] The use of any and all examples, or exemplary language (e.g., “such as”) provided herein, is intended merely to better illuminate the invention and does not pose a limitation on the scope of the present disclosure unless otherwise claimed. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the present disclosure.

[0118] All methods described herein are performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. In regard to any of the methods provided, the steps of the method may occur simultaneously or sequentially. When the steps of the method occur sequentially, the steps may occur in any order, unless noted otherwise. In cases in which a method comprises a combination of steps, each and every combination or subcombination of the steps is encompassed within the scope of the disclosure, unless otherwise noted herein.

[0119] Each publication, patent application, patent, and other reference cited herein is incorporated by reference in its entirety to the extent that it is not inconsistent with the present disclosure. Publications disclosed herein are provided solely for their disclosure prior to the fding date of the present disclosure. Nothing herein is to be construed as an admission that the present disclosure is not entitled to antedate such publication by virtue of prior disclosure. Further, the dates of publication provided may be different from the actual publication dates, which may need to be independently confirmed.

[0120] It is understood that the examples and embodiments described herein are for illustrative purposes only and that various modifications or changes in light thereof will be suggested to persons skilled in the art and are to be included within the spirit and purview of this application and scope of the appended claims.

[0121] Examples

[0122] EXAMPLE 1

[0123] Impact of Cryptic Pocket Prediction on Small Molecule Drug Design

[0124] Cryptic protein-ligand binding pocket prediction is a high-impact task because successful predictions can form the basis for novel structure-based small molecule pharmaceutical development programs. “Cryptic” or “non-obvious” pockets are so named because of the difficultly in recognizing such ligandable pockets with conventional tools. Whereas rigid, highly conserved, active sites and other non-cryptic pockets can commonly be identified by a structural biologist in receptor or enzyme protein structures determined via x-ray crystallography, cryptic pockets generally cannot. Instead, they are often discovered accidentally in experimentally solved protein structures in complex with wet-lab screening or fragment screening hits, or through extensive MD simulations using existing structures.

[0125] Finding a cryptic pocket on a pharmacologically validated protein target can motivate resource allocation to produce a novel, first-in-class medication. New binding sites for validated targets enable development of new chemistry with the potential for improvement in efficacy, dosing regimen, and reduction of side effects. Significant improvement in one or a combination of those three clinical properties can improve the standard of care for patients.

[0126] Cryptic pocket prediction and ligand binding pocket prediction, in general, also have an impact when engaging new targets for the first time. Because new targets may not have any reference compounds known to engage it, virtual screening and de novo molecular generation against a putative pocket may be required to find hits that medicinal chemists can turn into leads and eventually drug candidates.

[0127] Non-Cryptic Pocket Identification Algorithms

[0128] Computational identification of ligand binding sites on protein surfaces is a field with many mature tools, each with its own capabilities and limitations. They evaluate structures at the atomic level, the residue level, or arbitrary grid points in space. These methods perform best when there is a cavity on the protein surface that looks similar in volume and chemistry to common, known binding sites. They tend not to elucidate cryptic pockets because they only see structure snapshots in isolation and use relatively simple geometric characterizations augmented with basic chemical properties. Dynamic properties of proteins that might correlate to or imply a propensity to open cryptic pockets are not taken into account.

[0129] Two notable examples of methods of this type are LIGSITE (Hendlich, et al., Journal of Molecular Graphics and Modelling, 15(6):359- 363 , 1997) and Fpocket (Guilloux et al., BMC Bioinformatics, 10(1): 168, 2009). LIGSITE scans a protein for concave volumes and reports cavities above a minimum size. Fpocket uses Voronoi tessellation to define alpha spheres, which are then clustered prior to ranking clusters and scoring pockets. While dated by ML standards, these two algorithms are still relevant in computational chemistry and were used during label generation by the SOTA ML cryptic pocket algorithm discussed below. For a review of non-ML and ML pocket prediction approaches developed over the past three decades please see Zhao et al. (2020) and Di Palma et al. (2023).

[0130] Machine Learning-Based, Cryptic Pocket Identification Algorithms

[0131] Cryptic pocket identification algorithms aim to find hard-to-find areas of a protein where a drug can bind and achieve a disease-modifying effect. Simple geometric calculations to find concave surfaces will not suffice because cryptic pockets in experimentally-solved structures are generally not in a shape that allows for a drug to bind. Cryptic pockets tend to form by protein atom movement opening a pocket, or bringing distal parts close enough to form a pocket. Since the flexibility of a protein enables these phenomena, MD is a tool for finding cryptic pockets. Unfortunately, scaling MD for this purpose is time-consuming and cost-prohibitive. The SOTA ML approach to cryptic pocket prediction is PocketMiner (PM). It uses a protein’s backbone atom coordinates and sequence to predict whether or not each residue is associated with the formation of a cryptic pocket. It does not use information from SOTA PLMs. The use case is to feed PM protein structures absent any bound ligand, and predict where cryptic pockets are most likely to form. PM was trained using labels generated by LIGSITE, Fpocket, and conditional characterization of MD trajectories. It achieves improved accuracy and orders-of- magnitude improvement in inference compute cost compared to its predecessor CryptoSite (Cimermancic, et al., Journal of Molecular Biology, 428 (4):709-719, 2016), which works best when MD simulations are executed at inference time. The small number of structures used to generate training data was enough to produce meaningful predictions on structures that are completely different from the training structures. Also, it achieves this with only 736,155 trainable parameters and no PLM.

[0132] Results

[0133] This example presents four major results: (1) training using samples with less than 30% sequence similarity to validation or test samples, (2) training across all levels of sequence similarity, (3) projection of predictions onto structures and comparison with SOTA, and (4) the P- 2024 model.

[0134] Performance at the 30% Sequence Identity Limit

[0135] For the first set of results, training samples having greater than 30% sequence identity with any validation or testing sample were removed. This is done to reduce data leakage in the structure domain, because samples with relatively low sequence identity can still be structurally similar. Amino acids can sometimes be changed without altering the overall structure or function of the protein, for example, if the amino acids are very similar or solvent exposed. While structural similarity can occur even when sequence identity is below 30%, results at the 30% threshold are still a meaningful measure of generalization ability. Sensitivity to this threshold will be addressed in the next section. Further detail is provided in the Methods section (Example 2).

[0136] Figure 2 shows the best prediction head and task regime for the model with the highest APS and AUC for each PLM. Ankh-Large enabled the best AUC of 0.926, and the best AUC for all prediction head classes except MLP. ESM-2 15B enabled the best APS of 0.865, and the best MLP in both APS and AUC. ESM-2 15B also enables the best APS for MHA without [CLS] tokens. MHA using [CLS] tokens is the prediction head most common in Figure 2a, and PDBBind- label-only is the most common training regime. MLP was the top ESM-2 15B prediction head in terms of AUC, and for ProtT5-XL on both APS and AUC.

[0137] SOTA and PocketMiner achieved 0.81 and 0.87 APS and AUC on the test set, respectively. Ankh-Large, ESM22 15B, ESM-2 3B, and ProtT5-XL all produced prediction heads of each class (MLP, MHA with [CLS], and MHA without [CLS]) that outperformed SOTA in either or both APS and AUC. No prediction head atop ProtBert outperformed SOTA on either APS or AUC.

[0138] Ankh-Large has an embedding size of 1536, compared to 5120 and 2560 for ESM-2 15B and ESM-2 3B, respectively. Both ProtT5-XL and ProtBert have an embedding size of 1024. The benefit of the higher APS achievable via ESM-2 15B embeddings is offset by a significantly higher cost in terms of calculating the embeddings and training the prediction head. When a high AUC predictor is more appropriate, then Ankh-Large vastly outperformed both ESM-2 variants when computational cost is taken into account. If computational cost is the highest priority, users may wish to use ProtT5-XL embeddings with MLP prediction heads.

[0139] Ankh-Large

[0140] Performance metrics for ESP using Ankh-Large are summarized in Table 1. Table elements in bold indicate the best performance on either APC or AUC within a class of prediction heads. The prediction heads are classified as: LR / MLP, MHA without [CLS] tokens, and MHA using [CLS] tokens.

[0141] Ankh-Large achieved the best performance of all the PLMs under test in terms of AUC, and second best in terms of APS. Many Ankh-Large prediction heads outperformed SOTA for APS, and every prediction head outperformed SOTA for AUC. Of the Ankh-Large MHA results; the best came from those trained on the single task of predicting the PDBBind labels, and MHAs using 4 attention heads tended to underperform.

[0142] LR achieved APS and AUC of 0.805 and 0.889, respectively. This also outperformed SOTA on AUC. The high performance of LR indicates that the embeddings produced by Ankh- Large have a meaningful degree of linear correlation to the PDBBind-ligand-derived labels. Single layer MLPs of various widths struggled to make meaningful improvement beyond this baseline, with the best performing MLP having only 64 nodes (although this is not apparent with results represented by three significant figures).

[0143] Table 1: Average prediction score and ROCAUC results across architectures for ESP using Anhk- Large embeddings. Training samples with greater than 30% sequence identity with any member of validation or testing sets were removed.

[0144] Performance as a Function of Sequence Identity Limit

[0145] Figure 3 shows the best prediction head and task regime for the model with the highest APS and AUC for each sequence identity threshold. Ankh-Large was used for all models in this subsection.

[0146] Figure 3a shows that 7 of 12 best models were MHA using [CLS] tokens, whereas the remainder were MHA without use of [CLS] tokens. 4 of 12 best models were trained in the multitask setting using both PDBBind ligand-derived labels and ESM-2 SSP labels. The best- performing models were fairly consistent in terms of AUC until the 100% sequence identity threshold was used, when AUC rose to above 0.95. APS was less consistent, but trended upward overall with an increased sequence identity threshold, achieving an APS of nearly 0.90 at the sequence identity threshold of 100%.

[0147] Figure 3b shows performance as a function of sequence identity limit for LR / MLP prediction heads. The upward trending performance as sequence identity threshold is increase is smooth yet slight for AUC, and again more varied but overall uptrending for APS.

[0148] The relatively flat performance curve until the 100% sequence identity level is consistent with expectations of a well-generalized model. One potential confounder arises when multiple structures with different labels for the same residues enter the training set as the sequence identity limit increases. Detection, analysis, and mitigation of this possibility is left for future work.

[0149] Inference on Example Test Set Sequences

[0150] The PM test set offers two types of samples. The first has only positive labels and “unknown” labels, and the second has only negative labels and “unknown” labels. Positive labels indicate residues known to be associated with cryptic pocket formation, whereas negative labels indicate residues known to not be associated with cryptic pocket formation. “Unknown” labels indicate residues where cryptic pocket formation is neither known to occur nor known not to occur, and are masked when calculating APS and AUC.

[0151] E. coli Outer Membrane Transporter FecA (1KMO)

[0152] Figure 4 shows inference results and labels for E. coli outer membrane transporter FecA, which has the RCSB PDBID 1KM0. The ESP inference results for 1KM0, Figure 4a shows a positive signal focused in the area of the cryptic pocket positive labels shown in Figure 4c, which are the PM test set labels for this protein. Outside the area of the cryptic pocket prediction scores, ESP predicts that no cryptic pockets are present. This prediction is easy to interpret, and phenomenologically correct in the sense that the known pocket area stands out as such.

[0153] The 1KM0 PM inference result, as shown in Figure 4b, also has a positive signal in the area of the cryptic pocket positive labels. However, the PM inference result has a significant positive signal in many other places on the protein. While some of these positive signals in the unknown region may reveal new cryptic pockets, there are so many that it is difficult to motivate any particular starting point for drug design programs, and it is possible for there to be many false positives. Distillation of this result into actionable insights is, therefore, not as straightforward as with ESP.

[0154] Bovine Trypsin (1BTP)

[0155] Figure 5 shows inference results and labels for bovine trypsin, which has the RCSB PDBID 1BTP. The ESP inference results for 1KM0, Figure 5a show a negative signal focused in the area of the non-pocket, negative labels as shown in Figure 5c, which are the PM test set labels for this protein. ESP predicts that no cryptic pockets are present in the area of the non-pocket, negative labels. This prediction is also easy to interpret, and phenomenologically correct in the sense that the known non-pocket area stands out as such.

[0156] The 1BTP PM inference result, as shown in Figure 5b, has a positive signal broadly distributed across many residues with non-pocket negative labels. Whereas in the above example there were many positive predictions in an unknown area, here many positive predictions are in known non-pocket regions and are therefore clear false positives. Distillation of the PM inference result into actionable insights is therefore not possible because of false positives.

[0157] ESP achieves APS and AUC using Ankh-Large of 0.85 and 0.93 when trained using PDBBind-derived labels, respectively. ESM-2 15B achieves the best APS of all PLMs tested of 0.86, at a considerably higher computational cost due to the large PLM itself and embedding size of 5120, compared to Ankh’s embedding size of 1536. MHA prediction heads tend to outperform others except for ProtT5-XL, which favors MLP. ESM-2 SSP multitask training produced the best model in many cases, but not the majority.

[0158] SOTA, PocketMiner, achieves 0.81 and 0.87 APS and AUC on its test set. The PM test set was also used for all ESP APS and AUC calculations. All PLMs except ProtBert enabled models that outperformed SOTA on one or both of APS and ALIC. All Ankh-Large enabled models achieved the same.

[0159] Inference via ESP tends to produce positive signal in a more focused and spatially-localized manner, whereas inference via PM tends to produce positive signal broadly and smoothly distributed over many areas of a protein. While inference via ESP may miss some cryptic pockets, inference via PM includes non-trivial incidence of false positives. It was found that: (1) many PLMs enable ESP predictive efficacy beyond previous SOTA cryptic pocket prediction algorithms, and Ankh-Large and ESM-2 15B enable top AUC and APS, respectively, (2) that multitask learning using the ESM-2 secondary structure prediction dataset enhances predictive efficacy for many cases, and (3) that ESP outperforms PM by a significant margin (ROCAUC 0.93 vs 0.87) on its own test set.

[0160] Because mere logistic regression also outperformed SOTA on AUC, this result shows that Ankh-Large learned residue-level information from unsupervised training that linearly correlates with cryptic pocket formation propensity.

[0161] The P-2024 model

[0162] The P-2024 model has been developed utilizing a specialized pocket database, complemented by the ESM-2 model for protein sequence embedding and an MHA 16 network for the prediction head. This model is trained on a dataset comprising 84,803 unique protein structures (P-2024), which is more than four times the size of conventional models trained on the PDBBind dataset. The P-2024 demonstrates exceptional accuracy in the prediction of both normal ligandbinding pockets and cryptic pockets.

[0163] Performance metrics for the P-2024 model are summarized in Table 2, which includes: (1) a training-validation -testing split of the dataset, with 80% allocated for training, 10% for validation, and 10% for testing; (2) evaluation on the PocketMiner test set, comprising two scenarios: (i) training on the complete P-2024 dataset, and (ii) an ablation study where training data with sequence similarity greater than 30% to the testing set was excluded, resulting in the removal of 7,895 sequences from the training dataset. This methodology was implemented to mitigate data leakage within the structural domain, recognizing that low sequence identity may still correspond to structural similarity. It is acknowledged that amino acid substitutions may occur without altering the protein's overall structure or function, particularly when the amino acids are similar or solvent exposed.

[0164] Table 2: Performance of the P-2024 model in predicting ligand-binding pockets in protein structures.

[0165] The results indicate that structural similarity can persist even when sequence identity is below 30%, making this threshold a meaningful indicator of the model's generalization capability. Table 1 illustrates that the P-2024 model achieves superior accuracy in pocket prediction. The model recorded AUC values exceeding 0.98 and APS scores of 0.881 and 0.971 on the P-2024 and PocketMiner test sets, respectively. Remarkably, even after the removal of sequences with greater than 30% identity from the training set, the model sustained high performance levels on the PocketMiner test set, achieving AUC and APS scores of 0.939 and 0.901, respectively. The performance metrics of the P-2024 model surpass those of all other models referenced in this disclosure.

[0166] A small-molecule pharmaceutical development program invests significant capital in exactly one target. Therefore, prediction clarity and ease of interpretation are essential. Further, initiating drug development against a false positive cryptic pocket prediction would be very costly in terms of capital, time, and leadership bandwidth. False negative predictions are less costly in direct terms. Application of ESP at scale is, therefore, more valuable to small-molecule drug developers than SOT A.

[0167] EXAMPLE 2

[0168] This example describes additional methods used in EXAMPLE 1.

[0169] Datasets

[0170] Three datasets form the foundation for this work: (1) PDBBind (Su, et al., Journal of Chemical Information and Modeling, 59(2):895-913, 02 2019), (2) the ESM-2 SSP dataset, and (3) the PM validation and test sets. Training is conducted using the sequences and labels: (1) derived from the PDBBind dataset, and (2) directly from the ESM-2 SSP dataset. The PM validation and test sets serve those functions herein.

[0171] Since significant curation effort has been invested in the PDBBind dataset, it was used without further curation except for omission of proteins with synthetic residues. This results in a dataset of 17,986 complexes. Each protein’s amino acid sequence was extracted from its structure file. Positive labels were assigned to any residue containing at least one atom within 6 A of any ligand atom (Merveille Eguida and Didier Rognan. Tnt J Mol Sci, 23(20), Oct 2022. ISSN 1422- 0067). Negative labels are assigned everywhere else. Average sequence length, total number of pos labels, total number of labels.

[0172] The ESM-2 SSP dataset is used without modification, however since it is significantly larger than the PDBBind dataset we only use the x samples obtained using the “cv partition=0” and “split=train” options.

[0173] The PM validation and testing data are used with the same caveat applied by the PM authors in their work — namely that residues assigned a “unclassified” label are masked during loss calculation. This means that negative labels are only from rigid structures where the PM authors are confident no pocket can form, and positive labels are only associated with close proximity to ligands in resolved protein / ligand complexes. The validation and testing sets have 436 and 563 positive and 375 and 1,283 negative labels, respectively.

[0174] Data leakage is possible when identical sequences are present in the training set and either the validation or testing set. Because the PDB IDs are known for the PM validation and test sets, the authors used the RCSB APIs (Rose, et al., Journal of Molecular Biology, 433(11): 166704, 2021. ISSN 0022-2836) to identify sets of structures in the PDBBind dataset that are within arbitrary sequence identity thresholds to the PM validation and test structures. Sequence identity thresholds of 100%, 95%, 90%, 70%, 50%, and 30% were used. At training time, the structures within the desired identity threshold to the validation and test structures can be removed from the training set.

[0175] Table 3: Structures related to PM validation and test set structures at different thresholds of sequence identity.

[0176] In addition to data leakage prevention, training using different sequence identity thresholds offers insight into generalizability of ESP and dependence of generalizability on the specific PLM used. The number of structures from the RCSB PDB database (Berman, el al., The Protein Data Bank. Nucleic Acids Research, 28(1):235-242, 2000), PDBBind dataset, and PBDBind structures removed from the training dataset at different levels of sequence identity are reported in Table 3.

[0177] Architecture

[0178] Protein sequences extracted from PDBBind were input without modification into several PLMs: Ankh-Large, ESM-2 3B, ProtT5-XL, and ProtBert. Fine-tuning was not executed; embeddings are calculated and stored, and then used to train a prediction head using the PDBBind dataset and optionally the ESM-2 SSP dataset. For ProtT5-XL, the average embedding was used as a pseudo-[CLS] token.

[0179] Because the available datasets were small by ML standards, the prediction head architectures under test were limited in the number of trainable parameters they can justifiably have. Another key theme of this work was to prefer maximal simplicity, and introduce complexity only when and to the extent necessary. Results are presented for Logistic regression (LR), multilayer perceptron (MLP) with one hidden layer, and a single layer of multi-headed-attention (MHA). The number of learnable parameters per prediction head for each PLM is presented in Table 8.

[0180] Training Process

[0181] The training was executed using the PDBBind input data and labels derived via proximity to the ligand using binary cross-entropy loss. When training concurrently with the ESM-2 SSP data, loss for that task was evaluated via cross entropy loss. The total loss is a sum of the two, and a coefficient of SSP loss was used to adjust the relative significance of each loss. For the results presented here, the SSP loss coefficient used was 1.0. SGD has been used with no weight decay and a momentum value of 0.9.

[0182] Results for each specific PLM, prediction head, and task configuration are reported for the best model from 7 trials. Each trial was limited to a maximum of 40 epochs of training. A Figure of Merit (FOM) was defined using the validation APS and AUC as shown in Equation (1). Several early stopping criteria have been implemented: detection of any NaN FOM, identical FOM for two consecutive epochs, or FOM increasing beyond 105% of an individual trial’s lowest FOM. This strategy was chosen for convenience, since occurrence of any of these conditions tended not to lead to meaningful results.

[0183] FOM = 2 - APS - AUC (1)

[0184] The present disclosure is not to be limited in scope by the specific embodiments described herein. Indeed, various modifications of the invention, in addition to those described herein, will become apparent to those skilled in the art from the foregoing description and the accompanying figures. Such modifications are intended to fall within the scope of the appended claims.

[0185] Table 4: Average prediction score and ROCAUC results across architectures for ESP using ESM-2 15B embeddings. Training samples with greater than 30% sequence identity with any member of validation or testing sets has been removed.

[0186] Table 5: Average prediction score and ROCAUC results across architectures for ESP using ESM- 2 embeddings. Training samples with greater than 30% sequence identity with any member of validation or testing sets has been removed. Table 6: Average prediction score and ROCAUC results across architectures for ESP using Prot- Bert embeddings. Training samples with greater than 30% sequence identity with any member of validation or testing sets has been removed.

[0187] Table 7: Average prediction score and ROCAUC results across architectures for ESP using Prot- Bert embeddings. Training samples with greater than 30% sequence identity with any member of validation or testing sets has been removed. Table 8: Trainable parameters per prediction head for different architectures and PLMs, without concurrent SSP training.

Claims

CLAIMSWhat is claimed is:

1. A method of identifying a ligand binding site or ligand binding residues of a protein, comprising: inputting a sequence of the protein into a machine learning model comprising a trained protein language model and a trained prediction head; generating, by the protein language model (PLM), a contextual vector embedding for each of residues contained in the sequence of the protein, wherein the contextual vector embedding encodes functional and structural properties of a corresponding residue; and inputting contextual vector embeddings of the residues contained in the sequence into the prediction head.

2. The method of claim 1, further comprising after the step of inputting the contextual vector embeddings: determining a pocket prediction score for each of the residues contained in the sequence based on the contextual vector embeddings of the sequence; and identifying a residue as a ligand binding residue constituting a ligand binding site of the protein if a value of the pocket prediction score of the residue is equal to or greater than a threshold value.

3. The method of any one of the preceding claims, further comprising mapping the ligand binding residue on a three-dimensional structure model of the protein.

4. The method of any one of the preceding claims, further comprising inputting a classification token into the protein language model; and generating, by the protein language model, a summary or classification embedding.

5. The method of any one of the preceding claims, further comprising training the machine learning model by: retrieving a set of sequences from a database;determining a reference pocket prediction score of each of residues contained in each of the sequences from the database; inputting each of the sequences into the machine learning model comprising a protein language model and a prediction head; generating by the protein language model a contextual vector embedding for each of residues contained in each of the sequences, wherein the contextual vector embedding encodes functional and structural properties of a corresponding residue; inputting contextual vector embeddings of the residues contained in each of the sequence into the prediction head; determining a pocket prediction score for each of the residues contained in each of the sequences based on the contextual vector embeddings of the residues contained in each of the sequences; determining a value of a loss function associated with the prediction head based on deviation between the pocket prediction score for each of the residues and a reference pocket prediction score of the corresponding residue; and performing iterative adjustments of parameters of the prediction head until a stopping criterion is reached.

6. A method of training a machine learning model for identifying a ligand binding site or ligand binding residues of a protein, comprising: retrieving a set of sequences from a database; determining a reference pocket prediction score of each of residues contained in each of the sequences from the database; inputting each of the sequences into a machine learning model comprising a protein language model and a prediction head; generating by the protein language model a contextual vector embedding for each of residues contained in each of the sequences, wherein the contextual vector embedding encodes functional and structural properties of a corresponding residue; inputting contextual vector embeddings of the residues contained in each of the sequence into the prediction head;determining a pocket prediction score for each of the residues contained in each of the sequences based on the contextual vector embeddings of the residues contained in each of the sequences; determining a value of a loss function associated with the prediction head based on deviation between the pocket prediction score for each of the residues and a reference pocket prediction score of the corresponding residue; and performing iterative adjustments of parameters of the prediction head until a stopping criterion is reached.

7. The method of any one of claims 5-6, wherein the database comprises a protein ligandbinding pocket database.

8. The method of claim 7, wherein the protein ligand-binding pocket database is P-2024.

9. The method of any one of claims 5-6, wherein the database comprises a PDBBind dataset, an ESM-2 SSP dataset, or a combination thereof.

10. The method of any one of claims 5-9, wherein the step of retrieving the set of sequences from the database comprises removing sequences that have a greater than 30% sequence identity with another sequence in the set of validation and testing sequences.

11. The method of any one of the preceding claims, wherein the prediction head comprises a Multi-Head Attention (MHA) prediction head.

12. The method of claim 11, wherein the MHA prediction head utilizes a PLM’s classification token.

13. The method of claim 11, wherein the MHA prediction head does not utilize a PLM’s classification token.

14. The method of any one of the preceding claims, comprising tokenizing each of the residues contained in each of the sequences with an additional contextual vector embedding.

15. The method of any one of the preceding claims, wherein the step of inputting the contextual vector embeddings of each of the sequence into the prediction head further comprises inputting one or more additional features of each of the residues contained in the sequence.

16. The method of any one of the preceding claims, wherein the step of determining the pocket prediction score for each of the residues contained in each of the sequences comprising determining the pocket prediction score for each of the residues contained in each of the sequences based on the contextual vector embeddings of each of the sequences and the one or more additional features of each of the residues contained in the sequence.

17. The method of any one of claims 15-16, wherein the one or more additional features comprise one or more dynamic features or sequence features.

18. The method of claim 17, wherein the one or more dynamic features comprise one or more of b-factor features, square fluctuation features, cross-correlation features, and perturbation response scanning features.

19. The method of claim 17, wherein the one or more sequence features comprise one or more of residue identity features, conservation features, co-evolution features, and structure features.

20. The method of any one of the preceding claims, wherein the ligand binding site comprises a cryptic ligand binding pocket.

21. The method of claim 3, wherein the three-dimensional structure model of the protein is experimentally determined or generated by homology modeling.

22. The method of any one of the preceding claims, wherein the ligand binding site comprises an allosteric ligand binding site.

23. The method of any one of the preceding claims, wherein the sequence comprises an amino acid sequence, a DNA sequence, or a RNA sequence.

24. The method of any one of the preceding claims, wherein the machine learning model exhibits a receiver operating characteristic area under the curve (AUC) value or average prediction score (APS) of at least 0.8.

25. The method of any one of the preceding claims, wherein the machine learning model exhibits a receiver operating characteristic area under the curve (AUC) value of at least 0.9.

26. A system of identifying a ligand binding site or ligand binding residues of a protein, comprising one or more processors configured to: input a sequence of the protein into a machine learning model comprising a trained protein language model and a trained prediction head; generate, by the protein language model, a contextual vector embedding for each of residues contained in the sequence of the protein, wherein the contextual vector embedding encodes functional and structural properties of a corresponding residue; and input contextual vector embeddings of the residues contained in the sequence into the prediction head.

27. The system of claim 26, wherein the one or more processors are further configured to: determine a pocket prediction score for each of the residues contained in the sequence based on the contextual vector embeddings of the sequence; and identify a residue as a ligand binding residue constituting a ligand binding site of the protein if a value of the pocket prediction score of the residue is equal to or greater than a threshold value.

28. The system of any one of claims 26-27, wherein the one or more processors are further configured to map the ligand binding residue on a three-dimensional structure model of the protein.

29. The system of any one of claims 26-28, wherein the one or more processors are further configured to input a classification token into the protein language model and generate, by the protein language model, a summary or classification embedding.

30. The system of any one of claims 26-28, wherein the one or more processors are further configured to train the machine learning model by: retrieving a set of sequences from a database; determining a reference pocket prediction score of each of residues contained in each of the sequences from the database; inputting each of the sequences into the machine learning model comprising a protein language model and a prediction head; generating by the protein language model a contextual vector embedding for each of residues contained in each of the sequences, wherein the contextual vector embedding encodes functional and structural properties of a corresponding residue; inputting contextual vector embeddings of the residues contained in each of the sequence into the prediction head; determining a pocket prediction score for each of the residues contained in each of the sequences based on the contextual vector embeddings of the residues contained in each of the sequences; determining a value of a loss function associated with the prediction head based on deviation between the pocket prediction score for each of the residues and a reference pocket prediction score of the corresponding residue; and performing iterative adjustments of parameters of the prediction head until a stopping criterion is reached.

31. A system of training a machine learning model for identifying a ligand binding site or ligand binding residues of a protein, comprising one or more processors configured to: retrieve a set of sequences from a database; determine a reference pocket prediction score of each of residues contained in each of the sequences from the database; input each of the sequences into a machine learning model comprising a protein language model and a prediction head;generate by the protein language model a contextual vector embedding for each of residues contained in each of the sequences, wherein the contextual vector embedding encodes functional and structural properties of a corresponding residue; input contextual vector embeddings of the residues contained in each of the sequence into the prediction head; determine a pocket prediction score for each of the residues contained in each of the sequences based on the contextual vector embeddings of the residues contained in each of the sequences; determine a value of a loss function associated with the prediction head based on deviation between the pocket prediction score for each of the residues and a reference pocket prediction score of the corresponding residue; and perform iterative adjustments of parameters of the prediction head until a stopping criterion is reached.

32. The system of any one of claims 30-31, wherein the database comprises a protein ligandbinding pocket database.

33. The system of claim 32, wherein the protein ligand-binding pocket database is P-2024.

34. The system of any one of claims 30-31, wherein the database comprises a PDBBind dataset, an ESM-2 SSP dataset, or a combination thereof.

35. The system of any one of claims 30-34, wherein the step of retrieving the set of sequences from the database comprises removing sequences that have a greater than 30% sequence identity with another sequence in the set of validation and testing sequences.

36. The system of any one of claims 28-35, wherein the prediction head comprises a MultiHead Attention (MHA) prediction head.

37. The system of claim 36, wherein the MHA prediction head utilizes a PLM’s classification token.

38. The system of claim 36, wherein the MHA prediction head does not utilize a PLM’s classification token.

39. The system of any one of claims 26-38, wherein the one or more processors configured to tokenize each of the residues contained in each of the sequences with an additional contextual vector embedding.

40. The system of any one of claims 26-39, wherein the step of inputting the contextual vector embeddings of each of the sequence into the prediction head further comprises inputting one or more additional features of each of the residues contained in the sequence.

41. The system of any one of claims 26-40, wherein the step of determining the pocket prediction score for each of the residues contained in each of the sequences comprising determining the pocket prediction score for each of the residues contained in each of the sequences based on the contextual vector embeddings of each of the sequences and the one or more additional features of each of the residues contained in the sequence.

42. The system of any one of claims 40-41, wherein the one or more additional features comprise one or more dynamic features or sequence features.

43. The system of claim 42, wherein the one or more dynamic features comprise one or more of b-factor features, square fluctuation features, cross-correlation features, and perturbation response scanning features.

44. The system of claim 42, wherein the one or more sequence features comprise one or more of residue identity features, conservation features, co-evolution features, and structure features.

45. The system of any one of claims 26-44, wherein the ligand binding site comprises a cryptic ligand binding pocket.

46. The system of claim 28, wherein the three-dimensional structure model of the protein is experimentally determined or generated by homology modeling.

47. The system of any one of claims 26-46, wherein the ligand binding site comprises an allosteric ligand binding site.

48. The system of any one of claims 26-46, wherein the sequence comprises an amino acid sequence, a DNA sequence, or a RNA sequence.

49. The system of any one of claims 26-48, wherein the machine learning model exhibits a receiver operating characteristic area under the curve (AUC) value or average prediction score (APS) of at least 0.8.

50. The system of any one of claims 26-49, wherein the machine learning model exhibits a receiver operating characteristic area under the curve (AUC) value of at least 0.9.

Images