Analysis of antigen-binding proteins

Abstract

A method for predicting one or more features of an antigen-binding protein is provided, comprising: providing the amino acid sequence of at least part of one or more chains of the antigen-binding protein or an encoded sequence derived therefrom as input to a deep learning model comprising a sequence encoding module configured to take as input an amino acid sequence and to produce as output an embedding corresponding to the input sequence, and predicting the one or more features of the antigen-binding protein using the embedding produced by the sequence encoding module The sequence encoding module has been trained as part of a machine learning model comprising a structure encoding module, and using training sequence data and structure data from a plurality of antigen-binding proteins. Related methods, systems and products are also described.

Description

[0001] ANALYSIS OF ANTIGEN-BINDING PROTEINS

[0002] FIELD OF THE INVENTION

[0003] The present invention relates to methods for predicting features of antigen-binding proteins such as B cell receptors, antibodies and T cell receptors. The present invention also relates to methods of designing or providing an antigen-binding protein, such as a therapeutic antibody.

[0004] BACKGROUND TO THE INVENTION

[0005] An individual’s immunity against foreign molecules, or antigens, is governed by their repertoire of B cell receptors (BCRs). BCRs are proteins on the surfaces of B cells, comprising two pairs of heavy-light chain dimers. Each BCR has a unique amino acid sequence, which determines the BCR’s three-dimensional structure and thus its antigen recognition properties. Humans can produce up to 1015unique BCRs to cover the vast diversity of antigens. Once a BCR recognises its antigen, this activates the B cell, leading to the production of the BCR as a soluble protein, or antibody.

[0006] The BCR repertoire is a valuable resource for drug discovery. Several studies have shown that it is possible to extract therapeutic antibodies from the BCR repertoire using computational analyses. For example, Krawczyk et al. (2019, 2021) have shown that it is possible to find lead molecules from BCR repertoires by comparing each BCR to a known therapeutic antibody based on sequence homology, then selecting BCRs with high amino acid sequence homology. In Galson et al. (2020) another strategy using “clonotyping” is described, where the BCR repertoires of multiple individuals are clustered according to their sequence similarity. The BCR clonotypes, or clusters, that are enriched among individuals with ideal clinical characteristics are then selected. An advantage of clonotyping is its speed: clustering a typical repertoire dataset of 105-106BCRs only requires minutes to hours. All of these approaches are based on sequence similarity. However, antibodies with dissimilar amino acid sequences can adopt similar structures, suggesting that clonotyping can miss antibodies with convergent function (Kovaltsuk et al., 2017). Vice-versa, sequence-similar antibodies can sometimes fold into very different shapes, meaning that clonotyping can also encounter false positive errors.

[0007] More advanced machine learning methods can be used to refine this analysis. For example, “paratyping” can be used where a machine learning algorithm is first used to predict the amino acids that form the BCR binding site, or paratope (Richardson et al., 2021). Sequence identity is then calculated across the predicted paratope, as opposed to the whole BCR. Alternatively, machine learning can also be used to compute a numeric representation, or embedding, of the BCR amino acid sequence (Leem et al., 2022). Embeddings correspond to various aspects of BCR function; antibodies with similar properties tend to have close numeric representations. Furthermore, computing embeddings requires milliseconds of compute, making it practical for analysing BCR repertoires. However, these techniques also rely solely on sequences and therefore suffer from the above limitations to some extent.

[0008] One alternative strategy is to predict the three-dimensional structure of the BCR, then cluster the model structures (Robinson et al., 2021 ; Raybould et al., 2021). In principle, structural comparisons are ideal, but the computational overhead of structure prediction is huge; even the fastest structure prediction methods (e.g. Lee et al., 2023) require 1-10 seconds per BCR. For a BCR repertoire dataset (i.e. a single repertoire), this approximates to a minimum of one day of compute for generating the models before clustering.

[0009] Thus, there is still a need for improved methods for analysing antigen binding protein repertoires, such as BCR or TCR repertoires.

[0010] SUMMARY OF THE INVENTION

[0011] The present inventors previously developed (see Leem et al., 2022) a machine learning model capable of learning the “language” of antibodies, producing embeddings that can be used for a range of tasks such as tracing the B cell of origin of the antibody, quantifying immunogenicity and predicting the antibody’s binding site. The approach uses a transformer neural network pre-trained by masked language modelling (MLM). For antibodies, this involves randomly masking a portion of the amino acid sequence, then training the model to predict the original amino acid in place of the masked positions. This leads to the transformer learning representations that are optimised for MLM but are also suitable for other prediction tasks. However, the present inventors observed that antibody-specific transformer embeddings have a modest relationship to an antibody’s structure. They postulated that one way to improve these models would be to enhance the sequence representations with structural information using multimodal learning.

[0012] While some multimodal protein models have been proposed (e.g. Hsu et al., 2022; Yang et al., 2023), these tended to focus on how a protein language model can enhance structural representations, and were not specific to antigen binding proteins such as BCR or TCR, which have very specific properties. More importantly, these tended to still depend on using both protein structure and sequence as inputs even after training, thereby maintaining the drawbacks associated with the need to know or have the compute time to predict structure.

[0013] Thus, the inventors developed what they believe to be the first antigen-binding protein-specific (in particular, antibody-specific) multimodal model. This model is trained using both sequence information about antibodies (provided as text input) and structural information about antibodies (in these examples presented as graphs or tensors representing image-like inputs). In addition, after training the multimodal model, only the adjusted text encoder is used for generating representations, thus demonstrating that structural inputs are unnecessary at the time of inference. The inventors described several approaches to combine antibody sequence and structural representations. Specifically, they describe three different approaches to combine sequence and structural data, and also describe two different approaches to represent structural data that are amenable to combination of structural data with sequencebased methods. Through six experiments (structural correlation analysis, paratope prediction, HER2 binding, SARS-CoV-2 peptide binding prediction, data ablation test, viral / human antigen classification, see Examples Figure 6, Tables 1-3 and Figure 7), they showed that the multimodal training leads to a more refined antibody sequence representation model that encapsulates valuable functional information. They further showed that the multimodal training requires less task-specific training data to achieve similar performance as comparative sequence only models. This is particularly advantageous as task-specific training data is often difficult to acquire at scale, compared to the much more abundant non-task specific structure and especially sequence data.

[0014] The methods described herein generally use embeddings of a sequence only machine learning model trained in a multimodal manner using sequence and structure data for a plurality of antigen binding proteins. These embeddings are used to predict one or more features of antigen binding proteins, such as e.g. structural similarity as a proxy for functional similarity (e.g. by clustering a plurality of antigen binding proteins in embedding space), binding to a target, paratope, etc.

[0015] Thus, according to a first aspect, there is provided a method of predicting one or more features of antigen binding proteins. The method comprises: providing the amino acid sequence of at least part of one or more chains of the antigen-binding protein or an encoded sequence derived therefrom as input to a deep learning model comprising a sequence encoding module configured to take as input an amino acid sequence and to produce as output an embedding corresponding to the input sequence, wherein the sequence encoding module has been trained as part of a machine learning model comprising a structure encoder, and using training sequence data and structure data from a plurality of antigen-binding proteins; and predicting the one or more features of the antigen-binding protein using the embedding produced by the sequence encoding module.

[0016] The method may have one or more of the following features. Training sequence data may comprise amino acid sequences of at least part of one or more chains of the plurality of antigen-binding proteins, or encoded sequences derived therefrom. Training structure data may comprise 3D structure coordinates of at least part of one or more chains of the plurality of antigen-binding proteins, or information derived therefrom that characterises the relationship between pairs of amino acids in the structure.

[0017] The one or more chains may be selected from: an immunoglobulin light chain and an immunoglobulin heavy chain. The antigen-binding protein may be an antibody, a B cell receptor or a T cell receptor. The one or more chains may be selected from: an immunoglobulin light chain and an immunoglobulin heavy chain. Alternatively, the one or more chains may be selected from: a T cell receptor a chain, and a T cell receptor p chain. Alternatively, the one or more chains may be selected from: a T cell receptor y chain, and a T cell receptor 5 chain. The amino acid sequence may comprise, for each of the one or more chains, the amino acid sequence of the variable domain of the chain, or the amino acid sequence of the full chain. The amino acid sequence may comprise, for each of the one or more chains, the amino acid sequence of at least part of a variable domain of the chain, optionally wherein the amino acid sequence comprises, for each of the one or more chains, the amino acid sequence of at least: one or more CDR sequences, a junction sequence, a variable domain sequence. The amino acid sequence may further comprise one or more of a V-gene sequence or identifier, a J-gene sequence or identifier, and a D-gene sequence or identifier. The amino acid sequence may comprise the amino acid sequence of at least part of each of a pair of chains of the antigen-binding protein. The antigen binding protein may be a BCR or antibody and the pair of chains may be the heavy chain and the light chain of the antigen-binding protein. The methods of the present disclosure are expected to be particularly beneficial in the context of predicting features associated with pairs of chains. This is because many functional features of antigen-binding proteins such as those that are predicted using the methods described herein are believed to be associated with the structure of the antigenbinding protein, which is determined at least in part by both of the antigen-binding chains. As a consequence, structural Information that can be imbued in sequence based representations are expected to be most informative (in the context of functional features to be predicted), when the structure enhanced sequence model takes as input some sequence information about both chains. Providing the amino acid sequence as input to a deep learning model may comprise encoding the amino acid sequence using an encoding scheme wherein each amino acid corresponds to an individual token. Providing the amino acid sequence as input to a deep learning model may comprise providing a token for each of the one or more chains encoding the type of the chain. The type of the one or more chains may be encoded as a single token (e.g. a prefix token), such as e.g. “[!_]” for an antibody / BCR light chain and “[H]” for an antibody I BCR light chain, “[a]” for a TCR alpha chain and “[b]” for a TCR beta chain, etc. The method may start with the amino acid sequence or with the encoded sequence. Providing the amino acid sequence as input to a deep learning model may comprise encoding the amino acid sequence using an encoding scheme wherein: each gene sequence identifier corresponds to an individual token, each amino acid corresponds to an individual token, and / or sequences are encoded using tokens that each correspond to an individual k-mer or using byte-pair encoding, optionally wherein each sequence is encoded using overlapping k-mers. Providing the amino acid sequence or encoded sequence derived therefrom may comprise receiving the amino acid sequence or encoded sequence derived therefrom from a user interface, computing device or data store. Providing the amino acid sequence or encoded sequence derived therefrom may comprise retrieving the amino acid sequence or encoded sequence derived therefrom from a data store. Providing the amino acid sequence may comprise obtaining information from which the amino acid sequence can be derived and deriving the amino acid sequence. The information may comprise a nucleic acid sequence coding for the amino acid sequence. Deriving the amino acid sequence may comprise translating at least part of the nucleic acid sequence in silico. The one or more chains may comprise a first chain and a second chain. The first chain and the second chain or encoded sequences derived therefore may be provided as a concatenated string. The first chain and the second chain or encoded sequences derived therefore may be provided together with a token indicating the type of the respective chain.

[0018] The amino acid sequence may comprise the amino acid sequence of at least part of each of a pair of chains of the antigen-binding protein and wherein the amino acid sequences of both pairs of chains are provided as a single input to the deep learning model. Providing the amino acid sequence as input to a deep learning model may comprise encoding the amino acid sequence using an encoding scheme wherein amino acid sequences from different chains of the antigen-binding proteins are separated by a special token.

[0019] The one or more features of the antigen-binding protein may be functional features. The functional features may be selected from: an antigen binding property, a physicochemical property, and a functional similarity to a reference antigen binding protein. The one or more features of the antigen-binding protein may comprise the value of a similarity metric between the embedding for the antigen-binding protein and one or more embeddings produced by the deep learning model for respective one or more reference antigen-binding proteins. The one or more reference antigen-binding proteins may satisfy one or more predetermined criteria selected from: a therapeutic function, binding to a predetermined antigen, binding to a predetermined class of antigens. The one or more features of the antigen-binding protein may comprise a cluster of antigen-binding proteins to which the antigen-binding protein belongs, or a functional property derived therefrom, wherein the cluster has been obtained by clustering the embedding for the antigen-binding protein and one or more embeddings produced by the deep learning model for a plurality of further antigen-binding proteins. The plurality of further antigen-binding proteins may comprise one or more reference antigen-binding proteins with a known value for the functional property that is predicted by clustering.

[0020] The antigen-binding protein may be referred to as “query” antigen-binding protein (for which a feature is to be predicted), by contrast with “reference” antigen-binding proteins (for which the feature is known).

[0021] Antigen-binding proteins that have embeddings that are more similar to those of reference antigen-binding proteins that have one or more functional properties may be more likely to have these one or more functional properties than antigen-binding proteins that have embeddings that are less similar to those of the reference antigen-binding proteins. Thus, the methods of the present aspect may comprise predicting whether the antigen-binding protein is likely to have one or more functional features associated with a reference antigenbinding protein by comparing the embeddings produced by the deep learning model for the antigen-binding protein and the reference antigen-binding protein, respectively. Comparing the embeddings for the antigen-binding protein and the reference antigen-binding protein may comprise determining a similarity between said embeddings. Determining a similarity may comprise computing any similarity metric known in the art, such as e.g. a distance metric (e.g. Euclidian distance), a correlation metric, etc. Embeddings may be considered “similar” and therefore the antigen-binding protein may be considered likely to have the one or more functional features when the similarity is above a predetermined threshold (when the similarity metric increases with increasing similarity, such as e.g. correlation) or below a predetermined threshold (when the similarity metric decreases with increasing similarity, such as e.g. a distance metric).

[0022] Determining a similarity may comprise clustering embeddings for a plurality of antigen-binding proteins comprising one or more query antigen-binding protein and the one or more reference antigen-binding proteins. A query antigen-binding protein may be considered to be similar to a reference antigen-binding protein when the query and reference antigen-binding protein have embeddings in the same cluster. Any clustering method known in the art may be used in this context.

[0023] A therapeutic function may be a known or predicted therapeutic function. For example, a reference antigen-binding proteins may be an antigen-binding protein with known therapeutic function such as a known therapeutic antibody. As another example, a reference antigenbinding protein may be an antigen-binding protein that has been identified in an individual who has a desired phenotypic characteristic, such as resilience to a particular disease or one or more symptoms thereof. Resilience to a particular disease or a symptom of a disease may be determined to be present when an individual would be expected to show the disease or symptom thereof due to the presence of one or more risk factors, but does not show the disease or symptom thereof.

[0024] The one or more reference antigen-binding proteins may comprise positive reference antigenbinding proteins that have one or more desired properties, and negative reference antigenbinding proteins that do not have the one or more desired properties, and the method may comprise clustering the embedding for the antigen-binding protein and the reference antigenbinding proteins and identifying one or more clusters that are enriched for positive reference antigen-binding proteins. The method may comprise determining that the identified cluster comprises antigen-binding proteins that are likely to have one or more functional features. For example, the one or more desired properties may comprise having been identified in an individual who has a desired phenotypic characteristic. The one or more functional features may comprise the ability to induce the desired phenotypic characteristic in an individual. For example, the desired characteristic may be resilience to a particular disease or symptom of a disease. In such cases the one or more functional features may comprise the ability to treat the disease or alleviate the symptom. A query antigen-binding protein with an embedding that clusters with such an identified cluster may be considered likely to have the one or more functional features. The disease may be selected from a cancer (such as e.g. breast cancer), a neurodegenerative disease (such as e.g. amyotrophic lateral sclerosis), and an infectious disease (such as e.g. COVID-19).

[0025] The one or more features of the antigen-binding protein may comprise one or more features selected from: information identifying one or more positions of the amino acid sequence that are part of the paratope of the antigen-binding protein, information indicating whether the antigen-binding protein is likely to bind a predetermined antigen, and information indicating whether the antigen-binding protein is likely to have a predetermined physicochemical property (e.g. temperature stability, pH stability).

[0026] The deep learning model may comprise a classification model that is configured to classify an antigen-binding protein between a plurality of classes using as input the embedding from the sequence encoding module, a regression model that is configured to predict a continuous or semi-continuous parameter using as input the embedding from the sequence encoding module, and / or a generative model that is configured to predict an antigen-binding protein sequence using the embedding from the sequence encoding module. The deep learning model comprising the sequence encoding module and the classification model may have been trained using training data comprising sequence data for a plurality of antigen-binding proteins and data indicating which of the plurality of classes each of the plurality of antigen-binding proteins belongs to. The deep learning model comprising the sequence encoding module and the regression model may have been trained using training data comprising sequence data for a plurality of antigen-binding proteins and data indicating the value of the parameter for each of the plurality of antigen-binding proteins. The deep learning model comprising the sequence encoding module and the regression model may have been trained using training data comprising sequence data for a plurality of antigen-binding proteins and data indicating a sequence associated with the antigen-binding proteins, such as e.g. paired sequences. The deep learning model may comprise one or more sequence embedding modules. For example, the deep learning model may comprise a pair of sequence embedding modules, the outputs of which may be concatenated and used as input to a classification model, regression model or generative model. Thus, the predicted continuous or semi-continuous parameter may be associated with one or more antigen-binding proteins, where the amino acid sequence of at least part of one or more chains of a respective antigen-binding protein or an encoded sequence derived therefrom is provided as input to a respective sequence embedding module of the deep learning model. This may be useful to provide outputs that compare two antigen binding proteins, such as e.g. to predict structural similarity.

[0027] The method may comprise repeating, for a plurality of antigen-binding proteins, the step of providing the amino acid sequence of at least part of one or more chains of the antigen-binding protein or an encoded sequence derived therefrom as input to a deep learning model comprising a sequence encoding module, thereby obtaining an embedding for each of the plurality of antigen-binding proteins, wherein the plurality of antigen-binding proteins comprise at least 106antigen-binding proteins and / or wherein the plurality of antigen-binding proteins comprise a BCR repertoire from one or more individuals.

[0028] The sequence encoding module may be an encoder of a sequence-to-sequence deep learning model, encoder only deep learning model or decoder only deep learning model. The sequence encoding module may be a transformer-based encoder, such as a RoFormer, or a transformer-based decoder.

[0029] The sequence encoding module may have been trained in at least two steps: a first step in which the sequence encoding module is trained using training sequence data from a plurality of antigen-binding proteins, and a second step in which the sequence encoding module is trained as part of a machine learning model comprising a structure encoder, and using training sequence data and structure data from a plurality of antigen-binding proteins. The first step may be referred to as “unimodal learning” or “unimodal training” or “unimodal pretraining”. The second step may be referred to as “multimodal training”. The further training step may be referred to as “fine-tuning”, “task-specific training” or “task-specific fine-tuning”. The first step may use a masked language modelling task. The sequence encoding module may have been further trained as part of a machine learning model comprising the sequence encoding module and: (i) a classifier model that is configured to classify antigen-binding proteins between a plurality of classes, using training sequence data and associated class label data for a plurality of antigen-binding proteins; or (ii) a regression model that is configured to predict a property of an antigen-binding protein, using training sequence data and associated ground truth data comprising a known value for said property for a plurality of antigen-binding proteins. Alternatively, the sequence encoding module (after unimodal and multimodal training) may be fixed (frozen) when training the machine learning model comprising the trained sequence encoding module and a classifier or regression model.

[0030] The structure encoder may be configured to take as input structure data for an antigen-binding protein and provide as output an embedded representation of the structure data. The structure encoder may have been trained in at least two steps in which the structure encoder is trained using training structure data from a plurality of proteins, comprising a first substep in which the structure encoder is trained using training structure data from a plurality of proteins comprising proteins that are not antigen-binding proteins, and a second substep in which the structure encoder is trained using training structure data from a plurality of antigen-binding proteins. The structure encoder may additionally be trained in a further step in which the structure encoder is trained as part of a machine learning model comprising a sequence encoding module, and using training sequence data and structure data from a plurality of antigenbinding proteins. Alternatively, the parameters of the structure encoder may be frozen during training of the machine learning model comprising a sequence encoding module and the structure encoding module.

[0031] The structure data for an antigen-binding protein may comprise information characterising the relationship between pairs of amino acids or pairs of atoms in the structure of the antigenbinding protein. The structure encoder may comprise a graph neural network or an imagebased neural network. The structure encoder may be a deep learning model. The structure data may comprise information characterising the relationship between pairs of amino acids in the structure of the antigen-binding protein in the form of a graph. Any graph-based model for protein structure encoding may be used. The structure encoder may be or comprise a graph neural network. The structure encoder may comprise one or more geometric vector perceptron layers. The structure encoder may have the same architecture as any of the structure models described in Hsu et al. 2022 (e.g. models described as GVP-GNN, GVP- GNN-large and GVP-Transformer). The structure encoder may have been pretrained as described in Hsu et al. 2022. The structure encoder may have been instead or additionally pretrained using structures of antigen-binding proteins. The structure encoder may be a GVP- transformer.

[0032] The structure data may comprise information characterising the relationship between pairs of amino acids or atoms in the structure of the antigen-binding protein in the form of one or more matrices or flattened version thereof each comprising coefficients Cy that represent a physicochemical property of a pair of amino acids or atoms at respective positions i and j in the sequence corresponding to the structure. Any image based encoder may be used in such embodiments. For example, the structure encoder may be a transformer-based image model. The structure encoder may be a Swin transformer, e.g. Swin Transformer v2.

[0033] The structure encoder may have been trained using structure data comprising structure data that has been obtained experimentally. The structure encoder may have been trained using structure data that has been computationally predicted. The structure encoder may have been trained using structure data comprising both structure data obtained experimentally and structure data that has been computationally predicted. The present inventors have found it advantageous for the structure data comprising mostly (e.g. at least 50%, at least 60%, at least 70%, at least 80%, at least 90%) or substantially only structure data that has been obtained experimentally.

[0034] The first step may use any image analysis task, such as simple masked image modelling. For example, in embodiments the structure encoder may take as input information characterising the relationship between pairs of amino acids in the structure of the antigen-binding protein in the form of tensors (image like data). For example, the information may be in the form of one or more matrices or flattened version thereof each comprising coefficients Cy that represent a physicochemical property of a pair of amino acids at respective positions i and j in the sequence corresponding to the structure.

[0035] The first step may use a task where a decoder model generates protein sequences from structural embeddings from the structure encoder, or where a model comprising the structure encoder is trained to predict one or more properties of a protein from structural embeddings from the structure encoder. For example, graph-based models such as ESM (comprising a geometric vector perceptron), Equiformer or E(n)GNN may be trained for a particular task (sequence prediction, physicochemical property of a structure, etc) and the pretrained structural encoder model used in these models may then be used in a second (multimodal) training step as described herein.

[0036] The information characterising the relationship between pairs of amino acids in the structure of the antigen-binding protein may be in the form of a graph comprising a plurality of nodes connected by a plurality of edges, where each node represents an amino acid in the structure and each edge represent a relationship between a pair of amino acids. Each node may be associated with at least one scalar feature and at least one vector feature, and each edge may be associated with at least one scalar feature and at least one vector feature, and vector features may characterise the direction of vectors between atoms in the structure. Each node may be associated with one or more features selected from: one or more vectors representing the directions of bonds between backbone atoms at the amino acid, a scalar feature that specifies the identity of the amino acid, and more scalar features that represent a dihedral angle of the backbone structure at the amino acid. Each edge may be associated with one or more features selected from: a vector representing the direction of the vector between carbon alpha atoms of the pair of amino acids, a scalar feature representing the distance between carbon alpha atoms of the pair of amino acids, and a scalar feature representing the distance along the backbone structure between the amino acids in the pair. The information characterising the relationship between pairs of amino acids in the structure of the antigenbinding protein may be in the form of one or more matrices or flattened version thereof each comprising coefficients Cy that represent a physicochemical property of a pair of amino acids at respective positions i and j in the sequence corresponding to the structure, optionally wherein the one or more matrices comprise one or more of: a matrix with coefficients Cy that represent the (physical) distance between the amino acids at positions i and j in the 3D structure; a matrix with coefficients Cy that represent the electrostatic force between the amino acids at positions i and j in the structure; a matrix with coefficients Cy that represent the mean hydrophobicity of the amino acids at positions i and j in the structure; a matrix with coefficients Cy that represent the Boman index associated with the amino acids at positions i and j in the structure; a matrix with coefficients Cy that represent the mean isoelectric point of the amino acids at positions i and j in the structure; and a matrix with coefficients Cy that represent the mean solvent accessible surface area of the amino acids at positions i and j in the structure. The information characterising the relationship between pairs of amino acids in the structure of the antigen-binding protein may comprise at least a matrix with coefficients cij that represent the (physical) distance between the amino acids at positions i and j in the 3D structure. The distance between amino acids may be a Euclidian distance in cartesian coordinates. The distance between amino acids may be determined based on the position of the alpha carbons of the respective amino acids. The distance between amino acids may be determined based on the position of the centroids of the respective amino acids. Centroid positions may be determined based on the coordinates of all atoms of a residue or backbone atoms of a residue.

[0037] The sequence encoding module may have been trained as part of a machine learning model comprising a structure encoder, using training sequence data and structure data from a plurality of antigen-binding proteins, and at least one multimodal learning objective selected from: a contrastive loss, masked multimodal modelling and image-text matching. The sequence encoding module may have been trained as part of a machine learning model comprising a structure encoder, using training sequence data and structure data from a plurality of antigen-binding proteins, and one of: (i) CLIP contrastive loss; (ii) a combination of a contrastive loss and masked language modelling; and (iii) a global contrastive loss, masked multimodal modelling and image-text matching. The sequence encoding module may have been trained as part of a machine learning model comprising a structure encoder, using training sequence data and structure data from a plurality of antigen-binding proteins, and at least one multimodal learning objective selected from: a contrastive loss, masked multimodal modelling and image-text matching. The sequence encoding module may have been trained as part of a machine learning model comprising a structure encoder, using training sequence data and structure data from a plurality of antigen-binding proteins, and CLIP contrastive loss. For example, a structural encoder comprising a graph neural network (optionally a graph neural network comprising geometric vector perceptron layers, such as any of the structure models described in Hsu et al. 2022) or an image encoder (e.g. with a Swin Transformer v2 architecture) may have been trained together with a sequence encoding module (e.g. an encoder, e.g. with a RoFormer architecture) using a CLIP contrastive loss as described in Radford et al., 2021. The sequence encoding module may have been trained as part of a machine learning model comprising a structure encoder, using training sequence data and structure data from a plurality of antigen-binding proteins, and a combination of a contrastive loss and masked language modelling. For example, a structural encoder comprising a graph neural network (optionally a graph neural network comprising geometric vector perceptron layers, such as any of the structure models described in Hsu et al. 2022) or an image encoder (e.g. with a Swin Transformer v2 architecture) may have been trained together with a sequence encoding module (e.g. with a RoFormer architecture) using a contrastive loss and a masked language modelling loss as described in Yu et al., 2022. The contrastive loss and the masked language modelling loss may be combined using respective weights between 1 and 2 (e.g. the contrastive loss having a weight twice that of the MLM loss or vice versa). The sequence encoding module may have been trained as part of a machine learning model comprising a structure encoder in the form of an image encoder (e.g. with a Swin Transformer v2 architecture), using training sequence data and structure data from a plurality of antigenbinding proteins, and a global contrastive loss, a masked multimodal modelling loss and an image-text matching task., as described in Singh et al., 2022.

[0038] According to a second aspect, there is provided a method of providing an antigen-binding protein having a desired property, the method comprising: providing one or more query sequences, predicting one or more features of the one or more query sequences using a method according to the first aspect, wherein the features are indicative of the presence or absence of the desired property, and identifying a sequence of the one or more query sequences having the desired property using the predicted feature(s) and / or selecting one or more of the query antigen-binding proteins based on the one or more features.

[0039] The method may have any one or more of the following features.

[0040] The plurality of query antigen-binding proteins may each be associated with a respective amino acid sequence comprising at least part of a heavy and / or light chain sequences obtained by bulk B cell repertoire sequencing or single B cell sequencing. The plurality of query antigen-binding proteins may comprise at least 100, at least 1000, at least 10,000, at least 100,000 or at least 1 ,000,000 proteins (i.e. the method may comprise obtaining predicted one or more features for at least 100, at least 1000, at least 10,000, at least 100,000 or at least 1 ,000,000 corresponding amino acid sequences. The plurality of query antigen-binding proteins may each be associated with a respective amino acid sequence obtained by bulk B cell sequencing of the heavy or light chain repertoire in a sample, such as a sample from a subject. The plurality of sequences may be a subset of a set of sequences obtained by bulk B cell sequencing of the heavy or light chain repertoire in a sample. The method according to the present aspect may have any of the features described in relation to the first aspect.

[0041] The method may further comprise obtaining one or more candidate antigen-binding proteins each comprising the amino acid sequence of a selected query antigen-binding protein. The method may further comprise testing the one or more candidate antigen-binding proteins for the desired property. The one or more candidate antigen-binding proteins may be antibodies or fragment thereof. Obtaining a candidate antigen-binding protein may comprise identifying a coding sequence for the candidate antigen-binding protein and expressing the sequence in a suitable expression system (such as e.g. in a suitable host cell).

[0042] The desired property may be a desired binding property (such as e.g. an affinity (e.g. kD), the ability to bind one or more targets, the ability to bind one or more targets with an affinity (e.g. kD) above one or more respective thresholds, etc.), a desired expression property (such as e.g. an increased expression level compared to a standard in one or more expression systems, an expression level above a predetermined level in one or more expression systems, a yield above a predetermined level in one or more expression systems, etc.), a desired stability property (such as e.g. a stability above a certain threshold in one or more conditions), or a combination thereof. The desired property may include the ability to bind a predetermined target. Testing the one or more candidate antigen-binding proteins for the desired property may comprise identifying one or more antigens that the one or more candidate antigen-binding proteins bind(s) to, for example by testing for binding to one or more candidate antigens. Testing the one or more candidate antigen-binding proteins for the desired property may comprise identifying one or more antigens that the one or more candidate antigen-binding proteins is / are likely to bind to, for example by comparison with one or more antibodies with known targets. The antigen-binding protein may be a therapeutic antibody, and the desired property may comprise binding of a therapeutic target. An antigen-binding protein may also be referred to herein as “immune protein”.

[0043] Testing the one or more candidate antigen-binding proteins for the desired property may comprise identifying the presence or absence of a desired phenotype in an organism (such as e.g. an animal model) or cell expressing the one or more candidate antigen-binding proteins. Identifying the presence of a desired phenotype may comprise expressing the one or more candidate antigen binding proteins in one or more model cells (e.g. one or more cell lines) or organisms (such as e.g. one or more animal models).

[0044] The method may further comprise optimising the sequence of at least one of the one or more candidate antigen-binding proteins. Optimising the sequence of a candidate antigen-binding protein may be performed for example using any antibody optimisation technique known in the art.

[0045] The method may further comprise identifying one or more targets by screening antibodies from the same source(s) as the one or more samples against a plurality of candidate peptides. The plurality of candidate peptides may be selected based on the species from which the one or more samples originate. For example, the source of the one or more samples may be one or more human subjects and the antibody repertoire(s) from the same source(s) as the one or more samples may be screened against a set of candidate peptides representative of the human peptidome to select a plurality of candidate peptides.

[0046] Identifying an antigen that the one or more candidate antigen-binding proteins bind(s) to may comprise using one or more targets identified by screening antibodies from the same source(s) as the one or more samples against a plurality of candidate peptides. The method may further comprise filtering the set of identified query antigen-binding proteins based on one or more criteria. The one or more criteria may apply to the identity of an antigen or set of antigens that a candidate antigen-binding protein bind(s) to or is predicted to bind to.

[0047] According to a third aspect, there is provided a method of analysing a plurality of antigenbinding proteins each comprising a respective query sequence, such as e.g. a plurality of antigen binding proteins comprising respective query sequences from one or more B cell repertoires, the method comprising: performing the method of any embodiment of the first aspect for each of the query sequences, wherein the one or more features predicted for each query sequence are the sequence embedding themselves, and clustering the sequence embeddings of at least a subset of the query sequences, thereby identifying clusters of functionally similar antigen binding proteins.

[0048] The step of clustering the sequence embeddings of at least a subset of the query sequences may comprise clustering at least a subset of the query sequences together with sequence embeddings obtained for one or more reference antigen binding proteins. A reference antigenbinding protein may be an antigen-binding protein that has known specificity.

[0049] The method may further comprise predicting a specificity for at least one of the plurality of antigen binding proteins, based on the embeddings of the at least one of the plurality of antigen binding proteins clustering together with embeddings of a reference antigen-binding protein. The specificity of the at least one of the plurality of antigen binding proteins may be predicted to be the same as that of a reference antigen-binding protein that has embeddings clustering together with the embeddings of the at least one of the plurality of antigen binding proteins. The specificity of the at least one of the plurality of antigen binding proteins may be predicted to be the same as that which is most common amongst a plurality of reference antigen-binding proteins that have embeddings clustering together with the embeddings of the at least one of the plurality of antigen binding proteins. The specificity of the at least one of the plurality of antigen binding proteins may be predicted to be the same as that of a reference antigenbinding protein that has embeddings clustering together with the embeddings of the at least one of the plurality of antigen binding proteins, and that is most similar in embedding space to the embeddings of the at least one of the plurality of antigen binding proteins. Similarity may be measured using any similarity metric, such as e.g. a distance metric (e.g. Euclidean distance, cosine distance, Manhattan distance, etc.)

[0050] According to a fourth aspect, there is provided a method of providing a tool for predicting one or more features of antigen binding proteins. The method comprises: providing training data comprising sequence data and structure data from a plurality of antigen-binding proteins, wherein the sequence data for an antigen-binding protein comprises the amino acid sequence of at least part of one or more chains of the antigen-binding protein or an encoded sequence derived therefrom and the structure data for an antigen-binding protein comprises data characterising the 3D structure of the antigen-binding protein, and training, using the training data, a deep learning model that comprises: a sequence encoding module configured to take as input a query amino acid sequence and to produce as output an embedding corresponding to the input sequence, and a structure encoder; and providing the sequence encoding module of the deep learning model or a sequence only model including the sequence encoding module for predicting one or more features of an antigen-binding protein, wherein the prediction does not use the structure encoder. The method of the present aspect may have any of the features described in relation to the first aspect. The method may have any one or more of the following features.

[0051] The method may have any one or more of the features described in relation to the preceding aspects.

[0052] In particular, the method may comprise obtaining the training data from one or more databases, computing devices or user interfaces. The method may comprise encoding the amino acid sequences in the training data using an encoding scheme wherein each amino acid corresponds to an individual token. The method may comprise encoding the amino acid sequences in the training data using an encoding comprising a token for each of the one or more chains encoding the type of the chain. The method may comprise providing encoded amino acid sequences for one or more chains of each antigen-binding protein in the training data as a single string. In other words, encoded sequences from multiple chains of the same protein may be provided as a concatenated chain. Concatenated chains may be separated by a special token. Other encoding schemes may be used, as described herein including by reference to the first aspect.

[0053] The training data may comprise the value of one or more features of at least a subset of the antigen-binding proteins for which sequence data is provided.

[0054] The training data may comprise sequence data from a plurality of antigen-binding proteins for which structure data is not included in the training data. Providing training data may comprise obtaining training data and filtering the training data prior to using the data to train the deep learning model. Filtering the training data may comprise one or more of: filtering sequence data using one or more criteria that apply to the length of CDR regions, and filtering sequence data to exclude sequences that have over a predetermined level of sequence identity with another sequence. The structure training data may not be filtered to remove structures that have similar or identical sequences.

[0055] The method may further comprise training a deep learning model that comprises the sequence encoding module and not the structure encoder.

[0056] The method may comprise training a deep learning model comprising the sequence encoding module and a classification model that is configured to classify an antigen-binding protein between a plurality of classes using as input the embedding from the sequence encoding module. The method may comprise training a deep learning model comprising the sequence encoding module and the classification model using training data comprising sequence data for a plurality of antigen-binding proteins and data indicating which of the plurality of classes each of the plurality of antigen-binding proteins belongs to. The method may comprise training a deep learning model comprising the sequence encoding module and a regression model that is configured to predict a continuous or semi-continuous parameter using as input the embedding from the sequence encoding module, using training data comprising sequence data for a plurality of antigen-binding proteins and data indicating the value of the parameter for each of the plurality of antigen-binding proteins.

[0057] The method may comprise training a deep learning model comprising the sequence encoding module and a generative model that is configured to predict an antigen-binding protein sequence using as input the embedding from the sequence encoding module, using training data comprising sequence data for a plurality of antigen-binding proteins and data indicating a sequence associated with the antigen-binding proteins, such as e.g. paired sequences.

[0058] The deep learning model may comprise one or more sequence embedding modules. For example, the deep learning model may comprise a pair of sequence embedding modules, the outputs of which may be concatenated and used as input to a classification model, regression model or generative model. Thus, the deep learning model may be trained to provide an output associated with one or more antigen-binding proteins, where the amino acid sequence of at least part of one or more chains of a respective antigen-binding protein or an encoded sequence derived therefrom is provided as input to a respective sequence embedding module of the deep learning model. This may be useful to provide outputs that compare two antigen binding proteins, such as e.g. to predict structural similarity. In embodiments, the deep learning model comprises a pair of sequence encoding modules, the outputs of which are concatenated and fed into a regression model (e.g. a single regression layer) trained to predict a similarity metric. The similarity metric may be a structural similarity, such as e.g. RMSD between CDRH3 loops or constant chains, or a normalised (e.g. z-score) version thereof. The outputs of the two sequence encoding modules may be concatenated as (antibody_embedding_1 , antibody_embedding_2, |antibody_embedding_2 - antibody_embedding_1 |), where antibody_embedding_1 is the output of a first sequence embedding module, and antibody_embedding_2 is the output of a second sequence embedding module.

[0059] The method may comprise training the sequence encoding module in at least two steps: a first step in which the sequence encoding module is trained using training sequence data from a plurality of antigen-binding proteins, and a second step in which the sequence encoding module is trained as part of the deep learning model comprising the structure encoder, and using training sequence data and structure data from a plurality of antigen-binding proteins. Thus, the method may further comprise (in addition to training, using the training data, a deep learning model that comprises: a sequence encoding module configured to take as input a query amino acid sequence and to produce as output an embedding corresponding to the input sequence, and a structure encoder), a step of pretraining the sequence encoding module using training sequence data from a plurality of antigen-binding proteins. The pretraining step use e,g, a masked language modelling task.

[0060] The method may comprise training the structure encoder in at least two steps: a first step in which the structure encoder is trained using training structure data from a plurality of proteins, and a second step in which the structure encoder is trained as part of the deep learning model comprising the sequence encoding module, and using training sequence data and structure data from a plurality of antigen-binding proteins. Thus, the method may further comprise (in addition to training, using the training data, a deep learning model that comprises: a sequence encoding module configured to take as input a query amino acid sequence and to produce as output an embedding corresponding to the input sequence, and a structure encoder), a step of pretraining the structure encoder using training structure data from a plurality of antigenbinding proteins. The pretraining step use e,g, a masked image modelling task. For example, in embodiments the structure encoder may take as input information characterising the relationship between pairs of amino acids in the structure of the antigen-binding protein in the form of tensors (image like data). For example, the information may be in the form of one or more matrices or flattened version thereof each comprising coefficients Cy that represent a physicochemical property of a pair of amino acids at respective positions i and j in the sequence corresponding to the structure.

[0061] The first step may use a task where a decoder model generates protein sequences from structural embeddings from the structure encoder, or where a model comprising the structure encoder is trained to predict one or more properties of a protein from structural embeddings from the structure encoder. For example, graph based models such as ESM (comprising a geometric vector perceptron), Equiformer or E(n)GNN may be trained for a particular task (sequence prediction, physicochemical property of a structure, etc) and the pretrained structural encoder model used in these models may then be used in a second (multimodal) training step as described herein.

[0062] The structure data for an antigen-binding protein may comprise information characterising the relationship between pairs of amino acids or pairs of atoms in the structure of the antigenbinding protein. The method may comprise determining the information characterising the relationship between pairs of amino acids or pairs of atoms in the structure of the antigenbinding protein, from 3D structure data e.g. in the form of atom coordinates.

[0063] Training the deep learning model may comprise using at least one multimodal learning objective selected from: a contrastive loss, masked multimodal modelling and image-text matching. Training the deep learning model may comprise using one of: (i) CLIP contrastive loss; (ii) a combination of a contrastive loss and masked language modelling; and (iii) a global contrastive loss, masked multimodal modelling and image-text matching. For example, the structural encoder may comprise a graph neural network (optionally a graph neural network comprising geometric vector perceptron layers, such as any of the structure models described in Hsu et al. 2022) or an image encoder (e.g. with a Swin Transformer v2 architecture). Training the deep learning model may comprise training the sequence encoding module (e.g. an encoder, e.g. with a RoFormer architecture) in a model comprising the structure encoding module using a CLIP contrastive loss as described in Radford et al., 2021.

[0064] As another example, the deep learning model may comprise a structural encoder comprising a graph neural network (optionally a graph neural network comprising geometric vector perceptron layers, such as any of the structure models described in Hsu et al. 2022) or an image encoder (e.g. with a Swin Transformer v2 architecture) and a sequence encoding module (e.g. an encoder, e.g. with a RoFormer architecture), and the sequence encoding module may be trained as part of a model also comprising the structure encoding module using a contrastive loss and a masked language modelling loss as described in Yu et al. , 2022. The contrastive loss and the masked language modelling loss may be combined using respective weights between 1 and 2 (e.g. the contrastive loss having a weight twice that of the MLM loss or vice versa).

[0065] In any of these embodiments, the structure encoding module parameters may be trained or may be frozen (i.e. maintaining weights of the pretrained structure encoding module constant) when training the model comprising the sequence encoding module and the structure encoding module. For example, using contrastive loss (as demonstrated in the examples below) only backpropagate to the sequence encoder, i.e. only the sequence encoder is learning anything from the alignment of sequence and structure inputs. Without wishing to be bound by theory, it is believed that training the sequence encoding module as part of such a model while maintaining the weights of the structure encoding module constant (i.e. frozen) is both computationally efficient and in line with the aim of imbuing “structure information” into the sequence encoding module.

[0066] The method may further comprise providing the trained deep learning model or any part thereof, such as e.g. the sequence encoding module, to a user. The methods described herein are computer implemented unless context indicates otherwise, such as e.g. where a sample is obtained, processed, analysed, or a molecule or composition produced, tested or used for any other purpose. Indeed, the complexity of training and / or deploying deep learning models involving millions to billions of learnable parameters optimised over many training iterations from complex data including sequence and structure information is far beyond the capability of the human mind.

[0067] According to a fifth aspect, there is provided a system comprising: a processor; and a computer readable medium comprising instructions that, when executed by the processor, cause the processor to perform the steps of the method of any embodiment of any preceding aspect. The instructions may cause the processor to perform the steps of the method of any embodiment of the any of the preceding aspects.

[0068] According to a sixth aspect, there is provided one or more computer readable media comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the method of any embodiment of any preceding method aspect. The instructions may cause the processor to perform the steps of the method of any embodiment of the first to fourth aspects.

[0069] According to a seventh aspect, there is provided a computer program product comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the method of any embodiment of any preceding method aspect. The instructions may cause the processor to perform the steps of the method of any embodiment of the first to fourth aspects.

[0070] BRIEF DESCRIPTION OF THE FIGURES

[0071] Figure 1 is a flowchart illustrating schematically a method of predicting features of antigen binding proteins according to the disclosure.

[0072] Figure 2 is a flowchart illustrating schematically a method of providing a tool for predicting features of antigen binding proteins according to the disclosure.

[0073] Figure 3 shows an embodiment of a system for implementing methods of the disclosure.

[0074] Figure 4 shows a training process used in examples of the disclosure. A. Pretraining of AntiBERTa2, a unimodal sequence encoder based on the RoFormer architecture (Su et al., 2021). B. Pretraining of AntiSwin, a unimodal structure encoder based on the Swin Transformer v2 architecture (Liu et al., 2022). C. Multimodal training of ImmunoCLIP-ESM and ImmunoCoca-ESM: AntiBERTa2 + ESM-IF1 structure encoder, trained using CLIP or CoCa objectives. D. Multimodal training of ImmunoCLIP-Swin and ImmunoFLAVA-Swin: AntiBERTa2 + Swin Transformer V2 (either 1 -channel or 6-channel version), trained using CLIP or FLAVA objectives. E. Task specific model training for ImmunoCLIP-ESM: left: multimodal training, right: architecture and training of structure-enhanced sequence model (updated AntiBERTa2 model comprising encoder of ImmunoCLIP-ESM). ImmmuoCLIP is also referred as “ImmunoCSSP” herein (i.e. the names ImmunoCLIP-ESM and ImmunoCSSP- ESM are used interchangeably, as are the names ImmunoCLIP-Swin and ImmunoCSSP- Swin). ImmunoFLAVA is also referred to herein as “ImmunoSSA” (i.e. the names ImmunoFLAVA-Swin and ImmunoSSA-Swin are used interchangeably).

[0075] Figure 5 shows schematically the architecture of models used in examples of the disclosure. A. AntiBERTa2, a unimodal sequence encoder based on the RoFormer architecture (Su et al., 2021). B. AntiSwin, a unimodal structure encoder based on the Swin Transformer v2 architecture (Liu et al., 2022). C. Projection layer (used in the examples described using CLIP - top: general architecture, bottom specific embodiment used in the examples). D. ImmunoCSSP-ESM: AntiBERTa2 + ESM-IF1 structure encoder, length corresponds to number of residues in the antibody, 3 x 3 refers to the x, y, z coordinates of the backbone nitrogen, alpha carbon, and carbon atoms, respectively. E: ImmunoCSSP-Swin: AntiBERTa2 + Swin Transformer V2 (either 1-channel or6-channel version). F. ImmunoCoCa: AntiBERTa2 + ESM-IF1 using the CoCa multimodal training objective. G. ImmunoSSA: RoFormer AntiBERTa2 + Swin transformer Transformer V2 (either 1-channel or 6-channel version).

[0076] Figure 6 shows the results of comparison of antibody structure RMSD between pairs of antibody structures and embedding similarity (cosine distance) between the same pairs, using different models. A. Sequence-only model (AntiBERTa2). B. Structure-enhanced sequence only model: AntiBERTa2 encoder from the ImmunoCLIP-ESM (ImmunoCSSP-ESM) after multimodal training.

[0077] Figure 7 shows the results of an analysis of the impact of the amount of task-specific training data (x axis) on the performance (in terms of AUC, y axis) of a task-specific regression model comprising a sequence only encoder (AntiBERTa2, in blue), a structure-enhanced sequence only encoder (in green, where the AntiBERTa2 encoder from the ImmunoCSSP-ESM model is used after training with the multimodal objective), and a structure-enhanced sequence only encoder (in orange, where the AntiBERTa2 encoder from the ImmunoSSA-Swin model is used after training with the multimodal objective) for two tasks: prediction of HER2 binding (A) and prediction of SARS-CoV-2 peptide binding (B).

[0078] Figure 8 illustrates schematically the architecture of a heavy chain and a light chain of a BCR I antibody. A. Architecture of an Ig heavy chain. The approximate boundaries of the V, D, and J genes are marked, along with the boundaries of the junction. A segment of the V-gene toward the N-terminus is in a dotted boundary as the read length from many NGS methods are too short to cover this region and / or some primers used for NGS are slightly inset within the V region. However, it is still possible to infer the V-gene using the sequence within the solid boundaries. B. Same as A., but for the light chain. Figure 9 shows schematically a scheme for evaluating multimodal learning results used in examples of the disclosure, and selected results. A. The cosine distance between ImmunoCSSP embeddings, d, is compared to structural RMSD. B. Two structurally similar antibodies (PDB 7BEK, green, and 7LKA, cyan) that are grouped together by the ImmunoCSSP approach, but not by a corresponding unimodal language model, AntiBERTa2. C. Antibodies in PDB 4R26 (yellow) and 5CEX (magenta) that are grouped together by the ImmunoCSSP approach, but not by a corresponding unimodal language model, AntiBERTa2.

[0079] Figure 10 shows results obtained using examples of models of the disclosure (roformer-esm- sadbab-v2.7=lmmunoCSSP-ESM (ImmunoCLIP); esm2-esm-sabdab-esm2.650m=ESM2- 650M sequence encoder and ESM-IF1 structure encoder, trained with CSSP; roformer-esm- naturalantibody=AntiBERTa2 as sequence encoder, ESM-IF1 as structure encoder, predicted structures used for CSSP) and comparative models (AntiBERTa2 sequence only encoder, labelled as Ro Form er- v2.7; and ESM2-650M sequence only encoder), showing that multimodal training (here using contrastive pretraining) improves prediction performance. A. Contrastive pretraining with SAbDab structures progressively improves model performance on HER2 binding prediction up to 5 epochs. B. Contrastive pretraining with SAbDAB also improves few-shot performance. This plot shows the result of ablation studies in which parts of the training dataset were randomly removed. Roformer-v2.7=AntiBERTa2, unimodal sequence encoder; ESM2-650M=unimodal sequence encoder; Roformer- CSSP=immunoCSSP-ESM (ImmunoCLIP), Roformer-CSSP-Sim=roformer-esm- naturalantibody; ESM2-CSSP=esm2-esm-sabdab-esm2.650).

[0080] Figure 11 shows results of investigation of the effect of multimodal training epochs and training set size (ablation studies) on HER2 binding prediction, using various language models). A. Effect of the number of epochs of multimodal training on HER2 binding prediction performance (mean area under receiver operator characteristic curve, AU ROC) as a function of the number of multimodal training epochs, using different sequence models (AntiBERTa2- CSSP-lmmunoCSSP, see Example 2; AntiBERTy=antibody-specific transformer encoder model described in Ruffolo et al. 2021 ; ESM2-650M is a transformer protein language model described in Lin et al. 2022). B. Effect of structural pretraining using decreasing sizes of fine- tuning data. X-axis: size of the fine-tuning dataset (supervised training of language model for HER2 binding prediction). Y-axis: AUROC of HER2 binding prediction. Continuous line=language model with unimodal training. Dashed line=language model with structural pretraining. Points are average and shaded area are standard deviation across models optimised with 5 random seeds (i.e. selecting a slightly different random subset of data each time, and slightly different randomly initialised weights). Figure 12 shows Pearson correlation coefficients between predicted and actual structural similarity before and after multimodal training. Structural similarity is predicted from embeddings using a regression head as described in the Examples. The calculation of structural similarity for the CDRH3 loop and the entire antibody variable domain (Fv) is also described in the Examples. Multimodal training increases the structural information content of all unimodal language models’ embeddings.

[0081] Figure 13 shows a diagram illustrating schematically a clustering experiment performed using methods of the disclosure and comparative methods.

[0082] DETAILED DESCRIPTION OF THE INVENTION

[0083] In describing the present invention, the following terms will be employed, and are intended to be defined as indicated below.

[0084] A B cell receptor is a transmembrane protein expressed on the surface of B cells. A B cell receptor comprises a binding moiety (also referred to as “antigen-binding subunit” or “membrane immunoglobulin”, “mlg”) comprising a membrane bound immunoglobulin molecule (also referred to as antibody) that recognises a cognate antigen, and a signal transduction moiety. The membrane-bound immunoglobulin molecule comprises two immunoglobulin light chains and two immunoglobulin heavy chains, and is identical to a corresponding secreted antibody with the exception of an integral membrane domain. The signal transduction moiety is a heterodimer called lg-a / lg-p (CD79), bound together and to the immunoglobulin by disulphide bridges. An antibody (Ab) or immunoglobulin (Ig) is an immune protein, comprising an antigen binding site and a constant region belonging to one of a limited set of isotypes (IgA, IgD, IgE, IgG, or IgM) and mediating interactions with other components of the immune system. In humans and most mammals, antibodies comprise four polypeptide chains: two identical heavy chains and two identical light chains connected by disulphide bonds. Light chains typically consist of one variable domain VL and one constant domain CL, while heavy chains typically contain one variable domain H and three to four constant domains CH1 , CH2, CH3, CH4. The variable domains form the antigen binding region and can also be referred to as the Fv region. Each variable domain contains three hypervariable regions referred to as the complementarity-determining regions (CDRs), which together form an antigen binding site. These can be referred to as CDRH1 , CDRH2, CDRH3 (for the CDRs of the heavy chain) and CDRL1 , CDRL2, CDRL3 (for the CDRs of the light chain), or CDR1 , CDR2 and CDR3 (referring to either the heavy or light chain CDRs). The variable region of each immunoglobulin heavy or light chain is encoded in several pieces — known as gene segments (subgenes): Ig heavy chains comprise variable (V), diversity (D) and joining (J) segments, and Ig light chains comprise V and J segments. Multiple copies of the V, D and J gene segments are present in the genome and developing B cells assemble an Ig variable region by (nearly) randomly selecting and combining one V, one D and one J gene segment (or one V and one J segment in the light chain), in a process called V(D)J recombination. The process involves the formation of double-strand breaks between the required segments, which form hairpin loops that are then joined together. The joining process is inaccurate, resulting in the variable addition or subtraction of nucleotides between the V and J (light chain) or V and D and J (heavy chain) segments, producing a large diversity in the sequences at the junction between segments (referred to as “junction sequences”). The V(D)J recombination process produces novel amino-acid sequences in the antigen-binding regions of Igs, generating a vast diversity of antigen recognition capability. As a result of the process, each Ig heavy chain variable region comprises: a V segment, a D segment and a J segment, with a junction sequence that spans the join between these segments (as illustrated on Figure 8A). Similarly, each Ig light chain hypervariable region comprises: a V segment, and a J segment, and a junction sequence that spans the join between these segments (as illustrated on Figure 8B). Within the variable region, CDR1 and CDR2 are found in the V segment, and CDR3 includes some of the V, all of D (in the heavy chain) and some of the J segment.

[0085] The methods described herein relate primarily to antigen-binding regions of proteins. Therefore reference to “antibodies” or “BCR” sequences or structures encompass the corresponding BCR or antibody unless context indicates otherwise. Further, the methods described herein can make use of various parts of these antigen-biding proteins, typically including at least the variable region or a part thereof including one or more (or all of) the CDRs of at least one of the heavy and light chains. Therefore, reference to an antibody or BCR sequence or structure encompasses a portion of the antibody or BCR comprising at least the variable region of the antibody or BCR light and / or heavy chain. The term “antibody sequence” (interchangeably referred to as “BCR sequence” unless context indicates otherwise) may refer to a sequence comprising the VL and / or the VH of the antibody. The sequence may further comprise the CL (in the case of a light chain or a combination of a light chain and a heavy chain) and / or one or more of the CH1 , CH2, CH3, CH4 (in the case of a heavy chain or a combination of a light and heavy chain). In embodiments, an antibody sequence comprises both a portion of a heavy chain and a portion of a light chain. Such antibody sequences may be referred to as “paired sequences”. All of the above apply equally to structures, i.e. an antibody structure may refer to the structure of a portion of the antibody including at least the VL and / or H. A VL and / or VH sequence may be expressed as a fully specified sequence (including the identity of each amino acid in the sequence), as a combination of one or more fully specified sequences and one or more gene segment identifiers, or as a combination of one or more gene segment identifiers. When segment identifiers are used, the sequence may comprise a fully specified junction sequence. For example, a VL and / or VH may be expressed as: a V-gene sequence or identifier, a J-gene sequence or identifier, a junction sequence, and optionally a D-gene sequence or identifier.

[0086] A T cell receptor is a membrane anchored protein expressed on the surface of T cells. A T cell receptor comprises a pair of protein chains that together form binding moiety that recognises a cognate antigen. These are expressed in a complex with constant T cell coreceptor chains CD3, comprising a CD3y chain, a CD35 chain, and two CD3E chains in mammals. The constant chains associate with the T cell receptor and the constant ^-chain to form the TCR complex, which together is able to generate a signal upon antigen binding to the T cell receptor, through phosphorylation of the immunoreceptor tyrosine-based activation motifs (ITAMs) in the CD3 proteins by Lek, which is anchored to the membrane by association with the coreceptors CD4 or CD8. The TCR is a heterodimeric protein, comprising two highly variable chains, the a and p chains (in the majority of T cells), or the alternative y and 5 chains (in a minority of T cells). Each chain comprises two extracellular domains: a variable region (or variable domain) and a constant region (or constant domain, proximal to the cell membrane), a transmembrane region and a short cytoplasmic tail. The variable regions together bind to a peptide (antigen), within the context of a MHC (major histocompatibility complex) molecule in the case of op TCRs. Each variable domain contains three hypervariable regions referred to as the complementarity-determining regions (CDRs, respectively referred to as CDR1 , CDR2 and CDR3 on each of the chains), which together form an antigen binding site. The TCR is a member of the immunoglobulin superfamily, which comprises BCRs and antibodies. In a process similar to that explained above, the variable region of each TCR chain is encoded in several pieces — known as gene segments (subgenes): p and 5 chains comprise variable (V), diversity (D) and joining (J) segments, and a and y chains comprise V and J segments. Multiple copies of the V, D and J gene segments are present in the genome and developing T cells assemble a TCR chain variable region by (nearly) randomly selecting and combining one V, one D and one J gene segment (or one V and one J segment in the a I y chain), in a process called V(D)J recombination. The process involves the formation of doublestrand breaks between the required segments, which form hairpin loops that are then joined together. The joining process is inaccurate, resulting in the variable addition or subtraction of nucleotides between the V and J (a I y chain) or V and DJ and D and J (P / 6 chain) segments, producing a large diversity in the sequences at the junction between segments (referred to as “junction sequences”). The V(D)J recombination process produces novel amino-acid sequences in the antigen-binding regions of TCRs, generating a vast diversity of antigen recognition capability. As a result of the process, each p / 6 chain variable region comprises: a V segment, a D segment and a J segment, with a junction sequence that spans the join between these segments. Similarly, each a / y chain hypervariable region comprises: a V segment, and a J segment, and a junction sequence that spans the join between these segments. Within the variable region, CDR1 and CDR2 are found in the V segment, and CDR3 includes some of the V, all of D (in the beta chain) and some of the J segment. The considerations above in relation to the meaning of the terms “antibody sequence” and “antibody structure” apply equally to TCR sequences I structures.

[0087] As used herein, a “chain” of an antigen-binding protein refers to a chain of an antigen-binding protein that is involved in antigen recognition, or a part thereof that contains at least part of a variable region of the chain. A chain may be a BCR heavy or light chain, an antibody heavy or light chain, a TCR a or p chain, a TCR y or 5 chain, or any part of such chains that contains at least a part of one or more variable regions within these chains. As used herein, the terms “chain sequence”, encompasses the terms “heavy chain sequence”, “light chain sequence”, “a chain sequence”, “ chain sequence” , “y chain sequence” and “5 chain sequence” and refer to any information that can be obtained from B cell sequencing or T cell sequencing technologies, ranging from a combination of one or more gene segment identifiers and / or junction sequences at one end, to full chain sequences at the other end. In particular, the terms “heavy chain sequence” and “light chain sequence” refer to any information that can be obtained from B cell sequencing technologies, ranging from a combination of one or more gene segment identifiers and / or junction sequences at one end, to full chain sequences at the other end. Further, the terms “chain sequence”, “heavy chain sequence”, “light chain sequence” “a chain sequence”, “P chain sequence”, “y chain sequence” and “5 chain sequence” refer interchangeably to the amino acid sequence or the corresponding nucleic acid coding sequence unless context indicates otherwise. The machine learning models described herein typically use protein sequences as inputs, and therefore the term “sequences” refers to amino acid sequences by default. However, nucleic acid sequences can be used to derive corresponding amino acid sequences, for example using a predetermined translation scheme. Thus, methods of the present disclosure can also make use of nucleic acid sequences. A chain pairing or pair (such as heavy-light chain pairing or pair) refers to a combination of a heavy chain sequence and a light chain sequence, an a chain sequence and a p chain sequence, or a y chain sequence and a 5 chain sequence as defined herein, each ranging from a combination of one or more gene segment identifiers and / or junction sequences at one end, to full specified chain sequences (including at least the variable domain) at the other end. As used herein, a “variable chain” (also referred to herein simply as “chain” or “variable region”) of an antigen-binding protein refers to a chain of an antigenbinding protein that is involved in antigen recognition, or a part thereof that contains at least part of a variable region of the chain. Variable chains comprise variable regions that are responsible for the diverse repertoire of antigen recognition properties within antigen-binding proteins. A variable chain may be a BCR heavy or light chain, an antibody heavy or light chain, a TCR a or p chain, a TCR y or 5 chain, or any part of such chains that contains at least a part of one or more variable regions within these chains.

[0088] Within the context of providing a desired antibody or antigen-binding protein, such as e.g. a therapeutic antibody, the term "antibody" (Ab) includes monoclonal antibodies, polyclonal antibodies, multispecific antibodies (e.g., bispecific antibodies), and antibody fragments (e.g., scFv, nanobodies) that exhibit the desired biological activity and that comprise at least a heavy chain or a part thereof identified as described herein or a heavy chain or a part thereof derived from a heavy chain identified as described herein (for example by further optimisation, affinity maturation, etc.)

[0089] A “sample” as used herein may be a cell or tissue sample, a biological fluid (e.g. plasma or serum), an extract (e.g. a DNA or RNA extract obtained from the subject, or a proteomic extract obtained from a subject), from which B cell genomic material or T cell genomic material (e.g. RNA or DNA) can be obtained for genomic analysis, such as by sequencing (e.g. whole genome sequencing, whole exome sequencing, targeted I capture sequencing, RNA-seq, etc.) or antigen-binding proteins or fragments thereof can be obtained for proteomic analysis (e.g. mass spectrometry sequencing). The sample may be a cell, tissue or biological fluid sample obtained from a subject (e.g. a biopsy). Such samples may be referred to as “subject samples”. In particular, the sample may be a blood sample, a lymph node sample, a spleen sample, or a tumour sample, or a sample derived therefrom (such as e.g. by B cell purification, T cell purification, RNA extraction, protein purification, etc.). As used herein, the terms “genomic material”, “genomic sequencing” and the like encompasses both to the material I sequence present in the genome and the transcriptome of a sample, unless context indicates otherwise. The sample may be one which has been freshly obtained from a subject or may be one which has been processed and / or stored prior to genomic analysis (e.g. frozen, fixed or subjected to one or more purification, enrichment, or extraction steps). The sample may be a cell or tissue culture sample. As such, a sample as described herein may refer to any type of sample comprising B cells (or T cells) or genomic material derived therefrom, whether from a biological sample obtained from a subject, or from a sample obtained from e.g. a cell line. The sample is preferably from a mammalian (such as e.g. a mammalian cell sample or a sample from a mammalian subject, such as a cat, dog, horse, donkey, sheep, pig, goat, cow, mouse, rat, rabbit or guinea pig), preferably from a human (such as e.g. a human cell sample or a sample from a human subject). Further, the sample may be transported and / or stored, and collection may take place at a location remote from the sequence data acquisition (e.g. sequencing) location, and / or any computer-implemented method steps described herein may take place at a location remote from the sample collection location and / or remote from the genomic data acquisition (e.g. sequencing) location (e.g. the computer-implemented method steps may be performed by means of a networked computer, such as by means of a “cloud” provider).

[0090] The term “sequence data” in the context of information obtained by sequencing refers to information that is indicative of the presence of genomic material (DNA or RNA) or proteomic material in a sample that has a particular sequence. Thus, sequence data may comprise one or more nucleotide sequences and / or one or more amino acid sequences. Such information may be obtained using nucleic acid sequencing technologies, such as e.g. next generation sequencing (NGS), for example whole exome sequencing (WES), whole genome sequencing (WGS), whole transcriptome sequencing (RNAseq) or sequencing of captured genomic loci (targeted or panel sequencing). Instead or in addition to nucleic acid sequencing, sequences of antigen binding proteins may be obtained through mass spectrometry based protein sequencing. When NGS technologies are used, the sequence data may comprise a count of the number of sequencing reads that have a particular sequence. Sequence data may be mapped to a reference sequence, for example a reference genome, using methods known in the art (such as e.g. Bowtie - available at bowtie-bio.sourceforge.net / index.shtml). Thus, counts of sequencing reads or equivalent non-digital signals may be associated with a particular location or locus (where the “location” refers to a location in the reference genome or transcriptome to which the sequence data was mapped). Further, a location may contain a mutation, in which case counts of sequencing reads or equivalent non-digital signals may be associated with each of the possible variants (also referred to as “alleles”) at the particular location. The process of identifying the presence of a mutation at a particular location in a sample is referred to as “variant calling” and can be performed using methods known in the art (such as e.g. general purpose NGS variant callers such as the GATK HaplotypeCaller, gatk.broadinstitute.org / hc / en-us / articles / 360037225632-HaplotypeCaller or tools specifically designed for immune sequences such as IgBLAST, www.ncbi.nlm.nih.gov / igblast / ). Genomic sequence data may be converted to amino acid sequences by translating coding regions in silico (directly from an mRNA sequence or from identified coding regions in a genomic sequence), as known in the art. When proteomic sequence technologies are used, the sequence data may comprise, for each of one or more peptides, mass spectrometry spectra or amino acid sequences derived therefrom.

[0091] The term “sequence data” in the context of inputs of a machine learning model refers to data that is in the form of text I a string of characters, where the text comprises information about the sequence of an antigen-binding protein. Typically, this is in the form of a string of characters representing amino acids. Any predetermined representation scheme can be used, such as e.g. the amino acid one letter code. Predetermined additional characters may be used such as e.g. “X” for unknown or undetermined positions. In embodiments, the sequence data is processed using one or more sequence encoding modules. A sequence encoding module refers to a machine learning model that has been trained to take as input sequence data and produce as output an embedded representation of the sequence data, or to produce as output a decoded or generated sequence from a learned embedded representation of the sequence data. Thus, a sequence encoder may be an encoder model, a decoder model of an encoderdecoder model, or a decoder model of a decoder only model such as a generative model. For example, generative models with architectures such as those of the ProGen2 models described in Nijkamp et al. (2022), or alternative architectures such as LLaMA (Touvron et al., 2023), Falcon LLM (falconllm.tii.ae / ), and GPT (e.g. GPT-3, Brown et al. 2020) may be used. Alternatively, transformer encoder-based architectures such as AntiBERTa2 (see Examples or its predecessor described in Leem et al. 2022) or those described in Burbach and Briney 2023 (BALM), Lin et al. 2022 (ESM2), or Ruffolo et al. 2021 (AntiBERTy) may be used. Transformer encoder based architectures may advantageously use rotary position embedding (e.g. RoFormer architectures like AntiBERTa2). When using a decoder model, the embedded representation may be obtained as the embeddings from the final decoder layer of the decoder. A sequence encoding module may also be referred to herein as a “sequence encoder”, although its architecture is not limited to that of an encoder. The embedded representation is a representation in latent space, typically configured such that the original sequence data can be reconstructed from the embedded representation. A sequence encoding module is typically a deep learning model. A sequence encoding module may be trained as part of a sequence-to-sequence model. A sequence encoding module may be a transformer-based encoder or decoder. A sequence encoding module may have been trained as part of a machine learning model that only takes sequence data as input (through a process of unimodal learning) and / or as part of a machine learning model that takes both sequence data and structure data as inputs (through a process of multimodal learning). A sequence encoding module may have been trained both through unimodal learning and through multimodal learning. For example, a sequence encoding module may first be trained using sequence data only, then using both sequence data and structure data. Such a sequence encoding module may be referred to herein as a “structure enhanced sequence only encoder”, “structure enhanced sequence encoder”, “structure aware sequence only encoder”, “structure aware sequence encoder”, or simply “enhanced sequence encoder”. The present invention relates to the use of such sequence encoding modules. By contrast, a sequence encoding module that has only been trained using sequence data may be referred to herein as “sequence only encoder”. The term “structure data” in the context of inputs of a machine learning model refers to data that comprises information about the 3D structure of an antigen-binding protein. The 3D structure information may be in the form of spatial coordinates characterising the position of residues or atoms in an antigen binding protein. For example, the 3D structure information may be in the form of a PDB or mmCIF file, comprising textual information specifying for each of a plurality of atoms that make up the protein, their X-axis, Y-axis, and Z-axis coordinates. The 3D structure information may be in the form of information characterising the relationship between pairs of amino acids or atoms in the structure. The information may characterise the relationship between pairs of amino acids or atoms selected from all amino acids or atoms in the structure, or a subset thereof such as e.g. backbone atoms. Information characterising the relationship between pairs of amino acids or atoms in the structure may be derived from spatial coordinates characterising the position of residues or atoms in an antigen binding protein (see e.g. information in the form of a graph as described below). Information characterising the relationship between pairs of amino acids in a structure may be in the form of a graph or in the form of one or more matrices (or flattened version thereof) each comprising coefficients Cy that represent a physicochemical property of a pair of amino acids at respective positions i and j in the sequence corresponding to the structure. Information characterising the relationship between pairs of atoms in a structure may be in the form of a graph or in the form of one or more matrices (or flattened version thereof) each comprising coefficients Cy that represent a physicochemical property of a pair of atoms i and j in the structure. Each one of one or more matrices may be referred to as a “channel”. Each of the one or more matrices may be seen as a single channel image, where the value of the coefficients is equivalent to a pixel value. A plurality of matrices may be seen collectively as a multi-channel image. Each one of one or more matrices may comprise coefficients that represent a different physicochemical property. The one or more matrices may comprise one or more of: a matrix with coefficients Cy that represent the (physical) distance between the amino acids at positions i and j in the 3D structure; a matrix with coefficients Cy that represent the electrostatic force between the amino acids at positions i and j in the structure; a matrix with coefficients Cy that represent the mean hydrophobicity of the amino acids at positions i and j in the structure; a matrix with coefficients Cy that represent the Boman index associated with the amino acids at positions i and j in the structure (weight value assigned to dipeptides as described in Boman et al., 2003); a matrix with coefficients Cy that represent the mean isoelectric point of the amino acids at positions i and j in the structure; and a matrix with coefficients Cy that represent the mean solvent accessible surface area of the amino acids at positions i and j in the structure. The Boman index is an index that was originally developed to quantify protein-binding potential. It is calculated as the arithmetic mean of the free energies of the respective side chains for transfer from cyclohexane to water. The present inventors postulated that the Boman index may capture information that may be useful in relation to antigen-binding proteins such as e.g. for identification of paratope residues or CDRs. The one or more matrices may comprise at least a matrix with coefficients Cy that represent the (physical) distance between the amino acids at positions i and j in the 3D structure. Such a matrix may be referred to as a “residue distogram” or “distogram”. Information characterising the relationship between pairs of atoms in a structure may be in the form of a graph or one or more matrices (or flattened version thereof) each comprising coefficients Cy that represent a physicochemical property of a pair of atoms in the structure. The one or more matrices may comprise at least a matrix with coefficients Cy that represent the (physical) distance between any pairs of atoms i and j in the 3D structure. Such a matrix may be referred to as an “atom distogram” or “distogram”. The one or more matrices may comprise a matrix with coefficients Cy that represent bonds, interactions or angles between the atoms i and j in the structure Information characterising the relationship between pairs of amino acids in a structure in the form of a graph may comprise a plurality of nodes connected by a plurality of edges, where each node represents an amino acid in the structure and each edge represents a relationship between a pair of amino acids. The graph may be fully connected. Alternatively, edges may be present between each pair of amino acids that satisfy one or more predetermined criteria. For example, edges may be present between each pair of amino acids that are at or below a predetermined distance from each other, where a distance can be a physical distance in the 3D structure, a distance along the primary structure in terms of number of residues, or a physical distance along the backbone structure between the amino acids in the pair. Each node may be associated with one or more features selected from scalar and vector features. Each edge may be associated with one or more features selected from scalar and vector features. Each node may be associated with at least one scalar feature and at least one vector feature. Each edge may be associated with at least one scalar feature and at least one vector feature. Vectors may be geometric vectors. Geometric vectors may be expressed as absolute orientations in a common coordinate system. A node may be associated with one or more vectors capturing the directions of bonds between backbone atoms. A node may be associated with a scalar feature that specifies the identity of the amino acid. A node may be associated with one or more scalar features that each represent a dihedral angle of the backbone structure at the amino acid. A node may be associated with scalar features including the dihedral angles of the backbone structure at the amino acid and / or a representation of the amino acid identity. A node may be associated with a plurality of vectors each representing the directions of a bond between backbone atoms at the amino acid. An edge may be associated with a vector representing the direction of the vector between carbon alpha atoms of the pair of amino acids. An edge may be associated with one or more scalar features selected from: the distance between carbon alpha atoms of the pair of amino acids, and the distance along the backbone structure between the amino acids in the pair. For example, the ESM-IF1 structure encoder further described below uses a geometric vector perceptron which encodes the 3D geometry of a protein as a graph where each node (representing amino acids) and edge (representing relationships between amino acids) is represented using an embedding comprising a combination of geometric vectors (V, expressed as absolute orientations in a common coordinate system) and scalar features (s). Information characterising the relationship between pairs of atoms in a structure in the form of a graph may comprise a plurality of nodes connected by a plurality of edges, where each node represents an atom in the structure and each edge represents a relationship between a pair of atoms. Nodes and edges may be associated with invariant information such as angles (e.g. dihedral angles and bond angles) and distances (e.g. distances between pairs of atoms), respectively. Nodes may be associated with embeddings indicating the nature of an atom (e.g. a four dimensional one-hot node embedding that describes the atom type -H, C, N, O) and 3D coordinates for each atom (coordinate embeddings). Edges may be associated with relationships between atoms. Edges may be present between all pairs of atoms that are being modelled (i.e. fully connected network). All or a selected set of atoms may be modelled, such as e.g. backbone atoms (N, alpha C, C). Alternatively, edges may only be present between selected pairs of atoms, such as e.g. based on distances between atoms. For example, the E(n) equivariant graph neural network (E(n)GNN) further described below uses as inputs atom type embeddings and coordinate embeddings and fully connected networks (optionally subject to edge inference limiting the neighbourhood of each node). When edge inference limiting the neighbourhood of each node is used, a neighbourhood is defined for each node (typically based on distance between nodes) and edges from nodes that are not in a node’s neighbourhood are effectively removed (i.e. information from those nodes outside of a particular node’s neighbourhood is ignored when determining the value of the particular node). Alternatively, nodes may be associated with embeddings indicating the nature of an atom (e.g. a four dimensional one-hot encoded node embedding that describes the atom type -H, C, N, O) and edges may be associated with vectors indicating relative positions between atoms. For example, the Equiformer model further described below uses as input atom type embeddings and embeddings derived from relative positions and distances between atoms.

[0092] A structure encoder or “structure encoding module” refers to a machine learning model that has been trained to take as input structure data and produce as output an embedded representation of the structure data. The embedded representation is a representation in latent space, typically configured such that the original structure data can be reconstructed from the embedded representation. The structure encoder is not limited in its architecture and any machine learning model that is able to learn such a latent space representation may be used. A structure encoder is typically a deep learning model. A structure encoder may be trained in a supervised manner, for example as part of a structure-to-structure model or as part of a model trained to predict another property from structure (e.g. physicochemical or functional properties estimated through regression). A structure encoder may be trained in a selfsupervised manner, for example using masked image modelling or masked coordinate modelling. Masked coordinate modelling refers to a training task where the coordinates or one or more atoms or residues are altered and the model is trained to provide the original (correct) coordinates. A structure encoder may be a transformer-based encoder. In embodiments, a structure encoder is a structure encoder that has been obtained as described in Hsu et al., 2022 (also referred to herein as a “ESM-IF1” structural encoder). A structure encoder may be a convolutional graph neural network. In embodiments, a structure encoder is a structure encoder that has been obtained as described in Satorras et al., 2021 (E(n)GNN structural encoder). E(n)GNN is a deep learning model that comprises equivariant graph convolutional layers (EGCL) that take as input a set of node embeddings, coordinate embeddings and edge information and outputs a transformation on the nodes and coordinate embeddings which depends on embeddings and squared distances between coordinates of connected nodes. A structure encoder may be a transformer-based encoder, such as a graph attention transformer. In embodiments, a structure encoder is a structure encoder that has been obtained as described in Liao & Smidt (2023, Equiformer). A structure encoder may be an encoder of an image processing model, such as a transformer-based encoder for image processing (e.g. Swin Transformer v2, described in Liu et al. 2022). An encoder of an image processing model may be trained in a self-supervised manner, e.g. using masked image modelling, using as input tensors that capture structural information about proteins (i.e. image like inputs) such as distograms as explained above.

[0093] As used herein, the terms “computer system” includes the hardware, software and data storage devices for embodying a system or carrying out a method according to the above described embodiments. For example, a computer system may comprise a central processing unit (CPU), graphical processing unit (GPU), input means, output means and data storage, which may be embodied as one or more connected computing devices. Preferably the computer system has a display or comprises a computing device that has a display to provide a visual output display (for example in the design of the business process). The data storage may comprise RAM, disk drives or other computer readable media. The computer system may include a plurality of computing devices connected by a network and able to communicate with each other over that network. It is explicitly envisaged that computer system may consist of or comprise a cloud computer. The term “processor” encompasses any processing unit or combination of processing units, including in particular CPUs and GPUs. As used herein, the term “computer readable media” includes, without limitation, any non- transitory medium or media which can be read and accessed directly by a computer or computer system. The media can include, but are not limited to, magnetic storage media such as floppy discs, hard disc storage media and magnetic tape; optical storage media such as optical discs or CD-ROMs; electrical storage media such as memory, including RAM, ROM and flash memory; and hybrids and combinations of the above such as magnetic / optical storage media.

[0094] The term “multimodal learning” refers to a family of deep learning methods where two or more types of data, such as text and image data, or text and video data, are fed as input into one model at least during the process of training the model. Depending on the architecture of the model, some elements of such a multimodal trained model can then be isolated and used to make unimodal predictions or multimodal predictions. In the context of the present disclosure, the term “multimodal learning” refers to a deep learning method where one or more machine learning models are trained using both sequence data and structure data. The term also encompasses the resulting model(s). Examples of multimodal learning techniques include CLIP (Radford et al., 2021), FLAVA (Singh et al., 2021), and CoCa (Yu et al., 2022), all of which can combine both image and text data. Despite variations in multimodal architectures or training regimes, the core idea of updating the individual text and image models to maximise agreement remains constant. For example, CLIP is trained by feeding an image and its associated caption into two separate neural networks: one for embedding text, and another for embedding images. CLIP aligns the representations from each model using a contrastive loss; it stipulates that an image and its caption refer to the same concept, and all other possible combinations of images and captions in a given batch are different. Optimising the contrastive loss refines the text and image models such that the text model is imbued with visual information, and vice-versa. FLAVA is trained using both unpaired images and text (unimodal data) and image-text pairs (multimodal data), with multimodal pretraining objectives (global contrastive loss similar to CLIP, masked multimodal modelling where patches of the image and parts of the text are masked, and image-text matching loss using a classifier appended to the pretrained multimodal encoder to determine whether input image-text pairs are matched) and unimodal pretraining objectives (masked image modelling (MIM) for unimodal image encoder initialisation, MLM for unimodal text encoder initialisation). Joint unimodal and multimodal training is applied after pretraining by training the complete model jointly on all data with round-robin sampling, applying unimodal MIM or MLM or multimodal losses depending on which dataset was sampled at each training iteration. CoCa is trained end-to-end on unpaired text data and paired image-text data, using a modified encoder-decoder architecture (where the decoder includes unimodal decoder layers without cross-attention, encoding text only representations, and subsequent multimodal decoder layers cross-attending to image encoder outputs to learn multimodal image-text representations - the multimodal decoder predicting text tokens autoregressively) trained with both contrastive loss (applied between the outputs of the image encoder and the unimodal text decoder) and captioning (generative) loss (applied at the output of the multimodal decoder). Note that reference to “CoCa” herein refer to “CoCa” like multimodal learning processes, wherein a multimodal model cross-attends to unimodal outputs, and the model is trained using contrastive loss between unimodal models and a loss applied to the output of the multimodal model. For example, the multimodal model may be a multimodal encoder that uses cross-attention between a sequence encoder (text) and a structure encoder (e.g. image or graph). The unimodal models may both be encoders (e.g. a sequence encoder and a structure encoder). The loss applied to the output of the multimodal model may be a masked language modelling loss. The contrastive and multimodal losses may be combined in a weighted sum. For example, the model may be trained using a total loss that weights the multimodal loss twice as much as the contrastive loss.

[0095] Multimodal models have also been used in the context of protein analysis. For example, ESM- IF1 (Hsu et al., 2022) was developed to address the problem of predicting a protein sequence from its backbone atom coordinates (also known as “inverse protein folding”), for example to design novel amino acid sequences with desired properties. The approach uses a hybrid geometric vector perceptron graph neural network structural encoder followed by a generic transformer (GVP-Transformer). Geometric vector perceptrons are neural network units that can be used in graph neural networks (deep learning methods that are designed to process data described by graphs) instead of standard multi-layer perceptrons in aggregation and feed-forward layers. The geometric vector perceptron encodes the 3D geometry of a protein as a graph where each node (representing amino acids) and edge (representing relationships between amino acids) is represented using an embedding comprising a combination of geometric vectors (V, expressed as absolute orientations in a common coordinate system) and scalar features (s). In particular, each amino acid is represented as a combination of vectors capturing the directions of bonds between backbone atoms, together with scalar features including the dihedral angles of the backbone structure at the amino acid and a representation of the amino acid identity; each edge is represented as a combination of a vector capturing the direction of the vector between alpha carbon atoms of the pair of amino acids and scalar features capturing the distance between alpha carbon atoms of the pair of amino acids and distance along the backbone structure between the amino acids in the pair.

[0096] MIF-ST (Yang et al., 2023) is a masked inverse folding MLM, pretrained to reconstruct corrupted sequences conditioned on the backbone structure. The model is then fine-tuned for specific tasks which are also conditional on structure. The approach follows the spirit of VisualBERT in having one model that concatenates dihedral angles and a CNN-encoded sequence representation. Multimodal protein models tend to focus on how a protein language model can enhance structural representations. In addition, even after training, multimodal protein models tend to still depend on using both protein structure and sequence as inputs.

[0097] Analysis of antigen binding proteins

[0098] The present disclosure provides methods for predicting one or more features of antigen binding proteins, using embeddings of a sequence only machine learning model trained in a multimodal manner using sequence and structure data for a plurality of antigen binding proteins. An illustrative method will be described by reference to Figure 1 . At optional step 10, a sample comprising one or more antigen-binding proteins or coding sequences for such proteins may be obtained. At optional step 12, one or more chains of the antigen-binding proteins (ABP) present in the sample may be sequenced. These may be referred to as a the “repertoire” of antigen-binding proteins present in the sample. The sequencing may comprise obtaining sequences for a single chain of each antigen binding protein (e.g. through bulk BCR sequencing or bulk TCR sequencing), or for a pairs of chains for each antigen binding protein (e.g. through single cell TCR or BCR sequencing). At step 14, the amino acid sequence of at least part of one or more chains of at least one antigen-binding proteins identified at step 12 is / are provided. Step 14 may comprise encoding the amino acid sequences using a predetermined encoding scheme. When amino acid sequences for pairs of chains are identified, these may be provided together as a concatenated encoded sequence. At optional step 16, a deep learning model comprising a sequence encoding module configured to take as input an amino acid sequence and to produce as output an embedding corresponding to the input sequence is provided. This sequence encoding module has been trained as part of a machine learning model comprising a structure encoding module and the sequence encoding module, and using training sequence data and structure data from a plurality of antigen-binding proteins, as explained herein for example by reference to Figure 2. At step 18, one or more features of the antigen-binding protein for which a sequence was provided at step 14 are predicted using the embedding produced by the sequence encoding module. Step 18 may comprise providing the embedding for each antigen-binding protein sequence using the sequence encoding module at step 18A, and optionally predicting a functional feature based on the embedding at step 18B. Alternatively, the embeddings obtained at step 18A may be analysed at step 20 to predict one or more features. For example, the embeddings may be clustered together with embeddings produced using the sequence encoding module for one or more reference antigen binding proteins, as explained further below. At step 22, the results of one or more of the preceding steps is provided to a user, for example through a user interface. A method of obtaining a tool for use in a method according to Figure 1 will now be described by reference to Figure 2. At step 10’, training data is obtained comprising sequence data and structure data from a plurality of antigen-binding proteins, wherein the sequence data for an antigen-binding protein comprises the amino acid sequence of at least part of one or more chains of the antigen-binding protein or an encoded sequence derived therefrom and the structure data for an antigen-binding protein comprises data characterising the 3D structure of the antigen-binding protein. At optional step 12’, the training data may be filtered. The training sequence and / or structure data may be filtered. In embodiments, only the sequence data is filtered. At optional step 14’, a sequence encoding module and a structure encoding module are pretrained, respectively using training sequence data and training structure data. This may be referred to as “pretraining” or “unimodal training”. The sequence encoding module is configured to take as input a query amino acid sequence and to produce as output an embedding corresponding to the input sequence. The structure encoder is configured to take as input a protein structure and to produce as output an embedding corresponding to the input structure. At step 16’, the sequence encoding module is trained as part of a single multimodal deep learning model comprising both the sequence encoding module and the structure encoder, using both structure data and training data. At optional step 18’, a deep learning model comprising the trained sequence encoding module but not the trained structure encoder is trained to predict one or more features of an antigen binding protein using the embeddings produced by the sequence encoding module. This may be referred to as “fine tuning” or “taskspecific training”. The deep learning model may comprise the sequence encoding module and a classifier, regressor or may be a generative model comprising the sequence encoding module. The task-specific training may comprise training the complete model including fine- tuning the sequence encoding module, or training a model (that may not be a deep learning model) that uses the output of the sequence encoding module, where the sequence encoding module is not further trained. At optional step 20’, one or more of the models trained at any preceding step is provided to a user. Typically, at least the trained (optionally further pretrained and fine-tuned) sequence encoding module is provided to a user. This may be used in a method of predicting one or more features of antigen binding proteins, as illustrated by reference to Figure 1.

[0099] According to any embodiment of any aspect, the step of training a sequence encoding module as part of a multimodal model comprising the sequence encoding module and a structure encoder may use a contrastive loss. This may be referred to as “contrastive sequence structure pretraining” (CSSP). Training using a contrastive loss may minimise the following ■ loss. L ,CLIP_- Lstructure2+Lsequencewwhnperrpe Sequence = ~ represents the cosine similarity between the projection of the kth sequence embedding z and the projection of the kth structure embedding zk' , scaled by a learnable parameter T. The step of training a sequence encoding module as part of a multimodal model comprising the sequence encoding module and a structure encoder may comprise training the multimodal model for at least 3, at least 4 or at least 5 training epochs.

[0100] Systems

[0101] Figure 3 shows an embodiment of a system for predicting features of antigen binding proteins and / or analysing antigen binding proteins using such predicted features and / or providing a tool for predicting features of antigen binding proteins, according to the present disclosure. The system comprises a computing device 1 , which comprises a processor 101 and computer readable memory 102. In the embodiment shown, the computing device 1 also comprises a user interface 103, which is illustrated as a screen but may include any other means of conveying information to a user such as e.g. through audible or visual signals. The computing device 1 is communicably connected, such as e.g. through a network 6, to sequence data acquisition means 3, such as a sequencing machine, and / or to one or more databases 2 storing sequence data. The one or more databases 2 may also store associated structure data, for example when the system is used to provide a tool for predicting features of antigen binding proteins. The one or more databases may additionally store other types of information that may be used by the computing device 1 , such as e.g. reference sequences, parameters, etc. The computing device may be a smartphone, server, tablet, personal computer or other computing device. The computing device is configured to implement a method for predicting features of antigen binding proteins, as described herein. In alternative embodiments, the computing device 1 is configured to communicate with a remote computing device (not shown), which is itself configured to implement a method of predicting features of antigen binding proteins, as described herein. In such cases, the remote computing device may also be configured to send the result of the method to the computing device. Communication between the computing device 1 and the remote computing device may be through a wired or wireless connection, and may occur over a local or public network such as e.g. over the public internet or over WiFi. Communication between the computing device 1 and the one or more databases 2 may be through a wired or wireless connection, and may occur through a network 6 which may be a local or public network such as e.g. over the public internet or over WiFi. The sequence data acquisition 3 means may be in wired connection with the computing device 1 , or may be able to communicate through a wireless connection, such as e.g. through a network 6, as illustrated. The connection between the computing device 1 and the sequence data acquisition means 3 may be direct or indirect (such as e.g. through a remote computer). The sequence data acquisition means 3 are configured to acquire sequence data from nucleic acid samples, for example genomic DNA samples or RNA samples extracted from B cells or T cells purified from fluid and / or tissue samples (such as e.g. peripheral blood, spleen, lymph node, tumour tissue, or any other type of sample comprising B cells or T cells). In some embodiments, the sample may have been subject to one or more preprocessing steps such as DNA / RNA purification, fragmentation, library preparation, target sequence capture (such as e.g. exon capture and / or panel sequence capture). Any sample preparation process that is suitable for use in the determination of a B cell receptor sequence or repertoire may be used within the context of the present invention. The sequence data acquisition means is preferably a next generation sequencer. The sequence data acquisition means 3 may be in direct or indirect connection with one or more databases 2, on which sequence data (raw or partially processed) may be stored.

[0102] Applications

[0103] The methods described herein find uses in any context where it is desirable to characterise antigen-binding proteins based on sequence information. The characterisation may use embeddings produced using methods as described herein directly, for example for sequence clustering, annotation, search, and comparison. Instead or in addition to this, the characterisation may use features predicted from the embeddings produced using methods as described herein, such as when the deep learning model that produces the embeddings further comprises a classification, regression or generative model that takes as input the embeddings and produces as output one or more predicted features.

[0104] Thus, the present disclosure also describes methods comprising providing the amino acid sequence of at least part of one or more chains of an antigen-binding protein or an encoded sequence derived therefrom as input to a deep learning model comprising a sequence encoding module configured to take as input an amino acid sequence and to produce as output an embedding corresponding to the input sequence (wherein the sequence encoding module has been trained as part of a machine learning model comprising a structure encoder, and using training sequence data and structure data from a plurality of antigen-binding proteins). The methods may further comprise using the embedding produced to cluster the sequence of the antigen-binding protein together with further sequences of antigen-binding proteins (by clustering the sequence embeddings produced as described herein for all sequences subject to the clustering). The further sequences of antigen-binding proteins may be sequences of reference antigen-binding proteins, such as e.g. antigen-binding proteins with known specificity. A sequence that has an embedding that clusters together with the embedding of a reference sequence may be predicted to have the same specificity as the reference sequence. When a sequence clusters together with multiple reference sequences any proximity and / or voting based criterion may be used to predict the specificity of the sequence. For example, the sequence may be predicted to have the same specificity as the closest reference sequence that clusters together with the sequence, the same specificity as the most common known specificity amongst the reference sequences that cluster together with the sequence, the majority specificity based on a voting scheme that is weighted by distance between the reference sequence and the sequence, etc. Further, while the prediction of features of a sequence by clustering of embeddings obtained as described herein has been illustrated by reference to specificity, any other functional property may similarly be predicted based on known functional properties of reference antigen-binding proteins. The reference to a sequence that has an embedding that clusters together with the embedding of a reference sequence encompasses sequences that have embeddings in the same cluster as determined using any clustering method known in the art, or a sequence that has an embedding that has the reference sequence (or a plurality of such reference sequences) as its nearest neighbour or part of its k-nearest neighbours in embedding space (where k can be selected as e.g. 3, 5, 10, etc). Nearest neighbours may be identified using any distance or similarity metric known in the art. When using k nearest neighbours where k>1 , any voting scheme as described above may be used.

[0105] Thus, also described herein are methods of annotating one or more query antigen-binding protein sequences, including a plurality of antigen-binding protein sequences obtained from a sample or a plurality of samples (e.g. a BCR repertoire for a subject or a cohort of subjects), the methods comprising providing the amino acid sequence of at least part of one or more chains of the query antigen-binding protein(s) or encoded sequence(s) derived therefrom as input to a deep learning model comprising a sequence encoding module configured to take as input an amino acid sequence and to produce as output an embedding corresponding to the input sequence (wherein the sequence encoding module has been trained as part of a machine learning model comprising a structure encoding module, and using training sequence data and structure data from a plurality of antigen-binding proteins). The methods may further comprise using the embedding produced to cluster the sequence(s) of the query antigenbinding protein(s) together with sequences of reference antigen-binding proteins (by clustering the sequence embeddings produced as described herein for all sequences subject to the clustering). The methods may further comprise annotating the query antigen-binding protein sequences based on one or more known properties of a reference antigen-binding proteins that clusters together with the query antigen-binding proteins or that satisfies one or more criteria based on distance in embedding space with a query antigen-binding protein. Instead or in addition to this, the methods may comprise annotating the query antigen-binding protein sequences based on one or more known properties of one or more reference antigen-binding proteins that cluster in clusters different from the query antigen-binding proteins or that satisfies one or more criteria based on distance in embedding space with a query antigenbinding protein (e.g. distance above a predetermined threshold). In other words, a query antigen-binding protein sequence may be annotated based on what reference antigen-binding proteins it is not similar to. The term “distance” in relation to comparing antigen-binding protein sequences in embedding space is used broadly and encompasses any distance or similarity metric such as e.g. Euclidean distance, Pearson correlation, cosine similarity, etc.

[0106] The one or more reference antigen-binding proteins may be antigen-binding proteins with known specificity. The one or more reference antigen-binding proteins may be therapeutic antibodies. Proximity of a query antigen-binding protein to a reference antigen-binding protein in embedding space may be used as an indication that the query antigen-binding protein may have one or more property in common with the reference antigen-binding protein, and may thus represent a good candidate for drug discovery. Thus, also described herein are methods of screening one or more antigen-binding proteins for drug discovery, the methods comprising: providing the amino acid sequence of at least part of one or more chains of one or more query antigen-binding protein(s) or encoded sequence(s) derived therefrom as input to a deep learning model comprising a sequence encoding module configured to take as input an amino acid sequence and to produce as output an embedding corresponding to the input sequence (wherein the sequence encoding module has been trained as part of a machine learning model comprising a structure encoder, and using training sequence data and structure data from a plurality of antigen-binding proteins). The methods may further comprise using the embedding produced to select at least one of the query antigen-binding proteins that satisfies a criterion on the distance in embedding space between the query antigen-binding protein and a reference antigen-binding protein. Proximity of a query antigen-binding protein to a reference antigen-binding protein in embedding space may be used as an indication that the query antigen-binding protein may have a similar antigen specificity as the reference antigen-binding protein, and may thus represent a poor candidate for drug discovery when the reference antigen-binding protein binds a target that is not of therapeutic interest. Thus, proximity of a query antigen-binding protein to a reference antigen-binding protein in embedding space, where the reference antigen-binding protein does not have a property of interest, may be used to filter out candidate antigen-binding proteins as unlikely to have the property of interest. Conversely, distance from and / or clustering in a different cluster from a reference antigenbinding protein in embedding space, where the reference antigen-binding protein does not have a property of interest, may be used as an indication that the query antigen-binding protein may have the property of interest. For example, the property of interest may be binding to a particular antigen or therapeutic activity in a particular disease, and the reference antigenbinding protein may bind to a different target or to a target that is unlikely to be involved in the particular disease. Distance from and / or clustering in a different cluster from such a reference antigen-binding protein can be used as an indication that the query antigen binding protein may not bind the same (irrelevant) target.

[0107] The methods described herein may further be used to jointly analyse multiple repertoires of antigen-binding proteins, to identify antigen-binding proteins with one or more properties of interest, for example based on properties of samples or subjects associated with the repertoires. For example, the methods described herein may be used to cluster repertoires of antigen-binding proteins associated with a plurality of subjects, and identify clusters of antigenbinding protein sequences that are enriched for sequences from subjects that have one or more properties of interest, such as e.g. a clinical characteristic (e.g. resilience to a disease). Thus, also described herein are methods comprising obtaining a plurality of sets of one or more antigen-binding protein sequences, each set including antigen-binding protein sequences obtained from a respective sample or subject (e.g. each set representing a BCR repertoire for a subject and the plurality of sets representing BCR repertoires for one or more cohorts of subjects). The samples or subjects may be associated with one or more properties, such as e.g. one or more clinical or phenotypic properties (e.g. healthy I diseased status, resilient status, where resilience may be defined in a disease specific manner, such as e.g. based on survival for a predetermined period of time despite the subject being at risk of developing a disease, or based on a subject not developing one or more symptoms of a disease despite being at risk of developing the disease, etc). The methods may comprise providing the amino acid sequence of at least part of one or more chains of each of the antigenbinding proteins or encoded sequences derived therefrom as input to a deep learning model comprising a sequence encoding module configured to take as input an amino acid sequence and to produce as output an embedding corresponding to the input sequence (wherein the sequence encoding module has been trained as part of a machine learning model comprising a structure encoder, and using training sequence data and structure data from a plurality of antigen-binding proteins). The methods may further comprise using the embedding produced to cluster the sequence(s). The methods may further comprise identifying one or more clusters of sequences that are enriched for sequences from samples or subjects associated with one or more predetermined properties. For example, the methods may comprise identifying one or more clusters of sequences that are enriched for sequences from subjects that have resilient status. The method may further comprise selecting one or more sequences from one or more identified clusters as candidate for further development, investigation, etc. Thus, sequences selected as described may represent candidate antigen-binding proteins for therapeutic and / or diagnostic applications.

[0108] The methods described herein may further be used to jointly analyse multiple repertoires of antigen-binding proteins, to identify features of such repertoires associated with properties of samples or subjects associated with the repertoires. For example, the methods described herein may be used to cluster repertoires of antigen-binding proteins associated with a plurality of subjects, and identify clusters of antigen-binding protein sequences that are enriched for sequences from subjects that have one or more properties of interest, such as e.g. a clinical characteristic (e.g. resilience to a disease, or lack of resilience to disease). Thus, also described herein are methods comprising obtaining a plurality of sets of one or more antigenbinding protein sequences, each set including antigen-binding protein sequences obtained from a respective sample or subject (e.g. each set representing a BCR repertoire for a subject and the plurality of sets representing BCR repertoires for one or more cohorts of subjects). At least some of the samples or subjects may be associated with one or more properties, such as e.g. one or more clinical or phenotypic properties (e.g. healthy I diseased status, resilient status, where resilience may be defined in a disease specific manner, such as e.g. based on survival for a predetermined period of time despite the subject being at risk of developing a disease, or based on a subject not developing one or more symptoms of a disease despite being at risk of developing the disease, etc). Such samples may be referred to as “reference samples”. At least one of the samples or subjects (query subject or sample) may not have an unknown status in relation to the one or more clinical or phenotypic properties. The methods may comprise providing the amino acid sequence of at least part of one or more chains of each of the antigen-binding proteins or encoded sequences derived therefrom as input to a deep learning model comprising a sequence encoding module configured to take as input an amino acid sequence and to produce as output an embedding corresponding to the input sequence (wherein the sequence encoding module has been trained as part of a machine learning model comprising a structure encoder, and using training sequence data and structure data from a plurality of antigen-binding proteins). The methods may further comprise using the embedding produced to cluster the sequence(s). The methods may further comprise identifying one or more clusters of sequences that are enriched for sequences from samples or subjects associated with one or more predetermined properties. For example, the methods may comprise identifying one or more clusters of sequences that are enriched for sequences from subjects that have resilient status. The method may further comprise identifying a subject or sample (query subject or sample) as having the one or more clinical or phenotypic properties based on the pattern of clustering of the set of one or more antigen-binding protein sequences from said subject or sample. For example, when the set comprises antigen-binding protein sequence(s) that cluster in clusters that are enriched for samples that have a clinical or phenotypic property, the query subject or sample may be determined as likely to have the property. Conversely, when the set comprises antigen-binding protein sequence(s) that cluster in clusters that are enriched for samples that do not have a clinical or phenotypic property, or that do not cluster in clusters that are enriched for samples that have the property, the query subject or sample may be determined as unlikely to have the property.

[0109] The methods described herein may further be used to analyse a set of antigen-binding sequences, for example by using the embeddings obtained as described herein to search for sequences in a database of antigen-binding protein sequences. In particular, the methods may comprise providing the amino acid sequence of at least part of one or more chains of a query antigen-binding protein(s) or encoded sequence(s) derived therefrom as input to a deep learning model comprising a sequence encoding module configured to take as input an amino acid sequence and to produce as output an embedding corresponding to the input sequence (wherein the sequence encoding module has been trained as part of a machine learning model comprising a structure encoder, and using training sequence data and structure data from a plurality of antigen-binding proteins). The methods may further comprise using the embedding produced to search through a database of antigen-binding protein sequences for sequences similar to the query antigen-binding protein(s), using a metric of similarity (e.g. a distance metric such as Euclidean distance, Cosine similarity, etc.) between the embedding for the query sequence and embeddings produced as described herein for all sequences on the database of antigen-binding protein sequences.

[0110] The methods described herein may comprise predicting one or more features of an antigenbinding protein using a machine learning model that takes as input embeddings obtained as described herein. For example, the deep learning model that produces the embeddings may further comprise a classification, regression or generative model that takes as input the embeddings and produces as output one or more predicted features. Any functional property prediction from antigen-binding protein sequence may benefit from the methods described herein. The one or more features may be a property of the antigen-binding protein, such as e.g. a functional property. The one or more properties may be selected from: the localisation of paratope residues in the sequence of an antigen-binding protein, binding to a particular target (e.g. binder or non-binder, KD value prediction, etc.), a physicochemical property (such as e.g. thermostability, pH stability), specificity or class of specificity (such as e.g. whether an antigen binding protein binds to a first class of antigens, e.g. viral antigens, or a second class of antigens, e.g. non-viral, such as human antigens), B cell type from which the antigenbinding protein originates, a mutational load associated with the antigen-binding protein, a V gene family associated with the antigen-binding protein, and likelihood of forming a functional pair. Such embodiments may comprise training the machine learning model that performs the property prediction based on the structure enhanced sequence embeddings (separately or as part of a deep learning model comprising the structure enhanced sequence encoding module) using training data comprising sequences of antigen-binding proteins with ground truth data for the one or more properties. The choice of a classification, regression or generative model may depend on the property or properties to be predicted, and / or the training data available. When the machine learning model is trained as part of a deep learning model comprising the structure enhanced sequence encoding module, the parameters of the structure enhanced sequence encoding module may be frozen for at least part of the training.

[0111] Thus, also described herein are methods of identifying the paratope of an antigen-binding protein, the method comprising providing the amino acid sequence of at least part of one or more chains of the antigen-binding protein or an encoded sequence derived therefrom as input to a deep learning model comprising a sequence encoding module configured to take as input an amino acid sequence and to produce as output an embedding corresponding to the input sequence, wherein the sequence encoding module has been trained as part of a machine learning model comprising a structure encoder, and using training sequence data and structure data from a plurality of antigen-binding proteins; and predicting one or more features of the antigen-binding protein using the embedding produced by the sequence encoding module, wherein the one or more features of the antigen-binding protein comprise information identifying one or more positions of the amino acid sequence that are part of the paratope of the antigen-binding protein. Predicting one or more features of the antigen-binding protein may comprise providing the embedding produced by the sequence encoding module as input to a machine learning model trained to take as input an embedded sequence of an antigenbinding protein and to provide as output a prediction of whether each amino acid in the sequence is part of the paratope. Thus, the machine learning model may be a classifier model trained to classify each amino acid of a sequence between a first class and a second class, the first class comprising residues that are part of the paratope and the second class comprising residues that are not part of the paratope. The machine learning model may be trained using training data comprising sequences of antigen-binding proteins for which the location of the paratope is known. Also described herein are methods for providing a tool for identifying the paratope of an antigen-binding protein, the method comprising training a machine learning model to take as input an embedded sequence of an antigen-binding protein and to provide as output a prediction of whether each amino acid in the sequence is part of the paratope, wherein the embedded sequence is an embedded sequence produced by a sequence encoding module as described herein. The method may further comprise training the sequence encoding module. Predicting one or more features of the antigen-binding protein may comprise providing the embedding produced by the sequence encoding module as input to a machine learning model trained to take as input an embedded sequence of an antigenbinding protein and to provide as output a prediction of the distance or relative distance between an amino acid in the sequence and an antigen recognised by the antigen-binding protein. Thus, the machine learning model may be a regression model trained to predict for each amino acid of a sequence a continuous value that is indicative of the distance between the amino acid and the antigen. The machine learning model may be trained using training data comprising sequences of antigen-binding proteins for which the structure of the antigen- antigen-binding protein complex is known. Also described herein are methods for providing a tool for identifying the paratope of an antigen-binding protein, the method comprising training a machine learning model to take as input an embedded sequence of an antigen-binding protein and to provide as output a prediction of the distance between one or more amino acids of the sequence and an antigen bound to the antigen-binding protein , wherein the embedded sequence is an embedded sequence produced by a sequence encoding module as described herein. The method may further comprise training the sequence encoding module.

[0112] Further, also described herein are methods of predicting a physicochemical property of an antigen-binding protein (such as e.g. thermostability, pH stability), the method comprising providing the amino acid sequence of at least part of one or more chains of the antigen-binding protein or an encoded sequence derived therefrom as input to a deep learning model comprising a sequence encoding module configured to take as input an amino acid sequence and to produce as output an embedding corresponding to the input sequence, wherein the sequence encoding module has been trained as part of a machine learning model comprising a structure encoder, and using training sequence data and structure data from a plurality of antigen-binding proteins; and predicting one or more features of the antigen-binding protein using the embedding produced by the sequence encoding module, wherein the one or more features of the antigen-binding protein comprise the physicochemical property. Predicting one or more features of the antigen-binding protein may comprise providing the embedding produced by the sequence encoding module as input to a machine learning model trained to take as input an embedded sequence of an antigen-binding protein and to provide as output a predicted value of a physicochemical property (or a plurality of physicochemical properties). Thus, the machine learning model may be a regression model trained to predict a continuous or semi-continuous value for one or more physicochemical properties. Alternatively, the machine learning model may be a classifier model trained to classify an antigen-binding protein sequence between a plurality of categories associated with different ranges of values of the one or more physicochemical properties. The machine learning model may be trained using training data comprising sequences of antigen-binding proteins for which the value of the physicochemical property or properties is known. Also described herein are methods for providing a tool for predicting one or more physicochemical properties of an antigen-binding protein, the method comprising training a machine learning model to take as input an embedded sequence of an antigen-binding protein and to provide as output a predicted value of the one or more physicochemical properties, wherein the embedded sequence is an embedded sequence produced by a sequence encoding module as described herein. The method may further comprise training the sequence encoding module.

[0113] Also described herein are methods of predicting a binding property of an antigen-binding protein to a specific target (such as e.g. a continuous value such as KD or a category such as binder / non-binder), the method comprising providing the amino acid sequence of at least part of one or more chains of the antigen-binding protein or an encoded sequence derived therefrom as input to a deep learning model comprising a sequence encoding module configured to take as input an amino acid sequence and to produce as output an embedding corresponding to the input sequence, wherein the sequence encoding module has been trained as part of a machine learning model comprising a structure encoder, and using training sequence data and structure data from a plurality of antigen-binding proteins; and predicting one or more features of the antigen-binding protein using the embedding produced by the sequence encoding module, wherein the one or more features of the antigen-binding protein comprise the binding property. Predicting one or more features of the antigen-binding protein may comprise providing the embedding produced by the sequence encoding module as input to a machine learning model trained to take as input an embedded sequence of an antigenbinding protein and to provide as output a prediction of whether the antigen-binding protein binds the specific target or a predicted value of a binding parameter for the specific target. Thus, the machine learning model may be a regression model trained to predict a continuous or semi-continuous value for one or more binding parameters (e.g. KD). Alternatively, the machine learning model may be a classifier model trained to classify an antigen-binding protein sequence between a plurality of categories associated with different ranges of values of the one or more binding parameters, or different binding behaviours (e.g. binder I nonbinder). The machine learning model may be trained using training data comprising sequences of antigen-binding proteins for which the value of the binding parameter or binding behaviour is known (e.g. antigen-binding sequences associated with measured KD values and / or binding category labels such as binder I non-binder I unknown, for example assigned using predetermined ranges of measured KD values). Also described herein are methods for providing a tool for predicting a binding property of an antigen-binding protein to a specific target, the method comprising training a machine learning model to take as input an embedded sequence of an antigen-binding protein and to provide as output a predicted value of the binding property, wherein the embedded sequence is an embedded sequence produced by a sequence encoding module as described herein. The method may further comprise training the sequence encoding module.

[0114] Also described herein are methods of predicting whether a pair of chains of an antigen-binding protein are likely to form a functional antigen-binding protein, the method comprising providing the amino acid sequence of at least part of the chains of the antigen-binding protein or an encoded sequence derived therefrom as input to a deep learning model comprising a sequence encoding module configured to take as input an amino acid sequence and to produce as output an embedding corresponding to the input sequence, wherein the sequence encoding module has been trained as part of a machine learning model comprising a structure encoder, and using training sequence data and structure data from a plurality of antigenbinding proteins; and predicting one or more features of the antigen-binding protein using the embedding produced by the sequence encoding module, wherein the one or more features of the antigen-binding protein comprise the likelihood that the pair of chains will form a functional antigen-binding protein. Predicting one or more features of the antigen-binding protein may comprise providing the embedding produced by the sequence encoding module as input to a machine learning model trained to take as input an embedded sequence of an antigen-binding protein and to provide as output a prediction of whether the antigen-binding protein is likely to be functional. An antigen-binding protein comprising a pair of chains may be considered functional if a pair of chains can form a stable antigen-binding protein, Thus, the machine learning model may be a classifier model trained to classify a pair of chains from an antigenbinding protein sequence between pairs from functional antigen-binding proteins (e.g. pairs from known antigen-binding proteins) and pairs assumed not to form functional antigenbinding proteins (e.g. randomly paired chains). The machine learning model may be trained using training data comprising pairs of chain sequences of known antigen-binding proteins and pairs of chain sequences assumed not to form functional antigen-binding proteins (e.g. randomly paired chains from single chain repertoires or known antigen-binding proteins). Also described herein are methods for providing a tool for predicting whether a pair of chains of an antigen-binding protein are likely to form a functional antigen-binding protein, the method comprising training a machine learning model to take as input an embedded sequence of a pair of chains of an antigen-binding protein and to provide as output a prediction of whether the pair of chains is likely to form a functional antigen-binding protein, wherein the embedded sequence is an embedded sequence produced by a sequence encoding module as described herein. The method may further comprise training the sequence encoding module. The methods may use methods as described in UK patent application GB2215218.5 (which is incorporated herein by reference), except that the encoder module is replaced by a sequence encoding module as described herein.

[0115] Also described herein are methods of predicting a corresponding chain for a query chain of an antigen-binding protein, the method comprising providing the amino acid sequence of at least part of the query chain of the antigen-binding protein or an encoded sequence derived therefrom as input to a deep learning model comprising a sequence encoding module configured to take as input an amino acid sequence and to produce as output an embedding corresponding to the input sequence, wherein the sequence encoding module has been trained as part of a machine learning model comprising a structure encoder, and using training sequence data and structure data from a plurality of antigen-binding proteins; and predicting one or more features of the antigen-binding protein using the embedding produced by the sequence encoding module, wherein the one or more features of the antigen-binding protein comprise the sequence of a corresponding chain. A corresponding chain may be a chain that is likely to form a functional antigen-binding protein with the query chain. Predicting one or more features of the antigen-binding protein may comprise providing the embedding produced by the sequence encoding module as input to a machine learning model trained to take as input an embedded sequence of an antigen-binding protein and to provide as output a prediction of a corresponding chain. An antigen-binding protein comprising a pair of chains may be considered functional if a pair of chains can form a stable antigen-binding protein. Thus, the machine learning model may be a model trained to predict a corresponding chain sequence from a query chain sequence. The machine learning model may be trained using training data comprising pairs of chain sequences of known antigen-binding proteins and pairs of chain sequences assumed not to form functional antigen-binding proteins (e.g. randomly paired chains from single chain repertoires or known antigen-binding proteins). Also described herein are methods for providing a tool for predicting a corresponding chain for a query chain of an antigen-binding protein, the method comprising training a machine learning model to take as input an embedded sequence of a query chain of an antigen-binding protein and to provide as output a prediction of a corresponding chain, wherein the embedded sequence is an embedded sequence produced by a sequence encoding module as described herein. The method may further comprise training the sequence encoding module. The methods may use methods as described in WO2022 / 223451 (which is incorporated herein by reference), except that the encoder module is replaced by a sequence encoding module as described herein. The methods may use a sequence encoding module that is part of a generative model, as described herein.

[0116] Also described herein are methods of providing a variant antigen-binding protein using a query antigen-binding protein, the method comprising providing the amino acid sequence of at least part of one or more chains of the query antigen-binding protein or an encoded sequence derived therefrom as input to a deep learning model comprising a sequence encoding module configured to take as input an amino acid sequence and to produce as output an embedding corresponding to the input sequence, wherein the sequence encoding module has been trained as part of a machine learning model comprising a structure encoder, and using training sequence data and structure data from a plurality of antigen-binding proteins; and predicting one or more features of the antigen-binding protein using the embedding produced by the sequence encoding module, wherein the one or more features of the antigen-binding protein comprise the sequence of one or more chains of a variant antigen-binding protein. A variant may be an antigen-binding protein that is expected to have the same binding specificity as the query antigen-binding protein. Predicting one or more features of the antigen-binding protein may comprise providing the embedding produced by the sequence encoding module as input to a machine learning model trained to take as input an embedded sequence of an antigenbinding protein and to provide as output a prediction of a sequences of one or more chains of a variant antigen-binding protein. The machine learning model may be a model trained to predict a variant chain sequence or sequences from a query chain sequence or sequences. The machine learning model may be trained using training data comprising pairs of chain sequences of known antigen-binding proteins. Also described herein are methods for providing a tool for providing a variant antigen-binding protein for a query antigen-binding protein, the method comprising training a machine learning model to take as input an embedded sequence of at least a part of one or more chains of an antigen-binding protein and to provide as output a prediction of variant chain or chains, wherein the embedded sequence is an embedded sequence produced by a sequence encoding module as described herein. The method may further comprise training the sequence encoding module. The methods may use an encoder-decoder model, or a sequence encoding module that is part of a generative model, as described herein.

[0117] The above methods find applications in any context where it is desirable to predict a property of an antibody or BCR. This is frequently the case in the context of the discovery process of antibody therapeutics. Antibody therapeutics have been shown to be successful approaches for a wide range of diseases from neurodegenerative diseases to cancer. Thus, the approaches described herein find use in the context of providing therapeutics in each of these clinical contexts.

[0118] Thus, the invention also provides a method of providing an antibody therapeutic, the method comprising predicting one or more features of an antigen-binding protein using methods as described herein. The features / properties may be predicted for a query sequence or set of sequences that has been obtained by bulk BCR sequencing of the heavy chain repertoire in one or more samples, and / or by bulk BCR sequencing of the light chain repertoire in one or more samples (where the one or more samples may be the same samples used to analyse the heavy chain BCR repertoire, or different samples). The one or more samples may be from one or more subjects. The one or more subjects may have been identified as having a desired characteristic, such as e.g. a particular clinical phenotype or clinically relevant characteristic such as a biomarker profile. For example, the one or more subjects may be resilient to a particular disease or condition. The disease or condition may be selected from a cancer (such as e.g. breast cancer), a neurodegenerative disease (such as e.g. amyotrophic lateral sclerosis), or an infectious disease (such as e.g. COVID-19). The method may comprise providing the amino acid sequence of at least part of one or more of the query sequences or an encoded sequence derived therefrom as input to a deep learning model comprising a sequence encoding module configured to take as input an amino acid sequence and to produce as output an embedding corresponding to the input sequence, wherein the sequence encoding module has been trained as part of a machine learning model comprising a structure encoder, and using training sequence data and structure data from a plurality of antigenbinding proteins, where the antigen-binding proteins are antibodies. The method may further comprise identifying the target (or a putative target or sets of targets) of the query sequences by clustering the embeddings produced by a sequence encoding module trained as described herein for the one or more query sequences together with embeddings produced by a sequence encoding module trained as described herein for antibody sequences with known targets. The method may further comprise identifying the target (or a putative target or sets of targets) of at least one selected query sequence by screening an antigen binding protein (or a sample comprising said protein, such as e.g. a sample from the same source as the source from which the sequence was identified) comprising said sequence against a plurality of candidate peptides. The plurality of candidate peptides may be selected based on the species from which the one or more samples originate. For example, the source of the one or more samples may be one or more human subjects and the antibody repertoire(s) from the same source(s) as the one or more samples (i.e. comprising an antigen binding protein with the selected query sequence) may be screened against a set of candidate proteins and / or peptides representative of the human peptidome. The method may further comprise selecting one or more query sequences based on one or more criteria that apply to the one or more predicted features and / or one or more criteria that apply to the identity of the putative targets or sets of targets identified for a query sequence. The method may further comprise obtaining an antibody or fragment thereof which comprises a selected query sequence. Obtaining an antibody or fragment thereof may comprise identifying a coding sequence for the antibody or fragment thereof and expressing the sequence in a suitable expression system (such as e.g. in a suitable host cell). The method may further comprise identifying one or more antigens that the antibody or fragment thereof binds to, for example by testing for binding to one or more candidate antigens. The method may further comprise optimising the sequence of the antibody or fragment thereof. Optimising the sequence of the antibody or fragment thereof may be performed using any antibody optimisation technique known in the art. Optimising the sequence of the antibody or fragment thereof may be performed using information from the sequence data from which the query sequence was identified, for example by analysing sequences similar to the query sequence in sequence data from the same or other samples. Optimising the sequence of the antibody or fragment thereof may be performed using methods as described herein. For example, the methods described herein may be used to predict one or more features of an antigen-binding protein, for one or more candidate antigen-binding proteins that are obtained by modifying the sequence of the antibody to be optimised (e.g. in silico mutation). The one or more features that are predicted may be e.g. binding to a predetermined antigen, or a physicochemical property. Thus, the methods described herein may be used to optimise the sequence of the antibody or fragment thereof by predicting one or more features of interest for sequence variants of the antibody or fragment thereof, and selecting one or more sequence variants that are predicted to have one or more features of interest that, alone or together, improve on the original antibody or fragment thereof.

[0119] The invention also provides a method for providing an immunotherapeutic composition, the method comprising identifying an antigen binding protein having a desired property using a method as described herein and producing an immunotherapeutic composition that comprises an antibody comprising the query sequence associated with the antigen binding protein or an antibody that has been derived from such an antigen binding protein (such as e.g. by further optimisation, mutation, etc).

[0120] The methods described herein may also find uses in the context of antibody optimisation. For example, the methods described herein may be used to identify antigen binding proteins that has one or more advantageous properties (such as e.g. improved functional or developability properties) compared to an original antigen binding protein. For example, a sequence associated with an original antigen binding protein may be mutated in silico to obtain one or more query sequences and one or more features of the one or more query sequences may be predicted. The predicted features may be compared with corresponding features (predicted and / or measured) for the original sequence. The query sequences may be ranked or otherwise prioritised based on these predicted features and optionally other criteria that apply to functional or developability properties. Thus, the invention also provides a method of providing an improved antibody, the method comprising identifying a query sequence having one or more desired properties using any of the methods described herein, or an antibody that is derived from a query sequence that has been identified using any of the methods described herein (such as e.g. by further optimisation, mutation, etc). The method may comprise identifying a light chain for a candidate heavy chain sequence, using a method of predicting a corresponding chain for a query chain of an antigen-binding protein as described above.

[0121] The methods described herein also find applications in any context where it is desirable to identify a TCR that has one or more desired properties, such as a TCR that is likely to bind its target from information that is limited to the sequence of the p chain, and / or the a chain (or, in less common cases, the y or 5 chain) or parts thereof (such as e.g. the V-gene, J-gene and junction sequences). This is frequently the case in the context of the discovery process of cell therapeutics such as engineered T cells. Indeed, although the examples below focus on antibodies, TCRs are also antigen binding proteins comprising pairs of proteins and antigen recognition sites, and for which similar data is available (e.g. single chain data from bulk TCR sequencing or paired data from single cell TCR sequencing). Thus, the methods described herein can be applied to TCRs using TCR sequence and structure data, which is available including in public repositories such as e.g. VDJdb, Goncharov et al., 2022; TCRdb, Chen et al. 2021 and STCRDab, Leem et al., 2018.

[0122] Thus, the invention also provides a method of providing a TCR-based therapeutic, such as an engineered T cell expressing a particular TCR, the method comprising identifying a query sequence that has one or more desired properties using any of the methods described herein, wherein the query sequence comprises at least part of an a, p, y and / or 5. For example, the method may comprise obtaining one or more query sequences such as e.g. from bulk TCR sequencing of one or more samples, e.g. from one or more subjects that may have been identified as having a desired characteristic. Embeddings may be obtained for one or more of the query sequences by providing the amino acid sequence of at least part of one or more of the query sequences or an encoded sequence derived therefrom as input to a deep learning model comprising a sequence encoding module configured to take as input an amino acid sequence and to produce as output an embedding corresponding to the input sequence, wherein the sequence encoding module has been trained as part of a machine learning model comprising a structure encoder, and using training sequence data and structure data from a plurality of antigen-binding proteins, where the antigen-binding proteins are TCRs. The method may further comprise clustering these embeddings together with embeddings obtained for TCR sequences with known specificity (e.g. as available in e.g. the VDJdb database). Thus, the methods described herein may also find uses in the context of T cell receptor optimisation, in a similar way as described above for antibodies.

[0123] The following is presented by way of example and is not to be construed as a limitation to the scope of the claims. EXAMPLES

[0124] These examples describe a method of providing a sequence-based, structure aware antibody embedding model, according to the present invention, and demonstrate its use in the context of paratope prediction and target binding prediction.

[0125] Example 1 - Methods

[0126] Datasets

[0127] Overview. The models were trained in three steps (illustrated on Figure 4):

[0128] 1 . Pretraining of sequence and structure encoders: a unimodal pretraining step in which a bi-directional transformer model based on the RoFormer architecture (Su et al. 2021) was trained using MLM and antibody training sequences (Figure 4A). We refer to this model as AntiBERTa2. Depending on the particular case: an optional unimodal pretraining step in which a Swin transformer v2 model (Liu et al., 2022) was trained using Simple Masked Image Modelling (simMIM) and antibody training structures (Figure 4B); an optional multimodal pretraining step in which a pretrained multimodal encoder (pretrained on proteins that are not antibodies) was further pretrained using antibody structures and sequences;

[0129] 2. Multimodal training of models comprising pretrained encoders via multimodal training objectives as specified above (such as e.g. contrastive loss in CLIP) (Figures 4C, D, E-left) - this typically updates the parameters of both the sequence and structural encoders; and

[0130] 3. Fine tuning of models comprising the sequence encoder of the multimodal model obtained at step 2, for a specific task (Figure 4E, right). Note that in the present examples the fine tuning of the models for a specific task was performed using the sequence encoder of the multimodal model rather than the complete multimodal model. However, fine tuning for a specific task may also be performed (instead or in addition to this) on the complete multimodal model.

[0131] The models resulting from step 2 were tested by comparing the similarity of embeddings with structural similarity. The models of step 3 were tested using data related to the task for which the models were trained.

[0132] All models were compared to a sequence only model, AntiBERTa2, which is the bi-directional transformer model based on the RoFormer architecture pretrained using MLM and antibody training sequences. For comparison, all models are fine-tuned if applicable for the same specific task.

[0133] Sequence data. See Figure 4A. Antibody sequence data for pretraining of sequence-based model (AntiBERTa2): human antibody sequences spanning 61 studies were downloaded from the OAS database (Kovaltsuk et al., 2018), and paired sequencing data was obtained from (Jaffe et al., 2022). In addition, proprietary unpaired sequencing data from 7 cohorts (representing about 10% of the paired sequencing data used), and paired sequencing data from a proprietary set of single B cell receptor sequencing were used (representing about 50% of the paired data used). In total, the dataset contains 1188700549 unpaired sequences (heavy or light chains) and 2369267 paired sequences (heavy and light chains). However, due to practical limitations, a random selection of 10% of the unpaired data was used, while all paired data was used. Sequences were filtered by sequence identity across the variable domains, using a different threshold for unpaired sequences (90%) and paired sequences (99%) - i.e. the dataset was filtered to exclude sequences that have over the specified level of sequence identity with a sequence already in the dataset. For paired sequences the pair that is kept is randomly selected. For unpaired sequences the selection of representative sequences was performed using Linclust (Steinegger & Soding, 2018). Sequences were additionally filtered based on CDR region lengths. Sequences with CDR lengths in the following ranges were kept: CDRH1 and CDRL1 with lengths 5-12, CDRH2 and CDRL2 with lengths 1-10, and CDRH3 and CDRL3 with lengths 5-30. The entire collection of 71.75M unique sequences (121202 paired heavy and light chain sequences, 66906391 unpaired heavy chains and 4722751 unpaired light chains) was filtered then split into disjoint training and validation sets for MLM using a 99:1 ratio. In total, the MLM training set comprised 120781 paired heavy and light chain sequences, 66266448 unpaired heavy chains and 4646987 unpaired light chains, while the MLM validation set consisted of 421 paired heavy and light chain sequences, 639943 unpaired heavy chains and 75764 unpaired light chains. Alternative ratios of training and validation sets may also be used and have been tested, including e.g. 95:5. Relatively high training to validation ratios were used in order to keep as much of the data available for training.

[0134] Structure data. See Figure 4B. Structure data was obtained from SAbDab (Dunbar et al., 2014; 20thJanuary 2023 version). All structures include full paired heavy and light chains, and are X-ray crystal structures or cryo-electron microscopy comprising 3D coordinates for all atoms (including backbone and side chains, i.e. sequence data is inherently included in the structure data). Structures that were derived from single-chain variable fragments (scFvs), or had unusual binding patterns (e.g. PDB: 4IOI) were manually removed. Structures were then filtered for those with resolution of 4.0A or better (i.e. resolution of 1 ,0A, 2.0A, 3.0A, 4.0A). The data was not filtered to exclude similar or identical sequences because structural coordinates can vary slightly even for the same sequence, and including this information in the training data is believed to help improve the robustness of the model to noise in structural data. The structural dataset was then subdivided in two different sets. Set A refers to the set of structures that bind protein antigens, where an antigen is comprised of at least 50 amino acid residues. Set B refers to the entire collection of structures whether or not it is bound to an antigen of any length. Both set A and set B were split into training and validation sets such that 80% of the set was used for training. Set A was used to train ImmunoCSSP-ESM (referred to in figures as ImmunoCLIP-ESM), ImmunoCoCa-ESM, while Set B was used to train the ImmunoCSSP- Swin and ImmunoSSA-Swin - respectively referred to in figures as ImmunoCLIP-Swin and ImmunoFLAVA-Swin). Set A comprises 3227 structures (split into 2582 train, 322 eval, 323 test). Set B comprises 4226 structures (split into 3381 train, 845 eval). The sets were formed for practical reasons and either set could be used to train any model described.

[0135] TheraSAbDab set. This refers to a set of therapeutic antibodies obtained from TheraSAbDab (Raybould et al., 2021), accessed on the 27thof March, 2023. There are 885 therapeutic antibodies, of which 142 antibodies have a corresponding structure in the PDB. The 142 antibodies with both structural and sequence data was used for validating the multimodal training (ImmunoCSSP-ESM).

[0136] Task-specific data - paratope prediction. The structural data described above was used to obtain sequences and corresponding paratope information. Briefly, the paratope residues were identified by searching for antibody residues with at least one non-hydrogen atom within a 4.5A radius of any non-hydrogen atom on the antigen’s structure. The data was filtered to reduce redundancy because the task is ultimately on sequence inputs so inclusion of multiple highly similar sequences with similar paratopes may lead to unwanted specialization. The filter was implemented by clustering the sequences using CD-HIT (Li et al., 2001). This resulted in a set of 1701 unpaired heavy-light chain sequences with ground truth annotated paratope. This data was split between training, validation and test sets using a ratio of 80:10:10, leading to a training set of 1360 unpaired sequences, validation set of 170 sequences, and test set of 170 sequences. However, structural data is noisy and paratope labels may vary between antibody structures despite having identical sequences. Thus, in cases where an antibody sequence in the training set is represented in multiple structural data units (e.g. PDB 7CAC and 7E5S), the paratope labels for all instances are used, leading to a final training set size of 2342 sequences (1360 unique sequences).

[0137] Task-specific data - SARS-CoV-2 binding. Data was obtained from Engelhart et al. 2022 which contains quantitative binding scores of scFv-format antibodies against a SARS-CoV-2 target peptide. Sequences with no measurements of binding to the sars-cov-2 antigen were removed. KD values were averaged across replicate measurements. The resulting values were binarized to obtain binder / non-binder ground truth labels. In particular, 25726 sequences with KD<=3 were defined as binders, 25864 sequences with a KD>=4 were defined as non-binders and 20240 sequences 3<KD<4 were discarded as ambiguous. The resulting dataset was split between training, validation and test sets using a ratio of 80:10:10.

[0138] Task-specific data - Trastuzumab binding. Data was obtained from Mason et al. 2021. This contained HER2 binding data from a library of variants of the therapeutic antibody trastuzumab, expressed in a mammalian cell line through site-directed mutagenesis via CRISPR-Cas9-mediated homology-directed repair. The data was obtained by cell sorting of yeast displaying antibodies, and sequences are therefore categorised as binders / non-binders in this data. The data was filtered as described in Mason et al. 2021 and additionally filtered to remove duplicate sequences. The resulting dataset was split between training, validation and test sets using a ratio of 80:10:10 (18223, 2278, 2278 sequences respectively).

[0139] Task-specific data - viral / human antigen classification. A dataset comprising of 13412 paired antibody sequences with known target binding associations was manually curated from SAbDab, TheraSAbDab, and a selected list of publications. Each antibody was manually annotated as binding a human antigen target (e.g. APP protein), or a viral antigen (e.g. SARS- CoV-2 spike protein). The dataset was then filtered for sequences that are paired (VH and VL available), and the number of sequences per class was balanced, as the number of anti-viral antibodies outnumbers anti-human antibodies by an 8:1 ratio. After randomly under-sampling the anti-viral antibodies, a total of 2076 sequences remained (1038 anti-human, 1038 antiviral). Of these, 1328 were in the training set (664 anti-human, 664 anti-viral), 332 in the validation set (166 anti-human, 166 anti-viral) and 416 in the test set (208 anti-human, 208 anti-viral).

[0140] Sequence tokenisation

[0141] The sequence based transformer is able to accept both paired (corresponding heavy and light chain sequences) or unpaired sequences (heavy or light chain sequences) as input. In these examples a single sequence based encoder was used which takes as input the concatenated sequences of the heavy and light chains when available (or one of these). Although separate encoders for the heavy and light chain sequences could have been trained, using a single encoder advantageously simplified the training process as it did not require pretraining and harmonisation of the separate pretrained encoders. All sequence data was tokenised as single amino acids. The vocabulary used is composed of 28 tokens: the standard 20 amino acids and eight special tokens (X, [CLS], [PAD], [UNK], [MASK], [SEP], H, L). Each amino acid acts as a token, and no byte-pair encoding was used. [PAD] tokens are used to pad out tensors to the maximum sequence length of the mini-batch. [UNK] tokens are used for ambiguous amino acids which are not represented in the vocabulary. H and L tokens indicate the heavy or light chain types, respectively. The [SEP] token has a dual purpose of separating the heavy and light chains when using paired input, and is used at the end as a stop token for either paired or unpaired inputs. [CLS] is used at the start token. For example, for a paired sequence input with heavy chain sequence [QVQ...] and light chain sequence [DIV...] may be encoded as “[CLS] H. Q V Q ... [SEP] L. D I ... [SEP]”. For unpaired sequences a single chain with corresponding chain identifier is used and the rest of the tensor is padded. A maximum length of 256 is allowed, as it covers the maximum sequence length in the sequence pretraining dataset, while ensuring that the amount of unnecessary padding is minimised.

[0142] Note that if any of the data to be used (whether for training, testing or use of the model) did not contain full-length heavy chain and / or light chain sequences, e.g. because the sequencing method used to generate this data was not able to recover the full amino acid sequence of the light / heavy chain, it would still be possible to use the model trained using full length sequences tokenised as single amino acids. For example, some datasets contain: (i) for each heavy chain: the V gene identifier, the junction sequence the J gene identifier, and the D gene identifier, and (ii) for each light chain: the V gene identifier, the junction sequence, and the J gene identifier. In such cases, each entry may comprise a combination of gene identifiers and sequences such as e.g. IGHV3-23 / CAR...DYW / IGHJ6 - IGKV3-20 / CQQ... / IGKJ2. Such sequence can be converted to full amino acid sequences by imputing the germline sequences for the corresponding portions (e.g. the germline sequences of the corresponding V and J gene identifiers).

[0143] Other tokenisation schemes could be used and are explicitly envisaged. For example, the models could be trained using tokenised sequences corresponding to a V gene identifier, a junction sequence and a J gene identifier. This may be particularly useful in cases where only partial sequence information is available. In order to deal with this data format, a custom encoding method can be used, where each V-gene constitutes a single token, each J-gene constitutes a single token, and the junction amino acid sequence is tokenised as single amino acids as above. The junction sequence is the most diverse region of the sequence, and is believed to mediate most of the binding functionality, hence the increased granularity in tokenisation of this sequence. In such a scheme, tokens may be used if there was a minimum of e.g. 2 occurrences in the training set. Other possible schemes for the tokenisation of the junction amino acid sequences (or the full sequence) include for example byte-pair encoding (BPE), or tokens for overlapping k-mers (such as e.g. 3-mers). Schemes such as byte-pair encoding may be particularly useful if more full-sequence sequence data was available. Using byte-pair encoding, several non-overlapping amino acids are encoded as tokens using a dictionary that is defined automatically in a data-driven manner. Byte pair encoding is a subword tokenisation scheme that iteratively replaces common pairs of consecutive bytes (in the present case, consecutive amino acids at the start, then bytes corresponding to subwords) with a byte that does not appear in the data. All pairs that occur more than once in the data are replaced by a corresponding token. Full sequence single amino acid tokenisation or bytepair encoding are likely to be more advantageous as structure and function are likely to be influenced by the specific amino acid sequence. Both BPE and the tokenisation scheme described above with single amino acid encoding have been successfully tested, although results are only shown below for the latter.

[0144] Structure encoding

[0145] Depending on the model used, different structure encodings were used. A first approach used a graph-based description of antibody structures using graph visual perceptrons (described in Jing et al., 2020, as implemented in Hsu et al., 2022). The models using this scheme are referred to as “ESM” and derivatives, by reference to the fact that they use the pre-trained ESM-IF1 encoder described in Hsu et al., 2022.

[0146] A second approach used an image-based representation where each antibody is represented as either: (i) a distogram (a 2D x 1 channel representation comprising pairwise distance between residues) or (ii) a pairwise physicochemical matrix comprising the distogram and additional properties of each pair of amino acids in the antibody (in particular, this is a 2D x 6 channel representation comprising the following channels: pairwise distances, electrostatic force, mean hydrophobicity, Boman index, mean isoelectric point, mean solvent accessible surface area). This is referred to as “swinformer” because in the specific examples here a Swin Transformer v2 was used (however, any architecture that can take such tensors I imagelike inputs may be used instead). The pairwise distance between resides was calculated using the Euclidean distance of the coordinates of the alpha carbons of the two residues. Other methods (such as Euclidean distance of backbone centroids, etc.) are also explicitly envisaged. Electrostatic force between residues was calculated using Coulomb’s law based on the amino acid net charge and the pairwise distance calculated as above. All other properties were calculated using the “peptides” library in Python (pypi.org / project / peptides / ).

[0147] Model architecture

[0148] The architecture of the unimodal and multimodal models used in these examples is illustrated on Figure 5. Figure 5A illustrates the architecture of AntiBERTa2, which is a unimodal sequence encoder based on the RoFormer architecture (a bidirectional transformer encoder with rotary position embeddings). Figure 5B illustrates the architecture of AntiSwin, which is a unimodal structure encoder based on the Swin Transformer v2 architecture. Figure 5C shows a projection layer that is used across all models based on CLIP as a means of unifying the output representations from the unimodal models into the same dimensionality. In other words, the ‘projection layer’ is used to obtain representations with a common dimensionality from the unimodal models’ different representation dimensionalities. The top half of figure 5C shows a generic architecture and the bottom half shows a specific example with dimensionalities that have been used in the present examples. The specific configuration of the projection layer is tailored to the specific embodiment. Figure 5D shows the architecture of ImmunoCSSP-ESM: a multimodal model which comprises AntiBERTa2 and a ESM-IF1 structure encoder, where length corresponds to the number of residues in the antibody, and the 3 x 3 refers to the x, y, z coordinates of the backbone nitrogen, alpha carbon, and carbon atoms, respectively. Figure 5E show the architecture of ImmunoCSSP-Swin: a multimodal model which comprises AntiBERTa2 and a Swin Transformer V2 (either 1 -channel or 6-channel version). Figure 5F shows the architecture of ImmunoCoCa: a multimodal model comprising AntiBERTa2 and a ESM-IF1 structure encoder using the CoCa multimodal training objective. Figure 5G shows the architecture of ImmunoSSA: a multimodal model comprising AntiBERTa2 and a Swin Transformer V2 (either 1-channel or 6-channel version)).

[0149] Sequence-based encoder. AntiBERTa2 (architecture shown in Figure 5A, based on the RoFormer; Su et al. 2021) was pre-trained on NVIDIA’S DGX SuperCloud (Cambridge-1) using 48 A100 GPUs, using the MLM task with a 40% masking rate. Peak learning rate: 8e-5. 260k steps (40k warmup). Linear learning rate schedule. Optimizer AdamW, weight decay 0.01. The model used had 16 layers, 16 attention heads, an embedding dimension of 1024 and a feed-forward dimension of 4096. AntiBERTa2 can accept either unpaired (i.e. heavy chain only or light chain only) or paired antibody sequences, where chains are separated by a [SEP] token. The RoFormer is an enhanced encoder only transformer model that integrates positional information into the learning process using rotary position embedding. Specifically, the Rotary Position embedding (RoPE) encodes absolute positions with a rotation matrix and incorporates explicit relative position dependency in the self-attention formulation. Other sequence-to-sequence, encoder only or decoder only models could be used. Sequence-to- sequence, encoder I decoder models are preferably transformer-based, such as e.g. a bidirectional transformer model based on the RoBERTa architecture (Liu et al., 2019). This can be pre-trained as described in Leem et al., 2022. In the present examples, an encoder with a transformer architecture was used. However, alternative encoders may be used. For example, encoder modules of any encoder-decoder convolutional neural network (CNN) or recurrent neural network (RNN) model may be used, for example using the same vocabulary as for the transformer, and also providing a latent representation of an amino acid chain. Recurrent neural network (RNN) architectures such as LSTMs (long short term memory networks) with attention were the previous “state of the art” for neural machine translation before transformers became common for this task. CNN have shown good performance particularly in image analysis tasks and have also been demonstrated for encoding text data, particularly protein sequences (CARP; Yang et al. 2022). RNNs have an entirely different mechanism of how they process sequences compared to Transformers. Briefly, transformers use a series of “selfattention” mechanisms that make them not only faster, but more accurate, while recurrent nets do not have this at all. Decoder only models, and in particular decoder-only transformer models such as e.g. LlaMa2 (Touvron et al., 2023), Falcon LLM (falconllm.tii.ae / ), and GPT (e.g. GPT-3, Brown et al. 2020) may also be used.

[0150] Structure-based encoders. Two different types of structure encoders were used as explained above, one using a graph-based approach including GVPs, and one using an image-based approach including pairwise physicochemical properties descriptors. These are further described below:

[0151] Graph-based: This used a pre-trained ESM-IF1 encoder (GVP-Transformer, directly obtained from Hsu et al., 2022, pretrained on approximately 16k protein structure-sequence pairs from CATH (www.cathdb.info / ) and approximately 12 million structure-sequence pairs from UniRef50 sequences and AlphaFold2 predictions). This is referred to as “ESM-IF1” herein. In a variant of this, the pre-trained ESM-IF1 encoder can be additionally pretrained using antibody structures and sequences from SAbDab. Other graph based models such as Equiformer (Liao & Smidt, 2023) and E(n)GNN (Satorras et al., 2021) were also tested (data not shown). Here, the Equiformer and E(n)GNN can be pre-trained using a structural denoising task, where 15% of the antibody’s residues’ coordinates are perturbed. In addition to the initial 15% of positions, one adjacent position would also be perturbed to create a non-trivial task for the model. For example, if the 10thresidue of the protein is chosen for perturbation, then residues 9 and 11 would also be perturbed. For noising, a random value generated from a standard normal distribution (i.e. Gaussian noise) is added to the nitrogen, alpha-carbon, and carbon atoms’ coordinates of each residue.

[0152] Image-based: This used a model inspired from Swin Transformer V2 (see Figure 5B) pretrained using SimMIM (Simple Masked Image Modelling) on one of: [2D x 1 channel pairwise residue distogram, 2D x 6 channel pairwise residue physicochemical properties] (as described above) in order to create an antibody-specific structure encoder. Swin Transformer V2 (Liu et al., 2022) is a transformer-based computer vision model that includes a relative position bias (introducing an additional parametric bias term to encode geometric relationships in the self-attention calculation), with residual post normalisation, scaled cosine attention and log-spaced coordinates. Any other vision transformers could be used for this, such as e.g. ViT (Dosovitskiy et al., 2020). Any other vision deep neural network model may be used, in particular CNNs such as EfficientNet (Tan and Le, 2019), ResNets and U-nets, VAEs (variational autoencoders, see e.g. Kingma and Welling, 2013) and Diffusion models (e.g. DDPM, Denoising Diffusion Probabilistic Models, see e.g. Ho et al., 2020). Use of ResNEt-50 was also tested (data not shown). SimMIM is a self-supervised pre-training method described in Xie et al. 2022, where random masking is applied on image patches in the context of a raw pixel regression task using a simple prediction head such as a linear layer. Any other image pre-training task could be used instead of or in addition to SimMIM, such as e.g. Masked Autoencoding (MAE). A modified version of SimMIM was implemented to account for the symmetry in the histograms I PCP matrices. In the standard SimMIM implementation, image patches are masked out, and then predicted similar to Masked Language Modelling. The input matrices used in this case are symmetrical because they include pairwise properties. This means that the naive SimMIM task would be trivial as there would likely be an identical patch at j,i in the model input for masked patches, and it may be trivially easy to identify i,j from the other pairwise properties in row i and column j. To account for this, the method was modified such that instead of just patch i,j being masked out, residues are randomly selected from the sequence and the entire row and column associated with that residue is masked out. A further modification also masks out the residues on either side of the masked residue to make the task a little more difficult. For example, with a sequence of length 10, of which residues 2 and 7 are randomly selected for masking, rows and columns 1 ,2,3 and 6,7,8 are masked from the input. The above models are specific examples of the general concept of structure-based encoders. For the image-based approach, any computer vision encoder may be used, as explained above. For the graph-based approach, any other graph based deep neural network encoder may be used, as explained above.

[0153] Multimodal training - CUP. The sequence-based encoder and structural encoders were trained using the CLIP contrastive loss as described in Radford et al., 2021. The following multimodal models were trained: i. ImmunoCSSP-ESM: AntiBERTa2 (RoFormer) as the sequence encoder in all cases. ESM-IF1 as structural encoder, train for 50 epochs; learning rate (LR): 2e-4, weight decay (WD) 0.25, Dropout 0.3, with a 512 dimensions (512d) embedding. ii. ImmunoCSSP-Swin: AntiBERTa2 (RoFormer) as sequence encoder. Train for 50 epochs on CLIP; LR: 1e-3, SwinV2 LR: 1e-4, RoFormer LR: 1e-5, WD: 1e-3, Dropout: 0.1 , 256-d embedding. The learning rates for the two encoders were selected independently because they have different sizes and structures. a. Swin Transformer V2 model as structural encoder - 1 -channel version b. Swin Transformer V2 model as structural encoder - 6-channel version

[0154] The ImmunoCSSP models are trained by minimising a contrastive loss. During this process of training for the multimodal contrastive loss objective, the learnable parameters of each of the underlying sequence unimodal models are updated as well. This applies to all other multimodal models, where the underlying sequence unimodal models are updated during the optimisation process. The learnable parameters of the structure unimodal models are frozen during the multimodal training. All models were trained with the AdamW optimizer. The contrastive pre-training approach used in these examples follows the CLIP procedure (Radford et al. 2021), as further explained herein.

[0155] Multimodal training - CoCa. The sequence-based encoder, AntiBERTa2 (RoFormer) and the ESM-IF1 structural encoder were combined and trained with two losses as described in Yu et al., 2022: a contrastive loss similar to CLIP, and a masked language modelling (MLM) loss with different weights, where the total loss is equal to 2 x MLM loss + 1 x contrastive loss. This is referred to as ImmunoCoCa-ESM. This model was trained for 50 epochs; LR: 2e-4, WD 0.25, Dropout 0.3, with a 512d embedding for ESM-IF1. Other weightings of the MLM and contrastive losses are possible and explicitly envisaged. For example, a loss=1xMLM + 2xcontrastive was also tested successfully. For example, any weighting scheme that weights the two losses between 2:1 and 1 :2 may be used.

[0156] Note that although this wasn’t tested here, the CoCa approach could also be applied to the combination of AntiBERTa2 with either of the Swinformer structural encoders (i.e. 1 or 6 channels).

[0157] Multimodal training - FLAVA. The sequence-based encoder, AntiBERTa2 (RoFormer) and either of the Swinformer structural encoders (i.e. 1 or 6 channels) were combined and trained with multimodal pretraining as described in Singh et al. 2022. Both of these models are referred to as “ImmunoSSA-Swin” or “ImmunoSSA-Swin” herein (where “SSA” stands for “sequence and structure alignment”). Encoders are initialized with unimodal pre-training (MLM and SimMIM, respectively) and then trained with multimodal pre-training (GC - global contrastive loss, MMM - masked multimodal modeling, and ITM - image text matching). For the global contrastive loss training the same hyperparameters were used as CLIP-Swinformer (LR 1e-3, although this was trained for 10 epochs). Results are shown for the 1 -channel version below but both the 1 and 6 channel versions were tested successfully.

[0158] Task-specific training. The pretrained models were then used to construct a neural network with as illustrated on Figure 4E (where AntiBERTa2 encoder is the RoFormer encoder of the pretrained models trained in using multimodal learning as described above), trained for a specific task selected from: paratope identification (where the model is trained to label each amino acid of a sequence as either forming part of the paratope or not), human / viral antigen binding (where the model is trained to classify antibody sequences as binding a human antigen target ora viral antigen), and binding prediction (where the model is trained to classify antibody sequences as binders or non-binders to a particular target - the model being specific to the target). Multimodal training encourages the unimodal models’ parameters to update, leading to a sequence encoder that effectively learns the information embedded within the structure encoder, and vice-versa. The newly-optimised sequence encoder is then used in isolation for sequence-based tasks. After multimodal training, the sequence encoder is taken out of the multimodal models and connected to a single classification or regression layer.

[0159] The full classifier models were trained using the following regime. Paratope prediction: training for 10 epochs, learning rate (LR) 1e-4, warmup: 5%. Sars-cov-2: training for 6 epochs, LR 1e- 5, warmup 15%. Her2: training for 6 epochs, LR 1e-5, warmup 15%; human / viral classification: the unimodal AntiBERTa2 model and ImmunoCSSP models were fine-tuned using the following regime: peak learning rate of 1e-5, batch size of 24, and over 12 epochs. No freezing period was used in these examples, so the weights of the updated AntiBERTa2 were allowed to vary throughout the task-specific (fine tuning) process. However, it is also possible to maintain the parameters of the encoder module, which generates the embeddings, as “frozen” for a predetermined number of epochs after which for the remaining of the training, the parameters of the encoder module are allowed to vary. This approach enables the training to initially focus on training the classification part of the model, only allowing tweaks to the encoding part of the model at a later stage when learning rates are lower. The training for the classification tasks used binary cross-entropy as a loss function.

[0160] Evaluation

[0161] The sequence encoder from each of the models trained using multimodal learning was evaluated by comparing the correlation between pairwise structural RMSD (root mean square deviation) and embedding distance (cosine distance), over pairs of 142 structures in the TheraSAbDab set.

[0162] Structural similarity between two antibody variable regions (Fv) was define as the root-mean square deviation (RMSD) between backbone atoms (N, Co, C, and O). To align antibodies with different sequence lengths, we aligned antibodies based on their IMGT numbering. CDRH3 loop structure similarity was defined as the RMSD between backbone atoms in the IMGT-define CDRH3 loops. We implemented the dynamic time warp algorithm for lengthindependent CDRH3 loop alignment, as described in Nowak et al. 2016. Z-score normalization was then applied to the RMSDs.

[0163] The sequence encoder from each of the models trained using multimodal learning (collectively referred to as “structure enhanced sequence encoder” or “structure enhanced sequence-only model”) was used to build classifiers for paratope prediction, SARS-CoV-2 antigen binding prediction and trastuzumab binding prediction. Performance of these classifier models were compared to performance of classifier models built using the pretrained sequence encoder prior to further multimodal learning. The following metrics were used: F1 score (harmonic mean of precision and recall), MCC (Matthews correlation coefficient, (TPxTN- FPxFN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN)) where TP=true positives, TN=true negatives, FP=false positives, FN=false negatives), AUC (Area Under the ROC (receiver operating characteristic) Curve, only for the binding prediction tasks) or AUPR (Area Under the PR (Precision Recall) Curve, only for the paratope prediction task). For the paratope prediction task, a probability is output for each amino acid and the F1 score is calculated by aggregating the predictions across all amino acids. To enable a robust comparison of the resulting sequence embeddings, the inventors froze each model’s sequence encoder weights and only allowed weight updates in the classification head.

[0164] For each of the classification tasks, the classification models trained (i) with the sequence only encoder (AntiBERTa2) and (ii) the sequence encoder trained as part of a multimodal model (structure enhanced sequence encoder) were compared (in terms of classification AUC) when trained with progressively smaller amounts of task-specific training data (by sampling -with replacement- 5 different subsets of training and test data from the complete task-specific training data, all sampled sets representing the same proportion of the total dataset). The test set data size was the same in all cases. For each model, the average AUC and 95%confidence interval or standard deviation around the AUC was calculated. The full data set used for each tasked comprised the following number of observations: (i) Paratope prediction: Train: 2342; Eval: 170; Test: 171 ; (ii) Sars-cov-2: Train: 41787; Eval: 4644; Test: 5159; (iii) Her2:Train: 18223; Eval: 2278; Test: 2278; (iv) human / viral classification: train: 664 anti-human, 664 antiviral 332; validation: 166 anti-human, 166 anti-viral; test: 208 anti-human, 208 anti-viral.

[0165] Example 1 - Results

[0166] As explained above, after multimodal training, the structure enhanced sequence-only model is removed and tested in isolation across various tasks.

[0167] Figure 6 shows the distance in embeddings (Cosine distance in embedding space) from the multimodal models for pairs of sequences as a function of the pairwise backbone RMSD computed from the structure for the same antibody for the sequence-only model (AntiBERTa2 only, Fig. 6A) and the ImmunoCSSP-ESM model (Fig. 6B). This data suggests that the updated multimodal language model (structure enhanced sequence only model) can put two antibodies with similar structures closer in embedding space than a unimodal language model. This is the equivalent of a model knowing that a horse and zebra (visually similar, text dissimilar) are more alike than a horse and hearse (visually dissimilar, text similar). Given that antibody structures are generally more indicative of function than sequences, this increase in correlation suggests that multimodal models will be better for predicting functional properties than sequence only models. Even though the correlation in both cases is low, the correlation obtained with the structure enhanced sequence only model is more than twice that of the correlation obtained with the sequence only model, indicating significant improvement.

[0168] Table 1 below shows the results obtained for a paratope prediction task (training a language model to predict which positions in an antibody sequence comprise the antibody binding site) using the different models. Using three different metrics, the data shows that multimodal models are at least as good and frequently better than the sequence only model. The results in Table 1 are indicated for five different runs of the model using five different random seeds.

[0169] The data shows that it is likely that the multimodal models would be able to consistently outperform the sequence only model at this task given additional structure training data (as only about 1000 structures were used for training whereas the sequence model benefits from multiple millions of training data points - in other words the structure informed enhancement only has very few data points to contribute an improvement). One possibility to obtain additional structures for training would be to generate structural predictions for antibody sequences in the sequence training data set.

[0170] Table 1. Classification performance for paratope prediction class. F1=F1 score; AUPR=area under the precision recall curve; MCC= Matthews Correlation Coefficient. Values are indicated +- standard deviation.

[0171] Table 2 below shows the results obtained for a HER2 binding prediction task for Trastuzumab mutants (i.e. train a language model to predict which CDRH3 variants of trastuzumab are binders or non-binders of HER2). under the ROC (receiver-operator characteristic) curve; MCC=Matthews Correlation Coefficient. Values are indicated +- standard deviation.

[0172] Table 3 below shows the results obtained for a SARS-CoV-2 peptide binding prediction. This is similar to the HER2 binding task in principle, except that sequence variation can occur across all three CDRs as opposed to just the CDRH3.

[0173] Table 3. Classification performance for SARS-Cov2 binding prediction. F1=F1 score; AUc=area under the ROC curve; MCC=Matthews Correlation Coefficient. Values are indicated +- standard deviation.

[0174] The data in Tables 2 and 3 shows that the structure enhanced sequence only models can perform at least as well or better than the sequence only model. This again indicates that given larger amounts of antibody structure training data, the structure enhanced sequence only models (multimodal derived models) would likely outperform the sequence only model. Note that the data ablation studies described below corresponding to the tasks in Tables 2 (Figure 7A) and 3 (Figure 7B) show that the structure enhanced sequence models outperform the sequence only model in cases where low amounts of task-specific data are available.

[0175] In all tests the ImmunoCSSP-ESM model was found to perform best amongst the multimodal models. In general, the multimodal models were found to match or perform slightly better than the sequence-only models. More significant gains in performance would likely be possible using larger amounts of structural data. The volume of structural data used in these examples is far smaller than the sequence-only dataset, which may explain why the extent of gains is small. There are multiple avenues to add more structural information into the multimodal models. This includes collecting structural data from more sources, and / or adding predicted structure information, such as e.g. by using tools such as AlphaFold2 (Jumper et al., 2021) to increase the structural data corpus. AlphaFold2-predicted structures has been successfully used for creating deep learning models in Hsu et al., 2021.

[0176] Table 4 below shows the results obtained for viral / human antigen classification task. The data shows that the structure enhanced sequence only models can perform at least as well or better than the sequence only model.

[0177] Table 4. Classification performance for human / viral antigen binding prediction. F1=F1 score;

[0178] AUPR=area under the precision recall curve; AUC=area under the ROC curve.

[0179] The benefit of added structural information is particularly apparent in the ablation task. Figure 7A shows the results of data ablation studies where the classification task AUC for the HER2 binding prediction was compared for the sequence only classification model and the structure enhanced sequence only classification models trained with increasingly smaller amounts of data. Figure 7B shows the results of similar data ablation studies for the SARS-CoV-2 peptide binding prediction. For these experiments, the classifier model is initiated with five different random seeds to derive an estimate of how uncertain the model can be. The figures show that the multimodal approach required much less task specific training data to reach a good performance, compared to the sequence only model. Furthermore, the sequence models after multimodal training show a lower standard deviation (indicated by the shaded areas in the graph) even with low volumes of data, suggesting greater model stability. This is extremely significant because task-specific training data can be difficult to obtain and typically require extensive experiments. Therefore, the methods described herein enable prediction of functional properties using much smaller amounts of functional data (pairs of sequence and associated ground truth functional properties) than was necessary using antibody language models known in the art such as that in Leem et al., 2022.

[0180] Example 2 - Methods

[0181] This example further investigates the potential of the Contrastive Language-Image Pretraining (CLIP) architecture to update antibody sequence representations with structural information. This is referred to below as “contrastive sequence structure pretraining” (CSSP), and the resulting model is referred to as “ImmunoCSSP”.

[0182] Datasets. For pre-training the sequence encoder model, AntiBERTa2, a total of 779.4 million sequences was used for pre-training, and 44.3 million sequences were used for evaluation. For pre-training sequence encoders, we compiled 1.47 billion unpaired (i.e. heavy chain or light chain only) human antibodies from the Observed Antibody Space (OAS) database on 23rd February 2023. An additional 70 million unpaired human antibodies from a proprietary dataset. We used Linclust to cluster antibodies at 90% sequence identity, leading to a dataset of 820.1 million unpaired antibodies. In addition, we collected 1.5 million paired human antibody sequences from Jaffe et al. (2022), and a further 1.4 million from an in-house experiment. Due to the relative paucity of pairing data, we implemented a 99% sequence identity cutoff, leading to a total of 2.48 million paired antibodies.

[0183] For multimodal pre-training, 1554 experimental structures downloaded from SAbDab on July 18th, 2023 were used. An additional set of 1.5M model antibody structures, predicted by ABodyBuilder2, was used to complement the experimental data (106527 structures for training, 133166 used for validation and 133166 used for testing) - see “Multimodal training with predicted structures” below). All unimodal and multimodal models in this example were fine-tuned on an antibody-antigen binding dataset comprised of trastuzumab variants screened for human epidermal growth factor receptor 2 (HER2) binding. The dataset was framed as a binary classification task for binding prediction. Antibody structures were downloaded from SAbDab on 18thJuly 2023. Structures resolved by either X-ray crystallography or cryo-electron microscopy were filtered for resolution of 2.5A or better; structures with missing coordinates in the CDR residues were removed. We also removed antibodies that were single-chain variable fragments, and omitted non-human antibodies. Since antibodies with identical amino acid sequences can have slight variations in their structure, we retained a redundant dataset of structures.

[0184] AntiBERTa2 unimodal pre-training. AntiBERTa2 is based on the RoFormer architecture, a bidirectional transformer encoder model with rotary position embeddings. AntiBERTa2 can accept both unpaired (i.e. heavy chain only or light chain only) antibodies, or paired antibodies, where each chain is separated by the [SEP] token. AntiBERTa2 was pre-trained on NVIDIA’S DGX SuperCloud (Cambridge-1 environment) using 48 A100 GPUs on the masked language modelling (MLM) task. Hyperparameters for AntiBERTa2 are: number of layers: 16; number of attention heads: 16; embedding dimension: 1024; feed-forward dimension: 4096; peak learning rate: 8e-5; learning rate schedule: linear; number of steps: 260k (40k warmup); optimizer: AdamZ; Weight decay: 0.01. Contrastive sequence-structure pre-training. ImmunoCSSP was trained by following the CLIP procedure as described above. In brief, CLIP minimises the following loss: LCLIP=Limaae^Ltext, , , exp (sim(zkI,zk') / r') , , , exp (sim(zkI,zk') / r') where Liimmaagqee = ~ L!JUk--iIlo9 ^^jN=1exp , (sim(zk / ‘ ,Zj ) / ,r ,) and Ltext= - E ^tK--iL / off £W'v= ]exp , (sim ,(zk z,Zj ) / r ,) where sim(zk, zk) represents the cosine similarity between the projection of the kth text embedding zkand the projection of the kth image embedding zk, scaled by a learnable parameter T. For ImmunoCSSP, we analogised antibody sequence embeddings as the text embeddings, and antibody structure embeddings as the image embeddings. Sequence embeddings were generated using AntiBERTa2, using the [CLS] token embedding to represent the antibody sequence. Structure embeddings were generated using ESM-IF1 (Hsu et al. 2022). Briefly, ESM-IF1 is a pre-trained geometric vector perceptron model that is linked to a 12-layer transformer encoder-decoder model (6 encoder layers, 6 decoder layers). For our work, we froze the ESM-IF1 model, and average pooled the embeddings from the transformer encoder. For multimodal pre-training the following hyperparameters were used: embedding dimension: 512, dropout: 0.3, clamp value for 100; peak learning rate: 1e-3; learning rate schedule: linear; number of epochs: 10 (although models from earlier epochs were found to be equally suitable); optimiser: adamW; adamW parameters: beat1=0.9, beta2=0.98, epsilon=1e-6; weight decay=0.25.

[0185] Similarity calculations. To compare the output embeddings of unimodal protein language models or ImmunoCSSP, we used the [CLS] token embedding (or <|endoftext|> for decoderbased models - here just ProGen2). In a first approach, we then computed the cosine distance between [CLS] embeddings to represent sequence embedding similarity. Structural similarity was quantified as the root-mean square deviation (RMSD) between backbone atoms (N, Ca, C, and O) in the CDRH3 loops. We implemented the dynamic time warp (DTW) algorithm for length-independent alignment, as described in Nowak et al. 2016.

[0186] In a second approach, the latent structural information in language model embeddings before and after multimodal training was quantified by predicting pairwise structural similarities using a regression head applied to the [CLS] embeddings of antibody sequences. Specifically, the [CLS] token embeddings of both sequences were concatenated, along with the difference between them, i.e. concat(zj , zk, |zk - zj |), and this was used as input to a regression head that predicted a structural similarity.

[0187] Clustering. Sequence embeddings were clustered using the DBSCAN (density-based spatial clustering of applications with noise) algorithm. The neighbourhood radius for DBSCAN was determined by visual inspection of the OPTICS (ordering points to identify the clustering structure) reachability plot. We imposed a minimum of 5 samples to form a cluster. CDRH3 loop structures were also clustered by DBSCAN based on their length independent RMSD values from the DTW method. We used a neighbourhood radius of 1.0A to define a cluster.

[0188] In a further experiment, as illustrated on Fig. 13, the inventors combined the VH sequences of 398 antibodies known to bind a single known target (labelled target A) with the 103284 sequences of the BCR repertoire of an individual with a characteristic of interest (here resilience to a particular condition). They then obtained clusters using one of 3 methods: clonotyping, clustering of embeddings from the AntiBERTa2 antibody language model, and clustering of embeddings from the AntiBERTa2 antibody language model trained using multimodal learning (AntiBERTa2-CSSP, also referred to as ImmunoCSSP above, i.e. combination of AntiBERTa2 and ESM-IF1 trained using CLIP). Two different clustering methods were applied to the embeddings (k-means with Euclidian distance metric or HDBSCAN using Euclidian distance).

[0189] CSSP with alternative language models. The CSSP approach was additionally validated using alternative language models AntiBERTy and ESM2-650M. AntiBERTy is a transformer encoder model described in Ruffolo et al. 2021. It is based on the BERT architecture ad trained using MLM on non-redundant antibody sequences from OAS. ESM2-650M is a transformer protein language model described in Lin et al. 2022. It uses a BERT style encoder only transformer architecture trained with MLM. AntiBERTy is an encoder only model that only accepts unpaired sequences. It does not have the rotary embeddings used in AntiBERTa2. AntiBERTy is much smaller (26M parameters) than AntiBERTa2 (200M parameters). ESM2 is similar to AntiBERTa2 in that it’s also based on the RoFormer architecture. ESM2 however is trained on all proteins as opposed to being an antibody-specific model. It is much larger (650M parameters) than AntiBERTa2 (200M parameters).

[0190] Multimodal training with predicted structures. The CSSP approach was additionally applied (using AntiBERTa2 as sequence encoder and the ESM-IF1 encoder) with predicted structures. Here instead of crystal structures, we use a collection of over 1 million predicted structures obtained from naturalantibody.com. This model is referred to as “reformer-esm- naturalantibody”.

[0191] Example 2 - results

[0192] Multimodal training enhances structural awareness of language models. Previous work has shown that protein language models learn aspects of protein structure following pre-training. For example, self-attention scores of sequence based transformer models have been shown to correlate with inter-residue contacts in experimental protein structures. However, it is unclear if these scores lead to a ‘structurally aware’ sequence embedding, where structurally homologous antibodies reside in similar areas of the language models’ representation spaces. Thus, we measured the cosine distance between antibody sequence embeddings, then correlated the distances to antibody structural similarities (Figure 9A). The results of this are shown in Table 5 below, which additionally includes results from state of the art protein or antibody language models for comparison (in particular, ESM2-150M and ESM2-650M are transformer encoder protein language models described in Lin et al. 2022, based on the RoFormer; ProGen2-OAS and ProGen2-Base are autoregressive transformer language models trained with next token prediction objective described in Nijkamp et al. 2022, trained with unpaired antibody sequences from OAS or generic protein sequences ; BALM is a transformer encoder model (RoBERTa language model) trained with MLM on antibody sequence pairs described in Burbach and Briney 2023).

[0193] The embeddings were computed zero-shot in order to better compare the underlying representations of the unimodal and multimodal models. Among unimodal language models, ESM2-650M distances have the highest correlation to CDRH3 loop RMSD (p = 0.130), while BALM is the strongest antibody-specific language model (p = 0.069). Our multimodal model, ImmunoCSSP, is the strongest performer overall (p = 0.144), representing over two-fold improvement of our sequence-only AntiBERTa2 model (p = 0.064). However, we note that sequence embedding distances from large language models have weak correlation to structural similarity when calculated in this way. Together, the data show that structural pretraining can be beneficial, even with a dataset of only 1554 structures. Further, the improved embeddings can be useful in the context of finding structural homologues (Figures 9B-C). For instance, PDBs 7BEK and 7LKA fall in the same structural cluster, and two different AntiBERTa2 embedding clusters; however, these antibodies are part of one ImmunoCSSP cluster. Likewise, PDBs 4R26 and 5CEX are another pair of antibodies that belong in one structural cluster, one multimodal cluster, but two unimodal sequence clusters.

[0194] This approach is limited by the fact that CDRH3 loop structures are highly dynamic and can adopt various conformations. For example, an antibody with the same antibody sequence may have been crystallised with and without its target antigen, leading to identical sequence embeddings, but highly different structures. Second, it may be that an alternative distance metric is more suited to expressing the structural similarity of antibodies in the latent space. Cosine similarity assumes that each dimension in the latent space has equal contribution toward quantifying differences in antibody structures, though some dimensions may be more important. Thus, all of the correlations in Table 5 may be underestimated. Nevertheless, the effect of the structural pre-training is expected to still be apparent and possibly even more prominent if these points were addressed.

[0195] Table 5. Spearman’s correlation coefficients between language model embedding distance and CDRH3 loop structure similarity. These are values obtained by CLS pooling (using only the CLS token). Similar conclusions were reached by averaging the 3-dimensional tensor produced by the models over one axis for a specific subset of tokens (CDR3 pooling), or mean pooling (averaging over one axis across all tokens).

[0196] To address some of the limitations of the cosine similarity approach, a different approach was also tested in which a regression layer was trained to predict the RMSD based on the embeddings of two sequences. This approach allows us to hone in on specific dimensions of the embedding that could be more meaningful. The results of this are shown on Figure 12

[0197] Before multimodal training, the inventors find that embeddings from antibody-specific language models, specifically AntiBERTy and AntiBERTa2, contain more information about antibody structural similarity than the general protein language model ESM2-650M. For all models, applying the multimodal training increases the structural information content of the embeddings as measured by the correlation of predicted and actual structural similarity. Overall, AntiBERTa2-CSSP embeddings (ImmunoCSSP) contain the most antibody structure information, with predictions’ Pearson correlation coefficients of 0.706 to Fv similarity, and 0.576 to CDRH3 similarity. One interesting observation is that CSSP was most beneficial for ESM2-650M; this is to be expected as ESM2-650M is not antibody-specific to begin with, and thus CSSP with antibody structures and sequences is effectively making it more of an antibody-specific model.

[0198] Additionally, the benefits of using structure aware antibody language models in the context of exploring unknown sequences, for example to identify new binders to a known target was explored using a clustering approach as explained on Fig. 13. Briefly, sequences of known binders to a target were combined with sequences from a BCR repertoire of interest for which targets are not known. The combined set was analysed using clonotyping (baseline method, clustering on sequence similarity), and clustering of embeddings of the sequences from the sequence only antibody language model AntiBERTa2, and the structure enhanced antibody language model AntiBERTa2-CSSP (ImmunoCSSP). Three metrics were computed to evaluate results, and the values of these are provided in Table 7 below: (i) homogeneity - if a method groups antibodies into any one cluster, do all members of that cluster bind the same target? Essentially a perfect score (of 1.0) suggests that anything in one cluster should bind the same target. The higher the homogeneity, the ‘purer’ the signal. We see that clonotyping achieves highest homogeneity (or seen another way, is most conservative), suggesting that anything within a clonotype is likely to have the same target binding profile, (ii) Completeness - this is orthogonal to homogeneity; this is saying “do all binders of a particular target A belong to one single cluster?” In one extreme, if we have 398 target A binders as in our test case, we could theoretically make 398 unique clusters to achieve perfect homogeneity, but that would not be informative - ideally we would want all the target A binders in one cluster or in a few clusters if very different types of binders can be expected. Higher completeness essentially reflects parsimony, (iii) V-measure - this is a weighted average of homogeneity and completeness.

[0199] The results show that the machine learning embedding approaches, coupled with K-means or HDBSCAN, will lead to a reduction in homogeneity but greater completeness than a baseline clonotyping method (completeness was even higher with k-means, 0.0402 for AntiBERTa2 and 0.0400 for AntiBERTa2-CSSP, leading to v-measure 0.0712 and 0.0704, respectively). In other words, the data show that clustering on model embeddings, and especially on structure enhanced sequence embeddings should allow us to do things such as scaffold hopping - i.e., explore sequence-different antibodies with the same binding functionality.

[0200] Table 7. Results of clustering experiments

[0201] Contrastive pre-training improves antigen-binding prediction. In antibody discovery, the main property of interest is an antibody’s binding to its cognate antigen. It is often impractical to gather large volumes of antibody-antigen binding data for a single target due to cost and time restraints. One of the benefits of multimodal models is their capacity to improve data efficiency in fine-tuning. Thus, we implemented our sequence-structure pre-training with an antibody language model (AntiBERTa2) and a general protein language model (ESM2-650M) (see method), then fine-tuned each resulting language model for antigen binding prediction. We focus on HER2, an antigen for which there is a large, publicly available dataset of over 20,000 antibody-antigen interactions. ImmunoCSSP secondary pre-training improves both models’ predictions of HER2 binding with a 7% improvement in F1 for ESM2 and an 11% improvement in F1 for RoFormer-v2.7 (AntiBERTa2, see full results in Table 6 below). Figure 10A shows that the contrastive pretraining progressively improves the performance on HER2 binding prediction up to 5 training epochs. Figure 11A further investigates the generalisability of this conclusion by applying the proposed CSSP approach to 3 different langue models (all trained in a multimodal way with CLIP as explained above). This shows that longer structural pre-training (which we call CSSP) is beneficial no matter what the underlying sequence model is.

[0202] In a data ablation study where we subsample the HER2 training set from a range of 0.1 % - 100%, we find that ImmunoCSSP secondary pre-training improves both models’ predictive accuracy at all ablations (see Figure 10B), demonstrating the utility of this approach in the low-data settings common in antibody discovery and optimization. In effect this can reduce the burden on collaborations with wet lab scientists and allow quicker iterations in a wider protein engineering campaign. Figure 11 B further investigates the generalisability of this conclusion by applying the proposed CSSP approach to 3 different langue models (all trained in a multimodal way with CLIP as explained above). This shows that structural pretraining (indicated by the dotted lines) will increase accuracy at different data volumes regardless of the underlying sequence model.

[0203] Table 6. Accuracy of prediction of HER2 binding using a sequence model (ESM2-650M), the same model trained in a multimodal model as described herein (ESM2-650M-CSSP), a sequence model (AntiBERTa2) and the same model trained in a multimodal model as described herein (AntiBERTa2-CSSP). Mean and std (standard deviation) are provided over 5 random seeds.

[0204] Training multimodal language models with predicted structures. Due to the relative paucity of publicly available antibody structural data in the PDB, we also performed ImmunoCSSP secondary pre-training using a large collection of model structures predicted by ABodyBuilder2. Despite using over one million structures in the training set (i.e. 1000-fold larger than our experimental structural dataset), we find that ImmunoCSSP secondary pretraining with predicted structures degrades performance on binding prediction. This indicates that it is not simply the volume of structural data, but the quality of data that has a beneficial impact to language model performance.

[0205] Thus, the results above show that structural information can be imbued into language models, for example using a contrastive loss, resulting in antibodies that seem more structurally related in the latent space of a multimodal language model than a unimodal one. Further, these results demonstrate that multimodal learning improves few-shot prediction performance, making multimodal learning a practical solution for many, especially in environments where both experimental and computational resources limited.

[0206] Discussion

[0207] Together, the data presented in these examples demonstrate the benefits of using structure enhanced sequence only antibody functional property prediction models, in general and when compared to sequence only antibody functional property prediction models. In particular, the data demonstrates that the multimodal approach better captures structural relationships than sequence-based models alone, suggesting that it will be better at grouping structurally-related (and thus, functionally-related) antibodies.

[0208] Although the multimodal approach in this specific example showed modest improvement across various tasks of interest, such as paratope prediction or target binding prediction, the data very importantly shows that the multimodal method requires less training data than a standard sequence-only transformer. In other words, if smaller amounts of task-specific training data had been available for the tasks, the data in Tables 1-3 would show higher performance for each of the multimodal approaches compared to the sequence only (unimodal) approach. This means that much less experimental data is needed to get a model that can provide useful predictions for any new task, such e.g. discriminating between binders / non-binders for a specific target from a larger repertoire dataset. As a specific example, a relatively small dataset of binders I non-binders to a specific target could be used to train a model as disclosed herein that could be usefully deployed to screen full BCR repertoires for a cohort of individuals (including multiple millions of heavy chain sequences), at the same prediction speed as sequence only models (since prediction uses sequence only and can provide functional predictions in a matter of seconds per sequence). This is by comparison to sequence only models that would have the same speed but much lower accuracy with limited task specific training data, and predicted-structure based models that would require at least a day of compute per BCR repertoire (and hence would just not be possible to deploy for large sets of candidate sequences such as those obtained from BCR repertoires from cohorts of individuals).

[0209] The examples above clearly show the potential of updating an antibody sequence encoder module, while using a frozen, pre-trained structure encoder. Previous approaches in protein multimodal learning have updated both sequence and structure encoders, often investigating how sequence encoders can enrich structure encoders. Furthermore, protein multimodal models have traditionally relied on having both sequence and structure data as input during inference, which would necessitate using a predicted protein structure. Unlike previous approaches in multimodal learning, the inventors use the updated sequence encoder only, showing that structural pre-training leads to sequence representations that better correspond to structural properties. Further, they showed that their approach improves data efficiency for downstream modelling tasks.

[0210] References

[0211] Dunbar J, Krawczyk K, Leem J, Baker T, Fuchs A, Georges G, Shi J, Deane CM. SAbDab: the structural antibody database. Nucleic Acids Res. 2014 Jan;42(Database issue):D1140-6. Engelhart, E., Emerson, R., Shing, L. et al. A dataset comprised of binding interactions for 104,972 antibodies against a SARS-CoV-2 peptide. Sci Data 9, 653 (2022).

[0212] Galson Jacob D., Schaetzle Sebastian, Bashford-Rogers Rachael J. M., Raybould Matthew I. J., Kovalt. Deep Sequencing of B Cell Receptor Repertoires From COVID- 19 Patients Reveals Strong Convergent Immune Signatures. Frontiers in Immunology. Vol. 11. 2020

[0213] Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, Alexander Rives. Learning inverse folding from millions of predicted structures. bioRxiv 2022.04.10.487779. 2022.

[0214] Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael J.L. Townshend, Ron Dror. Learning from Protein Structure with Geometric Vector Perceptrons. arXiv:2009.01411v3 [q- bio.BM] 16 May 2021.

[0215] Kovaltsuk et al., 2018. Observed Antibody Space: A Resource for Data Mining Next- Generation Sequencing of Antibody Repertoires. J Immunol October 15, 2018, 201 (8) 2502- 2509.

[0216] Krawczyk et al., 2019. Looking for therapeutic antibodies in next-generation sequencing repositories. mAbs. Volume 11 , 2019 - Issue 7, Pages 1197-1205.

[0217] Konrad Krawczyk, Andrew Buchanan & Paolo Marcatili (2021) Data mining patented antibody sequences, mAbs, 13:1.

[0218] Jae Hyeon Lee, Payman Yadollahpour, Andrew Watkins, Nathan C. Frey, Andrew Leaver- Fay, Stephen Ra, Kyunghyun Cho, Vladimir Gligorijevic, Aviv Regev, Richard Bonneau. EquiFold: Protein Structure Prediction with a Novel Coarse-Grained Structure Representation. bioRxiv 2022.10.07.511322. 02 January 2023.

[0219] Leem, J., Mitchell, L.S., Farmery, James H.R., Barton, J., Galson, J.D. Deciphering the language of antibodies using self-supervised learning. Patterns 3, 100513. July 8, 2022. Li W, Jaroszewski L, Godzik A. Clustering of highly homologous sequences to reduce the size of large protein databases. Bioinformatics. 2001 Mar; 17(3): 282-3.

[0220] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, Kai-Wei Chang. VisualBERT: A Simple and Performant Baseline for Vision and Language. arXiv:1908.03557v1 [cs.CV], 9 Aug 2019.

[0221] Liu et al., 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692

[0222] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, Baining Guo. Swin Transformer V2: Scaling Up Capacity and Resolution. arXiv:2111.09883v2 [cs.CV] 11 April 2022

[0223] Mason, D.M., Friedensohn, S., Weber, C.R. et al. Optimization of therapeutic antibodies by predicting antigen specificity from antibody sequence via deep learning. Nat Biomed Eng 5, 600-612 (2021).

[0224] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.00020v1 [cs.CV], 26 Feb 2021.

[0225] Matthew I J Raybould and others, Thera-SAbDab: the Therapeutic Structural Antibody Database, Nucleic Acids Research, Volume 48, Issue D1 , 08 January 2020, Pages D383- D388.

[0226] Robinson SA, Raybould MU, Schneider C, Wong WK, Marks C, Deane CM (2021) Epitope profiling using computational structural modelling demonstrated on coronavirus-binding antibodies. PLoS Comput Biol 17(12): e1009675.

[0227] Raybould MU, Marks C, Kovaltsuk A, Lewis AP, Shi J, Deane CM (2021) Public Baseline and shared response structures support the theory of antibody repertoire functional commonality. PLoS Comput Biol 17(3): e1008781.

[0228] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, Douwe Kiela. FLAVA: A Foundational Language And Vision Alignment Model. arXiv:2112.04482v3 [cs.CV] 29 Mar 2022

[0229] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, Yunfeng Liu. RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv:2104.09864v4 [cs.CL] 9 Aug 2022

[0230] Kevin K. Yang, Hugh Yeh, Niccold Zanichelli. Masked Inverse Folding with Sequence Transfer for Protein Representation Learning. bioRxiv 2022.05.25.493516. March 19, 2023

[0231] Kevin K. Yang, Nicolo Fusi, Alex X. Lu. Convolutions are competitive with transformers for protein sequence pretraining. bioRxiv 2022.05.19.492714. Feb 23, 2023.

[0232] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, Yonghui Wu. CoCa: Contrastive Captioners are Image-Text Foundation Models. rXiv:2205.01917v2 [cs.CV] 14 Jun 2022

[0233] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, Han Hu. SimMIM: A Simple Framework for Masked Image Modeling. arXiv:2111.09886v2 [cs.CV], v17 April 2022. Boman, H.G. (2003), Antibacterial peptides: basic facts and emerging concepts. Journal of Internal Medicine, 254: 197-215.

[0234] Yi-Lun Liao, Tess Smidt. Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs. arXiv:2206.11990v2 28 Feb 2023.

[0235] Victor Garcia Satorras, Emiel Hoogeboom, Max Welling. E(n) Equivariant Graph Neural Networks. arXiv:2102.09844v3 16 Feb 2022.

[0236] Leem J, de Oliveira SHP, Krawczyk K, Deane CM. STCRDab: the structural T-cell receptor database. Nucleic Acids Res. 2018 Jan 4;46(D1):D406-D412.

[0237] Si-Yi Chen and others, TCRdb: a comprehensive database for T-cell receptor sequences with powerful search function, Nucleic Acids Research, Volume 49, Issue D1 , 8 January 2021 , Pages D468-D474.

[0238] Goncharov, M., Bagaev, D., Shcherbinin, D. et al. VDJdb in the pandemic era: a compendium of T cell receptors specific for SARS-CoV-2. Nat Methods 19, 1017-1019 (2022).

[0239] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, Baining Guo. Swin Transformer V2: Scaling Up Capacity and Resolution. arXiv:2111.09883v2. 11 Apr 2022.

[0240] Steinegger, M., Sdding, J. Clustering huge protein sequence sets in linear time. Nat Commun 9, 2542 (2018).

[0241] Jaffe, D.B., Shahi, P., Adams, B.A. et al. Functional antibodies exhibit light chain coherence. Nature 611 , 352-357 (2022).

[0242] Brown, T.B. et al. 2020. Language Models are Few-Shot Learners. arXiv:2005.14165v4. 22 Jul 2020.

[0243] Touvron H. et al. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models arXiv:2307.09288v2. 19 July 2023

[0244] Tan M. and Le Q.V. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. arXiv: 1905.11946v5. 11 Sep 2020

[0245] Dosovitskiy A. et al., 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv:2010.11929v2. 3 Jun 2021

[0246] Ho J. et al. Denoising Diffusion Probabilistic Models. arXiv:2006.11239v2. 16 Dec 2020.

[0247] Jumper, J., Evans, R., Pritzel, A. et al. Highly accurate protein structure prediction with AlphaFold. Nature 596, 583-589 (2021).

[0248] Diederik P Kingma, Max Welling. Auto-Encoding Variational Bayes. 10 Dec 2022. arXiv: 1312.6114v11.

[0249] J. Nowak et al., “Length-independent structural similarities enrich the antibody CDR canonical class model,” mAbs, vol. 8, no. 4, pp. 751-760, 2016.

[0250] Jeffrey A. Ruffolo, Jeffrey J. Gray, Jeremias Sulam. Deciphering antibody affinity maturation with language models and weakly supervised learning. arXiv:2112.07782v1 [q-bio.BM] 14 December 2021. Zeming Lin, et al. Evolutionary-scale prediction of atomic level protein structure with a language model. bioRxiv 2022.07.20.500902. 31 October 2022.

[0251] Erik Nijkamp, Jeffrey Ruffolo, Eli N. Weinstein, Nikhil Naik, Ali Madani. ProGen2: Exploring the Boundaries of Protein Language Models. arXiv:2206.13517v1 [cs.LG]. 27 June 2022.

[0252] Sarah M. Burbach and Bryan Briney. Improving antibody language models with native pairing. arxiv.org / ftp / arxiv / papers / 2308 / 2308.14300.pdf 28 Aug 2023.

[0253] All references cited herein are incorporated herein by reference in their entirety and for all purposes to the same extent as if each individual publication or patent or patent application was specifically and individually indicated to be incorporated by reference in its entirety.

[0254] The specific embodiments described herein are offered by way of example, not by way of limitation. Various modifications and variations of the described compositions, methods, and uses of the technology will be apparent to those skilled in the art without departing from the scope and spirit of the technology as described. Any sub-titles herein are included for convenience only and are not to be construed as limiting the disclosure in any way. Other aspects and embodiments of the invention provide the aspects and embodiments described above with the term “comprising” replaced by the term “consisting of” or ’’consisting essentially of’, unless the context dictates otherwise.

[0255] The methods of any embodiments described herein may be provided as computer programs or as computer program products or computer readable media carrying a computer program which is arranged, when run on a computer, to perform the method(s) described above.

[0256] Unless context dictates otherwise, the descriptions and definitions of the features set out above are not limited to any particular aspect or embodiment of the invention and apply equally to all aspects and embodiments which are described.

[0257] Throughout the specification and claims, the following terms take the meanings explicitly associated herein, unless the context clearly dictates otherwise. The phrase “in one embodiment” as used herein does not necessarily refer to the same embodiment, though it may. Furthermore, the phrase “in another embodiment” as used herein does not necessarily refer to a different embodiment, although it may. Thus, as described below, various embodiments of the invention may be readily combined, without departing from the scope or spirit of the invention.

[0258] It must be noted that, as used in the specification and the appended claims, the singular forms “a,” “an,” and “the” include plural referents unless the context clearly dictates otherwise. Ranges may be expressed herein as from “about” one particular value, and / or to “about” another particular value. When such a range is expressed, another embodiment includes from the one particular value and / or to the other particular value. Similarly, when values are expressed as approximations, by the use of the antecedent “about,” it will be understood that the particular value forms another embodiment. The term “about” in relation to a numerical value is optional and means for example + / - 10%.

[0259] Throughout this specification, including the claims which follow, unless the context requires otherwise, the word “comprise” and “include”, and variations such as “comprises”, “comprising”, and “including” will be understood to imply the inclusion of a stated integer or step or group of integers or steps but not the exclusion of any other integer or step or group of integers or steps, “and / or” where used herein is to be taken as specific disclosure of each of the two specified features or components with or without the other. For example “A and / or B” is to be taken as specific disclosure of each of (i) A, (ii) B and (iii) A and B, just as if each is set out individually herein.

Claims

CLAIMS1. A computer-implemented method for predicting one or more features of an antigen-binding protein, the method comprising:Providing the amino acid sequence of at least part of one or more chains of the antigenbinding protein or an encoded sequence derived therefrom as input to a deep learning model comprising a sequence encoding module configured to take as input an amino acid sequence and to produce as output an embedding corresponding to the input sequence, wherein the sequence encoding module has been trained as part of a machine learning model comprising a structure encoder, and using training sequence data and structure data from a plurality of antigen-binding proteins; andPredicting the one or more features of the antigen-binding protein using the embedding produced by the sequence encoding module.

2. The method of claim 1 , wherein the one or more chains are selected from: an immunoglobulin light chain and an immunoglobulin heavy chain.

3. The method of any preceding claim, wherein the amino acid sequence comprises, for each of the one or more chains, the amino acid sequence of the variable domain of the chain, or the amino acid sequence of the full chain.

4. The method of any preceding claim, wherein the amino acid sequence comprises the amino acid sequence of at least part of each of a pair of chains of the antigen-binding protein, optionally wherein the antigen binding protein is a BCR or antibody and the pair of chains are the heavy chain and the light chain of the antigen-binding protein.

5. The method of any preceding claim, wherein providing the amino acid sequence as input to a deep learning model comprises encoding the amino acid sequence using an encoding scheme wherein each amino acid corresponds to an individual token, and / or wherein providing the amino acid sequence as input to a deep learning model comprises providing a token for each of the one or more chains encoding the type of the chain.

6. The method of any preceding claim, wherein the amino acid sequence comprises the amino acid sequence of at least part of each of a pair of chains of the antigen-binding protein and wherein the amino acid sequences of both pairs of chains are provided as a single input to the deep learning model, optionally wherein providing the amino acid sequence as input to a deep learning model comprises encoding the amino acid sequence using an encoding schemewherein amino acid sequences from different chains of the antigen-binding proteins are separated by a special token.

7. The method of any preceding claim, wherein the one or more features of the antigen-binding protein are functional features, optionally wherein the functional features are selected from: an antigen binding property, a physicochemical property, and a functional similarity to a reference antigen binding protein.

8. The method of any preceding claim, wherein the one or more features of the antigen-binding protein comprise the value of a similarity metric between the embedding for the antigenbinding protein and one or more embeddings produced by the deep learning model for respective one or more reference antigen-binding proteins, optionally wherein the one or more reference antigen-binding proteins satisfy one or more predetermined criteria selected from: a therapeutic function, binding to a predetermined antigen, binding to a predetermined class of antigens; or wherein the one or more features of the antigen-binding protein comprises a cluster of antigenbinding proteins to which the antigen-binding protein belongs, or a functional property derived therefrom, wherein the cluster has been obtained by clustering the embedding for the antigenbinding protein and one or more embeddings produced by the deep learning model for a plurality of further antigen-binding proteins, optionally comprising one or more reference antigen-binding proteins with a known value for the functional property that is predicted by clustering.

9. The method of claim 8, wherein the one or more reference antigen-binding proteins comprise positive reference antigen-binding proteins that have one or more desired properties, and negative reference antigen-binding proteins that do not have the one or more desired properties, and wherein the method comprises clustering the embedding for the antigen-binding protein and the reference antigen-binding proteins and identifying one or more clusters that are enriched for positive reference antigen-binding proteins.

10. The method of any preceding claim, wherein the one or more features of the antigenbinding protein comprise one or more features selected from: information identifying one or more positions of the amino acid sequence that are part of the paratope of the antigen-binding protein, information indicating whether the antigen-binding protein is likely to bind a predetermined antigen, and information indicating whether the antigen-binding protein is likely to have a predetermined physicochemical property (e.g. temperature stability, pH stability).

11. The method of claim 10, wherein the deep learning model further comprises a classification model that is configured to classify an antigen-binding protein between a plurality of classes using as input the embedding from the sequence encoding module, a regression model that is configured to predict a continuous or semi-continuous parameter using as input the embedding from the sequence encoding module, and / or a generative model that is configured to predict an antigen-binding protein sequence using the embedding from the sequence encoding module, optionally wherein the deep learning model comprises a plurality of instances of the sequence encoding module, and / or wherein the deep learning model comprising the sequence encoding module and the classification model has been trained using training data comprising sequence data for a plurality of antigen-binding proteins and data indicating which of the plurality of classes each of the plurality of antigen-binding proteins belongs to, or wherein the deep learning model comprising the sequence encoding module and the regression model has been trained using training data comprising sequence data for a plurality of antigen-binding proteins and data indicating the value of the parameter for each of the plurality of antigen-binding proteins, or wherein the deep learning model comprising the sequence encoding module and the regression model has been trained using training data comprising sequence data for a plurality of antigen-binding proteins and data indicating a sequence associated with the antigen-binding proteins, such as e.g. paired sequences.

12. The method of any preceding claim, further comprising repeating, for a plurality of antigenbinding proteins, the step of providing the amino acid sequence of at least part of one or more chains of the antigen-binding protein or an encoded sequence derived therefrom as input to a deep learning model comprising a sequence encoding module, thereby obtaining an embedding for each of the plurality of antigen-binding proteins, wherein the plurality of antigenbinding proteins comprise at least 106antigen-binding proteins and / or wherein the plurality of antigen-binding proteins comprise a BCR repertoire from one or more individuals.

13. The method of any preceding claim, wherein the sequence encoding module is an encoder of a sequence-to-sequence deep learning model, encoder only deep learning model or decoder only deep learning model, and / or wherein the sequence encoding module is a transformer-based encoder, optionally a RoFormer, or a transformer-based decoder.

14. The method of any preceding claim, wherein the sequence encoding module has been trained in at least two steps: a first step in which the sequence encoding module is trained using training sequence data from a plurality of antigen-binding proteins, anda second step in which the sequence encoding module is trained as part of a machine learning model comprising a structure encoder, and using training sequence data and structure data from a plurality of antigen-binding proteins, optionally wherein the sequence encoding module has been further trained as part of a machine learning model comprising the sequence encoding module and: (i) a classifier model that is configured to classify antigen-binding proteins between a plurality of classes, using training sequence data and associated class label data for a plurality of antigen-binding proteins; or (ii) a regression model that is configured to predict a property of an antigen-binding protein, using training sequence data and associated ground truth data comprising a known value for said property for a plurality of antigen-binding proteins.

15. The method of any preceding claim, wherein the structure encoder is configured to take as input structure data for an antigen-binding protein and provide as output an embedded representation of the structure data.

16. The method of any preceding claim, wherein the structure encoder has been trained in at least two steps in which the structure encoder is trained using training structure data from a plurality of proteins: a first substep in which the structure encoder is trained using training structure data from a plurality proteins comprising proteins that are not antigen-binding proteins, and a second substep in which the structure encoder is trained using training structure data from a plurality of antigen-binding proteins; and / or wherein the parameters of the structure encoder are maintained constant when training the machine learning model comprising the sequence encoding module and the structure encoder, using training sequence data and structure data from a plurality of antigenbinding proteins.

17. The method of any preceding claim, wherein the structure data for an antigen-binding protein comprises information characterising the relationship between pairs of amino acids or pairs of atoms in the structure of the antigen-binding protein and / or wherein the structure encoder comprises a graph neural network or an image-based neural network.

18. The method of claim 17, wherein the information characterising the relationship between pairs of amino acids in the structure of the antigen-binding protein is in the form of a graph comprising a plurality of nodes connected by a plurality of edges, where each node representsan amino acid in the structure and each edge represent a relationship between a pair of amino acids.

19. The method of claim 18, wherein: each node is associated with at least one scalar feature and at least one vector feature, and each edge is associated with at least one scalar feature and at least one vector feature, and wherein vector features characterise the direction of vectors between atoms in the structure, and / or each node is associated with one or more features selected from: one or more vectors representing the directions of bonds between backbone atoms at the amino acid, a scalar feature that specifies the identity of the amino acid, and more scalar features that represent a dihedral angle of the backbone structure at the amino acid, and / or each edge is associated with one or more features selected from: a vector representing the direction of the vector between carbon alpha atoms of the pair of amino acids, a scalar feature representing the distance between carbon alpha atoms of the pair of amino acids, and a scalar feature representing the distance along the backbone structure between the amino acids in the pair.

20. The method of claim 17, wherein the information characterising the relationship between pairs of amino acids in the structure of the antigen-binding protein is in the form of one or more matrices or flattened version thereof each comprising coefficients Cy that represent a physicochemical property of a pair of amino acids at respective positions i and j in the sequence corresponding to the structure, optionally wherein the one or more matrices comprise one or more of: a matrix with coefficients Cy that represent the (physical) distance between the amino acids at positions i and j in the 3D structure; a matrix with coefficients Cy that represent the electrostatic force between the amino acids at positions i and j in the structure; a matrix with coefficients Cy that represent the mean hydrophobicity of the amino acids at positions i and j in the structure; a matrix with coefficients Cy that represent the Boman index associated with the amino acids at positions i and j in the structure; a matrix with coefficients Cy that represent the mean isoelectric point of the amino acids at positions i and j in the structure; and a matrix with coefficients Cy that represent the mean solvent accessible surface area of the amino acids at positions i and j in the structure.

21. The method of any preceding claim, wherein the sequence encoding module has been trained as part of a machine learning model comprising a structure encoder, using training sequence data and structure data from a plurality of antigen-binding proteins, and at least onemultimodal learning objective selected from: a contrastive loss, masked multimodal modelling and image-text matching, optionally wherein the sequence encoding module has been trained as part of a machine learning model comprising a structure encoder, using training sequence data and structure data from a plurality of antigen-binding proteins, and one of: (i) CLIP contrastive loss; (ii) a combination of a contrastive loss and masked language modelling; and (iii) a global contrastive loss, masked multimodal modelling and image-text matching.

22. A method of providing an antigen-binding protein having a desired property, the method comprising: predicting one or more features of each of a plurality of query antigen-binding proteins using the method of any of claims 1 to 21 , selecting one or more of the query antigen-binding proteins based on the one or more features, wherein the one or more features are indicative of the presence of the desired property.

23. A method of providing a tool for predicting one or more features of an antigen-binding protein, the method comprising: providing training data comprising sequence data and structure data from a plurality of antigen-binding proteins, wherein the sequence data for an antigen-binding protein comprises the amino acid sequence of at least part of one or more chains of the antigen-binding protein or an encoded sequence derived therefrom and the structure data for an antigen-binding protein comprises data characterising the 3D structure of the antigen-binding protein, and training, using the training data, a deep learning model that comprises: a sequence encoding module configured to take as input a query amino acid sequence and to produce as output an embedding corresponding to the input sequence, and a structure encoder; and providing the sequence encoding module of the deep learning model or a sequence only model including the sequence encoding module for predicting one or more features of an antigen-binding protein, wherein the prediction does not use the structure encoder.

24. A system comprising: a processor; and a computer readable medium comprising instructions that, when executed by the processor, cause the processor to perform the steps of the method of any of claims 1 to 23.

25. One or more computer readable media comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the method of any of claims 1 to 23.

Images