Predicting the thermal stability of immunoglobulin single variable domains using machine learning models.

Abstract

A method, computer system, and apparatus comprising a computer program coded on a computer storage medium for predicting the thermal stability of an immunoglobulin single variable domain (ISV). The system obtains data representing the amino acid sequence of the ISV, generates an input token vector by numerically encoding the amino acid sequence, generates an embedding feature vector by processing the input token vector using an embedding machine learning model having a first set of model parameters, and processes the input including the embedding feature vector using a predictive machine learning model to produce an output that predicts a thermal stability measurement of the ISV.

Description

[Technical Field]

[0001] Cross-reference of related applications This application claims priority to European Patent Application No. 23305893, filed on 5 June 2023, and European Patent Application No. 24305323, filed on 1 March 2024, both of which disclosures are incorporated herein by reference in their entirety.

[0002] This specification generally relates to predicting the thermal stability of immunoglobulin single variable domains (ISVs) using machine learning models. [Background technology]

[0003] The term “Immunoglobulin Single Variable Domain” (ISV) is used interchangeably with “single variable domain” and defines an immunoglobulin molecule formed by the antigen-binding site being located on a single immunoglobulin domain. Thus, an immunoglobulin single variable domain differs from “conventional” immunoglobulins (e.g., monoclonal antibodies) or their fragments (Fab, Fab', F(ab')2, scFv, di-scFv, etc.) which have two immunoglobulin domains, particularly two variable domains that interact to form an antigen-binding site. Typically, conventional immunoglobulins have a heavy chain variable domain (V H ) and light chain variable domain (V L ) interact to form an antigen-binding site. In this case, V H and V L Both complementarity-determining regions (CDRs) contribute to the antigen-binding site, meaning a total of six CDRs are involved in antigen-binding site formation.

[0004] In contrast, a single immunoglobulin variable domain can specifically bind to an antigen epitope without pairing with an additional immunoglobulin variable domain. The binding site of a single immunoglobulin variable domain is a single V H , single V HH or single V LIt is formed by a domain. Therefore, the antigen-binding site of an immunoglobulin single variable domain is formed by three or fewer CDRs.

[0005] Therefore, a single variable domain forms a single antigen-binding unit (i.e., a functional antigen-binding unit essentially consisting of a single variable domain, such that a single antigen-binding domain does not need to interact with another variable domain to form a functional antigen-binding unit), as long as it can form a light chain variable domain sequence (e.g., V L sequence) or a suitable fragment thereof or a heavy chain variable domain sequence (e.g., V H sequence or V HH sequence) or a suitable fragment thereof.

[0006] An immunoglobulin single variable domain (ISV) can be, for example, a heavy chain ISV such as camelized V H or humanized V HH including V H and V HH and the like. In one embodiment, it is V H including camelized V HH or humanized V HH . The heavy chain ISV can be derived from a conventional four-chain antibody or a heavy chain antibody.

[0007] For example, an immunoglobulin single variable domain can be a (single) domain antibody (or an amino acid sequence suitable for use as a single domain antibody), "dAb" or dAb (or an amino acid sequence suitable for use as dAb) or (registered trademark) ISV (defined herein and including, but not limited to, V HH ), or any other single variable domain or any suitable fragment of any one of them.

[0008] In particular, an immunoglobulin single variable domain can be a Nanobody (registered trademark) ISV (e.g., V HH including humanized V H or camelized V HH ) or a suitable fragment thereof.

[0009] V HH and V HH antibody fragments, and V HH the "V HH domain", also known as an "immunoglobulin", was first described as the antigen-binding immunoglobulin variable domain of a "heavy chain antibody" (i.e., an "antibody lacking a light chain") (Hamers-Casterman et al. 1993 (Nature 363: 446-448)). The term "V HH domain" is used to distinguish these variable domains from the heavy chain variable domains (referred to herein as "V H domains") present in conventional four-chain antibodies and the light chain variable domains (referred to herein as "V L domains") present in conventional four-chain antibodies. For further explanation of V HH please refer to the review by Muyldermans 2001 (Reviews in Molecular Biotechnology 74: 277-302).

[0010] For the terms "dAb" and "domain antibody", see, for example, Ward et al. 1989 (Nature 341: 544), Holt et al. 2003 (Trends Biotechnol. 21: 484) and also, for example, WO 2004 / 068820 pamphlet, WO 2006 / 030220 pamphlet, WO 2006 / 003388 pamphlet and other published patent applications of Domantis Ltd. It should also be noted that single variable domains, although less preferred in the context of the present invention because they are not of mammalian origin, can be derived from certain species of shark (e.g., the so-called "IgNAR domain", see, for example, WO 2005 / 18629 pamphlet).

[0011] Immunoglobulin sequences of different origins including those of mouse, rat, rabbit, camel, human and camelidae can be used herein. In the methods described herein, fully human type, humanized or chimeric sequences can also be used. For example, camel immunoglobulin sequences and humanized camel immunoglobulin sequences or camelized domain antibodies, such as those described by Ward et al. 1989 (Nature 341:544), WO 1994 / 04678 pamphlet and Davis and Riechmann (1994, Febs Lett., 339:285-290 and 1996, Prot. Eng., 9:531-537), can be used herein. Further, the ISVs can be fused to form multivalent and / or multispecific constructs (one or more V HH For multivalent and multispecific polypeptides containing domain(s) and their preparation, reference is made to Conrath et al. 2001 (J. Biol. Chem., Vol. 276, 10.7346-7350), and reference is also made to, for example, WO 1996 / 34103 pamphlet and WO 1999 / 23221 pamphlet).

[0012] "Humanized V HH " contains an amino acid sequence corresponding to the amino acid sequence of a naturally occurring V HH domain, but is "humanized", i.e., one or more of the amino acid residues in the amino acid sequence of the naturally occurring V HH sequence (especially in the framework sequence) are replaced by one or more of the amino acid residues present at the corresponding positions in the V H domain of a conventional four-chain antibody derived from human (e.g., as shown above). This can be carried out in a manner known per se, for example, based on the prior art (e.g., WO 2008 / 020079 pamphlet), as will be apparent to those skilled in the art. Further, such humanized V HHIt should be noted that these can be obtained by any suitable method known in itself, and are therefore not strictly limited to polypeptides obtained using naturally occurring polypeptides containing VHH domains as starting materials.

[0013] "Camelization V" H " is a naturally occurring V H This corresponds to the amino acid sequence of the domain, i.e., naturally occurring V from conventional 4-chain antibodies. H One or more amino acid residues in the amino acid sequence of the domain are used in the V of the (Camellidae) heavy chain antibody. HH The amino acid sequence is “camelized” by substitution with one or more amino acid residues located at corresponding positions in the domain. This can be carried out in methods that are well known, as will be apparent to those skilled in the art, for example, based on the descriptions in the prior art (e.g., Davies and Riechman 1994, FEBS 339:285, 1995, Biotechnol. 13:475, 1996, Prot. Eng. 9:531 and Riechman 1999, J. Immunol. Methods 231:25). Such “camelized” substitutions are defined as V as defined herein. H -V L It forms an interface and / or a so-called camelid hallmark residue and / or is inserted into the position of an amino acid present therein (see, for example, International Publication No. 1994 / 04678 and Davies and Riechmann (1994 and 1996)). In one embodiment, camelid V H V is used as a starting material or starting point for generating or designing. H The sequence is derived from mammalian V H Sequences, for example, human-derived V H Arrays, for example V H It has 3 sequences. However, such camelid V H V can be obtained in any suitable manner known by itself, and therefore, as a starting material, naturally occurring V HPlease note that this is not strictly limited to polypeptides obtained using polypeptides containing domains.

[0014] A machine learning model is a computational model that learns patterns and relationships in data and then uses that knowledge to make predictions or decisions on new data. A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a given input. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or output layer. Each layer of the network generates an output from the received input according to the current values ​​of each set of parameters. [Overview of the project] [Means for solving the problem]

[0015] This disclosure describes a method, computer system, and apparatus for predicting the thermal stability of immunoglobulin single variable domains (ISVs), including a computer program coded on a computer storage medium.

[0016] In one embodiment, the Disclosure provides a predictive method for predicting the thermal stability of an ISV. The method may be implemented by a system comprising one or more computers. The system generates an input token vector by acquiring data representing the amino acid sequence of an ISV and numerically encoding the amino acid sequence, and generates an embedding feature vector by processing the input token vector using an embedding machine learning model having a first set of model parameters. The first set of model parameters is updated using self-supervised learning of a first machine learning model comprising an embedding machine learning model and configured to perform a sequence reconstruction task. The system uses a predictive machine learning model to process an input comprising the embedding feature vector and generate an output that predicts a thermal stability measurement of an ISV. The predictive machine learning model has a second set of model parameters updated using supervised learning based on multiple training examples of a second machine learning model comprising the predictive machine learning model. Each training example comprises (i) a respective training input specifying a representation of each ISV, and (ii) a respective label specifying a thermal stability measurement of each ISV.

[0017] In some implementations, self-supervised learning involves training a first machine learning model on a first dataset containing an array representation of an exemplary set of ISVs. In some cases, the exemplary set of ISVs contains multiple heavy-chain single variable domains (VHs). In some cases, the exemplary set of ISVs contains multiple camelized VHs. In some cases, the exemplary set of ISVs contains multiple VHHs. In some cases, the exemplary set of ISVs contains multiple humanized VHHs.

[0018] In some implementations, self-supervised learning involves training a first machine learning model on one or more tasks, such as reconstruction, token demasking, or predicting the next token.

[0019] In some implementations, the thermal stability measurement is the melting temperature.

[0020] In some implementations, the input to the predictive machine learning model further includes data representing the amino acid sequence of the ISV.

[0021] In some implementations, the input further includes data characterizing the array length of the ISV.

[0022] In some implementations, the input further includes data characterizing the three-dimensional (3D) structure of the ISV.

[0023] In some implementations, the input processed by the predictive machine learning model further includes data characterizing the germline of ISVs. For example, the input may further include data characterizing mutations in ISVs from the corresponding wild-type ISVs from which the ISVs originated.

[0024] In some implementations, the input further includes a second embedding feature vector that is different from the embedding feature vector generated by the embedding machine learning model.

[0025] In some implementations, generating an input token vector involves mapping each amino acid in an amino acid sequence to its corresponding numerical value and then concatenating these numerical values ​​to generate the input token vector.

[0026] In some implementations, the embedded machine learning model includes a Large-Scale Language Model (LLM). In some cases, the embedded machine learning model includes a Variational Autoencoder (VAE). In some cases, the embedded machine learning model includes an Autoregressive Transformer. In some cases, the embedded machine learning model includes a Bidirectional Transformer.

[0027] In some implementations, predictive machine learning models include regression models. In some cases, the regression model is a ridge regression model. In other cases, the regression model is a lasso regression model. In some cases, the regression model is implemented by one or more of the following: a neural network, a K-nearest neighbor model, a support vector machine, a decision tree model, or a random forest model.

[0028] In some implementations, the first set of model parameters is fixed both after the self-supervised learning process and during the supervised learning process.

[0029] In some implementations, the first set of model parameters is further updated during the supervised learning process in which the embedded machine learning model and the predictive machine learning model are trained end-to-end together.

[0030] In some implementations, the system performs further operations to select an ISV from a set of candidate ISVs. These operations include predicting the respective thermal stability measurements of each candidate ISV using the methods described above, and selecting an ISV from the set of candidate ISVs based on the predicted thermal stability measurements.

[0031] In another aspect, the disclosure provides a training method for training a predictive model for predicting the thermal stability of ISVs. The method may be implemented by a system comprising one or more computers. The predictive model comprises (i) an embedded machine learning model configured to generate an embedded feature vector for a model input representing the amino acid sequence of an ISV, and (ii) a predictive machine learning model configured to process the input, which comprises the embedded feature vector, and generate an output that specifies one or more properties of an ISV. The system obtains a first dataset comprising a set of sequence representations of ISVs, and uses the first dataset to perform self-supervised training of the first machine learning model comprising the embedded machine learning model on a reconstruction task, and obtains a second dataset comprising a plurality of training examples. Each training example comprises (i) a respective training input specifying a representation of each ISV, and (ii) a respective label specifying a respective thermal stability measurement for each ISV. The system performs supervised training of a second machine learning model comprising predictive machine learning on the second dataset.

[0032] In some implementations, the system further refines the first machine learning model based on the amino acid sequences of the humanized VHH set.

[0033] In some implementations, the first machine learning model includes a Large-Scale Language Model (LLM). In some cases, the first machine learning model includes a Variational Autoencoder (VAE), an Autoregressive Transformer, or a Bidirectional Transformer.

[0034] In some implementations, predictive machine learning models include one or more neural networks, K-nearest neighbor models, support vector machines, decision tree models, random forest models, ridge regression models, or lasso regression models.

[0035] In some implementations, the first set of model parameters for the embedded machine learning model is fixed both after the self-supervised learning process and during the supervised learning process.

[0036] In some implementations, the first set of model parameters for the embedded machine learning model is further updated during the supervised learning process, and the embedded machine learning model and the predictive machine learning model are trained together end-to-end.

[0037] In some implementations, to perform self-supervised learning of a first machine learning model, the system initializes the parameter values ​​of the first machine learning model and updates the parameter values ​​of the first machine learning model by minimizing a loss function defined for the array reconstruction task. In one example, the loss function is:

number

[0038] The disclosure also provides a system including one or more computers and one or more storage devices that, when executed by one or more computers, store instructions causing one or more computers to perform the method described above.

[0039] The disclosure also provides one or more computer storage media that, when executed by one or more computers, store instructions causing one or more computers to perform the method described above.

[0040] The subject matter described herein may be implemented in specific embodiments to achieve one or more advantages.

[0041] Thermal stability is a crucial property for immunoglobulin monovariate domains (ISVs) due to its impact on their functionality and applicability in various biomedical and biotechnological applications. ISV stability under different temperature conditions is important for ensuring their integrity, maintaining binding specificity, and extending their shelf life. In applications such as diagnostics, therapeutics, and biotechnology, where ISVs are often used to target specific antigens, the ability of these molecules to withstand temperature changes is essential for storage, transport, and final use. Furthermore, thermal stability is particularly important in manufacturing processes such as bioprocessing and formulation, where maintaining the structural integrity of ISVs ensures consistent and reliable performance, ultimately contributing to the efficiency and success of downstream applications. Robust thermal stability not only enhances the utility of ISVs but also promotes their wider adoption across diverse fields, highlighting the importance of this parameter in optimizing the performance and reliability of these unique antibody fragments.

[0042] Therefore, a key objective of ISV engineering is to design, create, and / or select an ISV with optimized thermal stability for a particular application. One approach to ISV engineering involves generating a number of candidate ISVs, measuring the thermal stability of each ISV for a specific application, and selecting the ISV with optimal thermal stability. This process can be carried out iteratively by diversifying the selected ISVs to generate candidate ISVs for subsequent iterations.

[0043] Existing ISV engineering processes are fraught with numerous challenges, including multi-objective optimization (i.e., the need to optimize multiple thermal stabilities for specific applications) and large search spaces. For example, in the case of 14-amino acid CDRH3, the sequence space is 1.6 × 10⁻⁶. 18 It is of such magnitude that experimentally benchmarking such a search space becomes impossible.

[0044] The techniques described herein use deep learning to computationally predict the thermal stability of ISVs based on their amino acid sequences. Specifically, the techniques described involve pre-training a language model using self-supervised learning to generate effective embeddings for ISV sequences, followed by supervised learning using labeled data to predict the thermal stability (e.g., melting temperature) of ISVs. In particular, the language model is generally pre-trained on ISV sequences rather than on general protein sequences. This selection of training data is important to ensure that the embeddings generated by the language model are meaningful for downstream tasks predicting ISV properties. The self-supervised pre-training process enables the generation of high-performance embeddings when the labeled data is limited to downstream tasks.

[0045] Based on the predicted ISV thermal stability, the system described or another system can select the optimal ISV sequence for a particular application. For example, the system can generate an output indicating whether a particular ISV is suitable for a particular application, an output specifying the optimal ISV sequence for a particular application, or an output indicating where in the sequence mutations can be performed. The system can send the output to a manufacturing device that can operate to implement instructions for producing an ISV. Overall, by training a high-performance predictive model based on limited experimental data and using the trained model to predict ISV thermal stability, the described techniques can significantly improve the effectiveness and efficiency of ISV engineering.

[0046] Predicting the thermal stability of proteins remains a significant challenge, despite recent advances in structural biology, such as the breakthrough achieved with AlphaFold2, which significantly increased the number of elucidated protein structures. Existing approaches have shown limited success in various contexts. For example, challenges like Kaggle's Novozymes Enzyme Stability Prediction competition highlighted the difficulty of the task, with the highest achievable R-squared value reaching only 0.2–0.55. This underscores the inherent complexity of predicting protein stability and the need for innovative solutions.

[0047] The techniques described herein leverage a machine learning-based analytical framework comprising several components. Firstly, in some implementations, the embedding model is pre-trained on a carefully curated dataset encompassing a diverse range of ISVs, including, for example, VH, VHH, humanized VHH, and camelidized VH. This differs from conventional approaches to training embedding models for predicting protein properties, where training on larger, more general datasets, such as all proteins, is often considered advantageous. The choice to limit the training data to ISV sequences is based on the recognition that ISVs share different properties and thermal stability patterns compared to other proteins. Intensive pre-training ensures that the model captures ISV-specific features while filtering out irrelevant signals from other proteins, which is crucial for accurate thermal stability prediction of ISVs. Secondly, in some implementations, the framework employs a Bidirectional Encoder Representation from Transformer (BERT) model as the architecture for the embedding model. The use of BERT, a powerful transformer-based model known for its success in natural language processing, combined with a curated ISV training dataset, provides an effective approach to extracting high-quality features from ISV ​​sequences. Furthermore, a downstream regression model is trained to predict the melting temperature (Tm) of the ISVs, providing a quantitative measure of their thermal stability. These components work synergistically to produce a model that achieves predictive accuracy significantly exceeding current technological standards.

[0048] Details of one or more embodiments of the subject matter described herein are given in the accompanying drawings and the following description. Other features, aspects and advantages of the subject matter will become apparent from the description, drawings and claims. [Brief explanation of the drawing]

[0049] [Figure 1] This document describes an exemplary environment for screening immunoglobulin single variable domain (ISV) libraries using a thermal stability prediction system. [Figure 2] An example of a thermal stability prediction system is shown. [Figure 3] This flowchart illustrates an exemplary process for predicting the thermal stability of ISVs. [Figure 4] This flowchart illustrates an exemplary process for training a predictive model to forecast the thermal stability of an ISV (Independent Stability Vehicle). [Figure 5] This is a block diagram of an exemplary computer system. [Figure 6] This shows exemplary results for predicting the thermal stability of ISV sequences. [Modes for carrying out the invention]

[0050] Similar reference numbers and names in various drawings refer to the same elements.

[0051] Figure 1 shows an exemplary environment 100 for screening an immunoglobulin single variable domain (ISV) library 102 using a thermal stability prediction system 200. The ISV library 102 defines a set of ISVs, each ISV represented by its respective amino acid sequence.

[0052] The ISV library 102 may contain any appropriate number of ISVs, e.g., 100, 1,000, 10,000, or 1,000,000 ISVs. The ISV library 102 may be generated in any of the various possible ways. For example, some or all of the ISVs in the ISV library 102 may be variants of one or more "original" or "wild-type" ISVs. "Original" or "wild-type" may refer to the ISV that is the starting point for generating one or more ISV variants. More specifically, each ISV in the ISV library 102 may be generated by modifying the identity of each amino acid at one or more positions in the amino acid sequence of the original ISV. The positions in the amino acid sequence of the original ISV may be selected for mutation in any appropriate way, e.g., through random selection or selection according to a predetermined rule. The identity of the new amino acid to be substituted at the position in the amino acid sequence of the original ISV may be selected in any appropriate way, e.g., randomly selected from a probability distribution over a set of possible amino acids. The amino acid sequence of each ISV in the ISV library may differ from the amino acid sequence of the original ISV at any appropriate number of positions, for example, at position 1, position 3, or position 10.

[0053] The thermal stability prediction system 200 is configured to process an array of ISVs (e.g., from the ISV library 102) to generate a thermal stability measurement 104 of the ISVs that characterizes the predicted thermal stability of the corresponding ISVs. The thermal stability measurement 104 may be represented, for example, by the melting temperature (Tm) of the ISV. The melting temperature (Tm) of the ISV may be defined as the temperature at which the ISV undergoes a transition from a folded state to an unfolded state.

[0054] The thermal stability prediction system 200 can screen the ISV library 102 to identify ISVs that have desirable thermal stability measurements. More specifically, the thermal stability prediction system 200 can predict each thermal stability measurement 104 for each ISV in the ISV library 102. Based at least partially on the predicted thermal stability measurements 104, the thermal stability prediction system 200 can designate an appropriate subset of ISVs in the ISV library 102 as "target" ISVs 106.

[0055] The thermal stability prediction system 200 can select an appropriate subset of ISVs in the ISV library to be designated as target ISVs in any of the various possible methods. For example, the thermal stability prediction system 200 can designate any ISV having a thermal stability measurement value 104 that satisfies a predetermined threshold as a target ISV. Alternatively, the thermal stability prediction system 200 can designate a predefined number of ISVs having the best thermal stability measurement value 104 as target ISVs.

[0056] The thermal stability prediction system 200 can designate any appropriate number of ISVs from the ISV library 102, for example, 10 ISVs, 100 ISVs, or 1000 ISVs, as target ISVs 106. In some cases, the thermal stability prediction system 200 designates only a small fraction of the total number of ISVs in the ISV library (for example, less than <1%, <0.1%, or <0.01% of the total number of ISVs in the ISV library) as target ISVs.

[0057] In some cases, the above screening process can be repeated. That is, the target ISV 106 can be used as the "original" ISV to generate additional variants through mutations contained in the ISV library 102 for subsequent iterations. After the iterations are complete (for example, when a predefined number of iterations have been performed or when the target ISV has one or more predefined thermal stability measures that satisfy one or more predefined conditions), the target ISV 106 can be manufactured 108 using appropriate manufacturing techniques, i.e., physically produced.

[0058] The generated ISVs can be used in a variety of applications. For example, the generated ISVs can be applied to a target 112 as a therapeutic agent 110 to achieve a therapeutic effect in the target. In particular, the generated ISVs can target specific disease-associated proteins or cells, such as cells associated with cancer, inflammatory disorders, or infectious diseases. For example, ISVs can be used to interfere with disease pathways by binding to specific proteins and blocking their activity. This may be particularly useful in conditions where abnormal protein signaling contributes to the pathogenesis of the disease. In another example, ISVs can be conjugated into therapeutic agents or payloads to create targeted drug delivery systems. This approach enables specific delivery of drugs to the disease site, reduces off-target effects, and improves the therapeutic index. In yet another example, ISVs can modulate the immune system by targeting immune cells or modulating the immune response. They can be designed to enhance or suppress immune function depending on the therapeutic objective. In yet another example, ISVs can be incorporated into antibody-drug conjugates, where the ISV functions as an antigen-binding domain. This enables targeted delivery of cytotoxic drugs to cancer cells and enhances the specificity of the treatment. In another example, ISVs could be designed to bind to and neutralize pathogens, preventing them from infecting host cells.

[0059] Figure 2 shows an example of a thermal stability prediction system 200. System 200 is an example of a system implemented as a computer program on one or more computers in one or more locations, in which the systems, components, and technologies described below may be implemented.

[0060] The thermal stability prediction system 200 uses a machine learning model to predict a thermal stability measurement 240 of an ISV based on input data 210 specifying the amino acid sequence of the ISV. Generally, the machine learning model includes an embedded machine learning model 220 having a first set of model parameters 222 and a predictive machine learning model 230 having a second set of model parameters 232.

[0061] System 200 includes a sequence tokenizer 225 configured to generate an input token vector as input to an embedded machine learning model 220. The sequence tokenizer 225 generates the input token vector by numerically encoding the ISV sequence, i.e., the amino acid sequence of the ISV.

[0062] The embedded machine learning model 220 is configured to process the input token vector to generate an embedded feature vector 225. The embedded feature vector 225 is a numerical representation of the input data that captures essential information required for one or more tasks. In particular, the embedded feature vector 225 can be a high-dimensional vector of real numbers that captures features of the model input specifying the ISV.

[0063] In some implementations, the embedded machine learning model 220 is a neural network. The embedded neural network can employ any suitable architecture. In particular, the embedded neural network 220 may include at least a portion (e.g., the embedding portion) of a state-of-the-art large-scale language model (LLM).

[0064] For example, in some implementations, the embedded neural network 220 may include a variational autoencoder (VAE) encoder network. Examples of VAE implementations are described in “Auto-encoding variational Bayes,” Kingma et al., arXiv:1312.6114, 2013.

[0065] In some implementations, the embedded neural network 220 may include an embedded layer of an autoregressive transformer, such as a generative pre-trained transformer (GPT). An example of a GPT implementation is described in “Language Models are Few-Shot Learners,” Brown et al., Advances in Neural Information Processing Systems 33:1877-1901, 2020.

[0066] In some implementations, the embedded neural network 220 may include a bidirectional transformer, such as a bidirectional encoder representation (BERT) model from a transformer. An example of a BERT implementation is described in “Pre-training of Deep Bidirectional Transformers for Language Understanding,”Devlin et al., arXiv:1810.04805, 2018.

[0067] Using language models to generate embeddings for ISV sequences offers several advantages for predicting ISV thermal stability in downstream tasks. Due to evolutionary pressures, ISV sequences are not random. For example, ISVs may contain evolutionarily conserved regions, and parts of the ISV sequence may be reused. Furthermore, correlations and interactions may exist between pairs of positions within an ISV. Language models can learn complex patterns in ISV sequences and can be used to identify previously unknown patterns and correlations in ISV sequences. As will be discussed in more detail below, language models for generating ISV embeddings are trained specifically using ISV sequence data. This is important to obtain a high-performance model for ISV characteristic prediction, i.e., a model that has high predictive accuracy and avoids model bias or overfitting.

[0068] The system 200 further includes a predictive machine learning model 230 configured to process an input containing an embedded feature vector 225 and produce an output 240 that predicts a thermal stability measurement 240 of the ISV. The predicted thermal stability may be the melting temperature (Tm) of the ISV.

[0069] Generally, the predictive machine learning model 230 may be a regression model that processes the input to output a value of a thermal stability measurement 240, such as the melting temperature (Tm). The input to the predictive machine learning model 230 includes an embedding feature vector 225 generated by the embedding model 220. In some cases, the input to the predictive machine learning model 230 may include additional data to characterize the ISV sequence. For example, in some implementations, the input may further include data representing the amino acid sequence of the ISV, such as the one-hot encoding of the ISV sequence. In some implementations, the input may further include data characterizing the ISV sequence length. In some implementations, the input may further include data derived from the three-dimensional (3D) structure of the ISV (e.g., molecular properties). The 3D structure of the ISV can be obtained experimentally or using the predictive model. In some implementations, the ISV is a variant of wild-type ISV, and the input may further include data characterizing the germline of the ISV, i.e., the identity of the wild-type ISV from which the variant ISV mutated. In some implementations, the input further includes data characterizing the variant ISV. In some implementations, the input further includes a second embedding feature vector distinct from the first embedding feature vector 225. The second embedding feature vector can be generated, for example, by processing the ISV sequence using a general protein language model.

[0070] The predictive machine learning model 230 can be implemented using any suitable machine learning technique, and may include one or more of the following: neural networks, K-nearest neighbor models, support vector machines, decision tree models, random forest models, ridge regression models, or lasso regression models.

[0071] In some implementations, system 200 or another system includes a self-supervised learning engine 250 configured to update model parameters 222 of an embedded machine learning model using self-supervised learning based on a set of ISV array representations 255. The goal of self-supervised learning is to learn meaningful embeddings of ISV arrays without the need to use labeled data. The self-supervised learning engine 250 learns embeddings using data that specifies or represents each of a set of unlabeled ISV array data, i.e., ISV properties unlabeled ISV arrays. In other words, the self-supervised learning engine 250 can leverage a large number of known ISV arrays to learn embeddings without the need to obtain a large amount of experimental benchmark data on the thermal stability of known ISVs. Generally, the dataset 225 contains a large number of ISV arrays, e.g., hundreds of thousands, millions, tens of millions, or hundreds of millions of ISV arrays. The ISV sequences included in dataset 225 may include sequences of various ISVs and ISV variants, such as VHH variants, VH variants, and humanized VHH variants.

[0072] The choice to limit training data to ISV sequences deviates from conventional approaches to training embedding models for predicting protein properties, as training on larger, more general datasets, such as all proteins, is often considered advantageous. This choice is based on the recognition that ISVs share different properties and thermal stability patterns compared to other proteins. Pre-training focused on ISV data ensures that the model captures ISV-specific features while filtering out irrelevant signals from other proteins, which is crucial for downstream tasks that accurately predict the thermal stability of ISVs.

[0073] To effectively learn embeddings from unlabeled data, the self-supervised learning engine 250 may be configured to train a first machine learning model to perform a reconstruction task, i.e., the task of generating embeddings for an input ISV array representation and reconstructing the input ISV array representation from the embeddings. The first machine learning model includes an embedding machine learning model 220 as a subnetwork for generating embeddings. In some cases, the self-supervised learning engine 250 may train the first machine learning model to perform a token demasking task and / or a next token prediction task. The demasking task is the task of predicting a masked token from an unmasked token in an input token array. The next token prediction task is the task of predicting the next token in the token array based on the preceding token.

[0074] For example, training an embedding machine learning model 220 using self-supervised learning techniques to perform tasks such as reconstruction, token unmasking, and / or predicting the next token allows the embedding machine learning model 220 to learn to generate an embedding encoded with rich information characterizing the ISV sequence.

[0075] As described above, the embedded machine learning model 220 may be a neural network with an appropriate architecture. Generally, the neural network is a deep neural network (DNN) with multiple hidden layers. Each hidden layer is associated with an activation function, such as ReLU, Sigmoid, Tanh, leaky ReLU, Gaussian Error Linear Unit (GELU), or Softmax activation function. Similarly, as described above, the training dataset 225 is selected to contain a large number of ISV sequences. In a particular exemplary example, the neural network may include a transformer encoder having an encoder layer, a pooling layer with 768 dimensions, and a feedforward layer with 3072 dimensions. The transformer encoder may include 12 hidden layers. Each attention layer of the transformer encoder may have 12 attention heads.

[0076] After the neural network architecture has been appropriately selected, the parameter values ​​of the first machine learning model can be started (for example, randomly). The self-supervised learning engine 250 can update the parameters of the first machine learning model (including the model parameters 222 of the embedded machine learning model 220) by minimizing a loss function computed using the training data (for example, the reconstruction error between the input ISV sequence and the reconstructed ISV sequence in the reconstruction task).

[0077] In one example, the loss function includes a masked language modeling loss for an unmasking task that predicts masked tokens from unmasked tokens in an input token array x of ISV arrays in the training data. The masked language modeling loss is:

number

[0078] The self-supervised learning engine 250 can update model parameters using any suitable backpropagation-based machine learning technique, such as the Adam or AdaGrad optimizer.

[0079] System 200 or another system may further include a supervised learning engine 260 configured to update model parameters 232 of a predictive model 230 based on a labeled dataset 265. The labeled dataset 265 includes multiple labeled training examples. Each training example includes (i) a training input specifying a representation of each ISV, and (ii) a label specifying a thermal stability measurement of each ISV. ISV labels may be obtained based on experimental measurements of the thermal stability of the corresponding ISV. For example, the melting temperature (Tm) of each ISV can be measured using techniques such as circular dichroism (CD) spectroscopy, differential scanning fluorescence quantification (DSF), nano-DSF, or differential scanning calorimetry (DSC). Generally, the labeled dataset 265 contains a much smaller number of training sequences compared to the unlabeled dataset 255. In an exemplary example, the unlabeled dataset 255 may contain millions, tens of millions, or hundreds of millions of ISV sequences, while the labeled dataset 265 may contain thousands, tens of thousands, or hundreds of thousands of labeled training examples.

[0080] The supervised learning engine 260 is configured to perform supervised learning of a second machine learning model, including a predictive machine learning model 230, on a labeled dataset 265. Specifically, the supervised learning engine 260 is configured to update the parameters of the second machine learning model (including the model parameters 232 of the predictive machine learning model 230) based on the labeled dataset 265.

[0081] In some implementations, the second machine learning model further includes an embedded machine learning model 220. That is, the model parameters 222 of the embedded machine learning model 220 are further fine-tuned end-to-end with the predictive machine learning model 230 through supervised learning based on a labeled dataset 265.

[0082] In some other implementations, the model parameters 222 of the embedded machine learning model 220 are fixed during supervised learning while the model parameters 232 of the predictive machine learning model 230 are being updated.

[0083] The supervised learning engine 260 can update the parameters of a second machine learning model (including the model parameters 232 of the predictive machine learning model 230, and optionally the model parameters 222 of the embedding machine learning model 220) by minimizing the prediction error between the predicted ISV thermal stability measurement (e.g., Tm) and the thermal stability measurement specified by the label. The supervised learning engine 260 can update the model parameters using any suitable backpropagation-based machine learning technique, for example, using the Adam or AdaGrad optimizer.

[0084] Based on the predicted ISV thermal stability measurement 240, the system 200 can select an ISV sequence with desirable thermal stability from a candidate ISV sequence for a specific application. For example, the system 200 can generate an output indicating whether a particular ISV is suitable for a particular application or an output specifying the optimal ISV sequence for a particular application. The system can send the output to a manufacturing device that can implement instructions for generating an ISV.

[0085] Figure 3 is a flowchart illustrating an exemplary process 300 for predicting the thermal stability of an ISV. For convenience, the process 300 is described as being performed by a system of one or more computers located in one or more locations. For example, a thermal stability prediction system appropriately programmed in accordance with this disclosure, e.g., the thermal stability prediction system 200 in Figure 2, can perform the process 300.

[0086] In step 310, the system obtains an ISV sequence, i.e., a token sequence representing the amino acid sequence of the ISV. The ISV sequence may be the sequence of a candidate ISV intended for a particular application. The candidate ISV may be a variant that has mutated from the "original" or "wild-type" ISV.

[0087] In 320, the system generates an input token vector by numerically encoding the token sequence. For example, the system can generate a vector by concatenating numerical values ​​assigned to different amino acids to form the input token vector.

[0088] In 330, the system generates an embedded feature vector by processing an input token vector using an embedded machine learning model. The embedded machine learning model has a first set of model parameters. This first set of model parameters is updated using self-supervised learning of a first machine learning model that includes the embedded machine learning model and is configured to perform an array reconstruction task. In one particular example, the embedded machine learning model is a large-scale language model (LLM) based on a bidirectional transformer.

[0089] In 340, the system uses a predictive machine learning model to process an input containing embedding feature vectors and generate an output that predicts a thermal stability measure of the input ISV. In some cases, the input may further include additional data that supplements the embedding feature vectors generated by the embedding machine learning model. For example, the additional data may include one or more of the following: the encoding of the input ISV sequence, the length of the ISV sequence, data characterizing the 3D structure of the ISV, data specifying the germline of the ISV, and / or data characterizing mutations in the input ISV from the original ISV. In some implementations, the input further includes a second embedding feature vector generated by a general protein language model.

[0090] The predictive machine learning model has a second set of model parameters. This second set of model parameters is updated using supervised learning based on multiple training examples of the second machine learning model, which includes the predictive machine learning model.

[0091] Figure 4 is a flowchart illustrating an exemplary process 400 for training a predictive model to predict thermal stability measurements of an ISV. For convenience, process 400 is described as being performed by a system of one or more computers located in one or more locations. For example, a thermal stability prediction system appropriately programmed in accordance with this disclosure, e.g., the thermal stability prediction system 200 in Figure 2, can perform process 400.

[0092] Generally, a predictive model includes (i) an embedding machine learning model configured to generate an embedding feature vector for a model input representing the amino acid sequence of the ISV, and (ii) a predictive machine learning model configured to process the embedding feature vector to generate an output specifying a thermal stability measurement of the ISV.

[0093] In step 410, the system obtains a first dataset containing a set of sequence representations of ISVs. For example, a sequence representation could be a token vector that numerically codes a sample sequence of an ISV whose amino acid sequence is known.

[0094] In 420, the system uses a first dataset to self-supervise training a first machine learning model, including an embedded machine learning model, for a reconstruction task. The first machine learning model may be a large-scale language model and may include a variational autoencoder (VAE), an autoregressive transformer, or a bidirectional transformer.

[0095] In 430, the system acquires a second dataset containing multiple training examples. Each training example includes (i) a training input specifying a representation of each ISV, and (ii) a label specifying a thermal stability measurement for each ISV.

[0096] In step 440, the system performs supervised learning of a second machine learning model, including a predictive machine learning model, based on a second dataset. The second machine learning model may include one or more of the following: a neural network, a K-nearest neighbor model, a support vector machine, a decision tree model, a random forest model, or a ridge regression model.

[0097] Figure 5 is a block diagram of an exemplary computer system 500 that may be used to perform the operations described above. The system 500 includes a processor 510, memory 520, storage device 530, and input / output device 540. Each of the components 510, 520, 530, and 540 may be interconnected, for example, using a system bus 550. The processor 510 can process instructions to be executed within the system 500. In one implementation, the processor 510 is a single-threaded processor. In another implementation, the processor 510 is a multi-threaded processor. The processor 510 can process instructions stored in memory 520 or storage device 530.

[0098] Memory 520 stores information within the system 500. In one implementation, memory 520 is a computer-readable medium. In another implementation, memory 520 is a volatile memory unit. In yet another implementation, memory 520 is a non-volatile memory unit.

[0099] The storage device 530 can provide high-capacity storage to the system 500. In one implementation, the storage device 530 is a computer-readable medium. In various different implementations, the storage device 530 may include, for example, a hard disk drive, an optical disk drive, a storage device shared over a network by multiple computing devices (e.g., cloud storage), or some other high-capacity storage device.

[0100] The input / output device 540 provides input / output operation for the system 500. In one implementation, the input / output device 540 may include one or more network interface devices, such as an Ethernet card, a serial communication device, such as an RS-232 port, and / or a wireless interface device, such as a 502.11 card. In another implementation, the input / output device may include a driver device configured to receive data and transmit output data to other input / output devices, such as a keyboard, printer, and display device 560. However, other implementations such as mobile computing devices, mobile communication devices, and set-top box television client devices may also be used.

[0101] An exemplary processing system is illustrated in Figure 5, but the subject matter and functional operations described herein may be implemented in other types of digital electronic circuits or in computer software, firmware, or hardware, or a combination thereof, including the structures disclosed herein and their structural equivalents.

[0102] Figure 6 shows the predicted melting temperatures (Tm) for two sets of ISV sequences using the techniques described above, compared to the measured Tm. In particular, panel (a) shows the predicted Tm versus measured Tm for 102 wild-type ISVs. The correlation coefficient r is calculated to be 0.6 for wild-type ISVs. Panel (b) shows the predicted Tm versus measured Tm for 454 wild-type ISV variants. The correlation coefficient r is calculated to be 0.74 for wild-type ISVs. Figure 6 demonstrates the effectiveness of using the machine learning models described herein to predict the thermal stability of ISV sequences.

[0103] This disclosure uses the term “configured” in relation to systems and computer program components. One or more computer systems being configured to perform a particular operation or action means that the system has software, firmware, hardware, or a combination thereof installed that causes the system to perform the operation or action while in operation. One or more computer programs being configured to perform a particular operation or action means that one or more programs, when executed by a data processing device, contain instructions that cause the device to perform the operation or action. The subject matter and functional operation embodiments described in this disclosure may be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including structures disclosed in this disclosure and their structural equivalents, or one or more combinations thereof. Embodiments of the subject matter described in this disclosure may be implemented as one or more modules of computer program instructions encoded on a tangible non-temporary storage medium for execution by one or more computer programs, i.e., for control of the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage board, a random or serial access memory device, or one or more combinations thereof. Alternatively, or in addition, program instructions may be encoded on artificially generated propagating signals, such as mechanically generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0104] The term "data processing device" refers to data processing hardware and encompasses all types of devices, machines, and equipment for processing data, including, for example, programmable processors, computers, or multiple processors or computers. A device may be or may further include dedicated logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, a device may optionally include code that constitutes an execution environment for computer programs, such as processor firmware, protocol stacks, database management systems, operating systems, or one or more combinations thereof.

[0105] Computer programs may be called or described as programs, software, software applications, apps, modules, software modules, scripts, or code; they may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and they may be deployed in any form, such as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but is not required, correspond to a file in a file system. A program may be part of a file that holds one or more scripts stored in other programs or data, such as a markup language document; a single file or multiple collaborative files dedicated to the program in question, such as a file that stores one or more modules, subprograms, or parts of code. Computer programs may be deployed to run on one computer or located in one site, or distributed across multiple sites and interconnected by a data communication network.

[0106] In this disclosure, the term “database” is used broadly to refer to any collection of data. The data may not need to be structured in any particular way, or may not need to be structured at all, and may be stored on storage devices in one or more locations. Thus, for example, an index database may contain multiple collections of data, each of which may be organized and accessed differently.

[0107] Similarly, in this disclosure, the term “engine” is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers are dedicated to a particular engine, while in other cases, multiple engines can be installed and run on the same one or more computers.

[0108] The processes and logic flows described herein may be executed by one or more programmable computers that run one or more computer programs to perform functions by acting on input data and producing outputs. The processes and logic flows may be executed by dedicated logic circuits, such as FPGAs or ASICs, or by a combination of dedicated logic circuits and one or more programmed computers.

[0109] A computer suitable for running computer programs may be based on a general-purpose or dedicated microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory or random-access memory, or both. Essential elements of a computer are a central processing unit for implementing or executing instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory may be complemented by or incorporated into dedicated logic circuits. Generally, a computer may include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or be operable to receive data from them, transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer may be incorporated into another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, such as a Universal Serial Bus (USB) flash drive.

[0110] Computer-readable media suitable for storing computer program instructions and data include, as an example, all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0111] To enable interaction with a user, embodiments of the subject matter described herein may be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, on which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user, for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback or tactile feedback, and input from the user may be received in any form, including acoustic, verbal or tactile input. Furthermore, the computer may interact with the user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from a web browser. The computer may also interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving response messages from the user in return.

[0112] Data processing equipment for implementing machine learning models may include, for example, dedicated hardware accelerator units for processing the common computationally intensive parts of machine learning training or generation, i.e., inference workloads.

[0113] Machine learning models can be implemented and deployed using machine learning frameworks, such as PyTorch, Scikit-learn, Keras, or TensorFlow.

[0114] Embodiments of the subject matter described herein may be implemented in a computing system that includes, for example, a backend component as a data server, or a middleware component, such as an application server, or a frontend component, such as a client computer having a graphical user interface, a web browser, or an application that allows a user to interact with the implementation of the subject matter described herein, or any combination of one or more such backend, middleware, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0115] A computing system may include a client and a server. The client and server are generally geographically separated and typically interact via a communication network. The client-server relationship arises from computer programs running on each computer that have a client-server relationship with each other. In some embodiments, the server sends data, such as an HTML page, to a user device to display data to a user interacting with a device acting as a client and to receive user input from that user. Data generated by the user device, such as the results of user interaction, may be received by the server from the device.

[0116] This disclosure includes many details of specific implementations, but these should not be interpreted as limitations on the scope of any invention or claim, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this disclosure in relation to separate embodiments may also be implemented in combination in a single implementation. Conversely, various features described in relation to a single embodiment may be implemented separately or in any suitable partial combination in multiple embodiments. Furthermore, features are described above as acting in a particular combination, and may even be initially claimed as such; however, one or more features from a claimed combination may be removed from that combination in some cases, and the claimed combination may cover partial combinations or variations of partial combinations.

[0117] Similarly, while the operations are shown in the drawings and described in the claims in a specific order, this should not be understood as requiring that such operations be performed in a specific order or sequence shown, or that all shown operations be performed, in order to achieve the desired result. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged in multiple software products.

[0118] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions described in the claims may be performed in a different order and still achieve the desired results. As an example, the process shown in the accompanying drawings does not necessarily require the specific order or sequence shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A computer implementation method for predicting the thermal stability of an immunoglobulin single variable domain (ISV), To obtain data representing the amino acid sequence of the aforementioned ISV, The process involves generating an input token vector by numerically encoding the aforementioned amino acid sequence, Generating an embedding feature vector by processing the input token vector using an embedding machine learning model having a first set of model parameters, wherein the first set of model parameters is updated using self-supervised learning of a first machine learning model that includes the embedding machine learning model and is configured to perform an array reconstruction task. The present invention relates to processing an input containing the embedding feature vector using a predictive machine learning model to generate an output that predicts a thermal stability measurement of the ISV, wherein the predictive machine learning model has a second set of model parameters that are updated using supervised learning based on a plurality of training examples of a second machine learning model including the predictive machine learning model, each training example comprising (i) a training input specifying a representation of each ISV, and (ii) a label specifying a thermal stability measurement of each ISV. Computer implementation methods including

2. The method according to claim 1, wherein the self-supervised learning comprises training the first machine learning model on a first dataset containing an array representation of an exemplary set of ISVs.

3. The method according to claim 2, wherein the exemplary set of ISVs comprises a plurality of heavy chain single variable domains (VHs).

4. The method according to any one of claims 1 to 3, wherein the thermal stability measurement value is the melting temperature.

5. The method according to any one of claims 1 to 4, wherein the input to the predictive machine learning model further includes data representing the amino acid sequence of the ISV.

6. The method according to any one of claims 1 to 5, wherein the input further includes data characterizing the sequence length of the ISV.

7. The method according to any one of claims 1 to 6, wherein the input further includes data characterizing the three-dimensional (3D) structure of the ISV.

8. The method according to any one of claims 1 to 7, wherein the input processed by the predictive machine learning model further comprises data characterizing the germline of the ISV.

9. The method according to claim 12, wherein the input further comprises data characterizing the mutation of the ISV from a corresponding wild-type ISV which is the source of the mutation of the ISV.

10. The method according to any one of claims 1 to 9, wherein the embedded machine learning model includes a bidirectional transformer.

11. Performing the self-supervised learning on the first machine learning model described above means Initializing the parameter values ​​of the first machine learning model described above, The values ​​of the parameters of the first machine learning model are updated by minimizing the loss function defined for the array reconstruction task. The method according to any one of claims 1 to 10, including the method described in any one of claims 1 to 10.

12. The aforementioned loss function is, [Math 1] Defined as, in the formula, p(x ｉ | x Ｍ ) is the unmasked part x of the input array x Ｍ Given, token x ｉ The method according to claim 11, which represents the probability that the first machine learning model predicts that exists at a specific masked location.

13. A method for selecting an ISV from a set of candidate ISVs, Predicting each of the thermal stability measurements of each of the candidate ISVs using a method according to any one of claims 1 to 12, Based on the predicted thermal stability measurements, select the ISV from the set of candidate ISVs. A method that includes this.

14. One or more computers, When executed by the one or more computers, the one or more computers store one or more storage devices that store instructions causing the one or more computers to perform the operation of each of the methods described in any one of claims 1 to 13. A system that includes this.

15. One or more computer-readable storage media that, when executed by one or more computers, store instructions causing one or more computers to perform the operation of each of the methods described in any one of claims 1 to 13.

Images