Methods of generating candidates for amino acid sequences of epitopes

Abstract

This application relates to a method for generating candidate amino acid sequences of epitopes, the method comprising executing a neural network in one or more processing devices for generating candidate sequences of epitopes, the neural network being the result of fine-tuning a neural network trained for generating amino acid sequences, wherein the training has been performed with data of amino acid sequences including non-epitope amino acid sequences, and the fine-tuning has been performed with data of epitope amino acid sequences.

Description

Technical Field

[0001] This invention is within the field of candidate methods for generating amino acid sequences of epitopes. Background Technology

[0002] The immune system plays a central role in human health, protecting the body from infection. Immune responses involve antigen-antibody interactions, which are non-covalent chemical interactions between antibodies or immunoglobulins (proteins of the immune system) and antigens (molecules that trigger the response). In common cases, antigens are pathogens or internal threats, such as tumors. Chemically, proteins are composed of small chains of molecules called amino acids, linked by covalent bonds to form the primary structure of a protein. The antigenic region that specifically binds to an antibody recognition site is called an epitope. Proteins can contain more than one epitope, and the epitope-antibody relationship is not one-to-one, as a given epitope can bind to more than one antibody, and vice versa.

[0003] Given the role of epitopes in triggering immune responses, predicting antibody-antigen binding and identifying epitopes are key goals in immunology. However, testing candidate sequences can be a daunting task. Limited to sequences with n amino acids, there are... The probability of finding amino acids with approximately six or more is unattainable for any experimental setup, even those equipped with high-throughput screening technologies. Therefore, predicting amino acid sequences with a relatively high probability of being epitopes is desirable. Summary of the Invention

[0004] A first aspect of this disclosure relates to a method for generating candidate amino acid sequences of epitopes, the method comprising: executing a neural network in one or more processing devices for generating candidate sequences of epitopes, the neural network being the result of fine-tuning a neural network trained for generating amino acid sequences, wherein the training has been performed with data of amino acid sequences including non-epitope amino acid sequences, and the fine-tuning has been performed with data of amino acid sequences of epitopes.

[0005] In some implementations, the amino acid sequences used to train the neural network include the amino acid sequences of epitopes. Therefore, the training data may include data on the amino acid sequences of epitopes, although it is not required that the training data include the amino acid sequences of epitopes. It may be advantageous for the training data to include multiple types of amino acid sequences so that the neural network can learn patterns that typically influence amino acid sequences.

[0006] In some implementations, training is unsupervised. For example, the neural network to be trained and the training can be disclosed as in “Ferruz, N., Schmidt, S. & Höcker, B. ProtGPT2 is a deep unsupervised language model for protein design. Nat Commun 13, 4348 (2022). https: / / doi.org / 10.1038 / s41467-022-32007-7”, which is incorporated herein by reference in its entirety.

[0007] Fine-tuning is performed after training. Typically, fine-tuning is a type of training that involves adjusting the weight parameters of a neural network in a smaller way than the adjustments performed during previous training (e.g., previous training of a neural network with pseudo-random initialization).

[0008] In fine-tuning, the weight parameters of the neural network are adjusted to optimize the performance of a specific task performed by the network. The neural network generated from previous training is fine-tuned only using data on the amino acid sequence of epitopes. In particular, the neural network can be fine-tuned to perform tasks that predict subsequent amino acids or portions of an amino acid sequence based on previously predicted amino acids in that sequence (i.e., including based on the position of each of the previously predicted amino acids in the sequence).

[0009] In some implementations, the method is used to generate candidate amino acid sequences for linear epitopes.

[0010] In some implementations, the data for the amino acid sequence of the epitope used for fine-tuning is the amino acid sequence of a linear epitope.

[0011] In some implementations, the entire amino acid sequence of the data used for fine-tuning has fewer than 12 residues. Therefore, the probability that each amino acid sequence of the data used for fine-tuning refers to a linear epitope rather than a conformational epitope increases.

[0012] In some embodiments, the method includes: classifying candidates into candidates with a higher probability of being epitopes and candidates with a lower probability of being epitopes by performing at least one classifier in one or more processing devices, wherein the at least one classifier has been trained with data of amino acid sequences of epitopes and amino acid sequences of non-episodes to classify amino acid sequences of epitopes as amino acid sequences with a higher probability of being epitopes and amino acid sequences of non-episodes as amino acid sequences with a lower probability of being epitopes.

[0013] Because fine-tuning was performed using data with epitope sequences, the neural network was not fine-tuned to detect non-episode amino acid sequences. Therefore, to eliminate candidates with low probabilities of being epitopes, a classifier already trained with data of non-episode amino acid sequences could be used to process the candidates generated by the fine-tuned neural network.

[0014] In some embodiments, the at least one classifier includes a viral epitope classifier and a bacterial epitope classifier, and the classification step includes: classifying the generated candidates in at least one of the viral epitope classifier and the bacterial epitope classifier; wherein the viral epitope classifier is configured to classify the candidates as candidates with a high probability of being viral epitopes and candidates with a low probability of being viral epitopes, and the bacterial epitope classifier is configured to classify the candidates as candidates with a high probability of being bacterial epitopes and candidates with a low probability of being bacterial epitopes.

[0015] Using viral and bacterial epitope classifiers is advantageous because most known epitopes are at least one of bacteria and viruses, making it expected that most candidates generated by the neural network will be at least one of bacteria and viruses. Furthermore, training the classifier with other types of amino acid sequences, which are scarcer than bacterial and viral epitopes, can lead to better training of the classifier due to the larger amount of training data, thus enhancing the classification of amino acid sequences.

[0016] A bacterial epitope classifier is trained using amino acid sequence data of bacterial epitopes and amino acid sequence data of non-bacterial epitopes (i.e., non-bacterial epitopes) to classify amino acid sequences of bacterial epitopes as epitopes and amino acid sequences of non-bacterial epitopes as non-epithels.

[0017] A viral epitope classifier is trained using amino acid sequence data of viral epitopes and amino acid sequence data of non-viral epitopes (i.e., non-viral epitopes) to classify amino acid sequences of viral epitopes as epitopes and amino acid sequences of non-viral epitopes as non-epithels.

[0018] In some embodiments, the bias of at least one classifier is adjusted to increase recall at the expense of at least one of accuracy and precision.

[0019] Increasing recall allows maximizing the number of true positives, such that fewer candidates with a relatively high probability of being epitopes are misclassified as non-epithels by at least one classifier (i.e., fewer false negatives are obtained).

[0020] In some implementations, the bacterial (viral) classifier includes bacterial (viral) subclassifiers. Each bacterial (viral) subclassifier is configured to classify candidates as those with a higher probability of being a bacterial (viral) epitope and those with a lower probability of being a bacterial (viral) epitope. The bacterial (viral) or non-bacterial (non-viral) classification provided by the classifier is based on an aggregation of the classifications provided by the classifier's subclassifiers. Bias can be the weight assigned by the subclassifier to the classification as bacterial (viral) relative to the weight assigned by the subclassifier to the classification as non-bacterial (non-viral). Therefore, by increasing the weight assigned by the subclassifier to the classification as bacterial (viral) relative to the classification as non-bacterial (non-viral) by the subclassifier, fewer false negatives are obtained, thereby increasing recall. For example, the weight assigned to the bacterial classification is 0.5 times higher than the weight assigned to the non-bacterial classification, for example, more than 0.6 times. For example, the weight assigned to the viral classification is five times higher than the weight assigned to the non-viral classification, for example, more than six times.

[0021] In some embodiments, the subclassifier is an XGBoost classifier.

[0022] The recall rate of a classifier can be calculated using the following equation:

[0023]

[0024] In some embodiments, the recall of each classifier in at least one classifier is at least 60%, for example, at least 70%.

[0025] The accuracy of a classifier can be calculated using the following equation:

[0026]

[0027] In some embodiments, the accuracy of each classifier in at least one classifier is up to 70%, for example up to 60%, for example up to greater than 50%.

[0028] In some embodiments, the accuracy of each classifier in at least one classifier is at least 30%, for example, at least 40%.

[0029] The accuracy of a classifier can be calculated using the following equation:

[0030]

[0031] In some embodiments, the accuracy of each classifier in at least one classifier is at most 70%, for example at most 60%, for example at most greater than 50%.

[0032] In some embodiments, the accuracy of each classifier in at least one classifier is at least 10%, for example, at least 20%.

[0033] In some implementations, the method includes synthesizing candidate epitopes having sequences that are equal to the candidate sequences.

[0034] In some implementations, the step of synthesizing epitope candidates includes synthesizing epitope candidates having the same sequence as a candidate sequence that is classified by at least one classifier as having a higher probability of being an epitope, and not synthesizing epitope candidates having the same sequence as a candidate sequence that is classified by at least one classifier as having a lower probability of being an epitope.

[0035] In some implementations, synthesized epitope candidates are tested with antibodies to determine whether a candidate is an epitope based on epitope-antibody recognition; preferably, the determination is based on an epitope-antibody recognition readout (e.g., fluorescence readout). Thus, each candidate is tested to determine whether it is an epitope.

[0036] In some embodiments, the neural network is autoregressive. For example, the neural network is a generative pre-trained transformer, such as generative pre-trained transformer 2 (i.e., GPT-2).

[0037] In some embodiments, the hyperparameters of the fine-tuned neural network include temperature and a repetition penalty coefficient.

[0038] In some implementations, the neural network includes cascaded transformer blocks, with an initial transformer block configured to receive an embedding of an amino acid sequence, and each transformer block including an attention module and a feedforward module.

[0039] In some embodiments, the attention module is a multi-head attention module.

[0040] In some embodiments, the embedding vector is a combination of a token embedding vector and a positional embedding vector.

[0041] In some embodiments, each transformer block includes a first-layer normalization module and a second-layer normalization module, a first-drop module and a second-drop module, and a first-fast-connect module and a second-fast-connect module; wherein, the output of the first-layer normalization module is connected to the input of the attention module, the output of the attention module is connected to the input of the first-drop module, the first-fast-connect module combines the input of the first-layer normalization module with the output of the first-drop module, and the input of the second-layer normalization module is configured to receive the combination; the output of the second-layer normalization module is connected to the input of the feedforward module, the output of the feedforward module is connected to the input of the second-drop module, and the second-fast-connect module combines the input of the second-layer normalization module with the output of the second-drop module.

[0042] In some embodiments, the feedforward module includes a first linear layer, an activation layer, and a second linear layer, wherein the output of the first linear layer is connected to the input of the activation layer, and the output of the activation layer is connected to the input of the second linear layer; wherein the activation layer is preferably a Gaussian error linear unit.

[0043] In some embodiments, fine-tuning is performed by minimizing the following cost function:

[0044] ,

[0045] in:

[0046] - Given the values ​​of previous terms with the same amino acid sequence, the neural network adjusts the values ​​of the actual terms accordingly. The probability of obtaining it;

[0047] - It is the actual word value at the i-th position of the k-th sequence in the amino acid sequence data of the epitope;

[0048] - D is the total number of sequences in the data of the amino acid sequence of the epitope.

[0049] The second aspect of this disclosure relates to a computer program including instructions that, when executed by a computer, cause the computer to perform the method of the first aspect of this disclosure.

[0050] A third aspect of this disclosure relates to one or more processing devices configured to perform the method of the first aspect of this disclosure.

[0051] Advantages similar to those described with respect to the first aspect of this disclosure may also apply to the second and third aspects of this disclosure. Attached Figure Description

[0052] To complete the description and provide a better understanding of this disclosure, a set of accompanying drawings is provided. These drawings form an integral part of the description and illustrate embodiments of the present disclosure, which should not be construed as limiting the scope of the disclosure, but merely as examples of how the disclosure may be practiced. The drawings include the following figures:

[0053] Figure 1 A method for training and fine-tuning a neural network according to an embodiment is shown.

[0054] Figure 2 A converter block according to an embodiment is shown.

[0055] Figure 3 A feedforward module according to an embodiment is shown.

[0056] Figure 4 A method according to an embodiment is shown.

[0057] Figure 5 The distribution of sequence lengths according to the implementation scheme is shown.

[0058] Figure 6 The relative entropy of a nine-residue sequence according to the implementation scheme is shown.

[0059] Figure 7 The global preference for a nine-residue sequence according to the implementation scheme is shown.

[0060] Figure 8 The mutual information between amino acid distribution pairs at two residue positions according to the embodiment is shown. Detailed Implementation

[0061] The following description is not restrictive, but is given only for the purpose of describing the broad principles of the invention. Embodiments of the invention will be described by way of example with reference to the accompanying drawings.

[0062] Figure 1 A method 100 for training and fine-tuning neural networks is shown. Figure 4 Method 300 is illustrated, which involves generating candidate epitope sequences using a neural network generated by method 100. Methods 100 and 300 are executed in one or more processing devices. One or more processing devices 101 are, for example, at least one of the following: at least one CPU, at least one GPU, at least one FPGA, at least one ASIC, at least one personal computer, and at least one laptop computer.

[0063] like Figure 1 As shown, method 100 includes training a neural network 102 using training data 103 in one or more processing devices 101.

[0064] Training data 103 includes non-epitope amino acid sequences. Examples of training data are amino acid sequences included in Uniref 50 version 2021_04. For instance, training data could contain approximately 50 million amino acid sequences.

[0065] Each amino acid sequence contains characters, where each character corresponds to one amino acid.

[0066] In order to use training data 103 in the training of neural network 102, training data 103 is converted into a numerical vector commonly referred to as an embedding vector.

[0067] To obtain the lexical embedding vectors, the training data 103 was lexicalized. Each lexical embedding vector is a numerical vector representing a different character or character sequence (i.e., a portion of an amino acid sequence) of an amino acid sequence. For example, lexical embedding vectors are obtained through byte-pair encoding (BPE) lexicalization or through one-hot lexicalization.

[0068] To further enhance the training of neural networks, positional embedding vectors can be used in addition to word embedding vectors. A positional embedding vector is a numerical vector representing the position of a portion of an amino acid sequence (i.e., a character or character sequence) within the amino acid sequence. Positional embedding vectors can be independent of the specific amino acid sequence at the position they represent. Positional embedding vectors can represent absolute positions within the amino acid sequence (i.e., the first position, the second position, the third position, etc.) or relative positions within the amino acid sequence.

[0069] One or more processing devices 101 process the training data 103 to transform each amino acid sequence of the training data 103 in the embedding vector. The embedding vector is, for example, a lexical embedding vector or an embedding vector generated by combining (e.g., summing) lexical embedding vectors and positional embedding vectors. The embedding vector is stored, for example, in a non-transitory data memory for subsequent use in the training of the neural network 102.

[0070] For example, the number of lexical embedding vectors is approximately 50,000. Each lexical embedding vector can represent the average of four amino acids.

[0071] The neural network 102 is, for example, a transformer, such as a generative pre-trained transformer (GPT), such as generative pre-trained transformer 2 (GPT-2). The neural network 102 includes, for example, cascaded transformer blocks. Figure 2 An example of converter block 200 is shown in the figure.

[0072] A tensor, including the embedding vector of the training data or the output of the previous transformer block 200, is input to the first-layer normalization module 201. In the first-layer normalization module 201, one or more processing devices adjust the values ​​of the components of the input tensor to have a mean of zero and a variance of 1. The first-layer normalization module allows for enhanced training of the neural network (e.g., helps minimize vanishing and / or exploding gradient problems). Clearly, if transformer block 200 is the initial transformer block of the neural network (i.e., the transformer block to which the embedding vector of the training data is input) and the embedding vector of the training data has been previously normalized, then the transformer block is not required to include the first-layer normalization module 201.

[0073] The output of the first-layer normalization module 201 is input to the attention module 202 (e.g., a multi-head attention module). In the attention module 202, one or more processing devices 101 process the input to combine the latest embedding vector (i.e., the embedding vector immediately preceding the embedding vector to be predicted by the neural network) with previous embedding vectors of the same amino acid sequence. Thus, the neural network's prediction depends not only on the value of the latest embedding vector, but also, in its context, particularly on the values ​​of previous embedding vectors of the same amino acid sequence.

[0074] The first dropout module 203 and the second dropout module 207 are used to regularize the model and prevent / minimize overfitting. The first dropout module 203 and the second dropout module 207 are used during the training of the neural network and optionally during the fine-tuning of the neural network; typically, they are not used once the neural network has been trained and / or fine-tuned. In the first dropout module 203 and the second dropout module 207, the outputs of some neurons in the neural network are ignored. Specifically, in the first dropout module 203, the outputs of some neurons in the attention module are ignored (e.g., one or more processing devices 101 set the weight parameters of the ignored neurons in the attention module to zero). In the second dropout module 207, the outputs of some neurons in the feedforward module are ignored (e.g., one or more processing devices 101 set the weight parameters of the ignored neurons in the feedforward module to zero).

[0075] In the first shortcut connection module 204, one or more processing devices 101 add the output of the first discard module 203 to the input of the first layer normalization module 201. The shortcut connection module allows minimizing the vanishing gradient descent problem that occurs in deep neural networks.

[0076] Similar to the first-level normalization module 201, in the second-level normalization module 205, one or more processing devices 101 process the input tensor to adjust the tensor values ​​to have a mean of zero and a variance of 1.

[0077] In feedforward module 206, one or more processing devices 101 process the output of second-layer normalization module 205 by applying linear layers and activation functions. An example of a feedforward module is shown below. Figure 3 As shown. The feedforward module 206 includes a first linear layer 2061, an activation function 2062, and a second linear layer 2063. In each linear layer, one or more processing devices 101 perform the following operations:

[0078] ,

[0079] in:

[0080] Y is a tensor output by a linear layer;

[0081] X is the tensor input to the linear layer;

[0082] W is a tensor with linear layer weights as components; and

[0083] B is the bias tensor.

[0084] The tensor output by the first linear layer 2061 can have more components (e.g., higher dimensions) than the tensor input to the first linear layer 2061.

[0085] Activation functions are typically nonlinear functions, such as Gaussian Error Linear Units (GELUs), Rectified Linear Units (ReLUs), or Swiggluents. In an activation function, one or more processing devices 101 can process the components by applying activation function 2062 to each component of the input tensor. Therefore, the tensor output by activation function 2062 has the same dimension as the tensor input to activation function 2062.

[0086] The tensor output by the second linear layer 2063 can have fewer components (e.g., lower dimension) than the tensor input to the second linear layer 2063.

[0087] In the second quick-connect module 208, one or more processing devices 101 add the output of the second discard module to the input of the first quick-connect module 204 to generate the output of the converter block 200.

[0088] The output of transformer block 200 is the input of subsequent transformer blocks, which can have the same architecture as transformer block 200, until the last transformer block of the neural network is reached. The input and output of transformer block 200 can have the same dimension.

[0089] One or more processing devices 101 are configured to select a predicted character or character sequence based on the values ​​of the components of a tensor output by the last transformer block of the neural network. These output tensor components are commonly referred to as logit in machine learning. The logit is a raw score generated by the last transformer block. There exists a logit associated with each lexical embedding vector (i.e., associated with the character or character sequence represented by the lexical embedding vector), and the value of each logit gives the probability that the next predicted character / character sequence is the character / character sequence represented by the lexical embedding vector associated with the corresponding logit. Generally, a higher logit value indicates a higher probability.

[0090] During training, since the next actual embedding vector of the amino acid sequence is known, a cost function is computed by comparing the predicted embedding vector with the actual embedding vector by one or more processing devices 101, and the values ​​of the weight parameters of the neural network are adjusted by one or more processing devices 101 to optimize the result of the cost function (e.g., by applying gradient descent). The cost function is, for example, as follows:

[0091] ,

[0092] in:

[0093] - This refers to the neural network optimizing the actual word value given the values ​​of previous words with the same amino acid sequence. The probability of obtaining it;

[0094] - It is the actual word value at the i-th position of the k-th sequence in the amino acid sequence data of the epitope;

[0095] - D is the total number of sequences in the amino acid sequence data of the epitope.

[0096] Therefore, by optimizing the aforementioned cost function, the probability of the predicted character / character sequence matching the actual character / character sequence is increased. In this way, neural network 102 is trained using training data to obtain neural network 104 trained to generate amino acid sequences.

[0097] The values ​​of the weight parameters to be optimized can be the weight parameter values ​​of the feedforward network module and the weight parameter values ​​of the attention module.

[0098] The neural network 104 is then fine-tuned to generate candidate amino acid sequences for epitopes. Fine-tuning can be performed using the same processing device 101 used in training or a different processing device. The run of the fine-tuned neural network to generate candidates can be performed using the same processing device 101 used in training and / or fine-tuning or by a different processing device.

[0099] To fine-tune neural network 104, fine-tuning data 105 of the amino acid sequence of epitopes is used. Specifically, the fine-tuning data can include approximately 500,000 different amino acid sequences of epitopes, such as those retrieved from the immune epitope database (IEDB) (see, for example, Randi Vita et al., “The immune epitope database (IEDB) 3.0”. In: Nucleic Acids Research 43 (2014), pp. D405–D412. url: https: / / api.semanticscholar.org / CorpusID:7931138; and see, for example, Randi Vita et al., “The Immune Epitope Database (IEDB): 2018 update”. In: Nucleic Acids Research 47 (2018), pp. D339–D343. url: https: / / api.semanticscholar.org / CorpusID:53026444). Each amino acid sequence in the fine-tuning data can have fewer than 12 residues. Restricting the fine-tuning data to amino acid sequences with fewer than 12 residues is advantageous because it increases the chance that the epitopes in the fine-tuning data are linear epitopes (and thus reduces the chance that the epitopes in the fine-tuning data are conformational epitopes). The method disclosed herein has been found to be particularly advantageous for generating candidate amino acid sequences for linear epitopes.

[0100] To check whether fine-tuning data is suitable for fine-tuning, several characteristics of the fine-tuning data can be evaluated. For example, it can be determined that the fine-tuning data is experimentally validated data known to contain the basic facts. Additionally, it can be determined that the fine-tuning data does not contain unwanted biases. Furthermore, it can be checked that the sequence length has an appropriate range of values ​​and that no spurious correlations between residues are found, i.e., the mutual information between two residues is much smaller than the entropy of a single residue.

[0101] In fine-tuning, the values ​​of the neural network's weight parameters (e.g., the weight parameters of the feedforward module and the attention module) are optimized to predict the next lexical embedding vector. More specifically, as described above, the neural network generates a value for each lexical embedding vector that indicates the probability that the character / character sequence represented by the corresponding lexical embedding vector is the next prediction.

[0102] Fine-tuning can be performed using the same cost function used in training and by using fine-tuning data 105 instead of training data 103.

[0103] As a result of fine-tuning, a neural network 106 is obtained, which is fine-tuned to generate candidate amino acid sequences for epitopes. In each execution of the neural network, the network predicts subsequent lexical embedding vectors of the amino acid sequence based on previous embedding vectors predicted by the network. Thus, by iteratively executing the neural network, complete amino acid sequences of epitope candidates are obtained.

[0104] When executing neural network 106 to obtain epitope candidates, the dropout module of the neural network is usually not executed, because the dropout module can be used only to minimize overfitting during training and optionally for overfitting during fine-tuning.

[0105] The parameters of the repetition penalty coefficient and temperature parameter of the neural network can be adjusted to increase the number of different candidates generated by the neural network.

[0106] The repetition penalty coefficient is a parameter that reduces the probability of repeating characters represented by the same lexical unit in the same table position candidate by their embedding vectors. The repetition probability is reduced by increasing the repetition penalty coefficient. For example, the repetition penalty coefficient can be implemented in cases where character repetition exists using the following formula:

[0107] ,

[0108] in:

[0109] X i It is a value indicating the probability of the i-th word embedding vector output by the neural network (e.g., X). i (Logarithmic probability);

[0110] RP is the repetition penalty coefficient;

[0111] n i It is the number of previous occurrences of the embedding vector of the i-th term in the amino acid sequence of the embedding vector; and

[0112] Y i It is an updated value indicating the probability of the embedding vector of the i-th word.

[0113] In some implementation schemes, n i This refers to the number of times the embedding vector replaces or substitutes for the previous occurrence of the i-th word in the amino acid sequence previously generated by the neural network, except in the same amino acid sequence.

[0114] It has been found that a repetition penalty coefficient higher than 1.2 (e.g., at least 2 and at most 3) has lower perplexity than a lower repetition penalty coefficient.

[0115] In some embodiments, the repetition penalty coefficient is not used for training and / or fine-tuning.

[0116] In some implementations, a repetition penalty coefficient is used to generate candidate amino acid sequences for epitopes. Therefore, the repetition penalty coefficient reduces the probability of identical terms being repeated (e.g., repeated within the same amino acid sequence).

[0117] The repetition penalty coefficient adjusts the probability of selecting the next character / character sequence based on the number of previous occurrences of the corresponding character / character sequence. This is accomplished, for example, by adjusting the log odds based on the number of previous occurrences of the corresponding character / character sequence.

[0118] One or more processing devices 101 may apply a function (e.g., a softmax function) to a log-odds ratio to obtain the probability of a predicted portion of the amino acid sequence (i.e., a predicted character / character sequence). The probability of each lexical embedding vector is given by the relationship between the value of the neural network output associated with the lexical embedding vector (e.g., the value of the log-odds ratio associated with the lexical embedding vector) and the values ​​of the remaining lexical embedding vectors associated with the possible predicted portions of the amino acid sequence, as output by the neural network. One or more processing devices 101 select a prediction based on the probability of the lexical embedding vectors.

[0119] For example, one or more processing devices 101 may be configured to assign a selection probability other than zero to all token embedding vectors whose calculated probability based on the value output by the last transformer block is not zero. Alternatively, the selection probability other than zero may be assigned only to a determined number of k token embedding vectors that have the highest probability based on the value output in the last transformer block (e.g., to the k log odds with the highest value), rather than to all token embedding vectors.

[0120] The relationship between selection probability and calculated probability can be tuned using a parameter known as temperature in machine learning. Generally, a lower temperature value indicates a more uniform distribution of selection probabilities. Temperature can be applied to the values ​​of all components V that indicate the probability that a word embedding vector is a prediction, in the following way:

[0121] ,

[0122] in:

[0123] V: The value of the tensor component output by the last transformer block, indicating the probability that a particular token embedding vector is a prediction (e.g., V is the log odds); and

[0124] T: Temperature.

[0125] In some implementations, the temperature value is 1. Temperatures other than 1 have been shown to produce similar results to temperature 1 in terms of the probability that a candidate is an actual epitope. Therefore, choosing a temperature value of 1 is advantageous for reducing processing load without significantly affecting the quality of the obtained candidates.

[0126] Candidate amino acid sequences for 301 epitopes are generated by executing a neural network 106 in one or more processing devices 101. Of the 200,000 generated candidates, 192,222 candidates have been identified as distinct and were not included in the training data.

[0127] The statistics for candidate epitopes generated by the neural network are consistent with those for known epitopes (especially in the case of linear epitopes). In particular, the fine-tuned neural network is able to capture features present in the fine-tuned data, which are qualitatively interpreted as characterizing the epitopes (e.g., large relative entropy, small Shannon entropy at the last residue position, and low cysteine ​​propensity). Furthermore, no correlations were found between residue position pairs for a given sequence length, provided the number of sequences generated for that length is sufficiently large. This is a positive sign that the model has not introduced spurious correlations between amino acids. Another positive sign is that the cost function evaluated on the test set converges after several epochs.

[0128] Some statistical details of the generated candidates are in Figures 5 to 8 It is shown in the figure and explained below.

[0129] Figure 5 The distribution of sequence lengths is shown.

[0130] Figure 6 The relative entropy of a sequence with 9 residues is shown. The relative entropy is the relationship between the observed probability distribution of amino acids at each residue position and the probability distribution of the human proteome. For two discrete probability distributions P and Q defined on the sample space X, the relative entropy is defined as:

[0131] ,

[0132] This can be understood as a measure of how different the two distributions are.

[0133] Figure 7 The global preference for sequences with nine residues is shown, where the global preference is the amino acid at each residue position. Estimated probabilities are found in the human proteome. and reference probability The global preference of amino acid X at position y in a sample is defined by the following ratio:

[0134] .

[0135] Figure 8 This demonstrates the mutual information between multiple pairs of amino acid distributions at two residue positions. For two discrete probability distributions P and Q defined on the sample space X, the relative entropy is defined as:

[0136] ,

[0137] in, This is the Shannon entropy of the corresponding joint probability distribution. Note that if... ,but For a discrete probability P in the sample space X, the Shannon entropy of the amino acid distribution at each residue position can be defined as follows:

[0138] ,

[0139] This can be understood as an inverse measure of information or differences that have a random, uniform distribution without information.

[0140] The amino acid sequences of the generated epitopes are input into the viral epitope classifier and the bacterial epitope classifier.

[0141] A viral epitope classifier has been trained to classify amino acid sequences as viral or non-viral epitopes. A bacterial epitope classifier has been trained to classify amino acid sequences as bacterial or non-bacterial epitopes. The classifiers have been trained in one or more processing devices 101 using positive data (i.e., the amino acid sequence of a viral epitope in the case of the viral epitope classifier; and the amino acid sequence of a bacterial epitope classifier) ​​and negative data (i.e., the amino acid sequence of a non-viral epitope in the case of the viral epitope classifier; and the amino acid sequence of a non-bacterial epitope in the case of the bacterial epitope classifier).

[0142] The classifier can be an XGBoost classifier. Training the classifier can be performed by adjusting its weight parameters to optimize the cost function, for example, by increasing the number of amino acid sequences in the training data that are correctly classified by the classifier.

[0143] A classifier can classify each generated candidate based on the values ​​of all components of a defined vector of generated candidates. It has been shown that classifying different parts of a vector with each subclassifier of the corresponding classifier provides similar results to classifying the entire vector within each subclassifier. For example, if a candidate is represented by a vector with 1024 components, a classifier comprising eleven subclassifiers can be configured such that components 1 to 100 of the vector (i.e., the first 100 components of the vector) are input to the first subclassifier of the eleven subclassifiers, components 101 to 200 are input to the second subclassifier of the eleven subclassifiers, components 201 to 300 are input to the third subclassifier of the eleven subclassifiers, ... and components 1001 to 1024 are input to the eleventh subclassifier of the eleven subclassifiers. Therefore, to reduce processing load, each different part of the candidate can be classified by each subclassifier as having a higher probability of belonging to an epitope (e.g., viral epitopes and / or bacterial epitopes) or a lower probability of belonging to an epitope (e.g., viral epitopes and / or bacterial epitopes). The classification provided by each classifier is based on the aggregation of classifications provided by the corresponding classifier's subclassifiers.

[0144] For the same reason, classifier training can be performed using different portions of each amino acid sequence of the training data. Specifically, each sub-classifier may have been trained to process a portion of the candidate vector with a defined position. For example, a first sub-classifier can be trained to classify the portion containing components 1 to 100 of the candidate vector, a second sub-classifier can be trained to classify the portion containing components 101 to 200 of the candidate vector, and so on, and an eleventh sub-classifier can be trained to classify the portion containing components 1001 to 1024 of the candidate vector. Examples of statistical parameters for the classifiers are shown in Table 1 below:

[0145] Table 1

[0146]

[0147] Only candidates that are classified as viral epitopes by the viral epitope classifier 302 and as bacterial epitopes by the bacterial epitope classifier 302 proceed to the next stage of method 300. Candidates that are neither classified as viral nor bacterial epitopes are excluded.

[0148] Remove candidate 303 that is known to be an epitope but is moving to the next stage. Synthesize and test the remaining candidates for 304 to determine if they are epitopes.

[0149] Synthesized candidates can be screened using a system built on an inverted microscope, which has an optical setup capable of providing real-time visualization of experiments. Additionally, one or more processing devices can be configured to operate a droplet detector. Upon detection of a droplet by the droplet detector, an actuation signal is sent by one or more processing devices to a deflector arranged in the droplet's path to deflect the droplet to the desired channel for subsequent collection and sequencing. To achieve fluorescent readout for epitope-antibody recognition, the synthesized epitope can be fused to a fluorescent protein (e.g., to green fluorescent protein (GFP)). The excitation (488 nm) and emission (523 nm) peaks of GFP make it suitable for detection using a blue laser (473 nm) and a photomultiplier tube (PMT) with a bandpass filter in the range of 525 ± 25 nm. The microfluidic platform is based on the difference in fluorescence anisotropy within the system. Fluorescence polarization is the phenomenon that light emitted by a fluorophore has different intensities along its polarization axis. If the fluorophore binds to a molecule that is relatively large relative to its size, the rotational diffusion coefficient changes significantly, leading to variations in anisotropy. In this case, the antibody is significantly larger than GFP, thus anisotropy in the fluorescence of antibody-bound GFP was observed compared to the unbound state.

[0150] In this document, the term “including” and its derivatives (such as “contains”, etc.) should not be understood as having an exclusionary meaning; that is, these terms should not be interpreted as excluding the possibility that the content described and defined may include other elements, steps, etc.

[0151] On the other hand, this disclosure is obviously not limited to the specific embodiments described herein, but also covers any variations that can be considered by those skilled in the art within the general scope of the invention as defined by the claims (e.g., regarding the selection of materials, dimensions, components, configurations, etc.).

Claims

1. A method for generating candidate amino acid sequences of epitopes, the method comprising the steps of: A neural network is executed in one or more processing devices to generate candidate sequences of epitopes, the neural network being the result of fine-tuning a neural network trained to generate amino acid sequences, wherein the training has been performed with data of amino acid sequences including non-epitaxy amino acid sequences, and the fine-tuning has been performed with data of amino acid sequences of epitopes.

2. The method according to claim 1, wherein the method comprises the following steps: The candidates are classified into candidates with a higher probability of being epitopes and candidates with a lower probability of being epitopes by executing at least one classifier in one or more processing devices. The at least one classifier has been trained with data of amino acid sequences of epitopes and data of amino acid sequences of non-episodes to classify amino acid sequences of epitopes as amino acid sequences with a higher probability of being epitopes and amino acid sequences of non-episodes as amino acid sequences with a lower probability of being epitopes.

3. The method according to claim 2, wherein, The at least one classifier includes a viral epitope classifier and a bacterial epitope classifier, and the classification step includes: classifying the generated candidates in at least one of the viral epitope classifier and the bacterial epitope classifier; wherein the viral epitope classifier is configured to classify the candidates as candidates with a high probability of being viral epitopes and candidates with a low probability of being viral epitopes, and the bacterial epitope classifier is configured to classify the candidates as candidates with a high probability of being bacterial epitopes and candidates with a low probability of being bacterial epitopes.

4. The method according to any one of claims 2 and 3, wherein, The bias of the at least one classifier is adjusted to increase recall at the expense of reducing at least one of accuracy and precision.

5. The method according to any one of the preceding claims, the method comprising: Synthesize epitope candidates with sequences that are equal to the candidate sequences.

6. The method according to claim 5 when subordinate to claim 2, 3 or 4, wherein, The step of synthesizing epitope candidates includes: synthesizing epitope candidates having sequences that are equal to candidate sequences classified by the at least one classifier as having a high probability of being epitopes, and not synthesizing epitope candidates having sequences that are equal to candidate sequences classified by the at least one classifier as having a low probability of being epitopes.

7. The method according to any one of the preceding claims, wherein, The neural network is autoregressive.

8. The method according to any one of the preceding claims, wherein, The hyperparameters of the fine-tuned neural network include temperature and repetition penalty coefficient.

9. The method according to any one of the preceding claims, wherein, The neural network includes cascaded transformer blocks, with an initial transformer block configured to receive an embedding vector of an amino acid sequence, and each transformer block includes an attention module and a feedforward module.

10. The method according to claim 9, wherein, Each transformer block includes a first-layer normalization module and a second-layer normalization module, a first-drop module and a second-drop module, as well as a first-fast-connect module and a second-fast-connect module; wherein, the output of the first-layer normalization module is connected to the input of the attention module, the output of the attention module is connected to the input of the first-drop module, the first-fast-connect module combines the input of the first-layer normalization module with the output of the first-drop module, and the input of the second-layer normalization module is configured to receive the combination; the output of the second-layer normalization module is connected to the input of the feedforward module, the output of the feedforward module is connected to the input of the second-drop module, and the second-fast-connect module combines the input of the second-layer normalization module with the output of the second-drop module.

11. The method according to claim 10, wherein, The feedforward module includes a first linear layer, an activation layer, and a second linear layer, wherein the output of the first linear layer is connected to the input of the activation layer, and the output of the activation layer is connected to the input of the second linear layer; wherein the activation layer is preferably a Gaussian error linear unit.

12. The method according to any one of the preceding claims, wherein, The fine-tuning is performed by minimizing the following cost function: ， in: - Given the values ​​of previous terms with the same amino acid sequence, the neural network determines the values ​​of the actual terms based on the given values. The probability of obtaining it; - It is the actual token value at the i-th position of the k-th sequence in the amino acid sequence data of the epitope; and - It is the total number of sequences in the amino acid sequence data of the epitope.

13. The method according to claim 5 or 6 or according to any one of claims 7 to 12 dependent on claim 5 or 6, wherein, The synthesized epitope candidate is tested with an antibody to determine whether the candidate is an epitope based on epitope-antibody recognition; preferably, the determination is based on fluorescence readout of epitope-antibody recognition.

14. A computer program comprising instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 12.

15. One or more processing devices configured to perform the method according to any one of claims 1 to 12.

Images