A method for predicting bitter peptides based on peptide language model

Abstract

The present invention discloses a bitter peptide prediction method based on a peptide language model, which comprises the following steps: 1) selecting multiple protein sequences as original corpus; dividing the selected protein sequences into two subsets, performing enzyme digestion on each protein sequence in the first subset to obtain multiple peptide sequences; randomly segmenting each protein sequence in the second subset to obtain multiple amino acid sequences; and then generating peptide sequences represented by tokens from the obtained sequences; 2) training an ALBERT model using the peptide sequences represented by tokens to obtain a peptide language model; constructing a fine-tuning layer using Text‑CNN and training the model; 3) inputting the token representation of a target peptide into the peptide language model to obtain a vectorized representation of the target peptide; and 4) convolving the vectorized representation of the target peptide using the trained fine-tuning layer, concatenating the convolution results, and outputting the probability that the target peptide is a bitter peptide through a Softmax layer.

Description

Technical Field

[0001] The present invention relates to the field of computational recognition of bitter peptide prediction, and in particular to a bitter peptide recognition method based on a pre-trained peptide language model. Background Art

[0002] Bitter peptides are a class of small peptides produced by protein hydrolysis and possess a bitter taste. Research has shown that bitter peptides possess a variety of biological activities, including clinical studies demonstrating their potential applications in improving medication adherence, regulating blood sugar levels, and treating gastrointestinal ulcers. Furthermore, research on bitter peptides can provide a deeper understanding of the biological effects of bitter substances and the mechanisms of taste perception.

[0003] The identification of bitter peptides using wet lab methods, including gel separation, multiple rounds of liquid chromatography separation, and Fourier transform infrared spectroscopy, is complex, time-consuming, and costly. Therefore, using computer methods to predict and reveal the structure and function of bitter peptides is of great significance.

[0004] A pre-trained language model is a deep learning model trained on a large-scale corpus. By using a large-scale corpus for self-supervised learning, the pre-trained language model can learn some of the characteristics and regularities of the language and transfer this knowledge to downstream tasks. There are certain similarities between protein sequences and natural language, so it is possible to try to use methods from the field of natural language processing (NLP) to address problems in proteomics. For example, Zhang et al. trained an antimicrobial peptide identifier based on BERT; Phasit et al. combined BERT with a bidirectional long-short term memory (Bi-LSTM) to build a bitter peptide recognition network; Jiang et al. also developed a fresh peptide identifier based on BERT. In all of the above methods, the BERT (Bidirectional Encoder Representations from Transformers) model proposed by Google in 2018 was used as the pre-trained language model.

[0005] Current peptide discriminators based on pre-trained language models have certain shortcomings. For example, BERT is not the optimal choice for pre-trained language models due to its high number of parameters, large memory usage, high cost, and difficulty in model distribution and use. Furthermore, with technological advancements, numerous new BERT-based models have emerged with superior encoding capabilities. Regarding pre-trained model training strategies, current methods use protein sequences for pre-training while using peptides as prediction targets, resulting in discrepancies between the target task and the pre-trained task. Furthermore, protein data contains a large number of long protein sequences. The BERT model in these methods has a maximum length of 512, which may cause truncation during training. The truncated protein sequence may not exist, leading to biased model learning. Furthermore, regarding vocabulary selection, the method by Zhang et al., for example, uses a simple combination of amino acid letters, which may not conform to the natural laws of protein language. Regarding fine-tuning strategies, some methods simply use fully connected layers, ignoring the inherent connections that may exist in the encoding output of the pre-trained model. Summary of the Invention

[0006] In response to the above-mentioned problems in the prior art, the present invention provides a bitter peptide prediction method based on a peptide language model, which mainly includes: using the ALBERT model launched by Google in 2019 as a pre-trained language model to reduce model parameters and video memory costs, and lower the threshold for model distribution and use; improving the strategy of the pre-training stage to form a peptide-level pre-training language model, reducing the sample gap between the pre-training stage and the fine-tuning stage, avoiding unreasonable truncation at the protein level, using SentencePiece for word segmentation, and constructing an amino acid dictionary through this frequency-based statistical word segmentation method; using Text-CNN to construct a fine-tuning layer, and using Text-CNN to extract information at different levels in the ALBERT encoding to improve the accuracy of the model's prediction. The present invention first decomposes the pre-training corpus composed of protein sequences into peptide sequences, and uses SentencePiece for word segmentation to construct an amino acid dictionary based on statistical methods; then, the ALBERT model is used as the pre-trained language model to complete fine-tuning; finally, the Text-CNN layer is used to extract local mutual information, and bitter peptide prediction is completed through Softmax.

[0007] The method designed by the present invention mainly comprises the following steps:

[0008] Step 1: Data Preprocessing. First, for a single protein sequence, a feasible enzymatic digestion method is randomly selected with a 70% probability to generate multiple peptide sequences. This is then randomly segmented with a 30% probability to generate multiple amino acid sequences. Subsequently, SentencePiece is used for unigram automatic segmentation, and the amino acid terms are converted to token representations. During the fine-tuning and inference phases, no enzymatic digestion or segmentation is required; only the amino acid terms are converted to token representations to generate peptide sequences represented by tokens.

[0009] Step 2: Pre-training the peptide language model based on ALBERT. In the existing algorithms, protein sequences are used as pre-training corpus. The present invention uses the theoretical enzyme cutting tool and the SentencePiece word segmentation tool to decompose the protein sequence into peptides and automatically construct a statistical amino acid dictionary to replace amino acids with corresponding tokens (such as the amino acid sequence [MDA] is replaced by

[12] after the dictionary. It is mainly used in step 1 data pre-processing). The ALBERT model is introduced into the field of proteomics for the first time, and the first peptide-level language model based on ALBERT is proposed. In the pre-training stage, position information and sequence information are first embedded, and then sentence order prediction (sentence-order prediction) and masked vocabulary prediction (Masked Language Model) tasks are used for pre-training, which enables the model to perform self-supervised learning through a large amount of unlabeled data, mine mutual information between amino acids, and improve the high-dimensional encoding ability of the pre-trained language model.

[0010] Step 3: Process the target peptide using a pre-trained peptide language model to obtain a vectorized representation. This step still embeds positional information in the embedding layer, but no longer embeds sequence information. The pre-trained peptide language model then converts the peptide sequence, originally represented by tokens, into a high-dimensional vector representation.

[0011] Step 4: Use Text-CNN to build a fine-tuning layer, use convolution kernels of size 1, 3, and 5 to convolve the representation obtained in step 3, and concatenate the convolution results. Finally, the Softmax layer outputs the probability that the target sequence is bitter peptide.

[0012] The technical solution of the present invention is:

[0013] A method for predicting bitter peptides based on a peptide language model, comprising the following steps:

[0014] 1) Select multiple protein sequences as the original corpus; divide the selected protein sequences into two subsets. For each protein sequence in the first subset, randomly select a feasible enzymatic digestion method to obtain multiple peptide sequences; randomly split each protein sequence in the second subset to obtain multiple amino acid sequences; then perform word segmentation on the obtained peptide sequences and amino acid sequences, and replace the amino acid words obtained by word segmentation with tokens to obtain peptide sequences represented by tokens;

[0015] 2) Treating the peptide sequence represented by each token as a sentence and the amino acid combination obtained from the word segmentation as a vocabulary, the ALBERT model is trained; the trained ALBERT model is used as the peptide language model; using Text-CNN to construct a fine-tuning layer, the bitter peptide sample is segmented and the amino acid vocabulary obtained from the word segmentation is replaced with a token representation, the bitter peptide sample represented by the token is input into the peptide language model, and the vectorized representation of the bitter peptide sample is obtained to train the fine-tuning layer;

[0016] 3) Segmenting the target peptide segment and replacing the amino acid vocabulary obtained by the segmentation with a token representation, obtaining the target peptide segment represented by the token and inputting it into the peptide segment language model to obtain a vectorized representation of the target peptide segment;

[0017] 4) The trained fine-tuning layer is used to convolve the vectorized representation of the target peptide, and the convolution results are concatenated and output through the Softmax layer to determine the probability that the target peptide is a bitter peptide.

[0018] Furthermore, the fine-tuning layer includes three parallel convolution units, the first convolution unit includes four convolution kernels of size 1×1024, the second convolution unit includes two convolution kernels of size 3×1024, and the third convolution unit includes two convolution kernels of size 3×1024; the vectorized representation of the target peptide is convolved by the first convolution unit, the second convolution unit, and the third convolution unit, respectively, to obtain three feature maps; then the three feature maps are flattened and spliced to obtain a spliced feature map, and the spliced feature map is passed through the Softmax function to obtain the probability that the target peptide is a bitter peptide.

[0019] Furthermore, when the number of amino acids in the protein sequence is less than 30, random segmentation is adopted; if the current protein sequence cannot be enzymatically cut, random segmentation is selected.

[0020] Furthermore, SentencePiece was used to segment the obtained peptide sequences and amino acid sequences using the Unigram method.

[0021] Furthermore, the loss function used in training the fine-tuning layer is a cross entropy loss function.

[0022] A server, characterized in that it includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in the above method.

[0023] A computer-readable storage medium stores a computer program thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0024] The advantages of the present invention are as follows:

[0025] 1. For the first time, a peptide-based language model was proposed and applied to the field of bitter peptide recognition.

[0026] 2. For the first time, the ALBERT structure was applied as a pre-training model to the field of bitter peptide identification. Compared with BERT, it saved about 90% of parameters, made it possible to use mobile devices for local real-time peptide identification, and lowered the threshold for using BERT-like bitter peptide algorithms.

[0027] 3. For the first time, the amino acid vocabulary obtained by automatic training based on SentencePiece Unigram is used to pre-train the language model.

[0028] 4. In the pre-training phase, with only about 30% of the training time of Zhang et al.'s method, the accuracy rate reached 81.66% and the recall rate reached 84% when evaluated in the inference phase using a test set of the iBitter dataset with a length greater than 4. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 An overview of the peptide language model.

[0030] Figure 2 This is the data preprocessing flow chart.

[0031] Figure 3 This is the flow chart of the pre-training stage.

[0032] Figure 4 Flowchart for obtaining vectorized representation.

[0033] Figure 5 Flowchart for fine-tuning and inference. DETAILED DESCRIPTION

[0034] The present invention will be described in further detail below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0035] The pre-training strategy is an important part of the present invention. Figure 1Describes a pre-training process for self-supervised learning using large amounts of unlabeled protein sequence data. Figure 5 The present invention describes the fine-tuning and reasoning process for predicting bitter peptides for specific peptide sequences.

[0036] Step 1: Data preprocessing

[0037] In conjunction with the embodiment of the present invention, the specific data preprocessing stage in the pre-training stage is as follows: Figure 2 As shown. In the pre-training stage, the present invention uses about 500,000 protein sequences downloaded from the UniProt protein database (uniprot.org) as the original corpus. When the number of amino acids in the target protein sequence is less than 30, random segmentation is adopted; there is a 70% probability of forming peptide segments according to the five enzyme cleavage methods shown in Table 1. If the current protein sequence cannot be enzymatically cleaved, random segmentation is selected; there is a 30% probability of random segmentation to obtain an amino acid sequence representation; finally, the amino acid dictionary is constructed using the Unigram method through SentencePiece, and word segmentation is performed according to the dictionary to obtain the amino acid sequence represented by Token.

[0038] Taking the protein sequence [MDANIVSSSTIATYIDALAKNASELEQRSTAYEINNELEL] as an example, after simulated enzyme digestion, three peptides represented by amino acids [MDANIVSSSTIATYIDALA][KNASELEQ][RSTAYEINNELEL] are formed. After SentencePiece segmentation, the amino acid dictionary is used to obtain the tokenized representations [0,12,38,3,2,7,4,25,49,33,2,1], [0,7,31,17,36,28,1] and [0,21,14,18,1] of the three peptide sequences represented by Tokens, where 0 represents the start marker and 1 represents the separation and end marker.

[0039] Table 1 Simulated enzyme digestion method

[0040]

[0041] Step 2: Pre-training of peptide language model based on ALBERT

[0042] At present, large protein language models are usually pre-trained directly using protein sequences. However, in downstream tasks such as bitter peptides, the predicted content is peptides rather than complete proteins. The difference between the pre-trained content and the fine-tuned content may affect the performance of the model in downstream tasks and fail to give full play to the advantages of the pre-trained language model. Therefore, the present invention uses computational methods to simulate multiple enzyme cuts to cut protein sequences into multiple peptide sequences. In the pre-training stage, the present invention proposes for the first time to directly use peptides for model pre-training to ensure that the corpus category of the model learning is consistent with the downstream tasks, thereby enhancing the robustness of the pre-training model and reducing the learning difficulty of the fine-tuning stage. In this embodiment, the sentence length of ALBERT is set to 131 and the hidden layer size is set to 1024.

[0043] The pre-training stage mainly includes two tasks. The first is the masked language modeling task (MLM), whose purpose is to enable the model to learn to understand the context of natural language. During the training process, some tokens are replaced with specific mask symbols (such as "[MASK]"). The model needs to determine the masked words based on the context. The second is the sentence-order prediction task (SOP). The SOP task is an improvement on the next-sentence prediction task (NSP). First, two segmented peptides are extracted from the same protein sequence. There is a 50% probability of keeping the original order unchanged and a 50% probability of swapping the order of the two peptides. The model needs to determine whether the order is swapped.

[0044] The existing BERT-based methods all use the entire protein sequence as training corpus. The present invention regards the peptide sequence obtained in step 1 as a sentence and the amino acid combination obtained by SentencePiece as a vocabulary. In conjunction with the embodiment of the present invention, the ALBERT pre-training process is described with batchsize=1 as an example. Figure 3 The protein sequence is the peptide sequence obtained in step 1. Then two peptide sequences are selected and spliced, separated by a spacer (the spacer used in this example is Token 1), and the order of the two segments is swapped with a probability of 50% to complete the sentence order prediction task; and 15% of the tokens are randomly replaced with the masking symbol (the masking symbol in this example is 3) to complete the masked language model task.

[0045] The embedding is then completed through the embedding layer of ALBERT, where the word embedding layer maps the sequence represented by the token to a high-dimensional vector; since the Transformer model does not have the sequence position information of RNN or CNN, it is necessary to add position information through the position information embedding layer; sentence type embedding is used to indicate whether the two peptides have swapped positions.

[0046] After embedding is completed, pre-training is completed through the masked language model task and sentence order prediction task described above.

[0047] Step 3: Obtain vectorized representation of target peptide

[0048] After the pre-training in step 2, the ALBERT model can learn some intrinsic connections between peptide sequences and provide a vectorized representation for the target peptide. Figure 4 As shown in the figure, the target peptide [MNAKYDPSFSARYRY] first needs to be tokenized through SentencePiece to obtain the Token representation [0, 23, 11, 9, 37, 8, 1], where 0 represents the start token and 1 represents the separation and end tokens. At this time, it is sent to the ALBERT model to obtain the vectorized representation.

[0049] Step 4: Use fine-tuning layers for classification

[0050] In combination with the embodiment of the present invention, the target peptide segment [0, 23, 11, 9, 37, 8, 1] to be identified can be converted into a vector representation by step 3, where 0 represents the start mark and 1 represents the separation and end mark. Figure 5The target peptide is first vectorized by ALBERT. In order to extract the local features of the amino acid vocabulary, for peptides with a token length of less than 128, the vectorized representation provided by ALBERT is convolved with convolution kernels of sizes 1×1024, 3×1024, and 5×1024, respectively, to obtain three feature maps; 1024 is the hidden layer size (hidden_size=1024) set by the ALBERT model, the number of convolution kernels is 4, 2, and 2, and the padding parameters are (0, 0), (1, 0), and (2, 0), respectively. Taking batchsize=1 as an example, the feature map sizes obtained can be expressed as (4, 131), (2, 131), and (2, 131). The feature maps obtained by the three convolutions are then flattened and concatenated to obtain a concatenated feature map of (1, 1048). Finally, the concatenated feature map is passed through the Softmax function to obtain the probability value of the target sample being a bitter peptide. During the fine-tuning phase of training for downstream tasks, the model uses the probability values output by Softmax and the true labels to calculate a loss function, followed by backpropagation and gradient updates. In this example, the loss function used is the cross-entropy loss function. After training, during the inference phase, the model obtains the probability values and uses a threshold to determine whether the target peptide is a bitter peptide. In this example, a target peptide is considered a bitter peptide if the output probability value is greater than 50%.

[0051] While specific embodiments of the present invention have been disclosed for illustrative purposes, intended to facilitate understanding and implementation of the present invention, those skilled in the art will appreciate that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the disclosure of the preferred embodiments, and the scope of protection claimed in the present invention shall be determined by the scope of the claims.

Claims

1. A method for predicting bitter peptides based on a peptide language model, comprising the following steps: 1) Select multiple protein sequences as original corpus; The selected protein sequences are divided into two subsets. For each protein sequence in the first subset, a feasible enzymatic digestion method is randomly selected to obtain multiple peptide sequences; each protein sequence in the second subset is randomly segmented to obtain multiple amino acid sequences; then, each obtained peptide sequence and amino acid sequence is segmented, and the amino acid vocabulary obtained by segmentation is replaced with a token to obtain a peptide sequence represented by the token; 2) Treating the peptide sequence represented by each token as a sentence and the amino acid combination obtained from the word segmentation as a vocabulary, the ALBERT model is trained; the trained ALBERT model is used as the peptide language model; using Text-CNN to construct a fine-tuning layer, the bitter peptide sample is segmented and the amino acid vocabulary obtained from the word segmentation is replaced with a token representation, the bitter peptide sample represented by the token is input into the peptide language model, and the vectorized representation of the bitter peptide sample is obtained to train the fine-tuning layer; 3) Segmenting the target peptide segment and replacing the amino acid vocabulary obtained by the segmentation with a token representation, obtaining the target peptide segment represented by the token and inputting it into the peptide segment language model to obtain a vectorized representation of the target peptide segment; 4) The trained fine-tuning layer is used to convolve the vectorized representation of the target peptide, and the convolution results are concatenated and output through the Softmax layer to determine the probability that the target peptide is a bitter peptide.

2. The method according to claim 1, characterized in that The fine-tuning layer includes three parallel convolution units, the first convolution unit includes four convolution kernels of size 1×1024, the second convolution unit includes two convolution kernels of size 3×1024, and the third convolution unit includes two convolution kernels of size 3×1024; the vectorized representation of the target peptide is convolved by the first convolution unit, the second convolution unit, and the third convolution unit, respectively, to obtain three corresponding feature maps; then the three feature maps are flattened and spliced to obtain a spliced feature map, and the spliced feature map is passed through the Softmax function to obtain the probability that the target peptide is a bitter peptide.

3. The method according to claim 1, characterized in that When the number of amino acids in the protein sequence is less than 30, random segmentation is adopted; if the current protein sequence cannot be enzymatically cut, random segmentation is selected.

4. The method according to claim 1, 2 or 3, characterized in that: SentencePiece was used to segment the peptide sequences and amino acid sequences using the Unigram method.

5. The method according to claim 1, wherein The loss function used in training the fine-tuning layer is the cross entropy loss function.

6. A server, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the steps of the method according to any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images