A method for predicting protein disordered regions based on deep learning and language model
By using the pre-trained protein language model ProteinBert and integrated deep learning methods to build an IDP-EDL model, the problem of quickly and accurately identifying disordered regions of proteins was solved, and low-cost and efficient prediction effects were achieved.
Patent Information
- Application Number
- CN202411379641.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2025-09-12
- Estimated Expiration
- 2044-09-30
AI Technical Summary
Existing technologies have difficulty in identifying intrinsically disordered regions in proteins quickly, accurately, and at low cost, especially in high-throughput data processing.
The pre-trained protein language model ProteinBert is used to generate a universal feature matrix of protein sequences, and the IDP-EDL model is constructed by integrating deep learning methods. It combines a universal feature extraction module, a protein-specific representation module and a prediction layer, uses an attention mechanism and a convolutional neural network to capture disordered region features, and finally performs probabilistic prediction through a sigmoid function.
It enables rapid and accurate identification of intrinsically disordered regions in proteins at low cost and is suitable for high-throughput data processing.
Smart Images
Figure CN119252318B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of bioinformatics, and in particular to a method for predicting intrinsic disordered regions of proteins based on a pre-trained protein language model and integrated deep learning. Background Art
[0002] Intrinsically disordered regions of proteins refer to protein regions that do not have a stable three-dimensional structure under physiological conditions. Intrinsically disordered regions play important biological functions and are widely involved in important physiological processes, such as the regulation of transcription and translation, the storage of small molecules, cellular information transduction, protein phosphorylation, etc. In addition, some diseases are also associated with disordered regions. For example, 79% of cancer-related proteins contain disordered fragments with a length of more than 30 amino acids. Therefore, intrinsically disordered proteins are an important class of drug targets, and related research provides new opportunities for drug development. However, since most rational drug design strategies rely on high-resolution protein three-dimensional structure, the dynamic properties of intrinsically disordered proteins have created huge obstacles to related drug design. Therefore, accurately identifying intrinsically disordered regions in proteins is of great significance for studying protein functions and designing drugs.
[0003] Traditional experimental techniques have been used to identify intrinsically disordered proteins, including X-ray crystallography, nuclear magnetic resonance, and circular dichroism. However, due to the disadvantages of high cost and long cycle, experimental methods are not suitable for high-throughput data. Therefore, it is particularly important to develop fast and accurate computational methods. Computational methods consist of three essential parts: sufficient training data, effective protein features, and reliable models or algorithms. First, the massive protein sequence resources provide us with the opportunity to train a large model that can express the universal characteristics of proteins. In recent studies, many pre-trained protein language models have been invented. We can use pre-trained models to obtain universal expressions of proteins and thus predict protein disordered regions. We can use integrated deep learning methods to fuse several complementary models to obtain a stable and accurate model. Therefore, it is necessary to invent a method for predicting intrinsic disordered regions of proteins based on pre-trained protein language models. Summary of the Invention
[0004] The purpose of the present invention is to provide a prediction method with good prediction performance, low cost, and the ability to quickly and accurately identify intrinsic disordered regions in proteins.
[0005] In order to achieve the above object, the present invention provides the following technical solutions:
[0006] A method for predicting protein intrinsic disorder regions based on a pre-trained protein language model and integrated deep learning, characterized by comprising the following steps:
[0007] S1. Before the intrinsically disordered protein to be predicted is input into the pre-trained language model, the protein sequence needs to be embedded into a fixed-length integer token vector. For each sequence, 'START' and 'END' tokens are added before the first amino acid and after the last amino acid, respectively. If the sequence length is less than the fixed embedding length, the insufficient length is padded with 'PAD' tokens. If the sequence length is greater than the fixed embedding length, the original embedding length is multiplied by 2 until the length is met. Each amino acid is embedded as a unique integer token.
[0008] S2. Input the embedded integer vector into the BERT-based pre-trained model to generate the feature matrix of the intrinsically disordered protein to be predicted;
[0009] S3. Construct a model for the protein intrinsic disorder region prediction method IDP-EDL. This model integrates three specific models: IDP-EDL-G, IDP-EDL-L, and IDP-EDL-S. They predict the disordered regions, long disordered regions, and short disordered regions in proteins, respectively. The prediction results of each model are weighted summed to obtain the probability that each residue is a disordered region. Each specific model includes a general feature extraction module, a protein-specific representation extraction module, and a prediction layer. The general feature extraction module converts the intrinsic disordered protein sequence to be predicted into a general feature matrix through a pre-trained model, as described in steps S1 and S2. The protein-specific representation module further captures the feature information of different types of disordered regions through different deep neural networks. The working principle of the model is as follows:
[0010] By means of the universal feature module, the protein sequence is represented as a universal feature vector of the protein;
[0011] The specific representation module is used to further identify long disordered regions and short disordered regions in protein sequences, enrich the amino acid residue features of the protein sequences, use an attention mechanism to learn the global connections between residues in the protein, use a convolutional neural network to capture the local information of short disordered regions, and further learn the local contextual features and long-range dependencies of the amino acid residue features of the protein sequences through a bidirectional gated recurrent unit network layer to obtain a deep representation of the protein sequence.
[0012] Inputting the deep representation of the protein sequence into the output layer for processing, using a sigmoid function to obtain three sets of output values, and performing weighted summation on the three sets of output values to obtain a final probability value of each amino acid residue being disordered or ordered;
[0013] S4. Perform step S3 on the training set to train the IDP-EDL-G model; further divide the training set into two sub-training sets, one of which contains only short disordered proteins for training IDP-EDL-S, and the remaining portion is another training subset for training IDP-EDL-L, in which each protein contains at least one long disordered region;
[0014] S5. Execute steps S1, S2, and S3 on the test set to obtain the prediction results.
[0015] Furthermore, in step S2, the pre-trained protein language model ProteinBert is used to generate a universal feature matrix of proteins.
[0016] Furthermore, in step S4, the model is trained using the ADAM optimization algorithm.
[0017] Furthermore, in step S4, the model is trained using a Dropout layer with a probability of 0.5.
[0018] Furthermore, in step S4, a cross entropy function with weights is selected as the loss function in the training process.
[0019] Furthermore, in step S4, all pre-trained layers of the general feature extraction module are frozen, and only other layers are allowed to train for 40 cycles. Next, all layers are unfrozen and the model is trained for another 40 cycles.
[0020] The beneficial effects of the present invention are:
[0021] The present invention uses a pretrained protein language model, ProteinBert, to generate universal features of protein sequences. ProteinBert is a model trained on large protein datasets based on unsupervised learning. It learns the biochemical properties, spatial structure, and intrinsic functional patterns implicit in the input sequence, thereby obtaining universal features of protein sequences. The present invention uses an integrated deep learning approach to fuse three models for predicting long disordered regions, short disordered regions, and generally disordered regions to obtain a comprehensive and stable model. The present invention can quickly and accurately identify intrinsically disordered proteins, is low-cost, highly feasible, and easy to use and promote.
[0022] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 A framework diagram of the IDP-EDL prediction method shown in the present invention;
[0024] Figure 2 This is the prediction result of the present invention for protein P04637. DETAILED DESCRIPTION
[0025] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.
[0026] Unless otherwise specified, the technical solutions described in the present invention are all conventional solutions in the field; the reagents or materials described are all from commercial channels unless otherwise specified.
[0027] See Figure 1 The present invention discloses a method for predicting intrinsically disordered regions in proteins based on a pre-trained protein language model and integrated deep learning. This method can quickly and accurately identify intrinsically disordered regions in proteins with excellent prediction performance and low cost. The method operates as follows:
[0028] S1. The intrinsically disordered protein to be predicted is embedded as a fixed-length integer tag vector. For each sequence, 'START' and 'END' tags are added before the first amino acid and after the last amino acid, respectively. If the sequence length is less than the fixed embedding length, the insufficient length is padded with 'PAD' tags. If the sequence length is greater than the fixed embedding length, the original embedding length is multiplied by 2 until the length is satisfied. Each amino acid is embedded as a unique integer tag.
[0029] S2. Using the pre-trained language model ProteinBert, generate a feature matrix of the intrinsically disordered protein to be predicted, where the feature matrix can be represented as an L×1560 matrix, where L represents the fixed embedding length in step S1;
[0030] S3. Construct a model for the protein intrinsic disorder region prediction method IDP-EDL. This model integrates three specific models: IDP-EDL-G, IDP-EDL-L, and IDP-EDL-S. They predict the disordered regions, long disordered regions, and short disordered regions in proteins, respectively. The prediction results of each model are weighted summed to obtain the probability that each residue is a disordered region. Each specific model includes a general feature extraction module, a protein-specific representation extraction module, and a prediction layer. The general feature extraction module converts the intrinsic disordered protein sequence to be predicted into a general feature matrix through a pre-trained model, as described in steps S1 and S2. The protein-specific representation module further captures the feature information of different types of disordered regions through different deep neural networks. The working principle of the model is as follows:
[0031] By means of the universal feature module, the protein sequence is represented as a universal feature vector of the protein;
[0032] The specific representation module is used to further identify long disordered regions and short disordered regions in protein sequences, enrich the amino acid residue features of the protein sequences, use an attention mechanism to learn the global connections between residues in the protein, use a convolutional neural network to capture the local information of short disordered regions, and further learn the local contextual features and long-range dependencies of the amino acid residue features of the protein sequences through a bidirectional gated recurrent unit network layer to obtain a deep representation of the protein sequence.
[0033] The deep representation of the protein sequence is input into the output layer for processing, and a sigmoid function is used to obtain three groups of output values. The three groups of output values are weighted and summed to obtain the final probability value of each amino acid residue belonging to disorder or order.
[0034] IDP-EDL uses a general feature module and a protein-specific representation module. The features extracted by the general feature module are used as the basic features of the model, and the specific deep neural network in the protein-specific representation module learns the potential feature representations of different types of fixed disordered regions. The learned specific feature representations are input into the final classifier sigmoid function for classification.
[0035] S4. Execute step S3 on the training set to train the IDP-EDL-G model; further divide the training set into two sub-training sets, one of which contains only short disordered proteins for training IDP-EDL-S, and the remaining part is another training subset for training IDP-EDL-L, each of which contains at least one long disordered region. During training, the model is trained using the ADAM optimization algorithm to make the model simple to implement, computationally efficient, and low in memory requirements. At the same time, a Dropout layer with a probability of 0.5 is used for training to avoid overfitting. A weighted cross entropy function is selected as the loss function in the training process to solve the problem of extremely unbalanced ratios of ordered residues and disordered residues in the training set, so that the output of the model is closer to the true value and improves the predictive performance of the model.
[0036] S5. Execute steps S1, S2, and S3 for the protein sequence P04637 to obtain the prediction result. In step S1, the input sequence length is 393, and the embedded fixed vector length is set to 512. Therefore, the universal feature matrix obtained in step S2 can be represented as a 512×1560 matrix.
Claims
1. A protein disordered region prediction method based on deep learning and language model, characterized in that: The specific steps include: S1. Before inputting the intrinsically disordered protein to be predicted into the pre-trained language model, the protein sequence needs to be embedded into a fixed-length integer token vector. For each sequence, 'START' and 'END' tokens are added before the first amino acid and after the last amino acid, respectively. If the sequence length is less than the fixed embedding length, the insufficient length is padded with 'PAD' tokens. If the sequence length is greater than the fixed embedding length, the original embedding length is multiplied by 2 until the embedding length is no less than the sequence length. A unique integer is assigned to each amino acid to distinguish different amino acids for easy model input. S2. Input the embedded integer vector into the BERT-based pre-trained model to generate the feature matrix of the intrinsically disordered protein to be predicted; S3. Construct a model for the protein intrinsic disorder region prediction method IDP-EDL. This model integrates three specific models: IDP-EDL-G, IDP-EDL-L, and IDP-EDL-S. They predict the disordered regions, long disordered regions, and short disordered regions in proteins, respectively. The prediction results of each model are weighted summed to obtain the probability that each residue is a disordered region. Each specific model includes a general feature extraction module, a protein-specific representation extraction module, and a prediction layer. The general feature extraction module converts the intrinsic disordered protein sequence to be predicted into a general feature matrix through a pre-trained model, as described in steps S1 and S2. The protein-specific representation extraction module further captures the feature information of different types of disordered regions through different deep neural networks. The working principle of the model is as follows: By means of the universal feature module, the protein sequence is represented as a universal feature vector of the protein; The protein-specific representation extraction module further identifies long and short disordered regions in protein sequences, enriches the amino acid residue features of the protein sequences, uses an attention mechanism to learn the global connections between residues in the protein, uses a convolutional neural network to capture local information of short disordered regions, and further learns the local contextual features and long-range dependencies of the amino acid residue features of the protein sequences through a bidirectional gated recurrent unit network layer, thereby obtaining a deep representation of the protein sequence. The deep representation of the protein sequence is input and output layer processed, a sigmoid function is used to obtain three sets of output values, and the three sets of output values are weighted and summed to obtain the final probability value of each amino acid residue being disordered or ordered; S4. Perform step S3 on the training set to train the IDP-EDL-G model; further divide the training set into two sub-training sets, one of which contains only short disordered proteins for training IDP-EDL-S, and the remaining portion is another training subset for training IDP-EDL-L, in which each protein contains at least one long disordered region; S5. Execute steps S1, S2, and S3 on the test set to obtain the prediction results.
2. The method for predicting protein disordered regions based on deep learning and language model according to claim 1, characterized in that: In the step S1, the 'START', 'END', and 'PAD' marks each represent an integer.
3. The method for predicting protein disordered regions based on deep learning and language model according to claim 1, characterized in that: In the step S2, a pre-trained protein language model ProteinBert is used to generate a universal feature matrix of proteins.
4. The method for predicting protein disordered regions based on deep learning and language model according to claim 1, characterized in that: In the step S4, the model is trained using the ADAM optimization algorithm.
5. The method for predicting protein disordered regions based on deep learning and language model according to claim 1, wherein: In step S4, the model is trained using a Dropout layer with a probability of 0.
5.
6. The method for predicting protein disordered regions based on deep learning and language model according to claim 1, characterized in that: In the step S4, a cross entropy function with weights is selected as the loss function in the training process.
7. The method for predicting protein disordered regions based on deep learning and language model according to claim 1, characterized in that: In the S4 step, all pre-trained layers of the general feature extraction module are frozen, and only other layers are allowed to be trained for 40 cycles. Next, all layers are unfrozen and the model is trained for another 40 cycles.