Protein disorder region prediction method based on deep learning and language model

Abstract

The invention relates to a protein inherent disorder region prediction method based on a pre-trained protein language model and integrated deep learning. Based on a pre-trained protein language model, general characteristics of a protein sequence are constructed, different deep neural networks are combined, the characteristics of a long disordered region and a short disordered region in protein are captured, and the prediction method for the inherent disordered region of the protein is constructed by integrating three specific deep learning models. The prediction method provided by the invention not only can obtain the general expression of the protein through a deep learning method, but also can learn local context features and remote dependency relationships of amino acid residues in a protein sequence, and combines three complementary predictors through an ensemble learning method, so that the prediction accuracy is improved. The prediction performance of the inherent disordered protein is greatly improved; the method can quickly and accurately identify the inherent disordered protein, and is low in cost, high in feasibility and convenient to use and popularize.

Description

Technical Field

[0001] The present invention relates to the technical field of bioinformatics, and in particular to a method for predicting protein intrinsic disordered regions based on a pre-trained protein language model and integrated deep learning. Background Art

[0002] Intrinsically disordered regions of proteins refer to regions of proteins that do not have a stable three-dimensional structure under physiological conditions. Intrinsically disordered regions play important biological functions and are widely involved in important physiological processes, such as regulation of transcription and translation, storage of small molecules, cell information transduction, protein phosphorylation, etc. In addition, some diseases are also associated with disordered regions. For example, 79% of cancer-related proteins contain disordered fragments with a length of more than 30 amino acids. Therefore, intrinsically disordered proteins are an important class of drug targets, and related research provides new opportunities for drug development. However, since most rational drug design strategies rely on high-resolution three-dimensional protein structures, the dynamic properties of intrinsically disordered proteins have created huge obstacles to related drug design. Therefore, accurately identifying intrinsically disordered regions in proteins is of great significance for studying protein functions and designing drugs.

[0003] Traditional experimental techniques have been used to identify intrinsically disordered proteins, including X-ray crystallography, nuclear magnetic resonance, and circular dichroism. However, due to the disadvantages of high cost and long cycle, experimental methods are not suitable for high-throughput data. Therefore, it is particularly important to develop fast and accurate computational methods. The computational method consists of three essential parts: sufficient training data, effective protein features, and reliable models or algorithms. First, the massive protein sequence resources provide us with the opportunity to train a large model that can express the universal characteristics of proteins. In recent studies, many pre-trained protein language models have been invented. We can use pre-trained models to obtain universal expressions of proteins to predict protein disordered regions. We can use integrated deep learning methods to fuse several complementary models to obtain a stable and accurate model. Therefore, it is necessary to invent a method for predicting intrinsic disordered regions of proteins based on pre-trained protein language models. Summary of the invention

[0004] The object of the present invention is to provide a prediction method with good prediction performance, low cost, and the ability to quickly and accurately identify intrinsic disordered regions in proteins.

[0005] In order to achieve the above object, the present invention provides the following technical solutions: A method for predicting protein intrinsic disorder regions based on a pre-trained protein language model and integrated deep learning, characterized in that it comprises the following steps: S1. Before the intrinsic disordered protein to be predicted is input into the pre-trained language model, the protein sequence needs to be embedded into a fixed-length integer tag vector. For each sequence, the 'START' and 'END' tags are added before the first amino acid and after the last amino acid respectively. If the sequence length is less than the fixed embedding length, the insufficient length is filled with the 'PAD' tag. If the sequence length is greater than the fixed embedding length, the original embedding length is multiplied by 2 until it is satisfied. Each amino acid will be embedded as a unique integer tag. S2, input the embedded integer vector into the BERT-based pre-trained model to generate a feature matrix of the intrinsic disordered protein to be predicted; S3. Construct a model of protein intrinsic disorder region prediction method IDP-EDL, which integrates three specific models: IDP-EDL-G, IDP-EDL-L, IDP-EDL-S, which respectively predict the disordered regions, long disordered regions and short disordered regions in proteins under normal circumstances. The prediction results of each model are weighted summed to obtain the possibility of each residue being a disordered region. Each specific model includes a general feature extraction module, a protein specific representation extraction module and a prediction layer, wherein the general feature extraction module converts the intrinsic disordered protein sequence to be predicted into a general feature matrix through a pre-trained model, as described in steps S1 and S2, and the protein specific representation module further captures the feature information of different types of disordered regions through different deep neural networks. The working principle of the model is: By means of the universal feature module, the protein sequence is represented as a universal feature vector of the protein; Through the specific representation module, long disordered regions and short disordered regions in the protein sequence are further identified, the amino acid residue features of the protein sequence are enriched, the global connection between residues in the protein is learned using the attention mechanism, the local information of the short disordered regions is captured using the convolutional neural network, and the local context features and long-range dependencies of the amino acid residue features of the protein sequence are further learned through the bidirectional gated recurrent unit network layer to obtain a deep representation of the protein sequence; Inputting the deep representation of the protein sequence into the output layer for processing, using the sigmoid function to obtain three groups of output values, and performing weighted summation on the three groups of output values ​​to obtain the final probability value of each amino acid residue being disordered or ordered; S4, executing step S3 on the training set to train the IDP-EDL-G model; further dividing the training set into two sub-training sets, wherein one sub-training set contains only short disordered proteins for training IDP-EDL-S, and the remaining part is another training subset for training IDP-EDL-L, wherein each protein contains at least one long disordered region; S5. Execute steps S1, S2, and S3 on the test set to obtain the prediction result.

[0006] Furthermore, in step S2, a pre-trained protein language model ProteinBert is used to generate a universal feature matrix of proteins.

[0007] Furthermore, in step S4, the model is trained using the ADAM optimization algorithm.

[0008] Furthermore, in step S4, the model is trained using a Dropout layer with a probability of 0.5.

[0009] Furthermore, in step S4, a cross entropy function with weights is selected as the loss function in the training process.

[0010] Furthermore, in step S4, all pre-trained layers of the general feature extraction module are frozen, and only other layers are allowed to be trained for 40 cycles. Next, all layers are unfrozen and the model is trained for another 40 cycles. The beneficial effects of the present invention are: The present invention uses a pre-trained protein language model ProteinBert to generate universal features of protein sequences. ProteinBert is a model trained on a large protein data set based on unsupervised learning. It learns the biochemical properties, spatial structure, functional intrinsic laws, etc. implicit in the input sequence, thereby obtaining universal features of protein sequences. The present invention uses an integrated deep learning method to fuse three models for predicting long disordered regions, short disordered regions, and disordered regions under normal circumstances to obtain a comprehensive and stable model. The present invention can quickly and accurately identify intrinsically disordered proteins, has low cost, strong feasibility, and is easy to use and promote.

[0011] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a framework diagram of the IDP-EDL prediction method shown in the present invention; Figure 2 This is the prediction result of the present invention for protein P04637. DETAILED DESCRIPTION

[0013] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0014] The technical solutions described in the present invention, unless otherwise specified, are all conventional solutions in the art; the reagents or materials described, unless otherwise specified, are all from commercial channels.

[0015] See also Figure 1 The present invention discloses a method for predicting intrinsic disordered regions of proteins based on a pre-trained protein language model and integrated deep learning. The method can quickly and accurately identify intrinsic disordered regions in proteins, with good prediction performance and low cost. The operating steps of the method are as follows: S1. The intrinsic disordered protein to be predicted is embedded into a fixed-length integer tag vector. For each sequence, the 'START' and 'END' tags are added before the first amino acid and after the last amino acid respectively. If the sequence length is less than the fixed embedding length, the insufficient length is filled with the 'PAD' tag. If the sequence length is greater than the fixed embedding length, the original embedding length is multiplied by 2 until it is satisfied. Each amino acid is embedded as a unique integer tag. S2. Using the pre-trained language model ProteinBert, generate a feature matrix of the intrinsic disordered protein to be predicted, where the feature matrix can be represented as a matrix of L×1560, where L represents the fixed embedding length in step S1; S3. Construct a model of protein intrinsic disorder region prediction method IDP-EDL, which integrates three specific models: IDP-EDL-G, IDP-EDL-L, IDP-EDL-S, which respectively predict the disordered regions, long disordered regions and short disordered regions in proteins under normal circumstances. The prediction results of each model are weighted summed to obtain the possibility of each residue being a disordered region. Each specific model includes a general feature extraction module, a protein specific representation extraction module and a prediction layer, wherein the general feature extraction module converts the intrinsic disordered protein sequence to be predicted into a general feature matrix through a pre-trained model, as described in steps S1 and S2, and the protein specific representation module further captures the feature information of different types of disordered regions through different deep neural networks. The working principle of the model is: By means of the universal feature module, the protein sequence is represented as a universal feature vector of the protein; Through the specific representation module, long disordered regions and short disordered regions in the protein sequence are further identified, the amino acid residue features of the protein sequence are enriched, the global connection between residues in the protein is learned using the attention mechanism, the local information of the short disordered regions is captured using the convolutional neural network, and the local context features and long-range dependencies of the amino acid residue features of the protein sequence are further learned through the bidirectional gated recurrent unit network layer to obtain a deep representation of the protein sequence; The deep representation of the protein sequence is input into the output layer for processing, and the sigmoid function is used to obtain three groups of output values. The three groups of output values ​​are weighted and summed to obtain the final probability value of each amino acid residue belonging to disorder and order.

[0016] IDP-EDL adopts a general feature module and a protein-specific representation module. The features extracted by the general feature module are used as the basic features of the model, and the specific deep neural network in the protein-specific representation module learns the potential feature representations of different types of fixed disordered regions. The learned specific feature representations are input into the final classifier sigmoid function for classification.

[0017] S4, perform step S3 on the training set to train the IDP-EDL-G model; further divide the training set into two sub-training sets, one of which contains only short disordered proteins for training IDP-EDL-S, and the remaining part is another training subset for training IDP-EDL-L, each of which contains at least a long disordered region. During training, the model is trained using the ADAM optimization algorithm to make the model simple to implement, efficient in calculation, and low in memory requirements. At the same time, a Dropout layer with a probability of 0.5 is used for training to avoid overfitting. A weighted cross entropy function is selected as the loss function in the training process to solve the problem of extremely unbalanced ratios of ordered residues and disordered residues in the training set, so that the output of the model is closer to the true value and improves the prediction performance of the model.

[0018] S5. Execute steps S1, S2, and S3 for the protein sequence P04637 to obtain the prediction result. In step S1, the input sequence length is 393, and the embedded fixed vector length is set to 512, so the general feature matrix obtained in step S2 can be expressed as a 512×1560 matrix.

Claims

1. A method for predicting protein intrinsic disorder regions based on a pre-trained protein language model and integrated deep learning, characterized in that: The specific steps include: S1. Before the intrinsic disordered protein to be predicted is input into the pre-trained language model, the protein sequence needs to be embedded into a fixed-length integer tag vector. For each sequence, 'START' and 'END' tags are added before the first amino acid and after the last amino acid respectively; if the sequence length is less than the fixed embedding length, the insufficient length is filled with 'PAD' tags, and if the sequence length is greater than the fixed embedding length, the original embedding length is multiplied by 2 until it is satisfied; each amino acid will be embedded as a unique integer tag; S2, input the embedded integer vector into the BERT-based pre-trained model to generate a feature matrix of the intrinsic disordered protein to be predicted; S3. Construct a model of protein intrinsic disorder region prediction method IDP-EDL, which integrates three specific models: IDP-EDL-G, IDP-EDL-L, IDP-EDL-S, which respectively predict the disordered regions, long disordered regions and short disordered regions in proteins under normal circumstances. The prediction results of each model are weighted summed to obtain the possibility of each residue being a disordered region. Each specific model includes a general feature extraction module, a protein specific representation extraction module and a prediction layer, wherein the general feature extraction module converts the intrinsic disordered protein sequence to be predicted into a general feature matrix through a pre-trained model, as described in steps S1 and S2, and the protein specific representation module further captures the feature information of different types of disordered regions through different deep neural networks. The working principle of the model is: By means of the universal feature module, the protein sequence is represented as a universal feature vector of the protein; Through the specific representation module, long disordered regions and short disordered regions in the protein sequence are further identified, the amino acid residue features of the protein sequence are enriched, the global connection between residues in the protein is learned using the attention mechanism, the local information of the short disordered regions is captured using the convolutional neural network, and the local context features and long-range dependencies of the amino acid residue features of the protein sequence are further learned through the bidirectional gated recurrent unit network layer to obtain a deep representation of the protein sequence; Inputting the deep representation of the protein sequence into the output layer for processing, using the sigmoid function to obtain three groups of output values, and performing weighted summation on the three groups of output values ​​to obtain the final probability value of each amino acid residue being disordered or ordered; S4, executing step S3 on the training set to train the IDP-EDL-G model; further dividing the training set into two sub-training sets, wherein one sub-training set contains only short disordered proteins for training IDP-EDL-S, and the remaining part is another training subset for training IDP-EDL-L, wherein each protein contains at least one long disordered region; S5. Execute steps S1, S2, and S3 on the test set to obtain the prediction result.

2. The method for predicting intrinsic disordered regions based on a pre-trained protein language model and integrated deep learning according to claim 1, characterized in that: In the step S1, the 'START', 'END', and 'PAD' marks represent an integer respectively.

3. The method for predicting protein disordered regions based on deep learning and language model according to claim 1, characterized in that: In the step S2, a pre-trained protein language model ProteinBert is used to generate a universal feature matrix of proteins.

4. The method for predicting protein disordered regions based on deep learning and language model according to claim 1, characterized in that: In step S4, the model is trained using the ADAM optimization algorithm.

5. The method for predicting protein disordered regions based on deep learning and language model according to claim 1, characterized in that: In the step S4, the model is trained using a Dropout layer with a probability of 0.

5.

6. The method for predicting protein disordered regions based on deep learning and language model according to claim 1, characterized in that: In the step S4, a cross entropy function with weights is selected as the loss function in the training process.

7. The method for predicting protein disordered regions based on deep learning and language model according to claim 1, characterized in that: In the S4 step, all pre-trained layers of the general feature extraction module are frozen, and only other layers are allowed to be trained for 40 cycles. Next, all layers are unfrozen and the model is trained for another 40 cycles.

Images