Transcription factor identification method based on transformer-encoder and multi-scale convolutional neural network

Abstract

The application discloses a transcription factor identification method based on a Transformer-Encoder and a multi-scale convolutional neural network. The method first extracts global features of a protein sequence using the Transformer-Encoder, then further extracts multi-scale local features from the global features using the multi-scale convolutional neural network, and finally fuses the extracted multiple features to output a probability of the protein sequence being a transcription factor. The transcription factor identification method based on the Transformer-Encoder and the multi-scale convolutional neural network can accurately identify whether an unknown protein sequence is a transcription factor, can quickly determine the transcription factor only by relying on the protein sequence, and greatly improves the protein labeling efficiency.

Description

Technical Field

[0001] This invention relates to the field of protein functional annotation, specifically to a transcription factor identification method based on Transformer-Encoder and multi-scale convolutional neural networks. Since transcription factors are a class of proteins with special functions, this invention belongs to the application of deep learning in the field of protein functional annotation. Background Technology

[0002] Transcription factors are protein molecules with specific structures that regulate gene expression. They regulate the expression of target genes by specifically binding to DNA sequences, promoting or inhibiting the transcription process from specific DNA to RNA.

[0003] Traditionally, methods for identifying and recognizing transcription factors through biochemical experiments are time-consuming and costly, making them unsuitable for large-scale use. BLAST-based homology searches cannot identify whether proteins from different sources within the same database are transcription factors. Traditional machine learning prediction methods can identify transcription factors based on protein structure or sequence information, but require manually designed transcription factor-related features, strong domain knowledge, and low prediction accuracy. Deep learning has the advantage of directly learning protein sequence features, but most existing methods are based on convolutional neural networks to build prediction models. Due to the limitations of convolutional kernels, while these methods can automatically learn feature representations, they can only learn local features of relationships between closely spaced amino acids, failing to learn global features of relationships between distant amino acids, thus affecting the model's prediction accuracy. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a transcription factor identification method based on Transformer-Encoder and multi-scale convolutional neural networks. This method can simultaneously extract global and local information from protein sequences, automatically obtain comprehensive representational features of transcription factors, and thus further improve prediction accuracy.

[0005] The technical solution provided by this invention is as follows:

[0006] A transcription factor identification method based on Transformer-Encoder and multi-scale convolutional neural network, comprising the following steps:

[0007] Step 1: Constructing the training set: Collect protein sequences from protein databases, and label each protein sequence as a transcription factor or a non-transcription factor according to the corresponding protein annotation information; preprocess all sequences to obtain the training dataset;

[0008] Step 2: Building the Network Structure: Construct a transcription factor prediction model by building a network structure combining Transformer-Encoder and multi-scale convolutional neural network; where Transformer-Encoder is used to obtain the i-th protein sequence X. i global features Multi-scale convolutional neural networks are used for... Transcription factor prediction and identification;

[0009] Step 3: Train the prediction model: Use the training set obtained in Step 1 to train the network built in Step 2 to obtain the trained transcription factor prediction model.

[0010] Step 4: Transcription factor prediction: Using the prediction model obtained in Step 3, predict whether the unknown protein sequence is a transcription factor and output the prediction results.

[0011] Furthermore, step 1 includes the following sub-steps:

[0012] 1.1 Select protein sequences from the protein database that do not contain non-standard amino acids, namely B, O, U, and Z, to form dataset S1;

[0013] 1.2 Remove sequences longer than 1000 from S1, keeping only sequences shorter than or equal to 1000; for protein sequences shorter than 1000, pad with zeros to a length of 1000; finally, obtain the protein sequence dataset S2.

[0014] 1.3 Based on the GO annotation information of each protein in the protein database, each protein sequence in S2 is assigned a label of transcription factor "1" or non-transcription factor "0"; finally, the training dataset S = (X i ,c i |i = 1,..,N; where X i c represents the i-th protein sequence in the dataset; i For X i The tag, c i ∈{0,1}; N is the size of S.

[0015] Furthermore, in step 1.3, if the GO annotation of a protein contains the GO term "transcription factor", or contains both the GO terms "transcription regulation" and "DNA binding", then the protein sequence is classified as a transcription factor and assigned a value of "1"; otherwise, the protein sequence is classified as a non-transcription factor and assigned a value of "0".

[0016] Furthermore, the network structure in step 2 includes a Transformer-Encoder structure composed of cascaded components and a multi-scale convolutional neural network structure;

[0017] This Transformer-Encoder structure retains only the Encoder part of the Transformer, and consists of 6 stacked Encoder blocks, each containing 12 attention heads; the Transformer-Encoder is used to extract global features from the input protein sequence;

[0018] A multi-scale convolutional neural network consists of four parallel convolutional subnetworks with different one-dimensional convolutional kernels, two fully connected layers, and an output layer. The convolutional layers contain multiple one-dimensional convolutional operations corresponding to convolutional kernels of different sizes to obtain multiple convolutional features of different sizes. The pooling layers pool the multiple convolutional features to obtain features with reduced dimensions. The pooled features are concatenated and fed into the fully connected layers. The prediction results obtained by the fully connected layers are output by the output layer.

[0019] Furthermore, in step 2, let a protein sequence be X. i =x i1 ,x i2 ,…,x ij ,…x i1000 x ij Represents protein sequence X i The amino acid at position j is used to obtain X using a Transformer-Encoder. i global features The specific steps are as follows:

[0020] 2.1 Obtain X through embedding operation i The embedding vector, and the specific embedding method are as follows:

[0021] 2.1.1 First, randomize the different amino acid types, then assign X according to the corresponding amino acid type. i Each amino acid x ij Embedding generates the corresponding vector;

[0022] 2.1.2 Position encoding is used to extract the positional information of amino acids in a protein sequence. Position encoding identifies the different positions of amino acids in the protein using sine and cosine functions. The formula for encoding the position of the j-th amino acid is shown below:

[0023]

[0024] Where pos represents the position of the amino acid in the protein sequence, d represents the dimension of the embedding vector, and k is a natural number;

[0025] 2.1.3 Each amino acid x ij The embedding and the corresponding positional encoding are added together to obtain protein X. i The embedding vector of the sequence;

[0026] 2.2 Obtaining the protein sequence X i After obtaining the embedding vector, it is used as input to the Transformer-Encoder. Its attention mechanism is used to mine the attention score between every two amino acids. The attention score is then cross-multiplied with the embedding vector to obtain the entire protein sequence X. i global features

[0027] Furthermore, in step 2, the convolutional subnetwork is composed of convolutional layers, normalization layers, dropout layers, and max-pooling layers in sequence;

[0028] The formula for calculating the output of the i-th sub-network is:

[0029] F i (x)=MaxPooling(ReLU(Norm(ConV(x)))

[0030] Where x is the input to the convolutional layer;

[0031] The concatenated output of the four subnetworks is:

[0032] output = Concat(F i (x)), i = 1, 2, 3, 4.

[0033] Furthermore, in step 2, the multi-scale convolutional neural network uses one-dimensional convolutional kernels of different sizes to extract features between local protein sequences of different lengths.

[0034] Furthermore, in step 2, the length of the convolution kernel is the same as the embedding dimension of the protein, the width is set to 4-20, and the size of the convolution kernel is 4, 8, 12, 16 respectively.

[0035] Furthermore, in step 2, based on The method for predicting and identifying transcription factors is as follows:

[0036] Based on protein sequence X i global features X is obtained after performing convolution operations at different scales. i Local features Where j represents the local features corresponding to different convolutional kernels, all local features are then concatenated and finally input into the fully connected layer, where the loss function is chosen as cross entropy, as shown in the following formula:

[0037]

[0038] in y is the probability that the model predicts a sample is a transcription factor, and y is the sample label. If the sample is a positive example, the value is 1, otherwise the value is 0.

[0039] For all protein sequences, the final output probability obtained after passing through the SoftMax function is [p 1j ,p 2j ,…,p nj The predicted category y of protein i is calculated using the following formula. i :

[0040]

[0041] For example, y i =1 indicates that protein sequence i is a transcription factor, otherwise it is a non-transcription factor.

[0042] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0043] 1. This invention uses a novel identification and classification method that solves the problems of high experimental costs and complex processes in traditional biochemical experiments. It can quickly identify transcription factors based solely on protein sequences, greatly improving protein labeling efficiency.

[0044] 2. In terms of feature extraction, this invention increases the dimension of extracted sequence features compared to previous methods proposed by scholars. It increases the extraction dimension of features from only 3-16 amino acid sequences to the entire protein sequence, effectively increasing the dimension of extracted protein features and improving the prediction accuracy of transcription factors.

[0045] 3. This invention introduces a transformer-encoder module into the model, which can calculate the attention score between each amino acid and other amino acids, thereby discovering amino acid pairs with high correlation in the protein sequence and better exploring the interrelationships between amino acids within the protein sequence. Attached Figure Description

[0046] Figure 1 This is a flowchart of the model of the present invention;

[0047] Figure 2 This is a protein data preprocessing workflow;

[0048] Figure 3 This is a structural diagram of the Transformer-Encoder model used in this invention;

[0049] Figure 4 This is the attention score calculation process in the Transformer-Encoder of this invention;

[0050] Figure 5 This is a schematic diagram of the multi-scale convolutional neural network model in this invention; Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0052] Example 1

[0053] Please see the appendix Figure 1-5 A transcription factor identification method based on Transformer-Encoder and multi-scale convolutional neural network includes the following steps:

[0054] Step 1: Obtain all protein sequences from the UniProtKB 2021_04Swiss-Prot database, preprocess the data, and construct a transcription factor training set S.

[0055] The data preprocessing operations described in step 1, such as Figure 2 As shown, it includes the following sub-steps:

[0056] S1.1 First, all protein sequences, totaling 565,928, were downloaded from the UniProtKB 2021_04Swiss-Prot dataset. The dataset's attributes include "Sequence", "Length", "Gene ontology (GO)", "Entryname", and "Protein names". Then, abnormal protein data containing non-standard amino acids (i.e., B, O, U, Z) were removed from the dataset.

[0057] S1.2 Then, based on the "Length" attribute, filter out sequences with a length less than or equal to 1000, and then zero-padded them to a length of 1000.

[0058] S1.3 Term collection from protein annotation databases yielded terms containing three functions: "transcription factor," "transcription regulation," and "DNA binding." Since GO terms are a protein gene annotation function category, their significance lies in creating a tag index for transcription factors. The GO terms for transcription factor included "GO: 0000976; GO: 0000977; GO: 0000978; GO: 0000979; GO: 0000981; GO: 0000984; GO: 0000985; GO: 0000986; GO: 0000987; GO: 0000992," etc.; the GO terms for transcription regulation included "GO: 0001228; GO: 0006351; GO: 0006355; GO: 0043433," etc.; and the GO terms for DNA binding... Terms include "GO: 0003677; GO: 0008301; GO: 0043565; GO: 0050692".

[0059] S1.4 The GO terms corresponding to the protein sequences obtained in step (1) are filtered. If the GO terms contain terms related to "transcription factor" from step 2, or both "transcription regulation" and "DNA binding" terms, they are labeled as transcription factors; otherwise, they are labeled as non-transcription factors. This constructs a dataset containing 124,316 cleaned and preprocessed protein sequences and their corresponding transcription factor tags.

[0060] Step 2: Build the network model M, constructing a Transformer-Encoder with a 2-layer, 8-head attention mechanism (e.g., ...). Figure 3-4 Simultaneously, multi-scale convolutional neural networks are constructed, including convolutional neural networks with one-dimensional convolutional kernels of kernel sizes of 4, 8, 12, and 16 (e.g., Figure 5 The two are linked together to form model M.

[0061] The network structure described in step 2 consists of a Transformer-Encoder structure and a multi-scale convolutional neural network structure connected in series. The Transformer-Encoder structure is derived from the Transformer network structure in the field of natural language processing. The original Transformer uses an Encoder-Decoder architecture; in step 2, only the Encoder part of the Transformer is retained, consisting of 6 stacked Encoder blocks, each containing 12 attention heads. The Transformer-Encoder is used to extract global features from the input protein sequence.

[0062] A multi-scale convolutional neural network consists of four parallel convolutional subnetworks with different one-dimensional convolutional kernels, two fully connected layers, and an output layer. The convolutional layers contain multiple one-dimensional convolutional operations corresponding to convolutional kernels of different sizes to obtain multiple convolutional features of different sizes. The pooling layers pool the multiple convolutional features to obtain features with reduced dimensions. The pooled features are concatenated and fed into the fully connected layers. The prediction results obtained by the fully connected layers are output by the output layer.

[0063] Let a protein sequence be X. i =x i1 ,x i2 ,…,x ij ,…x i1000 x ij Represents protein sequence X i The amino acid at position j. In step 2, X is obtained using a Transformer-Encoder. i global features The specific steps are as follows:

[0064] 2.1 Obtain X through embedding operation i The embedding vector, and the specific embedding method are as follows:

[0065] 2.1.1 First, randomize the different amino acid types, then assign X according to the corresponding amino acid type. i Each amino acid x ij Embedding generates the corresponding vector.

[0066] 2.1.2 Position encoding is used to extract the positional information of amino acids in a protein sequence. Position encoding identifies the different positions of amino acids in the protein using sine and cosine functions. The formula for encoding the position of the j-th amino acid is shown below:

[0067]

[0068] Where pos represents the position of the amino acid in the protein sequence, d represents the dimension of the embedding vector, and k is a natural number.

[0069] 2.1.3 Each amino acid x ij The embedding and the corresponding positional encoding are added together to obtain protein X. i The embedding vector of the sequence.

[0070] 2.2 Obtaining the protein sequence X i After obtaining the embedding vector, it is used as input to the Transformer-Encoder. Its attention mechanism is used to mine the attention score between every two amino acids. The attention score is then cross-multiplied with the embedding vector to obtain the entire protein sequence X. i global features

[0071] Multi-scale convolutional neural networks use one-dimensional convolutional kernels of different sizes to extract features between local protein sequences of different lengths. The length of the kernel is the same as the embedding dimension of the protein, and the width is set to 4-20, with kernel sizes of 4, 8, 12, and 16 respectively.

[0072] The convolutional subnetwork consists of convolutional layers, normalization layers, dropout layers, and max-pooling layers in sequence.

[0073] The formula for calculating the output of the i-th sub-network is:

[0074] F i (x)=MaxPooling(ReLU(Norm(ConV(x)))

[0075] Where x is the input to the convolutional layer;

[0076] The concatenated output of the four subnetworks is:

[0077] output = Concat(F i (x)), i = 1, 2, 3, 4.

[0078] based on The method for predicting and identifying transcription factors is as follows:

[0079] Based on protein sequence X i global features X is obtained after performing convolution operations at different scales. i Local features Here, j represents the local features corresponding to different convolutional kernels. All local features are then concatenated and finally input into the fully connected layer. The loss function chosen is cross-entropy, as shown in the following formula.

[0080]

[0081] in y is the probability that the model predicts a sample is a transcription factor, and y is the sample label. If the sample is a positive example, the value is 1, otherwise the value is 0.

[0082] For all protein sequences, the final output probability obtained after passing through the SoftMax function is [p 1j ,p 2j ,…,p nj The predicted category y of protein i is calculated using the following formula. i :

[0083]

[0084] For example, y i =1 indicates that protein sequence i is a transcription factor, otherwise it is a non-transcription factor.

[0085] Step 3: Use the dataset S from Step 1 to train model M, obtaining the trained transcription factor prediction model T. The model training in Step 3 employs stochastic gradient descent, characterized by the use of the Adam optimizer with an initial learning rate of 0.0001. During model training, the batch size is set to 20.

[0086] Step 4: Input the protein sequence to be identified into model T (e.g., ... Figure 1 This allows us to determine whether the sequence is a transcription factor.

[0087] Addressing the limitations of existing methods mentioned in the background section on transcription factor prediction, this invention improves upon existing deep learning methods by utilizing global feature extraction from a Transformer-Encoder and local feature extraction from a multi-scale convolutional neural network to identify transcription factors. This effectively enhances the prediction accuracy of the original deep learning method. Table 1 shows a performance comparison between the model constructed using Transformer-Encoder and multi-scale convolutional neural network and other comparative models. The results in Table 1 demonstrate that the proposed method exhibits the best overall recognition performance.

[0088] Table 1 Comparison of recognition performance of different methods

[0089] Method Acc Prec Recall F1 Example 1 0.95 0.92 0.87 0.89 DeepTfactor 0.94 0.93 0.84 0.88 GRU 0.80 0.67 0.45 0.52

[0090] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0091] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the invention.

Claims

1. A method for identifying transcription factors based on Transformer-Encoder and multi-scale convolutional neural networks, characterized by the following steps: Step 1: Constructing the training set: Collect protein sequences from protein databases, and label each protein sequence as a transcription factor or a non-transcription factor according to the corresponding protein annotation information; preprocess all sequences to obtain the training dataset; Step 1 includes the following sub-steps: 1.1 Select protein sequences from the protein database that do not contain non-standard amino acids, namely B, O, U, and Z, to form dataset S1; 1.2 Remove sequences longer than 1000 from S1, keeping only sequences shorter than or equal to 1000; for protein sequences shorter than 1000, pad with zeros to a length of 1000; finally, obtain the protein sequence dataset S2. 1.3 Based on the GO annotation information of each protein in the protein database, each protein sequence in S2 is labeled with either transcription factor "1" or non-transcription factor "0"; thus, the training dataset is obtained. ;in Represents the first in the dataset A protein sequence; for The tag, {0,1}; for Size; Step 2: Building the Network Structure: Construct a network structure combining Transformer-Encoder and multi-scale convolutional neural networks to build a transcription factor prediction model; whereby Transformer-Encoder is used to obtain the first... protein sequence global features Multi-scale convolutional neural networks are used for... Transcription factor prediction and identification; In step 2, let a protein sequence be... Represents protein sequence The Middle The amino acids at each position are obtained using a Transformer-Encoder. global features The specific steps are as follows: 2.1 Obtaining through embedding operation The embedding vector, and the specific embedding method are as follows: 2.1.1 First, randomize the different amino acid types, then classify them according to their corresponding amino acid types. Each amino acid Embedding generates the corresponding vector; 2.1.2 Positional encoding is used to extract the positional information of amino acids in a protein sequence. Positional encoding identifies the different positions of amino acids in the protein using sine and cosine functions. The positional coding formula for each amino acid is shown below: Where pos represents the position of the amino acid in the protein sequence, d represents the dimension of the embedding vector, and k is a natural number; 2.1.3 Each amino acid The embedding and the corresponding positional encoding are added together to obtain the protein. The embedding vector of the sequence; 2.2 Obtaining the protein sequence After obtaining the embedding vector, it is used as input to the Transformer-Encoder. Its attention mechanism is used to mine the attention score between every two amino acids. The attention score is then cross-multiplied with the embedding vector to obtain the entire protein sequence. global features ; Step 3: Train the prediction model: Use the training set obtained in Step 1 to train the network built in Step 2 to obtain the trained transcription factor prediction model. Step 4: Transcription factor prediction: Using the prediction model obtained in Step 3, predict whether the unknown protein sequence is a transcription factor and output the prediction results.

2. The method according to claim 1, characterized in that: In step 1.3, if the GO annotation of a protein contains the GO term "transcription factor", or contains both "transcription regulation" and "DNA binding" GO terms, then the protein sequence is classified as a transcription factor and assigned a value of "1"; otherwise, the protein sequence is classified as a non-transcription factor and assigned a value of "0".

3. The method according to claim 1, characterized in that: The network structure in step 2 includes a Transformer-Encoder structure composed of cascaded components and a multi-scale convolutional neural network structure. This Transformer-Encoder structure retains only the Encoder part of the Transformer, and is composed of 6 stacked Encoder blocks, each Encoder block containing 12 attention heads; Transformer-Encoder is used to extract global features from the input protein sequence; A multi-scale convolutional neural network consists of four parallel convolutional subnetworks with different one-dimensional convolutional kernels, two fully connected layers, and an output layer. The convolutional layers contain multiple one-dimensional convolutional operations corresponding to convolutional kernels of different sizes to obtain multiple convolutional features of different sizes. The pooling layers pool the multiple convolutional features to obtain features with reduced dimensions. The pooled features are concatenated and fed into the fully connected layers. The prediction results obtained by the fully connected layers are output by the output layer.

4. The method according to claim 3, characterized in that: In step 2, the convolutional sub-network consists of convolutional layers, normalization layers, and so on. Layers and Layer composition; Among them, the first The formula for calculating the output of each sub-network is: in, x This is the input to the convolutional layer; The concatenated output of the four subnetworks is: 。 5. The method according to claim 1, characterized in that: In step 2, the multi-scale convolutional neural network uses one-dimensional convolutional kernels of different sizes to extract features between local protein sequences of different lengths.

6. The method according to claim 5, characterized in that: In step 2, the length of the convolution kernel is the same as the embedding dimension of the protein, the width is set to 4-20, and the size of the convolution kernel is 4, 8, 12, 16 respectively.

7. The method according to claim 1, characterized in that: In step 2, based on The method for predicting and identifying transcription factors is as follows: Based on protein sequence global features After performing convolution operations at different scales, we obtain Local features ,in The local features corresponding to different convolutional kernels are then concatenated and finally input into the fully connected layer. The loss function chosen is cross entropy, as shown in the following formula: in It is the probability that the model predicts a sample is a transcription factor. y This is the sample label. If the sample is a positive example, the value is 1; otherwise, the value is 0. For all protein sequences, the final output probability obtained after applying the SoftMax function is [ The protein content can be calculated using the following formula. i Prediction category : like This indicates a protein sequence. If it is a transcription factor, then it is a non-transcription factor.

Images