Multi-task learning framework for accurate protein secondary structure prediction

Abstract

The invention belongs to the technical field of biological information, and particularly relates to a multi-task learning framework for accurate protein secondary structure prediction. According to the method, a highly targeted autoencoder (HTA) is designed as a drive to encode a sequence irrelevant to homologous protein, so that the three-state and eight-state prediction performance of a protein secondary structure is improved. In highly targeted coding features generated by HTA, interference of homologous protein can be reduced, and the feature space utilization rate can be improved. Secondly, under the guidance of biological knowledge, the invention provides a protein secondary structure prediction model (PSSP-MTL) based on a multi-task learning strategy; the PSSP-MTL model can flexibly balance global and multi-scale local features, and full information exchange among multiple tasks is realized based on a multi-gate hybrid expert structure.

Description

Technical Field

[0001] The present invention belongs to the field of bioinformatics, and in particular relates to a multi-task learning framework for accurate protein secondary structure prediction. Background Art

[0002] Protein is the main bearer of life activities and occupies an important position in organisms. Protein structure is closely related to protein function. Proteins that perform similar functions in organisms usually have similar structures. The study of protein structure helps to understand the process of protein performing biological functions, and is of great significance in genetic engineering, protein engineering, drug development and disease diagnosis. The structure of protein is divided into four levels. The task of protein secondary structure prediction refers to predicting the secondary structure category of each residue from the primary structure of the protein, that is, the amino acid residue sequence. The secondary structure plays a connecting role in the above structure. Its accurate prediction not only helps to understand the initial folding rules of protein sequences, but can also be used for the prediction of tertiary structure, interaction, domain and category of proteins.

[0003] Each amino acid residue in the primary structure of a protein is represented by its single-letter abbreviation (A, D, E, F, G, H, I, K, L, M, N, P, Q, R, S, T, V, W, Y and X), where A to Y represent the 20 standard amino acids and X represents a non-standard amino acid. The DSSP program defines eight categories of protein secondary structure: α-helix (H), 3 10 -Helix (G), π-helix (I), extended chain (E), β-bridge (B), turn (T), bend (S) and other secondary structures (L). Usually, these eight categories are divided into three categories: H, G and I are divided into helices (H), E and B are divided into strands (E), and T, S and L are divided into coils (C). Therefore, the prediction of protein secondary structure is divided into coarse-grained three-state prediction and fine-grained eight-state prediction. Recent studies have shown that the accuracy limit of three-state prediction is about 92%, and the accuracy limit of eight-state prediction is 84%-87%, which means that the performance of protein secondary structure prediction still has a lot of room for improvement.

[0004] Existing traditional experimental methods for predicting protein secondary structure include nuclear magnetic resonance spectroscopy, X-ray diffraction, and circular dichroism. Although relatively accurate results can be obtained through experimental methods, the disadvantages of these methods, such as being time-consuming and costly, cannot be ignored. The scale of protein sequence data is growing exponentially, and it is challenging to narrow the scale gap between sequence and structural information by relying solely on experimental methods. Therefore, it is urgent to develop fast and accurate computational methods to predict protein secondary structure.

[0005] The encoding representation of amino acid sequences is a key issue in computational methods. Sequences need to be encoded to generate features that can be understood by computers in order to further extract deep information and make predictions. Currently, two main encoding features are used: one is the one-hot encoding feature that represents the amino acid type information at each position, and the other is the profile feature that contains the position-specific scoring matrix (PSSM) and the hidden Markov model (HMM) to represent sequence evolution information. However, the one-hot encoding features are sparse, and the profile features are obtained through multiple sequence alignment (MSA) encoding, which cannot fairly represent sequences with different numbers of homologous proteins. In addition, the biomolecular information contained in the features is too extensive, so that the feature space utilization rate of knowledge related to protein secondary structure prediction is low.

[0006] In recent years, many pre-trained protein language models based on massive training data have been proposed, aiming to extract deep embedding features that can be effectively used by downstream tasks. Unlike contour features, the quality of embedded features is independent of protein homology. The embedded features extracted by pre-trained protein language models have shown excellent results in tasks such as protein design and binding site prediction, and have achieved the best prediction performance for protein secondary structure prediction tasks. In short, it is worth developing a protein secondary structure prediction method that incorporates embedded features to weaken the interference of the number of homologous proteins on model performance.

[0007] Statistical computational methods such as Chou-Fasman and GOR only analyze the shallow features of amino acid residues, and their prediction performance is very limited. At present, machine learning methods based on big data analysis have become a research hotspot in the field of protein secondary structure prediction, among which methods based on deep neural networks have achieved the most advanced prediction performance. The relationship between neighboring amino acid residues and the relationship between long-range amino acid residues may affect the secondary structure category. For example, helices are mainly composed of neighboring residues, while folding involves long-range residues. Therefore, deep neural network methods mainly use convolutional neural networks (CNNs) and their improvements and recurrent neural networks (RNNs) and their improvements to capture local and global dependency information, respectively. Secondly, many deep multi-scale feature extraction models improve prediction performance by obtaining sequence information from different receptive fields. However, the simple stacking of diversified deep networks requires a lot of computing resources, and ordinary learning strategies are difficult to improve the learning ability of the model. In addition, stacked networks and multi-scale architectures without knowledge guidance cannot balance local and global features well. This illustrates that it is necessary to design models based on knowledge and use an advanced learning strategy to improve prediction performance.

[0008] In summary, accurate protein secondary structure prediction plays a vital role in biopharmaceuticals and disease diagnosis. Current prediction methods are mainly based on multiple sequence alignment encoding and diversified networks working together. However, the existing encoding methods lead to low feature space utilization of protein secondary structure prediction-related knowledge, and the encoding quality decreases with the decrease of homologous proteins. In addition, the performance of simple stacked networks is greatly limited by complex feature extraction capabilities and learning strategies. Summary of the invention

[0009] The present invention proposes a multi-task learning framework for accurate protein secondary structure prediction, which solves the above problems. The technical problem solved by the present invention is that in the prior art, when predicting protein secondary structure, the existing encoding method leads to low feature space utilization of protein secondary structure prediction related knowledge, the encoding quality decreases as the number of homologous proteins decreases, and the performance of simple stacked networks is greatly limited by complex feature extraction capabilities and learning strategies.

[0010] The technical solution of the present invention is achieved in this way:

[0011] A multi-task learning framework for accurate protein secondary structure prediction, named MHTAPred-SS, which includes the HTA model and the PSSP-MTL model;

[0012] The HTA model includes a DY-CNN module and a BiLSTM module. The DY-CNN module is used as the encoder part of HTA to generate highly targeted coding features for each sequence. It is highly targeted to sample information. The BiLSTM module uses a BiLSTM network to learn the bidirectional global features of the sequence and outputs the reconstruction results of the primary structure of the protein through the output layer.

[0013] The PSSP-MTL model includes a multi-scale residual convolution module, a parameter sharing module and a combined time series module. The multi-scale residual convolution module is used to extract the multi-scale local features of the sequence. The parameter sharing module and the combined time series module constitute a complete MMoE structure, which is used to supplement the single-task features through information exchange between tasks and extract deep global features.

[0014] Optionally, the encoder of the HTA model sets a smaller C out , in order to improve the utilization rate of the feature space of knowledge, the specific calculation process is as follows:

[0015] MAT ω =Softmax(Conv(GAP(X)));

[0016] MAT ω=[[ω1],[ω2],[ω3],…[ω B ]];

[0017]

[0018]

[0019] Among them, GAP is the global average pooling operation, Conv is the one-dimensional convolution operation, BN is the batch normalization operation, MAT is ω It is the matrix composed of the weights of each filter, MAT FLT is the matrix composed of the parameters of each filter, is the transposed matrix of the matrix composed of each filter parameter, ω i is the weight of the ith filter, is the internal parameter of the i-th filter, n is the number of parameters in each filter, Comb_FLT is the parameter of the Combined Filter, is element-wise addition, Conv Comb_FLT It is a convolution operation using Combined Filter.

[0020] Optionally, the BiLSTM network contains forward and backward connections of LSTM cells. Each LSTM cell is equipped with three gating mechanisms: input gate, forget gate, and output gate. The calculation process of a sequence of length L in the LSTM cell is as follows:

[0021] f t =σ(W f [h t―1 ,x t ]+b f );

[0022] i t =σ(W i [h t―1 ,x t ]+b i );

[0023] o t =σ(W o [h t―1 ,x t ]+b o );

[0024] C t =f t ☉C t-1 +i t ⊙tanh(W C [h t-1 , x t ]+bC );

[0025] h t =o t ⊙tanh(C t );

[0026] Where σ is the sigmoid function, tanh is the hyperbolic tangent function, ⊙ is the element-wise multiplication, t∈{1,2,…,L}, x t 、h t and C t are the input features, output features and cell states of the LSTM cell at the tth time step, W and b are the weights and bias parameters of the corresponding gating mechanism in the LSTM cell, respectively. t 、i t and t are the outputs of the forget gate, input gate, and output gate, respectively, and b f 、b i 、b o are the bias parameters of the forget gate, input gate, and output gate, respectively, and b C Bias parameter representing the cell state, W o , W C Represent the weight parameters of the output gate and cell state respectively.

[0027] Optionally, the multi-scale residual convolution module contains multiple single-scale convolutions and residual connections. Based on the knowledge of the length pattern of the secondary structure fragment, the input feature tensor is convolved in one dimension using convolution kernels of different scales. The embedded features extracted by ESM-2 are concatenated with the multi-scale local features to generate the concatenated feature tensor as follows:

[0028]

[0029] Among them, s is the number of scales, F i is the output feature of the i-th single-scale convolution, F s represents the features transmitted by the sth residual connection, is element-by-element addition, Res s is the feature transmitted by the i-th residual connection, F embed is the embedding feature, Concat is the concatenation operation, and F C is the concatenated feature tensor.

[0030] Optionally, the parameter sharing module uses the MMoE structure to realize information exchange between tasks. As the bottom part of the MMoE structure, the parameter sharing module can realize soft sharing of underlying parameters. The parameter sharing module comprehensively considers the output features of all expert networks and finally outputs fusion features;

[0031] Each residual connection uses a Conv1D and BN to process the concatenated feature tensor as follows:

[0032]

[0033] Among them, j∈{1,2,…N T}, is the feature transmitted by the jth residual connection;

[0034] Each gating network consists of Conv1D, BN, LeakyReLU activation function and Softmax operation, and the final output is the weight assigned to the output features of the expert network as follows:

[0035]

[0036] in, is the output feature tensor of the jth gating network, is the weight tensor assigned by the j-th gating network to the output features of the k-th expert network, is the weight value assigned by the j-th gating network to the output feature of the k-th expert network at the t-th position in the sequence, and L is the length of the sequence;

[0037] The LeakyReLU activation function is used to activate the obtained features. The activated features are the fusion features. The calculation process is as follows:

[0038]

[0039]

[0040] Where k∈1,2,…E,EX k is the sequential operation inside the k-th expert network, is the output feature tensor of the k-th expert network, is the fused feature tensor extracted by the parameter sharing module for the jth task, and CH is The number of channels, They respectively represent the features transmitted by the j-th residual connection and the weight tensor assigned by the j-th gating network to the output features of the k-th expert network.

[0041] Optionally, the combined time series module is the top part of the MMoE structure, which cascades a time series convolutional neural network and a bidirectional gated recurrent unit to process sequence data of length L;

[0042] The GRU cell contains two gating mechanisms, namely the reset gate and the update gate. The calculation process is as follows:

[0043] r t=σ(W r [h t―1 ,x t ]+b r );

[0044] z t =σ(W z [h t―1 ,x t ]+b z );

[0045] h t =z t ⊙h t―1 +(1―z t )⊙tanh(W h [r t ⊙h t―1 ,x t ]+b h );

[0046] where t∈{1,2,…,L}, x t and h t are the input and output features of the BiGRU cell at the tth time step, W and b are the weight and bias parameters of the corresponding gating mechanism of the GRU cell, r t and z t are the outputs of the reset gate and the update gate, respectively, r 、b z 、b h Represent the bias parameters of reset gate, update gate and cell features respectively.

[0047] After adopting the above technical solution, the beneficial effects of the present invention are:

[0048] The MHTAPred-SS framework proposed in the present invention is used for accurate prediction of three and eight states of protein secondary structure. Specifically, MHTAPred-SS includes a strongly targeted autoencoder (HTA) and a protein secondary structure prediction (PSSP-MTL) model based on multi-task learning. The input of HTA is the sequence features after masking of random secondary structure fragments. The encoder sets an attention mechanism on the sample dimension to generate specific network parameters for each sample. Secondly, the long short-term memory (LSTM) network, which is good at extracting global features, is used as a decoder to restore the primary structure of the sequence. The HTA encoder outputs the strongly targeted encoding features of the sequence, which is independent of protein homology and has a very high feature space utilization. In addition, the PSSP-MTL model is designed based on the knowledge of task relevance and secondary structure fragment length patterns, and is committed to balancing local and global dependency information. The PSSP-MTL model uses a multi-gate mixed expert (MMoE) structure to softly share parameters based on a multi-task learning (MTL) strategy, thereby performing information exchange between tasks. Then, the role of auxiliary tasks can be fully utilized to promote more accurate protein secondary structure prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0050] Figure 1 This is the workflow of MHTAPred-SS;

[0051] Figure 2 It is the operating mechanism of the HTA model;

[0052] Figure 3 It is the length law of protein secondary structure;

[0053] Figure 4 is the prediction performance of the model when using residual convolution modules of different scales;

[0054] Figure 5 is the prediction performance of protein secondary structure under different learning strategies;

[0055] Figure 6 It is a visualization of protein secondary structure prediction results using different methods. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] The embodiments of the present application disclose a multi-task learning framework for accurate protein secondary structure prediction.

[0058] Example

[0059] according to Figures 1 to 6 Shown, a multi-task learning framework for accurate protein secondary structure prediction.

[0060] 1. Introduction to related work

[0061] Methods based on artificial intelligence algorithms can avoid time-consuming and costly experimental processes and improve prediction performance, and have become the mainstream research direction in the field of protein secondary structure prediction. They can be roughly divided into traditional machine learning methods, deep learning methods, and methods based on pre-trained protein language models.

[0062] Traditional machine learning methods such as hidden Markov, support vector machine, nearest neighbor and neural network have been widely used in the field of protein secondary structure prediction, which are trained using rich sequence features such as PSS and HMM profile features, sequence context features and bigram features in the field of natural language processing. However, it is challenging to use traditional machine learning models to capture the complex relationship between amino acid residue features and secondary structure categories.

[0063] The expansion of protein databases and the improvement of equipment computing power have made it possible to use deep learning methods with strong learning capabilities to predict protein secondary structures. For example, DNSS, SPIDER, and SSREDNs are all based on deep neural networks to predict protein secondary structures. Since CNN and RNN have advantages in extracting sequence context features, deep learning models based on the above two neural networks are widely used in the field of protein secondary structure prediction. For CNN-based prediction methods, DeepCNF combines CNN with conditional neural fields and uses PSSM to predict protein secondary structures. CNNHPSS improves traditional CNN into a high-speed convolutional neural network, which saves computing resources and improves prediction accuracy. IGPRED combines CNN with graph convolutional networks, but it does not extract global features of sequences. For RNN-based prediction methods, SPIDER3 captures global features based on long short-term memory (LSTM) networks and bidirectional recurrent neural networks (BRNN). SPOT-1D uses an integrated model of LSTM-BRNN and ResNet for protein structure prediction. Secondly, DCRNN, CNN_BIGRU, DeepACLSTM and OPUS-TASS use CNN and RNN together to learn local and global features of the sequence at the same time. Multi-S3P builds three independent models based on CNN, BiLSTM and self-attention mechanism to predict different types of protein secondary structures. DeepCNN, MUFOLD-SS and SAINT are based on multi-scale convolution to extract multi-scale features. In addition, CFLM uses a transfer learning model based on residual dense network to improve the prediction accuracy. Recently, some methods have focused on sequence feature selection and used the final feature subset to predict the secondary structure of proteins.

[0064] Pre-trained protein language models can generate embedding features that are independent of the number of homologous proteins. Therefore, many methods based on pre-trained protein language models have been proposed to achieve as close a prediction effect as possible for different sequences. DML_SS uses the pre-trained protein language model ProtT5-XL-U50 to generate embedding features and uses deep metric learning to predict the model, avoiding the time-consuming search process of the K nearest neighbor algorithm. AttSec also uses the pre-trained protein language model ProtT5-XL-U50 and the attention map obtained by the Transformer encoder to predict protein secondary structure. NetSurfP-3.0 uses the embedding features extracted by the pre-trained protein language model ESM-1b to replace the sequence features used by NetSurfP-2.0 for protein secondary structure prediction. SPOT-1D-LM combines one-hot encoding features with embedding features extracted from two pre-trained protein language models, ProtTrans and ESM-1b, to achieve performance comparable to that of protein secondary structure prediction methods based on contour features. In addition, LIFT_SS focuses on fine-tuning the pre-trained protein language model, and uses models based on seven different fine-tuning methods for protein secondary structure prediction tasks.

[0065] In short, compared with traditional machine learning methods, deep learning methods can efficiently process large amounts of high-dimensional data and have more powerful modeling and expression capabilities, thereby obtaining more accurate prediction results. The prediction method based on the pre-trained protein language model has achieved the current advanced performance, but it mainly uses only embedded features, without fusion with features such as contours and physicochemical properties. Considering the advantages of multi-feature fusion in the comprehensive expression of protein knowledge, the present invention fuses six features of different types and uses HTA to improve the utilization of feature space. In addition, the present invention designs a network architecture based on knowledge, and uses a multi-task learning strategy to improve prediction performance, breaking through the constraints of a simple stacked network architecture on the ability to extract complex features.

[0066] 2. Methods

[0067] The workflow of MHTAPred-SS proposed in the present invention is as follows Figure 1As shown in the figure, it contains four key parts: (1) Data acquisition: Sequence samples are screened from the protein database according to homology, resolution and other conditions to obtain two sets of data sets for model training and testing. Each sample in the data set consists of a protein sequence and a secondary structure label of the amino acid residues at each position of the sequence; (2) Multi-feature fusion: Five methods are used to obtain six features, including one-hot encoding features, PSSM features, HMM features, physical and chemical property features, embedded features extracted from the pre-trained protein language model, and strong targeted encoding features; among them, the one-hot encoding feature is used as the input of HTA to extract strong targeted encoding features, and the other features are the input features of PSSM-MTL, and PSSM and HMM are both from MSA encoding. In particular, the mutual fusion of the five features except the embedded feature is used as the input of the PSSP-MTL model, and the embedded feature is fused with other features in the subsequent process; (3) PSSP-MTL model: The PSSP-MTL model includes three modules: multi-scale residual convolution module, parameter sharing module and combined time series module. The multi-scale residual convolution module is used to extract the multi-scale local features of the sequence. The parameter sharing module and the combined time series module constitute a complete MMoE structure, which aims to supplement the single-task features through information exchange between tasks and extract deep global features; (4) Output predictor: output the prediction possibility for each category and obtain the final prediction result. The auxiliary task used in the present invention is the relative solvent accessibility (RSA) prediction task, which simultaneously outputs the secondary structure and RSA prediction category of each amino acid residue in the sequence.

[0068] 2.1HTA model

[0069] The operating mechanism of the HTA model is as follows Figure 2 As shown, the training set and test set used by HTA contain N tr and N te protein sequences. In the model training phase, a residue segment with a single secondary structure type is randomly selected for each sequence to be masked, and the one-hot features of the masked sequence are input into HTA. The output of the model is the reconstruction result of the primary structure. In the testing phase, HTA uses the unmasked one-hot features as input to reconstruct the primary structure of the protein. In the HTA model, the dynamic convolutional neural network (DY-CNN) module is used as an encoder and the BiLSTM module is used as a decoder. The HTA model is used as a driver to map the one-hot features into low-dimensional, highly targeted encoding features, improve the focus of the features on the protein secondary structure prediction task, and improve the feature space utilization of the knowledge related to protein secondary structure prediction.

[0070] 2.1.1 DY-CNN Module

[0071] The DY-CNN module is built based on dynamic convolution technology. Unlike traditional convolution technology, it sets an attention mechanism on the sample dimension and can assign specific filter parameters to each sample to comprehensively consider the context information requirements of different sequences. The DY-CNN module designed in this invention includes A residual dynamic one-dimensional convolution (DyConv1D) blocks. In each residual DyConv1D block, the three-dimensional input feature The global average pooling result of is passed through a one-dimensional convolution layer (Conv1D) and a Softmax operation to obtain the weights corresponding to B filters. The weighted addition of the B filters can generate a combined filter. Please note that N bs is the batch size, C in is the number of channels of the input feature, L max is the longest sequence length in the same batch. The features obtained by convolving the input features with the combined filter are batch normalized (BN) and then added element by element with the features transmitted by the residual connection. Then the LeakyReLU activation function is used for nonlinear mapping and the output features are obtained. That is, a strong targeted coding feature, where C out is the number of channels of the output feature. In particular, the present invention sets a smaller C for the encoder of the HTA model out , in order to improve the utilization rate of the feature space of knowledge. The specific calculation process is as follows:

[0072] MAT ω =Softmax(Conv(GAP(X))) (1)

[0073] MAT ω =[[ω1],[ω2],[ω3],…[ω B ]] (2)

[0074]

[0075]

[0076] Among them, GAP is the global average pooling operation, Conv is the one-dimensional convolution operation, BN is the batch normalization operation, MAT is ω It is the matrix composed of the weights of each filter, MAT FLT is the matrix composed of the parameters of each filter, is the transposed matrix of the matrix composed of each filter parameter, ω i is the weight of the ith filter, is the internal parameter of the i-th filter, n is the number of parameters in each filter, Comb_FLT is the parameter of the Combined Filter, is element-wise addition, Conv Comb_FLT It is a convolution operation using Combined Filter.

[0077] 2.1.2 BiLSTM Module

[0078] The BiLSTM module uses a BiLSTM network to learn the bidirectional global features of the sequence and outputs the reconstruction results of the primary structure of the protein through the output layer. The output layer contains two linear layers, each of which is connected to a LeakyReLU activation function.

[0079] The BiLSTM network contains forward and backward connections of LSTM cells. Each LSTM cell has three gating mechanisms: input gate, forget gate, and output gate. The calculation process of a sequence of length L in the LSTM cell is as follows:

[0080] f t =σ(W f [h t-1 ,x t ]+b f ) (6)

[0081] i t =σ(W i [h t―1 ,x t ]+b i ) (7)

[0082] o t =σ(W o [h t―1 ,x t ]+b o ) (8)

[0083] C t =f t ⊙C t―1 +i t ⊙tanh(W C [h t―1 ,x t ]+b C ) (9)

[0084] h t =o t ⊙tanh(C t ) (10)

[0085] Where σ is the sigmoid function, tanh is the hyperbolic tangent function, ⊙ is the element-wise multiplication, t∈{1,2,…,L}, x t 、h t and C tare the input features, output features and cell states of the LSTM cell at the tth time step, W and b are the weights and bias parameters of the corresponding gating mechanism in the LSTM cell, respectively. t 、i t and t are the outputs of the forget gate, input gate, and output gate, respectively, and b f 、b i 、b o are the bias parameters of the forget gate, input gate, and output gate, respectively, and b C Bias parameter representing the cell state, W o , W C Represent the weight parameters of the output gate and cell state respectively.

[0086] Since HTA can generate specific Combined Filters for different sequence samples, it is highly targeted at sample information. In addition, the masked residue fragments have a single secondary structure category, and HTA can learn information related to protein secondary structure, so it is highly targeted at protein secondary structure.

[0087] 2.2 PSSP-MTL model

[0088] like Figure 1 As shown, the PSSP-MTL model is the main part of MHTAPred-SS for predicting protein secondary structure, which includes three modules. First, the concatenated features of the mixed features and the strongly targeted coding features are input into the multi-scale residual convolution module to extract the multi-scale local features of the sequence. Since the embedded features extracted by ESM-2 contain information between related residues, the multi-scale residual convolution module has little significance for acting on the embedded features. Therefore, the embedded features are directly spliced ​​with the multi-scale local features and input into the MMoE structure composed of the parameter sharing module and the combined time series module, and then the secondary structure and RSA of the protein are predicted simultaneously using a multi-task learning strategy. In particular, the present invention introduces residual connections in each module of the PSSP-MTL model, which can effectively alleviate the problems of gradient vanishing and gradient explosion in the model training process and reduce the loss of information.

[0089] 2.2.1 Multi-scale residual convolution module

[0090] like Figure 1As shown in the figure, the multi-scale residual convolution module contains multiple single-scale convolutions and residual connections, which aims to perform one-dimensional convolution on the input feature tensor using convolution kernels of different scales based on the knowledge of the secondary structure fragment length pattern to extract the multi-scale local features of the sequence. Each single-scale convolution contains α Conv1D and BN, and each residual connection contains one Conv1D and BN. The output features of different single-scale convolutions and the features transmitted by the residual connection are added element by element and then spliced ​​after activation using the LeakyReLU activation function to obtain the multi-scale local features of the sequence. The embedded features extracted by ESM-2 (a pre-trained protein language model) are spliced ​​with the multi-scale local features to generate the spliced ​​feature tensor as follows:

[0091]

[0092] Among them, s is the number of scales, F i is the output feature of the i-th single-scale convolution, F s represents the features transmitted by the sth residual connection, is element-by-element addition, Res s is the feature transmitted by the i-th residual connection, F embed is the embedding feature, Concat is the concatenation operation, and F C is the concatenated feature tensor.

[0093] 2.2.2 Parameter Sharing Module

[0094] Multi-task learning aims to complete multiple tasks using one model and achieve better generalization performance through information exchange between tasks. It is mainly divided into hard parameter sharing models such as MT-DNN and soft parameter sharing models such as MMoE. The former refers to the complete sharing of the underlying parameters of the model by different tasks, which is suitable for situations where the correlation between tasks is strong. The latter refers to the sharing of some underlying parameters between different tasks, which does not require high correlation between tasks and can better adapt to the characteristics of different tasks. Based on the knowledge of task correlation, the present invention adopts the MMoE structure (multi-gated expert hybrid structure) to realize information exchange between tasks.

[0095] The MMoE structure consists of two parts, the top and the bottom. The bottom part is responsible for filtering out the information required for each task, and the top part performs specific analysis for each task. The parameter sharing module, as the bottom part of the MMoE structure, can realize the soft sharing of the underlying parameters. It consists of three parts: residual connection, gating network and expert network. The number of residual connections and gating networks is the same as the number of tasks, and the number of expert networks is a hyperparameter that needs to be adjusted. Assume that the number of tasks is N T , the number of expert networks is E.

[0096] The input of the parameter sharing module is the concatenated feature tensor of multi-scale local features and embedded features, which is simultaneously input into the residual connection, gated network and expert network. The parameter sharing module comprehensively considers the output features of all expert networks and finally outputs the fusion features.

[0097] Each residual connection uses a Conv1D and BN to process the concatenated feature tensor as follows:

[0098]

[0099] Among them, j∈{1,2,…N T}, is the feature transmitted by the jth residual connection.

[0100] Each gating network consists of Conv1D, BN, LeakyReLU activation function and Softmax operation, and the final output is the weight assigned to the output features of the expert network as follows:

[0101]

[0102] in, is the output feature tensor of the jth gating network, is the weight tensor assigned by the j-th gating network to the output features of the k-th expert network, is the weight value assigned by the j-th gating network to the output feature of the k-th expert network at the t-th position in the sequence, and L is the length of the sequence.

[0103] Each expert network contains a combination of β Conv1D and BN. The output features of the expert network are weighted and added, and then element-by-element with the features transmitted by the residual connection.

[0104] The present invention uses the LeakyReLU activation function to activate the obtained features. The activated features are fused features, and the calculation process is as follows:

[0105]

[0106] Where k∈1,2,…E,EX k It is the sequential operation inside the k-th expert network. is the output feature tensor of the k-th expert network, is the fused feature tensor extracted by the parameter sharing module for the jth task, and CH is The number of channels, They respectively represent the features transmitted by the j-th residual connection and the weight tensor assigned by the j-th gating network to the output features of the k-th expert network.

[0107] 2.2.3 Combined time series module

[0108] The combined time series module is the top part of the MMoE structure, which cascades the time series convolutional neural network (TCN) and the bidirectional gated recurrent unit (BiGRU) to process sequence data of length L. The TCN unit consists of M residual blocks, and the i-th residual block contains two layers with a hole rate of 2 (i―1) One-dimensional dilated causal convolution, each layer of one-dimensional dilated causal convolution is connected to a LeakyReLU activation function and a dropout layer.

[0109] BiGRU unit uses BiGRU network to extract global features of the sequence. Similar to BiLSTM network, BiGRU network contains forward and reverse connections of GRU (gated recurrent unit) cells, but it has fewer parameters than BiLSTM network and requires less computing resources and memory. GRU cells contain two gating mechanisms, namely reset gate and update gate, and their calculation process is as follows:

[0110] r t =σ(W r [h t―1 ,x t ]+b r ) (17)

[0111] z t =σ(W z [h t―1 ,x t ]+b z ) (18)

[0112] h t =z t ⊙h t―1 +(1―z t )⊙tanh(W h [r t ⊙h t―1 ,x t ]+b h ) (19)

[0113] where t∈{1,2,…,L}, x t and h t are the input and output features of the BiGRU cell at the tth time step, W and b are the weight and bias parameters of the corresponding gating mechanism of the GRU cell, r t and z t are the outputs of the reset gate and the update gate, respectively, r 、b z 、b h Represent the bias parameters of reset gate, update gate and cell features respectively.

[0114] TCN can process sequence information in parallel, greatly improving data processing efficiency without increasing model parameters. In addition, the residual connection inside TCN enhances the robustness of the model, making the model less prone to gradient vanishing or exploding problems. BiGRU unit can enhance the learning ability of the model, which is conducive to capturing complex global features, and can supplement the unidirectional features extracted by TCN unit based on internal bidirectional connections. In short, the combination of two time series models with different advantages can effectively avoid the limitations of a single model to fully extract the global features of the sequence.

[0115] 3. Experiment

[0116] 3.1 Dataset

[0117] In order to make a fair comparison with other models, the PSSP-MTL model uses the same two sets of data sets as DML_SS. The detailed description of the six independent test sets is shown in Table 1. For convenience, this paper refers to these two sets of data sets as DS1 and DS2, respectively. DS1 contains four test sets: CASP12, CASP13, CASP14, and CB513. CASP12-14 are from the CASP competition. Sequences with homology greater than 25% in each test set are deleted. CB513 is a test set constructed by Cuff and Barton and widely used in the field of protein secondary structure prediction. The training set and validation set of DS1 are compiled using the PISCES server

[64] according to the sequence homology less than 25% and resolution less than and R-value less than 1.0. Sequences without secondary structure information or with a length of less than 50 or greater than 800 were deleted from this data set, and 12,510 sequences were retained, which are denoted as CullPDB in this article. Sequences with a homology greater than 25% relative to each test set in this data set were deleted, and finally there were sets containing 12,376, 12,453, 12,504 and 12,012 sequences for CASP12, CASP13, CASP14 and CB513, respectively. 512 sequences were randomly selected from the intersection of the four sets as the validation set, and the remaining sequences of each set were used as the training set. DS2 contains two test sets, TEST2016 and TEST2018, and uses the default training set containing 10,029 sequences and the validation set containing 983 sequences of these two test sets, denoted as SPOT_1D_Train and SPOT_1D_Valid, respectively.

[0118] The first set of data sets for the HTA model uses the intersection of the four training sets in DS1 as the training set, and sets the same validation set and test set as DS1. The second set of data sets for the HTA model is DS2. The highly targeted coding features extracted by the HTA model using the first and second sets of data sets are fused with the hybrid features used for the PSSP-MTL model using DS1 and DS2, respectively.

[0119] Table 1 Detailed description of the six independent test sets

[0120]

[0121]

[0122] 3.1.1 Protein secondary structure length patterns

[0123] The present invention counts the number of secondary structure fragments of different interval lengths in the used data set, and learns the knowledge of the secondary structure fragment length pattern from the visualization results. Figure 3 As shown, the length of the secondary structure fragments is mostly between 1 and 11. Therefore, the present invention uses filters of three scales of 1, 5, and 9 for multi-scale convolution. Since a few secondary structure fragments are long and different secondary structure fragments may affect each other, the present invention designs a combined time series module to extract the global features of the sequence so that the global features and multi-scale local features can be balanced.

[0124] 3.1.2 Correlation between secondary structure and RSA

[0125] The accessible surface area (ASA) refers to the surface area of ​​the residue in contact with the solvent, and the RSA of each residue is defined as the ratio of its ASA to the maximum ASA of this residue currently found. The present invention analyzes the correlation between the secondary structure of the protein and RSA, and uses this as the knowledge basis for using a multi-task learning strategy.

[0126] Cramer's V correlation coefficient can evaluate the correlation between discrete variables. The contingency table of secondary structure and RSA is shown in Table 2. The process of calculating Cramer's V correlation coefficient based on the contingency table is as follows:

[0127]

[0128] Among them, sum i is the sum of the data in the i-th row, sum j is the sum of the data in the jth column, sum is the sum of all data, E ij is the expected value of the data in row i and column j, O ijis the true value of the data in the i-th row and j-th column, m is the number of rows in the contingency table, and n is the number of columns in the contingency table (the above calculations do not include Sum rows and Sum columns).

[0129] The Cramer's V correlation coefficient between the secondary structure and RSA was calculated to be 0.2264, so it can be considered that there is a weak correlation between the two, and it is suitable to use the MMoE structure to achieve soft sharing of the underlying parameters.

[0130] Table 2 Contingency table of secondary structure and RSA

[0131]

[0132] 3.2 Feature Representation

[0133] In order to comprehensively express protein knowledge, the present invention uses six protein sequence features: PSSM profile features, HMM profile features, physicochemical properties, embedded features, one-hot features and strongly targeted coding features. Assuming that the protein sequence length is L, the PSSM and HMM feature sizes are L×20 and L×30, respectively. Each PSSM feature is generated by two iterative searches of the Uniref50 database using the PSI-BLAST program, with an E value of 0.001. Each HMM feature is generated by four iterations using the HHblits v3.3.0 program based on the default parameters of the uniprot20_2013_03 database. It should be noted that the profile features generated based on MSA coding contain the evolutionary information of the sequence, and the profile features of sequences with fewer homologous proteins have lower quality. Secondly, the present invention uses four features that are not related to homologous sequences: physicochemical properties, embedded features, one-hot features and strongly targeted coding features. The physicochemical properties of the sequence are represented by seven characteristics of the amino acid residues at each position, and an embedded feature of size L×1280 is generated using ESM-2 trained on the UniRef database. In addition, a one-hot feature of size L×21 is used as the input of HTA, and the output feature of the HTA encoder is a strongly targeted encoding feature of the sequence, which has a size of L×4. Since the RSA information extracted by the DSSP program is incomplete, the ASAquick program is used to determine the RSA label of each sequence. The present invention assigns a label of 0 to amino acid residues with an RSA less than or equal to 0.15, and a label of 1 to amino acid residues with an RSA greater than 0.15.

[0134] 3.3 Evaluation indicators

[0135] The present invention uses the accuracy index (Q3 and Q8) calculated for a single amino acid residue and the segment overlap (SOV) index (SOV3 and SOV8) calculated for an amino acid residue segment to evaluate the prediction performance of the model. The calculation process of the accuracy value calculated for a single residue is as follows:

[0136]

[0137] Where SS is the set of secondary structure categories, |SS| is the number of elements in SS (3 or 8), and T i is the number of residues with secondary structure class i that are predicted correctly, and sum is the total number of residues.

[0138] The SOV index is calculated for amino acid residue fragments. For the subsequent tertiary structure prediction task, correctly predicting the secondary structure fragment is more meaningful than correctly predicting the secondary structure category of a single residue. The calculation process of the SOV value is as follows:

[0139]

[0140] Where SS is the set of secondary structure categories, |SS| is the number of elements in SS (3 or 8), s(i) is the set of fragment pairs for which the experimental results overlap with the predicted results for secondary structure category i, len(s1) is the length of s1, and len(s2) is the length of s2. is the set of fragments for which the experimental results and the predicted results do not overlap for secondary structure category i, minov(s1,s2) is the number of residues whose secondary structures overlap with those of s1 and s2, and maxov(s1,s2) is the total number of amino acid residue positions in s1 and s2.

[0141] 3.4 Experimental Details

[0142] The present invention uses NVIDIA Corporation Device 2684 GPU and 12th Gen Intel(R) Core(TM) i7-12700F CPU, and conducts experiments based on the PyTorch framework. The cross entropy loss function and the AdamW optimizer with a weight decay of 0.05 are used. The batch size is set to 32, and all sequences are padded to the length of the longest sequence in the batch. The initial learning rate is set to 0.0005, and the learning rate decays to half of the previous one every five epochs of training.

[0143] 3.5 Ablation Experiment

[0144] In order to explore the effects of model hyperparameters, PSSP-MTL model components, feature combinations and learning strategies on prediction performance, the present invention has conducted extensive experiments on two validation sets and test sets (CB513 and TEST2016) with the largest number of sequence entries and residues. In the present invention, the first validation set and test set are respectively denoted as validation set 1 and test set 1, and the second validation set and test set are respectively denoted as validation set 2 and test set 2.

[0145] 3.5.1 Impact of Model Hyperparameters

[0146] (1) Impact of single-scale and multi-scale residual convolution modules

[0147] like Figure 3 As shown in Table 3, the lengths of protein secondary structure fragments are mostly between 1 and 11. Therefore, the present invention constructs a single-scale residual convolution module based on six scales of 1, 3, 5, 7, 9 and 11 to replace the multi-scale residual convolution module in the PSSP-MTL model. The experimental results are shown in Table 3. The experimental results of the three scales of 1, 5 and 9 almost cover the best prediction performance of the single-scale model on all data sets in the table, so the present invention uses these three scales to construct a multi-scale residual convolution module. Figure 4 As shown in the figure, the prediction performance of the model based on the multi-scale residual convolution module on all datasets in the figure is better than the average prediction performance of the six single-scale models. The reason may be that the multi-scale residual convolution module is designed based on the knowledge of the length pattern of the secondary structure fragment, which can learn the local features related to the secondary structure at multiple important receptive field scales.

[0148] Table 3 Prediction performance of the model at different convolution scales

[0149] ((1,5,9) means constructing a multi-scale residual convolution module with three scales of 1, 5 and 9.

[0150] The underlined data indicate the best prediction performance at a single scale, and the best prediction performance at all scales is in bold.

[0151] The unit of Q3 and Q8 is "%")

[0152]

[0153]

[0154] (2) The impact of the number of expert networks in the parameter sharing module E

[0155] The present invention changes the number of expert networks of E in the range of {1, 2, 3, 4}, and the experimental results are shown in Table 4. When E=1, the prediction performance of the model is the worst, while when E=3, the model achieves the best prediction performance on most of the data sets in the table. The reason may be that the number of tasks involved is small, and one expert network is difficult to integrate the requirements of different tasks, while three expert networks are sufficient to achieve the communication of features between different tasks.

[0156] Table 4 Prediction performance of the model when the parameter sharing module contains different numbers of expert networks (the best performance is in bold, and the unit of Q3 and Q8 is "%")

[0157]

[0158] 3.5.2 Impact of key components of the PSSP-MTL model

[0159] In order to demonstrate the necessity of the key components in the PSSP-MTL model and explore their contributions to the predictive performance of the model, the present invention conducts experiments using three variants of the PSSP-MTL model.

[0160] The experimental results are shown in Table 5. When the parameter sharing module or the combined time series module is disabled, the prediction performance decreases significantly, about 0.34%, while when the residual connection is disabled, the decrease is only about 0.05%. The reason may be that when the combined time series module is disabled, the model cannot effectively extract the global features of the sequence, and when the parameter sharing module is disabled, the parameters at the bottom of the model are hard shared, which is not conducive to the feature exchange between tasks. Therefore, these two components have a greater impact on the prediction performance, while the slight increase in the loss of the intermediate layer features caused by disabling the residual connection has a smaller impact on the prediction performance.

[0161] In addition, the full PSSP-MTL model consistently outperforms its variants, which demonstrates the effectiveness and necessity of its internal key components, which enable the PSSP-MTL model to balance the global and multi-scale local features of the sequence, achieve information exchange between tasks, and be relatively stable to train.

[0162] Table 5 Prediction performance under different PSSP-MTL model components (#Time represents the combined time series module, #Sharing represents the parameter sharing module, #Residual represents all residual connections in the multi-scale residual convolution and parameter sharing modules, √ represents the use of this component, × represents the disabling of this component, the best performance is indicated in bold, and the unit of Q3 and Q8 is "%")

[0163]

[0164] 3.5.3 Impact of feature combinations

[0165] The present invention explores the prediction performance under different input feature combinations of the PSSP-MTL model, and the experimental results are shown in Table 6. Since the dimension of the embedded feature is much larger than that of other features, the strongly targeted coding feature is derived from the deep mining of the one-hot coding feature, so only the combination of the other four features except these two features needs to be considered. When the strongly targeted coding feature or the profile feature is disabled, the decrease in prediction performance is similar. When the physical and chemical properties are disabled, the prediction performance only decreases slightly, and even the Q3 accuracy on the Validation set2 has a slight improvement of 0.09%. Although the length of the strongly targeted coding feature is only 4, it can play roughly the same role as the PSSM and HMM features with lengths of 20 and 30 respectively. This shows that the strongly targeted coding features generated under the drive of HTA have extremely high feature space utilization of knowledge related to protein secondary structure prediction.

[0166] It can be seen from Table 8 that when the PSSP-MTL model does not disable any features, its prediction performance is optimal. Therefore, it can be proved that multi-feature fusion can comprehensively reflect protein knowledge, and the highly targeted coding features proposed in the present invention are based on the strong targeting of sample information and secondary structure, which can effectively supplement other general features and further improve the prediction performance.

[0167] Table 6 Prediction performance under different feature combinations (#HTA indicates strong targeted coding features, #PSSM indicates PSSM features, #HMM indicates HMM features, #Phys indicates physical and chemical properties, √ indicates use of this feature, × indicates disabling this feature, the best performance is indicated in bold, and the unit of Q3 and Q8 is "%")

[0168]

[0169] 3.5.4 Impact of Model Learning Strategy

[0170] In order to prove the superiority of the multi-task learning strategy, the present invention uses the single-task learning (STL) strategy and the multi-task learning strategy to conduct experiments. Figure 5 As shown in the figure, the prediction performance of the multi-task learning strategy is always better than that of the STL strategy on all data sets, and the difference between the two ranges from 0.04% to 0.22%. This proves that the multi-task learning strategy can effectively realize the information exchange between tasks, making the RSA prediction task play an auxiliary role in the protein secondary structure prediction task, thereby improving the prediction performance.

[0171] 3.6 Comparison with Existing Deep Learning Models

[0172] The present invention compares MHTAPred-SS with 10 recent advanced deep learning methods on six independent test sets. Among the 10 baseline models, DeepCNN uses multi-scale convolution to extract features of different receptive fields. MUFOLD-SS is a predictor based on the inception-inside-inception architecture. SPIDER3 and SPOT-1D jointly use two RNN architectures, LSTM and BRNN, to extract bidirectional global features of sequences. DCRNN, CNN_BIGRU, DeepACLSTM and Multi-S3P enable CNN and RNN to work together, effectively alleviating the limitations of a single model. In addition, DML_SS and LIFT_SS use pre-trained protein language models to embed sequences independently of homologous proteins, and improve prediction performance based on dense vectors of amino acid residues. DML_SS using embedded features is represented as DML_SS embed , DML_SS using the integrated model is represented as DML_SS ensemble It should be noted that in order to make a fair comparison on each method, the present invention shows the prediction performance of the method based on the same dataset in each experimental result. Due to the limitations of these baselines in terms of training platform, computing resources, and input feature factors, the prediction performance of DCRNN, DeepCNN, CNN_BIGRU, MUFOLD-SS, DeepACLSTM, and DML_SS is not significantly different from that of DCRNN. embed For the six methods, their prediction performance on the test sets of DS1 and DS2 is reported, while for other predictors based on complex factors, their prediction performance on the test sets of DS2 (TEST2016 and TEST2018) is reported.

[0173] Tables 7 to 10 report the three-state and eight-state prediction performance of MHTAPred-SS and baselines on six independent test sets. The three-state and eight-state prediction performance of MHTAPred-SS on the six test sets are significantly better than the baselines, with strong generalization ability and stable prediction effect, achieving the most advanced performance in protein secondary structure prediction tasks.

[0174] It can be seen from Tables 7 and 8 that the Q3 accuracy values ​​of MHTAPred-SS on the CASP12, CASP13, CASP14, and CB513 test sets are 86.73%, 86.04%, 83.21%, and 87.43%, respectively, which is higher than DML_SS. embed 0.65%-2.46% higher, and the SOV3 values ​​are 78.54%, 74.04%, 73.05% and 83.61% respectively, which is higher than DML_SS embedThe Q8 accuracy of MHTAPred-SS on the CASP12, CASP3, CASP14, and CB513 test sets is 77.3%, 76%, 72.54%, and 76.53%, respectively, which is 1.22%-7.24% higher than DML_SS. embed 0.99%-2.32% higher, and the SOV8 values ​​are 65.97%, 62.01%, 66.25% and 74.71% respectively, which is higher than DML_SS embed 1.49%-4.02% higher.

[0175] As shown in Tables 9 and 10, the Q3 accuracy of MHTAPred-SS on the TEST2016 and TEST2018 test sets is 88.14% and 87.42%, respectively, which is 0.3% and 0.29% higher than LIFT_SS, and the SOV3 values ​​are 84.89% and 82.81%, respectively, which is 0.13% and 0.49% higher than LIFT_SS. The Q8accuracy values ​​of MHTAPred-SS on the TEST2016 and TEST2018 test sets are 78.74% and 77.28%, respectively, which is 0.04% and 0.42% higher than LIFT_SS, and the SOV8 values ​​are 77.15% and 74.79%, respectively, which is 0.36% and 0.55% higher than LIFT_SS. It can be seen that MHTAPred-SS can more accurately assign secondary structure categories to each amino acid residue, which is closest to the theoretical performance limit of protein secondary structure prediction.

[0176] MHTAPred-SS significantly improves the more critical SOV index on six independent test sets, which shows that the prediction results of MHTAPred-SS can fully reflect the distribution of secondary structure fragments and can play a great supporting role in subsequent tasks such as tertiary structure prediction. In particular, it can be seen from Tables 9 and 10 that compared with DML_SS embed , DML_SS ensemble The prediction effect of MHTAPred-SS is better, which shows that multi-model integration has a positive contribution to the accurate prediction of protein secondary structure. However, whether it is three-state prediction or eight-state prediction, the prediction performance of MHTAPred-SS is better than that of DML_SS. ensemble This also reflects the strong competitiveness of MHTAPred-SS.

[0177] Table 7 Comparison of three-state prediction performance on the independent test set of DS1 (the best performance is in bold, and the unit of Q3 and SOV3 is "%")

[0178]

[0179] Table 8 Comparison of eight-state prediction performance on the independent test set of DS1 (the best performance is in bold, and the unit of Q8 and SOV8 is "%")

[0180]

[0181] Table 9 Comparison of three-state prediction performance on the independent test set of DS2 (the experimental results of LIFT_SS are the best results of fine-tuning the pre-trained protein language model, the best performance is in bold, and the unit of Q3 and SOV3 is "%")

[0182]

[0183] Table 10 Comparison of eight-state prediction performance on the independent test set of DS2 (the experimental results of LIFT_SS are the best results of fine-tuning the pre-trained protein language model, the best performance is in bold, and the unit of Q3 and SOV3 is "%")

[0184]

[0185]

[0186] The present invention refers to the amino acid residues at the secondary structure change point as boundary amino acid residues. The accuracy of the prediction of boundary amino acid residues and the overall secondary structure is shown in Table 11. The boundary amino acid residues are at the critical point of the secondary structure, and it is extremely difficult to predict their secondary structure categories. Although the boundary prediction accuracy of each model is lower than the corresponding overall prediction three-state prediction accuracy or eight-state prediction accuracy, MHTAPred-SS still achieves the best boundary prediction performance. The boundary prediction accuracy of MHTAPred-SS has been improved by about 4.3% compared with MUFOLD-SS and by about 1.3% compared with DML_SS. embed An improvement of about 1% was achieved. This proves that the features captured by MHTAPred-SS are more detailed and have significant advantages in predicting boundary secondary structures.

[0187] Table 11 Boundary prediction and overall prediction performance on TEST2016 (the best performance is in bold)

[0188]

[0189] 3.7 Case Study

[0190] like Figure 6 As shown in the figure, in order to intuitively compare the prediction effects of MHTAPred-SS with other methods, the present invention randomly selected three protein sequences and used PyMol to visualize the prediction results of different methods, among which Q SRepresents the accuracy of secondary structure prediction on a single sequence.

[0191] The first sequence selected is the "B" chain of a protein with PDB ID "5F29", which is expressed in E. coli O103. The second sequence selected is the "S" chain of a protein with PDB ID "5B1A", which is an oxidoreductase present in Bos taurus. Secondly, the third sequence selected is the "A" chain of a protein with PDB ID "5tkz", which is expressed in E. coli BL21 (DE3). Figure 6 It can be seen that the prediction results of MHTAPred-SS are most similar to the experimental results, and can maintain extremely high stability under different secondary structure changes. The prediction accuracy of MHTAPred-SS on the three sequences reached 97.18%, 98.98% and 98.88% respectively. embed MUFOLD-SS easily confuses coil and strand, and MUFOLD-SS is insensitive to the interaction of some neighboring residues, thus mistakenly predicting helix as coil. It can be proved that MHTAPred-SS based on the knowledge of secondary structure fragment length pattern can effectively balance the global and multi-scale local characteristics of the sequence, so as to accurately predict the distribution of protein secondary structure fragments. In short, MHTAPred-SS can flexibly and accurately perceive the changes in secondary structure, and its prediction results can be used as the basis for the determination of the overall protein structure.

[0192] In summary, the present invention proposes a sequence-to-sequence deep framework MHTAPred-SS, which can accurately predict three-state and eight-state protein secondary structures. In the proposed MHTAPred-SS, the designed HTA is used as a driver, which can extract sequence features that are highly targeted to sample information and protein secondary structures through the attention mechanism on the sample dimension and the random secondary structure fragment masking technology. HTA does not require a complex and time-consuming multiple sequence alignment process, and improves the feature space utilization of knowledge related to protein secondary structure prediction. Secondly, based on the task relevance and the knowledge of secondary structure fragment length patterns, a multi-feature fusion PSSP-MTL model is designed as the main part of MHTAPred-SS. The PSSP-MTL model balances global and multi-scale local features based on multi-scale residual convolution and combined time series networks, and uses the MMoE structure for information exchange between tasks. As far as we know, this is the first time that the MMoE structure has been applied to the field of protein secondary structure prediction. Through extensive ablation experiments, it is proved that the strongly targeted coding features that are not related to homologous proteins can play a considerable role based on very short length and contour features, with extremely high feature space utilization, and the MMoE structure enables the auxiliary task to fully assist the protein secondary structure prediction task. In addition, MHTAPred-SS is compared with 10 baselines such as DeepACLSTM, DML_SS, Multi-S3P and LIFT_SS on six independent test sets. The experimental results show that MHTAPred-SS has achieved 0.3%-2.46%, 0.13%-7.24%, 0.04%-2.32% and 0.36%-4.02% improvement in Q3, SOV3, Q8 and SOV8 indicators respectively for the suboptimal methods, achieving excellent prediction performance. MHTAPred-SS also has significant advantages in the prediction of secondary structure of boundary amino acid residues. From the visualization results of the case study, it can be intuitively seen that MHTAPred-SS can flexibly and accurately predict the distribution of secondary structure fragments at different scales, which also confirms the fact that MHTAPred-SS performs well in the SOV indicator.

[0193] The present invention has conducted extensive ablation experiments to explore the impact of model hyperparameters on prediction performance, and proved the necessity of model components, the efficiency of strongly targeted encoding, and the advancement of multi-task learning strategies. Experiments on six independent test sets have shown that MHTAPred-SS has achieved state-of-the-art performance, intuitively demonstrating that MHTAPred-SS can finely capture the distribution of secondary structure fragments, and it also has significant advantages in boundary secondary structure prediction. The values ​​of Q3, SOV3, Q8, and SOV8 indicators on the TEST2016 dataset reached 88.14%, 84.89%, 78.74%, and 77.15%, respectively. In addition, the present invention proves that MHTAPred-SS has significant advantages in boundary secondary structure prediction, and can finely capture the distribution of secondary structure fragments, thereby contributing to subsequent tasks such as protein tertiary structure prediction.

Claims

1. A multi-task learning framework for accurate protein secondary structure prediction, characterized in that: The name of this framework is MHTAPred-SS. MHTAPred-SS includes the HTA model and the PSSP-MTL model; The HTA model includes a DY-CNN module and a BiLSTM module. The DY-CNN module is used as the encoder part of HTA to generate highly targeted coding features for each sequence. It is highly targeted to sample information. The BiLSTM module uses a BiLSTM network to learn the bidirectional global features of the sequence and outputs the reconstruction results of the primary structure of the protein through the output layer. The PSSP-MTL model includes a multi-scale residual convolution module, a parameter sharing module and a combined time series module. The multi-scale residual convolution module is used to extract the multi-scale local features of the sequence. The parameter sharing module and the combined time series module constitute a complete MMoE structure, which is used to supplement the single-task features through information exchange between tasks and extract deep global features.

2. A multi-task learning framework for accurate protein secondary structure prediction according to claim 1, characterized in that: The encoder setting of the HTA model is smaller C out , in order to improve the utilization rate of the feature space of knowledge, the specific calculation process is as follows: MAT ω =Softmax(Conv(GAP(X))); MAT ω =[[ω1],[ω2],[ω3],…[ω B ]]; Among them, GAP is the global average pooling operation, Conv is the one-dimensional convolution operation, BN is the batch normalization operation, MAT ω It is the matrix composed of the weights of each filter, MAT FLT is the matrix composed of the parameters of each filter, is the transposed matrix of the matrix composed of each filter parameter, ω i is the weight of the ith filter, is the internal parameter of the ith filter, n is the number of parameters in each filter, Comb_FLT is the parameter of the Combined Filter, is element-wise addition, Conv Comb_FLT It is a convolution operation using Combined Filter.

3. A multi-task learning framework for accurate protein secondary structure prediction according to claim 2, characterized in that: The BiLSTM network contains forward and backward connections of LSTM cells. Each LSTM cell has three gating mechanisms: input gate, forget gate, and output gate. The calculation process of a sequence of length L in the LSTM cell is as follows: f t =σ(W f [h t―1 ,x t ]+b f ); i t =σ(W i [h t―1 ,x t ]+b i ); the t =σ(W o [h t―1 ,x t ]+b o ); C t =f t ⊙C t―1 +i t ⊙tanh(W C [h t―1 ,x t ]+b C ); h t =o t ⊙tanh(C t ); Where σ is the sigmoid function, tanh is the hyperbolic tangent function, ⊙ is the element-wise multiplication, t∈{1,2,…,L}, x t 、h t and C t are the input features, output features and cell states of the LSTM cell at the tth time step, W and b are the weights and bias parameters of the corresponding gating mechanism in the LSTM cell, respectively. t 、i t and t are the outputs of the forget gate, input gate, and output gate, respectively, and b f 、b i 、b o are the bias parameters of the forget gate, input gate, and output gate, respectively, and b C Bias parameter representing the cell state, W o , W C Represent the weight parameters of the output gate and cell state respectively.

4. A multi-task learning framework for accurate protein secondary structure prediction according to claim 3, characterized in that: The multi-scale residual convolution module contains multiple single-scale convolutions and residual connections. Based on the knowledge of the secondary structure fragment length pattern, the input feature tensor is convolved in one dimension using convolution kernels of different scales. The embedded features extracted by ESM-2 are concatenated with the multi-scale local features to generate the concatenated feature tensor as follows: Among them, s is the number of scales, F i is the output feature of the i-th single-scale convolution, F s represents the features transmitted by the sth residual connection, is element-by-element addition, Res s is the feature transmitted by the i-th residual connection, F embed is the embedding feature, Concat is the concatenation operation, and F C is the concatenated feature tensor.

5. A multi-task learning framework for accurate protein secondary structure prediction according to claim 4, characterized in that: The parameter sharing module uses the MMoE structure to realize information exchange between tasks. As the bottom part of the MMoE structure, the parameter sharing module can realize soft sharing of underlying parameters. The parameter sharing module comprehensively considers the output features of all expert networks and finally outputs fusion features. Each residual connection uses a Conv1D and BN to process the concatenated feature tensor as follows: Among them, j∈{1,2,…N T }, is the feature transmitted by the jth residual connection; Each gating network consists of Conv1D, BN, LeakyReLU activation function and Softmax operation, and the final output is the weight assigned to the output features of the expert network as follows: in, is the output feature tensor of the jth gating network, is the weight tensor assigned by the j-th gating network to the output features of the k-th expert network, is the weight value assigned by the j-th gating network to the output feature of the k-th expert network at the t-th position in the sequence, and L is the length of the sequence; The LeakyReLU activation function is used to activate the obtained features. The activated features are the fusion features. The calculation process is as follows: Where k∈1,2,…E,EX k is the sequential operation inside the k-th expert network, is the output feature tensor of the k-th expert network, is the fused feature tensor extracted by the parameter sharing module for the jth task, and CH is The number of channels, They respectively represent the features transmitted by the j-th residual connection and the weight tensor assigned by the j-th gating network to the output features of the k-th expert network.

6. A multi-task learning framework for accurate protein secondary structure prediction according to claim 5, characterized in that: The combined time series module is the top part of the MMoE structure, which cascades the time series convolutional neural network and the bidirectional gated recurrent unit to process the sequence data of length L; The GRU cell contains two gating mechanisms, namely the reset gate and the update gate. The calculation process is as follows: r t =σ(W r [h t―1 ,x t ]+b r ); z t =σ(W z [h t―1 ,x t ]+b z ); h t =z t ⊙h t―1 +(1―z t )⊙tanh(W h [r t ⊙h t―1 ,x t ]+b h ); where t∈{1,2,…,L}, x t and h t are the input and output features of the BiGRU cell at the tth time step, W and b are the weight and bias parameters of the corresponding gating mechanism of the GRU cell, r t and z t are the outputs of the reset gate and the update gate, respectively, r 、b z 、b h Represent the bias parameters of reset gate, update gate and cell features respectively.

Images