A protein secondary structure prediction method, device and computer device

Abstract

This invention relates to the field of protein secondary structure prediction, specifically to a method, apparatus, and computer device for protein secondary structure prediction, which improves the accuracy of protein secondary structure prediction. The scheme includes: obtaining the sequence embedding features of a given protein through a pre-trained protein language model; and obtaining a high-quality PSSMX through the calculation of a position-specific scoring matrix. H For high-quality PSSMX H Perform downsampling to obtain low-quality PSSMX L ; Incorporating sequence embedding features and low-quality PSSMX L Input sequence embedding of an enhanced PSSM module to obtain enhanced PSSM features X E A knowledge distillation strategy is introduced to transfer the knowledge of the protein secondary structure prediction teacher model trained with a high-quality PSSM to the student model trained with an enhanced PSSM. The teacher model uses a high-quality PSSM. H The student model uses the sequence encoding matrix and position encoding matrix as input to enhance PSSM features X. E The sequence coding matrix and position coding matrix are used as inputs. This invention is applicable to protein secondary structure prediction.

Description

Technical Field

[0001] This invention relates to the field of protein secondary structure prediction, and specifically to a method, apparatus, and computer device for predicting protein secondary structure. Background Technology

[0002] The occurrence and development of diseases are often closely related to abnormal protein function or structural changes. Therefore, researchers need to identify suitable protein targets for intervention in order to design drugs with therapeutic potential. Protein secondary structure can reveal the functional and structural characteristics of proteins, providing a basis for target selection. For example, in the study of specific diseases, certain secondary structural elements may be related to the disease progression, and these structural elements may become important targets for drug intervention. However, experimental methods for resolving protein structures are time-consuming and resource-intensive, and their speed is far from meeting the needs of a deeper understanding of life processes and the development of complex drugs. Therefore, developing efficient and accurate computational PSSP (Protein Secondary Structure Prediction) methods is of significant research value. It can not only accelerate the process of protein functional annotation and structural prediction, but also has important applications in bioinformatics, drug development, and disease diagnosis.

[0003] Existing technologies, such as CN112365921B, disclose a protein secondary structure prediction method based on a long short-term memory network. First, based on the protein sequence information with L input residues for secondary structure prediction, feature matrices PSFM (Position-Specific Frequency Matrix) and PSSM (Position-Specific Scoring Matrix) are obtained using HHblits and PSI-BLAST programs, respectively. Then, these two matrices are combined into a feature matrix F and converted into a feature tensor. Next, the protein sequence is processed into residue samples. Then, a long short-term memory network and a conditional random field model are constructed, and a dataset is built using protein sequences with known secondary structures to train the network. Finally, the residue samples of the protein sequence for secondary structure prediction are input into the trained model to obtain the secondary structure of the sequence.

[0004] This approach improves the efficiency and accuracy of protein secondary structure prediction by combining long short-term memory networks and conditional random field models. However, the following problems still exist:

[0005] 1. Sequence profiles are derived from multiple sequence alignment (MSA). It is very difficult to generate high-quality MSA without sufficient homologous protein sequences.

[0006] 2. Non-local interactions between residues that are far apart in sequence position but close in three-dimensional space are not effectively considered, and the dependence of local interaction features and long-range interaction features is not taken into account.

[0007] Therefore, the accuracy of the above-mentioned scheme is not high. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, apparatus and computer device for predicting protein secondary structure, thereby improving the accuracy of protein secondary structure prediction.

[0009] The present invention achieves the above-mentioned objective by adopting the following technical solution: Firstly, the present invention provides a method for predicting the secondary structure of proteins, the method comprising:

[0010] Sequence embedding features of a given protein are obtained by pre-training a protein language model;

[0011] High-quality PSSM X is obtained through the calculation of a location-specific rating matrix. H ;

[0012] For high-quality PSSM X H Perform downsampling to obtain low-quality PSSM X L ;

[0013] The sequence embedding features and low-quality PSSM X L Input sequence embedding of an enhanced PSSM module to obtain enhanced PSSM features X E ;

[0014] A knowledge distillation strategy is introduced to transfer the knowledge of the teacher model trained using a high-quality PSSM to the student model trained using an enhanced PSSM. During training, the teacher model is trained using a high-quality PSSM. H The student model uses the protein sequence coding matrix and protein position coding matrix as inputs to enhance PSSM features X. E The protein sequence coding matrix and the protein position coding matrix are used as inputs;

[0015] The trained student model is used to predict the secondary structure of proteins.

[0016] Furthermore, the inputs to the pre-trained protein language model include:

[0017] Original amino acid sequence of protein Where a i Represents the i-th amino acid in the sequence, l prot This indicates the length of the amino acid sequence.

[0018] Furthermore, obtaining the sequence embedding features of a given protein through a pre-trained protein language model specifically includes:

[0019] The encoder output of the pre-trained protein language model ProtT5-XL-U50 is used as the embedding feature. The amino acid sequence of a given protein is used as the input to ProtT5-XL-U50. The model is then run at half precision, and the encoder output is extracted as the sequence embedding. Among them l prot It is the length of the amino acid sequence, d fse It is the dimension of each amino acid embedding.

[0020] Furthermore, the specific methods for calculating the protein sequence coding matrix include:

[0021] Define a surjective mapping f: A pr ×A pr →N pr , where A pr A represents a set of amino acids. pr ×A pr ={<a,b> ; a, b∈A pr} represents set A pr Cartesian product, N pr Represents a set of integers.<a,b> and<b,a> To map to set N pr For the same element, the protein sequence coding matrix is ​​obtained according to the surjective mapping.

[0022] Furthermore, the specific methods for calculating the protein position coding matrix include:

[0023] The amino acid sequence is positionally encoded using a position encoding submodule, where the amino acid position at position p in the sequence is encoded as follows:

[0024] d fx The dimension of the amino acid position encoding vector is i, where i is the i-th dimension of the vector, thus yielding the protein position encoding matrix.

[0025] Furthermore, the specific calculation method for the location-specific rating matrix includes:

[0026] Multiple sequence alignment involves comparing the amino acid sequences of multiple proteins with phylogenetic relationships, then arranging identical amino acid residues in the same column, and using PSI-BLAST software for multiple sequence alignment.

[0027] Calculate the frequency matrix, location-specific frequency matrix. It represents the relative frequency of various amino acids at a specific position.

[0028] k is a certain amino acid, j∈(1,…,l) prot ), N MSA This is the number of homologous sequences in the MSA. Fcount = 1 is a spurious count to avoid certain amino acid types appearing 0 times. k，j It represents the number of times amino acid k appears at position j in the MSA. I is an indicator function, when MSA i，j The value is 1 when the value is equal to k, otherwise it is 0.

[0029] Background frequency matrix The frequency of each amino acid in all proteins involved in the MSA is represented as:

[0030] Constructing a PSSM matrix, where the position-specific matrix of a protein sequence represents the relative score of a specific amino acid at a particular position relative to the background frequency, then...

[0031] Furthermore, the sequence embedding enhancement PSSM module specifically includes two sub-modules: sequence embedding-assisted enhancement and self-attention enhancement;

[0032] The sequence embedding auxiliary enhancement submodule is used to embed sequences into X. SE Feature mapping to Query vector, transferring low-quality PSSM features X L Mapping to Key and Value vectors, using multi-head attention, the linear mapping for the i-th head is as follows:

[0033] These are the learnable weight matrices for the i-th head query vector, key vector, and value vector, respectively.

[0034] For each head, an attention score is calculated using dot product attention, and then weighted and summed with the value vector to obtain the attention feature matrix:

[0035] d KK is the dimension of the feature space; the attention feature matrices of each head are concatenated and then fused through a linear transformation to obtain the final multi-head attention representation, which is the PSSM feature representation enhanced by the semantic information of the sequence embedding features, as shown in the following formula:

[0036] W o，L It is a learnable weight matrix used for fusing multi-head attention, kH l This refers to the number of attention heads, and [·] represents the splicing operation;

[0037] The PSSM features are enhanced by introducing initial sequence embedding features using residual connections and batch normalization operations, as follows: Then it is fed into a feedforward network, and after a residual connection and normalization operation, the enhanced PSSM features are obtained:

[0038]

[0039]

[0040] Among them, W e1 W e2 It is a learnable weight matrix, b e1 b e2 It is a learnable bias matrix.

[0041] Furthermore, the self-attention enhancement submodule is used to enhance the PSSM features X obtained by the sequence embedding-assisted enhancement submodule. L_E Short-range and long-range feature extraction is performed, and a self-attention feature matrix is ​​obtained through a multi-head attention module. Specifically, this includes enhancing the PSSM feature X. L_E Perform a linear transformation as follows:

[0042] W Q，L_E W K，L_E W V，L_E It is a learnable weight matrix;

[0043] Extraction of short-range and long-range interactions of features is achieved through multi-head attention operations: X′ E =MultiHeadAtt(Q L_E K L_E V L_E Finally, through a feedforward network, residual connections and batch normalization operations are introduced simultaneously for feature enhancement, ultimately yielding enhanced PSSM features.

[0044] Furthermore, knowledge transfer is achieved by minimizing distillation loss, which is: L d =σLce +(1-σ)L kl L ce The student model predicts the label F. S (X E S CP The cross-entropy loss between PE and the true PSSP label Y, L ce =CE(F S (X E S CP ,PE),Y),F s This represents the student model, where CE is the cross-entropy loss function, and L... kl L is the Kullback-Leibler divergence used to measure the difference in the predicted probability distributions between the student model and the teacher model. kl =KL marginal (F S (X E S CP ,PE),F T (X H S CP ,PE)),F T This represents the teacher model, where σ is the weighted L... ce and L kl The hyperparameters are obtained by minimizing the distillation loss L. d Update the PSSM augmentation network and student model.

[0045] Furthermore, the student model includes:

[0046] The multi-source information fusion module integrates the protein sequence coding matrix S CP and enhanced PSSM features X E The data is then concatenated and fused, and then added to the positional encoding matrix PE to obtain a new representation that simultaneously contains amino acid semantic information, evolutionary information, and positional information. This is used as the input to the PSSP prediction model: X = [S CP X E ]+PE;

[0047] The self-attention submodule is used to perform a linear transformation on the input vector x to generate representations of the Q, K, and V feature spaces for different heads. The linear transformation for the i-th head can be expressed as:

[0048] It is a learnable weight matrix;

[0049] For each head, calculate the self-attention feature matrix:

[0050] d KLet K be the dimension of the feature space. The attention feature matrices of each head are concatenated and then fused through a linear transformation to obtain the final multi-head attention representation, as shown in the following formula:

[0051] W o，P It is a learnable weight matrix used for fusing multi-head attention, kH p This refers to the number of attention heads, and [·] represents the splicing operation;

[0052] The feature extraction module is used to balance the initial augmented PSSM input features and attention features through a scaling factor α for effective fusion: Z′=BatchNorm(α×MAtt+(1-α)×X) E The scaling factor α is a learning scalar, and then the final protein feature vector Z is obtained through a feedforward neural network and residual connections.

[0053] W f1 W f2 It is a learnable weight matrix, b f1 b f2 It is a learnable bias matrix;

[0054] The prediction module is used to compute prediction results using a multilayer perceptron model. W p1 W p2 It is a learnable weight matrix.

[0055] In a second aspect, the present invention provides a protein secondary structure prediction device, the device comprising:

[0056] The sequence embedding feature acquisition module is used to obtain the sequence embedding features of a given protein through a pre-trained protein language model.

[0057] PSSMX L The acquisition module is used to obtain high-quality PSSM X through the calculation of a location-specific rating matrix. H For high-quality PSSM X H Perform downsampling to obtain low-quality PSSM X L ;

[0058] X E The acquisition module is used to embed the sequence features and low-quality PSSM X. L Input sequence embedding of an enhanced PSSM module to obtain enhanced PSSM features X E ;

[0059] The protein secondary structure prediction module takes PSSM features, protein sequence coding matrix and position coding matrix as input to obtain the predicted probability of protein secondary structure;

[0060] The knowledge distillation strategy module introduces a knowledge distillation strategy, which transfers knowledge from a teacher model of protein secondary structure prediction trained using a high-quality PSSM to a student model of protein secondary structure prediction using an enhanced PSSM. The PSSM features of the teacher model are used with high-quality PSSM X. H The PSSM features of the student model were enhanced using PSSM X. E ;

[0061] Finally, the trained student model is used to predict the secondary structure of proteins.

[0062] Thirdly, the present invention provides a computer device, wherein the memory stores program instructions, and when the program instructions are executed, the protein secondary structure prediction method described above is performed.

[0063] The beneficial effects of this invention are as follows:

[0064] 1. This invention proposes a sequence embedding enhancement PSSM module, which enhances low-quality PSSM by using sequence embeddings generated by a pre-trained protein language model. This helps to more accurately capture the evolutionary information of protein sequences, providing a more robust feature representation for subsequent structure prediction.

[0065] 2. This invention integrates multi-source protein information (sequence information, location information, evolutionary information) through the Position Aware Attention (PSSP) prediction model, while also focusing on short-range and long-range feature dependencies, thereby enhancing the model's ability to understand and represent proteins.

[0066] 3. The knowledge distillation strategy introduced in this invention transfers the knowledge of the teacher model obtained by training with a high-quality PSSM to the student model using an enhanced PSSM, thereby achieving the goal of obtaining a high-performance model with low computing resources. Attached Figure Description

[0067] Figure 1 This is a flowchart of a protein secondary structure prediction method provided in an embodiment of the present invention. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0069] The data used in the technical solution of this invention comes from the CullPDB dataset, which includes 6125 proteins.

[0070] This invention provides a method for predicting protein secondary structure, such as... Figure 1 As shown, the specific steps are as follows:

[0071] 1. Model Input

[0072] The original amino acid sequence of the protein was set as follows: Where a i Represents the i-th amino acid in the sequence, l prot This indicates the length of the amino acid sequence.

[0073] (1) Protein sequence encoding: Define a surjective mapping f: A pr ×A pr →N pr , where A pr A represents a set of amino acids. pr ×A pr ={<a,b> ; a, b∈A pr} represents set A pr Cartesian product, N pr Represents a set of integers.<a,b> and<b,a> To map to set N pr For the same element, the protein sequence coding matrix is ​​obtained according to the surjective mapping.

[0074] (2) Protein position coding: In order to focus on the relative position information of residues in the protein sequence, the amino acid sequence is position-coded. The amino acid position code at position p in the sequence is as follows:

[0075] d fx The dimension of the amino acid position encoding vector is i, where i is the i-th dimension of the vector, thus yielding the protein position encoding matrix.

[0076] (3) Location-specific rating matrix

[0077] Position-Specific Score (PSSM) is an important matrix representation used in bioinformatics to describe protein sequences. It is calculated using multiple sequence alignment (MSA) based on evolutionary information between sequences. It represents the relative frequencies of various amino acids at a given position and their interactions with other amino acids, reflecting the conservation and variability of a protein sequence at different positions—that is, evolutionary information. Generally, a higher score indicates that the amino acid at the corresponding position is more conserved in evolution, while a lower score may indicate a position with higher variability. The generation process of a PSSM typically includes the following steps:

[0078] ① Multiple sequence comparison

[0079] Multiple sequence alignment (MSA) involves comparing the amino acid sequences of three or more phylogenetically related proteins, aligning identical amino acid residues in the same column whenever possible. MSA can be visualized as a two-dimensional table, where each row represents a sequence and each column represents the position of a residue. PSI-BLAST software is used for MSA.

[0080] ② Calculate the frequency matrix

[0081] Location-specific frequency matrix The relative frequency of various amino acids at a specific position is defined as:

[0082]

[0083] Where k is a certain amino acid, j∈(1,...,l) prot ), N MSA This is the number of homologous sequences in the MSA. Fcount = 1 is a spurious count to avoid certain amino acid types appearing 0 times. k，j The number of times amino acid k appears at position j in the MSA is defined as:

[0084]

[0085] Here, I is an indicator function, when MSA i，j If the value is k, take 1; otherwise, take 0.

[0086] Background frequency matrix This represents the frequency of various amino acids in all proteins involved in the MSA (Massively Orthogonal Syndrome), reflecting the overall distribution of amino acids. It does not consider sequence variability at specific positions and is defined as follows:

[0087]

[0088] ③ Construct the PSSM matrix

[0089] The position-specific matrix of a protein sequence in PSSM represents the relative score of an amino acid at a specific position relative to its background frequency, i.e., the ratio of its frequency to its frequency in the overall background. This can help capture the evolutionary conservation and relative role of amino acids at specific positions. Defined as:

[0090]

[0091] 2. Transformer-based pre-trained protein language model

[0092] Protein language models are trained on large-scale protein sequence data to learn the relationships between amino acids and the semantic information of the sequence. Sequence embeddings obtained from pre-trained protein language models can learn more about the connections and co-occurrence patterns between different amino acids, containing the evolutionary history and structure-function relationships of the protein sequence, thus providing richer information and feature representations for tasks such as protein structure prediction. By applying these pre-trained sequence embeddings to protein secondary structure prediction models, richer and more informative feature representations can be provided for low-quality protein secondary structure prediction models, helping to improve the model's performance and accuracy.

[0093] This invention uses the encoder output of the pre-trained model ProtT5-XL-U50 as embedded features. ProtT5-XL-U50 is a Transformer-based language model with 300 million parameters, which is first trained on the BFD database and then fine-tuned on the UniRef50 database.

[0094] To obtain the sequence embedding features of a given protein, this invention uses its amino acid sequence as input to the ProtT5-XL-U50, then runs the model at half precision, and extracts the encoder output as the sequence embedding. Among them l prot It is the length of the amino acid sequence, d fse This is the dimension of each amino acid embedding, d in this scheme. fse =1024.

[0095] 3. Sequence Embedding Enhancement PSSM

[0096] First, based on the prior distribution, high-quality PSSM X H Downsampling yields low-quality PSSM X L Next, the sequence embedding enhancement PSSM module is trained, based on the sequence embedding X obtained from the pre-trained model ProtT5-XL-U50. SE Assisted enhancement of low-quality PSSM X L Ultimately, this resulted in the enhanced PSSM X. E .

[0097] (1) Low-quality PSSM based on prior distribution downsampling

[0098] This invention defines a low-quality PSSM as one where the number of homologous sequences in a protein is less than 60, and a high-quality PSSM otherwise. First, a high-quality PSSM is obtained by calculating the position-specific scoring matrix in the model input of Part 1. H Then, for high-quality PSSM X H Downsampling is performed to obtain low-quality PSSMX. LThis is an input to subsequent modules. During the training phase, the low-quality PSSM X obtained through downsampling... L There should be no domain gaps between it and the original low-quality PSSM. Therefore, this invention employs a PSSM downsampling strategy based on the prior native distribution.

[0099] First, the MSA count distribution χ is calculated based on the original sequences of low-quality PSSMs in the training set. Specifically, when the number of homologous sequences of MSA is less than 60, the frequency of MSA counts is calculated to obtain the prior distribution χ of the low-quality original PSSMs. During training, a batch of MSAs with counts sampled from the prior distribution χ are randomly selected from the original MSAs, and the low-quality PSSMX is calculated based on this. L .

[0100] Low-quality PSSMX obtained through domain-aligned downsampling L This makes the low-quality PSSM obtained by downsampling closer to the natural low quality.

[0101] (2) Sequence Embedding Enhancement PSSM Module

[0102] The architecture of the sequence embedding enhancement PSSM module is as follows: Figure 1 As shown in (b), it includes two sub-modules: sequence embedding-assisted enhancement and self-attention enhancement.

[0103] ① Sequence Embedding Auxiliary Enhancement Submodule

[0104] This module aims to achieve the effect of sequence embedding-assisted enhancement of PSSM by embedding sequence X. SE Feature mapping to Query vector, transferring low-quality PSSM features X L Mapping to key and value vectors. Multi-head attention is used here, and the linear mapping for the i-th head is as follows:

[0105]

[0106]

[0107]

[0108] in, These are the learnable weight matrices for the i-th head query vector, key vector, and value vector, respectively.

[0109] In this way, the Query vector will contain sequence embedding features to guide the subsequent attention given to low-quality PSSM features, while the Key and Value vectors will contain low-quality PSSM features to measure the interaction between sequence embeddings and low-quality embeddings. Through the attention mechanism, sequence embeddings can guide the weight allocation during the low-quality PSSM enhancement process, allowing more important sequence information to have a stronger impact on the enhanced PSSM. This is equivalent to modifying and weighting the low-quality PSSM based on the information from the sequence embeddings to better reflect evolutionary relationships and achieve more effective information transfer and feature enhancement.

[0110] For each head, an attention score is calculated using dot product attention, and then weighted and summed with the value vector to obtain the attention feature matrix:

[0111]

[0112] Where, d K It is the dimension of the feature space K.

[0113] The attention feature matrices of each head are concatenated and then fused through a linear transformation to obtain the final multi-head attention representation, which is a PSSM feature representation enhanced by semantic information of sequence embedding features, as shown in the following formula:

[0114]

[0115] Among them, W o，L It is a learnable weight matrix used for fusing multi-head attention, kH l [·] represents the number of attention heads, and [·] represents the splicing operation.

[0116] The PSSM features are further enhanced by incorporating initial sequence embedding features using residual connections and batch normalization operations.

[0117]

[0118] Then it is fed into a feedforward network, and after a residual connection and normalization operation, the enhanced PSSM features are obtained:

[0119]

[0120]

[0121] Among them, W e1 W e2 It is a learnable weight matrix, b e1 b e2 It is a learnable bias matrix.

[0122] ② Self-attention enhancement submodule

[0123] For the sequence embedding-assisted enhancement of PSSM features X obtained from the previous submodule L_E Further short-range and long-range feature extraction is performed, and a self-attention feature matrix is ​​obtained through a multi-head attention module.

[0124] First, the PSSM feature X L_E A linear transformation can be expressed as:

[0125] Q L_E =W Q，L_E X L_E

[0126] K L_E =W K，L_E X L_E

[0127] V L_E =W V，L_E X L_E

[0128] Among them, W Q，L_E W K，L_E W V，L_E It is a learnable weight matrix.

[0129] Then, multi-head attention operations are used to extract short-range and long-range interactions of features:

[0130] X′ E =MultiHeadAtt(Q L_E K L_E V L_E )

[0131] Finally, through a feedforward network, residual connections and batch normalization operations are introduced simultaneously for further feature enhancement, ultimately yielding enhanced PSSM features. The formulas in this part are similar to those in the sequence embedding auxiliary enhancement submodule, so they will not be repeated here.

[0132] 4. PSSP Prediction (Teacher / Student) Model

[0133] The teacher and student models predicted by PSSP have the same architecture, with the teacher model using high-quality PSSMX. H The sequence coding matrix and position coding matrix are used as inputs, and the predicted probability output of the protein secondary structure is expressed as F. T (X H S CP ,PE), while the student model enhances PSSM features X E The sequence encoding matrix and the position encoding matrix are taken as inputs, and the output is represented as F. S (XE S CP ,PE).

[0134] The following description uses a student model as an example to illustrate the model details.

[0135] The student model structure includes:

[0136] (1) Multi-source information fusion module, which integrates the protein sequence coding matrix S CP and enhanced PSSM features X E The data is then concatenated and fused, and then added to the positional encoding matrix PE to obtain a new representation that simultaneously contains amino acid semantic information, evolutionary information, and positional information. This is used as the input to the PSSP prediction model: X = [S CP X E ]+PE, in this embodiment d fx =l prot +20;

[0137] (2) The self-attention submodule performs a linear transformation on the input vector X to generate representations of the Q, K, and V feature spaces for different heads. The linear transformation for the i-th head can be expressed as:

[0138] It is a learnable weight matrix;

[0139] For each head, calculate the self-attention feature matrix:

[0140] d K Let K be the dimension of the feature space. The attention feature matrices of each head are concatenated and then fused through a linear transformation to obtain the final multi-head attention representation, as shown in the following formula:

[0141] W o，P It is a learnable weight matrix used for fusing multi-head attention, kH p This refers to the number of attention heads, and [·] represents the splicing operation;

[0142] (3) Feature extraction: The initial enhanced PSSM input features and attention features are balanced by a scaling factor α to achieve effective fusion: Z′=BatchNorm(α×MAtt+(1-α)×X) E The scaling factor α is a learning scalar, and then the final protein feature vector Z is obtained through a feedforward neural network and residual connections.

[0143] W f1 W f2 It is a learnable weight matrix, b f1 b f2It is a learnable bias matrix;

[0144] (4) Prediction module, which uses a multilayer perceptron model to calculate the prediction results: W p1 W p2 It is a learnable weight matrix;

[0145] 5. Knowledge distillation strategy

[0146] The model also incorporates a knowledge distillation strategy to enable the student prediction model to achieve better performance with a relatively lightweight approach.

[0147] First, use a high-quality PSSM X H The teacher model is pre-trained, and then enhanced PSSM X is used. E The student model is trained, while the teacher model guides the student model's training, transferring its feature representations and predictive capabilities to the student model. Knowledge transfer is achieved by minimizing the distillation loss, which is defined as follows:

[0148] L d =oL ce +(1-σ)L kl

[0149] L ce The student model predicts the label F. S (X E S CP The cross-entropy loss between the PE and the true PSSP label Y is:

[0150] L ce =CE(F S (X E S CP ,PE),Y)

[0151] Among them, F s Let represent the student model, and CE be the cross-entropy loss function.

[0152] L kl It is the Kullback-Leibler divergence used to measure the difference in the predicted probability distributions between the student model and the teacher model, i.e.:

[0153] L kl =KL marginci (F S (X E S CP ,PE),F T (X M S CP ,PE))

[0154] Among them, FT This represents the teacher model, where σ is the weighted L... ce and L kl The hyperparameters are obtained by minimizing the distillation loss L. d Update the PSSM augmentation network and student model.

[0155] 6. Loss Function: This invention also uses mean squared error loss to minimize the difference between enhanced PSSM and high-quality PSSM, i.e.:

[0156] L m =||X E -X H 2. Combining distillation loss and mean square error loss, the overall loss function of the framework of this invention is as follows: It is a weighted hyperparameter.

[0157] The present invention also provides a protein secondary structure prediction device for implementing the protein secondary structure prediction method described herein, the device comprising:

[0158] The sequence embedding feature acquisition module is used to obtain the sequence embedding features of a given protein through a pre-trained protein language model.

[0159] PSSM X L The acquisition module is used to obtain high-quality PSSM X through the calculation of a location-specific rating matrix. H For high-quality PSSM X H Perform downsampling to obtain low-quality PSSM X L ;

[0160] X E The acquisition module is used to embed the sequence features and low-quality PSSM X. L Input sequence embedding of an enhanced PSSM module to obtain enhanced PSSM features X E ;

[0161] The protein secondary structure prediction module takes PSSM features, protein sequence coding matrix and position coding matrix as input to obtain the predicted probability of protein secondary structure;

[0162] The knowledge distillation strategy module introduces a knowledge distillation strategy, which transfers knowledge from a teacher model of protein secondary structure prediction trained using a high-quality PSSM to a student model of protein secondary structure prediction using an enhanced PSSM. The PSSM features of the teacher model are used with high-quality PSSM X. H The PSSM features of the student model were enhanced using PSSM X. E ;

[0163] Finally, the trained student model is used to predict the secondary structure of proteins.

[0164] The present invention also provides a computer device, including a memory storing program instructions, which, when executed, perform the protein secondary structure prediction method described in the present invention.

[0165] This invention guides the enhancement of low-quality protein language models (PSSMs) using sequence embeddings generated by a pre-trained protein language model. Then, a position-aware attention-based PSSP prediction model is employed, integrating multi-source protein information (sequence, position, and evolutionary information) while also considering short- and long-range feature dependencies. Furthermore, a knowledge distillation strategy is introduced to further optimize and enhance the PSSM, while simultaneously training a lightweight, high-performance student model. This significantly improves prediction accuracy.

[0166] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A method for predicting protein secondary structure, characterized in that, The method includes: Sequence embedding features of a given protein are obtained by pre-training a protein language model; High-quality PSSM X is obtained through the calculation of a location-specific rating matrix. H ; For high-quality PSSM X H Perform downsampling to obtain low-quality PSSM X L ; The sequence embedding features and low-quality PSSM X L Input sequence embedding of an enhanced PSSM module to obtain enhanced PSSM features X E ; A knowledge distillation strategy is introduced to transfer the knowledge of the teacher model trained using a high-quality PSSM to the student model trained using an enhanced PSSM. During training, the teacher model uses a high-quality PSSM X... H The student model uses the protein sequence coding matrix and protein position coding matrix as inputs to enhance PSSM features X. E The protein sequence coding matrix and the protein position coding matrix are used as inputs; The trained student model predicts the secondary structure of proteins. The Sequence Embedding Enhancement (PSSM) module specifically includes two sub-modules: Sequence Embedding Assisted Enhancement and Self-Attention Enhancement. The sequence embedding auxiliary enhancement submodule is used to embed sequences into X. SE Feature mapping to Query vector, transferring low-quality PSSM features X L Mapping to Key and Value vectors, using multi-head attention, the linear mapping for the i-th head is as follows: , , , These are the learnable weight matrices for the i-th head query vector, key vector, and value vector, respectively. For each head, an attention score is calculated using dot product attention, and then weighted and summed with the value vector to obtain the attention feature matrix: d K K is the dimension of the feature space; the attention feature matrices of each head are concatenated and then fused through a linear transformation to obtain the final multi-head attention representation, which is the PSSM feature representation enhanced by the semantic information of the sequence embedding features, as shown in the following formula: , It is a learnable weight matrix used for fusing multi-head attention. This refers to the number of attention heads, and [·] represents the splicing operation; The PSSM features are enhanced by introducing initial sequence embedding features using residual connections and batch normalization operations, as follows: Then it is fed into the feedforward network, and after a residual connection and normalization operation, the enhanced PSSM features are obtained: in, , It is a learnable weight matrix. , It is a learnable bias matrix.

2. The protein secondary structure prediction method according to claim 1, characterized in that, The inputs to a pre-trained protein language model include: Original amino acid sequence of protein , where a i This represents the i-th amino acid in the sequence. This indicates the length of the amino acid sequence.

3. The protein secondary structure prediction method according to claim 2, characterized in that, Obtaining sequence embedding features of a given protein through a pre-trained protein language model specifically includes: The encoder output of the pre-trained protein language model ProtT5-XL-U50 is used as the embedding feature. The amino acid sequence of a given protein is used as the input to ProtT5-XL-U50. The model is then run at half precision, and the encoder output is extracted as the sequence embedding. ,in It is the length of the amino acid sequence, d fse It is the dimension of each amino acid embedding.

4. The protein secondary structure prediction method according to claim 1, characterized in that, The specific methods for calculating the protein sequence coding matrix include: Define a surjective mapping , where A pr Represents a set of amino acids. Let set A be pr Cartesian product, N pr Represents a set of integers.<a,b> and<b,a> To map to set N pr For the same element, the protein sequence coding matrix is ​​obtained according to the surjective mapping. .

5. The protein secondary structure prediction method according to claim 1, characterized in that, The specific methods for calculating the protein position coding matrix include: The amino acid sequence is positionally encoded using a position encoding submodule, where the amino acid position at position p in the sequence is encoded as follows: d fx The dimension of the amino acid position encoding vector is i, where i is the i-th dimension of the vector, thus yielding the protein position encoding matrix. .

6. The protein secondary structure prediction method according to claim 1, characterized in that, The specific methods for calculating the location-specific rating matrix include: Multiple sequence alignment involves comparing the amino acid sequences of multiple proteins with phylogenetic relationships, then arranging identical amino acid residues in the same column, and using PSI-BLAST software for multiple sequence alignment. Calculate the frequency matrix, location-specific frequency matrix. It represents the relative frequency of various amino acids at a specific position. ; k is a certain amino acid. , It is the number of homologous sequences in the MSA. It's a pseudo-count to avoid certain amino acid types appearing zero times. It represents the number of times amino acid k appears at position j in the MSA. I is an indicator function, when The value is 1 if the condition is met, otherwise it is 0. Background frequency matrix The frequency of each amino acid in all proteins involved in the MSA is represented as: ; Constructing a PSSM matrix, where the position-specific matrix of a protein sequence represents the relative score of a specific amino acid at a particular position relative to the background frequency, then... .

7. The protein secondary structure prediction method according to claim 1, characterized in that, The self-attention enhancement submodule is used to enhance the PSSM features obtained by the sequence embedding auxiliary enhancement submodule. Short-range and long-range feature extraction is performed, and a self-attention feature matrix is ​​obtained through a multi-head attention module, specifically including enhanced PSSM features. Perform a linear transformation as follows: , , , It is a learnable weight matrix; Extraction of short-range and long-range interactions of features is achieved through multi-head attention operations: Finally, through a feedforward network, residual connections and batch normalization operations are introduced simultaneously for feature enhancement, ultimately yielding enhanced PSSM features. .

8. The protein secondary structure prediction method according to claim 1, characterized in that, Knowledge transfer is achieved by minimizing distillation loss, which is: L ce Student model predicted labels Cross-entropy loss between the PSSP true label Y and the actual label Y. , This represents the student model, where CE is the cross-entropy loss function. The Kullback-Leibler divergence is used to measure the difference in the predicted probability distributions between the student model and the teacher model. , Representing the teacher model, For weighted and The hyperparameters are obtained by minimizing the distillation loss L. d Update the PSSM augmentation network and student model.

9. The protein secondary structure prediction method according to claim 1, characterized in that, The student model includes: The multi-source information fusion module integrates the protein sequence coding matrix S CP and enhanced PSSM features X E The data is then concatenated and fused, and then added to the positional encoding matrix PE to obtain a new representation that simultaneously contains amino acid semantic information, evolutionary information, and positional information. This serves as the input to the PSSP prediction model: ; The self-attention submodule is used to perform a linear transformation on the input vector X to generate representations of the Q, K, and V feature spaces for different heads. The linear transformation for the i-th head can be expressed as: , , , It is a learnable weight matrix; For each head, calculate the self-attention feature matrix: d K Let K be the dimension of the feature space. The attention feature matrices of each head are concatenated and then fused through a linear transformation to obtain the final multi-head attention representation, as shown in the following formula: , It is a learnable weight matrix used for fusing multi-head attention. This refers to the number of attention heads, and [·] represents the splicing operation; The feature extraction module is used to balance the initial augmented PSSM input features and attention features through a scaling factor α for effective fusion. The scaling factor α is a learning scalar, which is then passed through a feedforward neural network and residual connections to obtain the final protein feature vector Z. W f1 W f2 It is a learnable weight matrix, b f1 b f2 It is a learnable bias matrix; The prediction module is used to compute prediction results using a multilayer perceptron model. W p1 W p2 It is a learnable weight matrix.

10. A protein secondary structure prediction device, used to implement the protein secondary structure prediction method as described in any one of claims 1-9, characterized in that, The device includes: The sequence embedding feature acquisition module is used to obtain the sequence embedding features of a given protein through a pre-trained protein language model. PSSM X L The acquisition module is used to obtain high-quality PSSM X through the calculation of a location-specific rating matrix. H For high-quality PSSM X H Perform downsampling to obtain low-quality PSSM X L ; X E The acquisition module is used to embed the sequence features and low-quality PSSM X. L Input sequence embedding of an enhanced PSSM module to obtain enhanced PSSM features X E ; The protein secondary structure prediction module takes PSSM features, protein sequence coding matrix and position coding matrix as input to obtain the predicted probability of protein secondary structure; The knowledge distillation strategy module introduces a knowledge distillation strategy, which transfers knowledge from a teacher model of protein secondary structure prediction trained using a high-quality PSSM to a student model of protein secondary structure prediction using an enhanced PSSM. The PSSM features of the teacher model are used with high-quality PSSM X. H The PSSM features of the student model were enhanced using PSSM X. E ; Finally, the trained student model is used to predict the secondary structure of proteins.

11. A computer device comprising a memory storing program instructions, characterized in that, When the program instructions are executed, the protein secondary structure prediction method as described in any one of claims 1-9 is performed.

Images