A Multi-Perspective Hybrid Expert Protein Function Prediction Method Based on Cross-Attention
By constructing a multi-view hybrid expert model and dynamically allocating weights using a cross-attention routing module, the problems of insufficient utilization of multimodal features and overly smooth deep graph networks in existing technologies are solved, achieving high-precision protein function prediction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JIANGNAN UNIV
- Filing Date
- 2026-02-12
- Publication Date
- 2026-05-26
AI Technical Summary
Existing protein function prediction methods fail to effectively utilize multimodal features, have rigid fusion mechanisms, and deep graph networks are prone to oversmoothing, making it difficult to accurately predict the function of proteins with low sequence similarity.
A multi-perspective hybrid expert model based on cross-attention is constructed. By combining an expert network with pre-trained sequence features, amino acid primary and secondary structure features, and protein topology features, and dynamically allocating weights using a cross-attention routing module, high-precision prediction is achieved.
It improves the accuracy and generalization ability of protein function prediction, reduces false positive predictions, effectively handles proteins with low sequence similarity, and enhances the accuracy and robustness of function prediction.
Smart Images
Figure CN122090954A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the interdisciplinary field of bioinformatics and artificial intelligence, specifically involving a multi-perspective hybrid expert protein function prediction method based on cross-attention. Background Technology
[0002] Protein function prediction is one of the core tasks in the post-genomic era, aiming to infer the biological function of proteins from their amino acid sequences or structural information, such as Gene Ontology (GO) terminology or Enzyme Committee (EC) numbers. Accurate function prediction is of great significance for drug discovery, disease mechanism research, and synthetic biology.
[0003] Traditional protein function prediction methods mainly rely on homology comparison, such as using the BLAST algorithm to find similar sequences with known functions in a database to infer the function of the target protein. However, traditional methods often fail for "dark matter" proteins with low sequence similarity.
[0004] In recent years, deep learning-based methods have made significant progress. Existing methods are mainly divided into three categories: (1) Sequence-based methods: using recurrent neural networks (RNNs) or convolutional neural networks (CNNs) to extract amino acid sequence features, or using pre-trained language models such as ProtT5 and ESM to extract evolutionary features. (2) Structure-based methods: using graph neural networks to process protein contact maps or three-dimensional structures. (3) Multimodal fusion methods: simply splicing sequence features and structural features together.
[0005] However, existing technologies still have the following shortcomings: (1) Insufficient feature utilization: Many methods fail to simultaneously and effectively utilize the pre-training sequence features containing evolutionary information, the physicochemical features reflecting secondary structures, and the topological features characterizing the contact graph, resulting in information silos between different modalities. (2) Rigid fusion mechanism: Existing multimodal fusion usually adopts simple splicing or weighted summation, ignoring the differences in the degree of dependence of different proteins on the features of different modalities when predicting different functions. For example, the functions of some enzymes depend more on the local structure of the active site, while some binding functions depend more on global sequence features. (3) Bottleneck of deep graph networks: When processing protein contact graphs, traditional graph convolutional networks (GCNs) are prone to oversmoothing, making it difficult to capture deep topological information.
[0006] Therefore, developing a protein function prediction method that can adaptively fuse multi-view features and effectively extract deep graph structural information has important application value. Summary of the Invention
[0007] To overcome the limitations of existing technologies, this invention proposes a multi-perspective hybrid expert protein function prediction method based on cross-attention. This method constructs a hybrid model comprising three expert networks, extracting information from three perspectives: pre-trained sequence features, amino acid primary and secondary structure features, and protein topology features. Expert weights are dynamically allocated through a cross-attention-based routing module to achieve high-precision function prediction.
[0008] The technical solution of the present invention is as follows: A multi-perspective hybrid expert protein function prediction method based on cross-attention includes the following four stages: multi-source data preprocessing and feature construction, multi-perspective expert feature extraction, cross-attention routing fusion, and functional classification prediction. The specific steps are as follows: Phase 1: Multi-source data preprocessing and feature construction. This phase aims to transform protein sequences into tensor forms that can be input into the model, and includes the following steps: Step 1: Sequence Feature Extraction. For the target protein sequence, a pre-trained protein language model is used to extract sequence embedding features, denoted as... The pre-trained protein language model includes, but is not limited to, models based on the Transformer architecture.
[0009] Step 2: Construction of primary and secondary structure features. The amino acid sequence is encoded, for example using one-hot encoding, denoted as... Protein secondary structure prediction algorithms or tools are used to obtain protein secondary structure features that reflect the probability distribution of helices, folds, etc., denoted as... .
[0010] Step 3: Contact map construction. Based on the experimentally determined or predicted three-dimensional structural information of the protein, characteristic atoms of each amino acid residue are extracted (e.g., ...). (Carbon atom) coordinates. Calculate the Euclidean distance between any two residue feature atoms in the sequence. When the distance is less than or equal to a preset threshold, it is determined that there is spatial contact between the two residues, thus establishing a connection edge, and finally transforming it into an edge index tensor of a graph structure.
[0011] Phase 2: Multi-view expert feature extraction. This phase involves extracting deep features from different perspectives using three expert networks in parallel.
[0012] Step 4: Pre-trained Feature Expert 1 Extraction. This expert module employs a multi-scale one-dimensional convolutional network. First, the input is processed through linear projection and ReLU activation. Then, channel transformation is performed. Subsequently, the feature map is processed through three parallel paths, using one-dimensional convolutions with different kernel sizes (1, 6, and 12 respectively) to extract local features from different receptive fields. Finally, adaptive max pooling is used to compress and concatenate the features from each path to obtain the feature vector of expert 1. and category raw score .
[0013] The pre-trained feature expert 1 includes a linear projection layer, a channel transformation layer, and a multi-scale one-dimensional convolutional module; given the input sequence features After linear projection and channel transformation, the result is obtained The feature extraction process of this expert module can be represented as:
[0014] In the formula, Indicates the kernel size as One-dimensional convolutional layers are used to capture local features of different receptive fields; This indicates an adaptive max-pooling layer. Indicates a splicing operation; This is the output expert feature vector.
[0015] Step 5: Physicochemical Feature Extraction by Expert 2. This expert module aims to integrate native sequence information with predicted secondary structure information. First, the native sequence is encoded... The sequence implicit vector is obtained through embedding layer mapping. Secondary structure features The structural implicit vector is obtained through linear layer mapping. To adaptively filter out potential noise in secondary structure predictions, this step introduces a gated fusion mechanism:
[0016]
[0017] in For the Sigmoid function, For learnable weight matrices, This indicates a concatenation operation. Through this gating mechanism, the model can automatically adjust its dependence on the predicted structure based on the specific input. The fused features... The input then enters a multi-scale convolution module, where it is processed by one-dimensional convolutional layers with kernel sizes of 6 and 12 to capture the contextual dependencies of the local physical and chemical environment. The convolutional output is then subjected to adaptive max pooling and concatenation to obtain the feature vector of Expert 2. and category raw score .
[0018] Features after fusion The input is further fed into the convolutional feature extraction module, and the process is represented as follows:
[0019] in For the Sigmoid function, For learnable weight matrices, This indicates a splicing operation. Represents the complementary gating coefficients. Indicates the kernel size as One-dimensional convolutional layer This represents the global pooling layer.
[0020] Step 6: Graph Structure Expert 3 Extraction. This expert module uses a multi-track graph neural network. The input is... And protein contact map. This module contains parallel and serial graph convolution operations: (1) GCNII path. Use deep graph convolution with initial residuals and identity mapping to prevent over-smoothing and extract deep topological features of nodes. (2) GAT path. Use graph attention network to capture the importance of neighboring nodes using multi-head attention mechanism. (3) Pooling path. Use ASAPooling to coarse the graph and extract hierarchical structure features. Finally, the features of different levels and paths are globally averaged and stacked and then fused through self-attention mechanism to obtain the fused features of expert 3. and category raw score .
[0021] The graph structure expert 3 includes a multi-layer graph convolutional network GCNII path and a graph attention network GAT path; given input node features And graph structure edge index, the first in the GCNII path The node update formula for the layer is as follows:
[0022] In the formula, and For hyperparameters, For the normalized adjacency matrix, The layer weight matrix, The identity matrix; the final output features of the graph structure expert. It is obtained by attention-weighted fusion and pooling of the outputs of the GCNII path and the GAT path.
[0023] Phase 3: Cross-attention routing fusion.
[0024] Step 7: Construct a cross-attention router. To dynamically determine the importance of each expert, this invention designs a router. Define the query vector. For expert 1 characteristics Linear projection (as anchor points for sequence semantics); define the key vector. Stacking of three expert features Calculate the cross-attention weights. :
[0025] in The dimension is , represents the trust weight of each sample to the three experts, and Batch represents the number of samples processed in a batch; Key vector The feature dimension is used to scale the dot product result to prevent gradient vanishing.
[0026] Phase 4: Functional classification prediction.
[0027] Step 8: Weighted Prediction. Calculate the classification Logits from the three expert outputs. According to weight Perform weighted summation:
[0028] Final output Used to calculate loss or the probability of obtaining protein functional class through Sigmoid.
[0029] The beneficial effects of this invention: (1) Asymmetric routing mechanism based on semantic anchors: Existing technologies typically employ simple concatenation or symmetric self-attention mechanisms. This invention innovatively uses pre-trained sequence semantic features (expert 1) as the query vector and multi-perspective features containing physicochemical and topological information as the key vector. This design simulates the process of "dynamically retrieving the required structural and physicochemical environmental information with evolutionary semantics as the core index." This not only ensures that the model remains robust with sequence features as anchors even when high-precision structural data is lacking, but also effectively solves the "modal competition" problem caused by sparse structural features or excessive noise in multimodal fusion.
[0030] (2) Refined feature extraction design: In sequence processing, multi-scale convolution is introduced, which can simultaneously capture the point features, short-range motifs, and long-range dependencies of amino acids. In structure processing, a combination of GCNII and GAT is introduced, which effectively solves the over-smoothing problem of deep graph neural networks, while ASAPOooling is used to capture the hierarchical structural domain information of proteins.
[0031] (3) Introduction of gating mechanism: When processing the original sequence and secondary structure, gating fusion is used to automatically balance the contribution of the sequence itself and the predicted secondary structure information, thereby reducing noise interference. Attached Figure Description
[0032] Figure 1 This is a diagram illustrating the overall model architecture of the protein function prediction method of this invention. Figure 2 This is a flowchart illustrating the pre-trained feature expert module; Figure 3 This is the flowchart of the physicochemical characteristics expert module; Figure 4 This is a flowchart of the graph structure expert module. Detailed Implementation
[0033] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.
[0034] Figure 1 This invention presents a protein function prediction model framework based on a multi-perspective hybrid expert and cross-attention mechanism; Figure 2 , Figure 3 and Figure 4 The structures of the pre-trained feature expert module, the physicochemical feature expert module, and the graph structure expert module are shown respectively. For example... Figure 1 As shown, the workflow of this invention is as follows: In the model input stage, the acquired target protein amino acid sequence is first transformed and constructed using multimodal features. Specifically, for sequence features, this embodiment uses the pre-trained protein language model ProtT5-XL-U50. The amino acid sequence is input into this model, and its last hidden state is extracted to obtain a sequence embedding tensor with a dimension of 1024, in order to capture deep evolutionary semantic information. For physicochemical features, the sequence is scanned using NetSurfP-3.0 (or similar algorithms such as DSSP), predicting the probability distribution of secondary structure of each residue (including three states: Helix, Sheet fold, and Coil) and relative solvent accessible surface area (RSA), and concatenating them into a physicochemical feature vector. For topological structure, based on the protein's PDB three-dimensional structure file (or AlphaFold predicted structure), the topological features of all residues are extracted. -carbon atom( Coordinates. In this embodiment, the spatial contact distance threshold is set to 10 Å, and the coordinates of any two residues in the sequence are calculated. The Euclidean distance is used to establish connection edges when the distance is less than a threshold, thereby constructing a topological graph that reflects the three-dimensional folding information of the protein.
[0035] The constructed multi-source features are then fed into three parallel-designed expert modules to extract deep representations from different perspectives: such as Figure 2 As shown, the pre-trained feature expert directly receives sequence embeddings generated by ProtT5. The module incorporates multi-scale one-dimensional convolutional layers with kernel sizes set to 1, 6, and 12, aiming to simultaneously capture single-residue features, local motifs, and long-range dependencies in amino acid sequences. Figure 3 As shown, to effectively utilize the predicted secondary structure information and reduce its potential error, the physicochemical feature expert introduces a gated fusion mechanism. The network learns an adaptive gating weight to dynamically adjust the fusion ratio of the native one-hot encoding and the NetSurfP-3.0 predicted features. The fused features are further extracted through one-dimensional convolutional layers containing kernel sizes of 6 and 12, and finally pooled and concatenated to achieve a deep representation of the local physicochemical environment. Figure 4 As shown, the graph structure expert employs a hybrid architecture combining GCNII (Graph Convolution with Initial Residual Connections and Identity Mapping) and GAT (Graph Attention Network) to address the characteristics of topological graphs. The network consists of 3 GCNII layers and 1 GAT layer in parallel. This design effectively solves the oversmoothing problem that often occurs in traditional GCNs, ensuring that the model can extract high-order neighborhood information of nodes.
[0036] In the feature fusion stage, this embodiment constructs a cross-attention router. This router innovatively uses the sequential semantic features output by expert 1 as the query vector and the stacked output features of the three expert modules as the key vector. By calculating the attention score between the query and the key, the model dynamically generates an expert weight distribution for each sample. Finally, the Logits outputs of the three experts are weighted and summed according to this weight, and after Sigmoid activation, the final protein function prediction probability is output.
[0037] Example 1 In Example 1, this invention constructs and evaluates a model on the publicly available PDBset dataset and compares it with five mainstream methods: BLAST, DeepGO, DeepFRI, DeepGO-SE, and HEAL. BLAST is a traditional method based on sequence homology alignment; DeepGO is a deep learning method based on sequence features combined with the GO hierarchical structure; DeepFRI is a method based on graph convolutional neural networks for processing protein structures; DeepGO-SE uses a pre-trained language model to extract sequence features; and HEAL is a relatively new multimodal learning method. This invention uses ProtT5 to extract sequence features, NetSurfP-3.0 to extract secondary structure features, and combines these with protein contact maps, using a multi-view hybrid expert model for prediction.
[0038] To comprehensively evaluate model performance, this invention employs methods widely used in protein function prediction tasks. The maximum F1 score was used as the evaluation metric. This metric comprehensively considers the model's precision and recall. The experimental results are shown in Table 1.
[0039] Table 1. On the PDBset dataset Compare
[0040] By observing the data in Table 1, we can draw the following conclusions: (1) This invention achieves optimal performance in all three functional domains (MFO, BPO, CCO). Compared with the second-best performing method HEAL, this invention achieves superior performance in molecular function (MFO). The accuracy was improved by approximately 7.9%, with an improvement of approximately 11.8% in biological processes (BPO) and approximately 5.7% in cellular components (CCO). This demonstrates that the hybrid expert architecture proposed in this invention can effectively improve the accuracy of predictions.
[0041] (2) The deep learning-based method significantly outperforms the traditional homology alignment method (BLAST). BLAST relies on the existence of highly similar sequences in the database, and its predictive ability for proteins with low sequence similarity (such as "dark matter" proteins) is limited. In contrast, this invention learns the deep semantics and physicochemical rules of amino acids, thus exhibiting better generalization ability.
[0042] (3) Multi-view fusion methods are superior to single-modal methods. DeepGO-SE mainly relies on sequence features, DeepFRI mainly relies on structural features, while this invention (Ours) simultaneously utilizes the evolutionary sequence features of ProtT5, the secondary structure physicochemical features of NetSurfP-3.0, and the contact map topological features. This shows that fusing multi-source information can complement the shortcomings of single-view methods (e.g., in cases where sequences are similar but structures are different), thereby more accurately describing protein function.
[0043] Example 2 To further evaluate the model's ability to handle imbalanced data and exclude false positive predictions, Example 2 used AUPR (Area Under Precision-Recall Curve) as an evaluation metric. AUPR is particularly important for predicting function labels with a small number of positive samples, reflecting the model's predictive ability while maintaining high precision.
[0044] The AUPR comparison results of the present invention and the five methods in Example 1 on the PDBset dataset are shown in Table 2.
[0045] Table 2. AUPR Comparison on the PDBset Dataset
[0046] Based on the data in Table 2 and the technical principles of this invention, the analysis is as follows: (1) This invention demonstrates significant robustness in the AUPR metric. In the most challenging biological process (BPO) prediction task, the AUPR of this invention reaches 0.480, which is more than 2.5 times that of DeepGO (0.189) and far exceeds that of the suboptimal model HEAL (0.339). This is attributed to the cross-attention router designed in this invention. This mechanism can dynamically reduce the weight of irrelevant expert features according to the characteristics of the input data, thereby effectively suppressing noise interference and reducing false positive predictions.
[0047] (2) Structural information is crucial for improving AUPR. As shown in Table 2, DeepFRI and HEAL, which incorporate structural information, generally outperform DeepGO, which relies solely on sequences, in terms of AUPR. Building on this, this invention further introduces a graph neural network expert, extracting deep topological features through GCNII and GATConv, enabling the model to distinguish subtle differences in the microenvironment of functional sites, thereby achieving a high score of 0.573 in cellular component (CCO) prediction.
[0048] (3) Comprehensive performance analysis. Combining Tables 1 and 2, this invention not only... It achieves high coverage (high recall) while maintaining high accuracy on AUPR. This demonstrates that the multi-view hybrid expert network of this invention can both broadly discover potential functions and accurately remove mislabeled markers, making it an efficient and reliable method for protein function prediction.
Claims
1. A multi-perspective hybrid expert protein function prediction method based on cross-attention, characterized in that... This includes the following steps: Step 1: Sequence Feature Extraction; For the target protein sequence, sequence embedding features are extracted using a pre-trained protein language model, denoted as X; the pre-trained protein language model includes, but is not limited to, models based on the Transformer architecture; Step 2: Construction of primary and secondary structure features; encoding the amino acid sequence, for example using one-hot encoding, denoted as... Protein secondary structure prediction algorithms or tools are used to obtain protein secondary structure features that reflect the probability distribution of helices, folds, etc., denoted as... ; Step 3: Contact graph construction. Based on the three-dimensional structural coordinates of the protein, the Euclidean distance between the feature atoms of any two residues in the sequence is calculated. When the distance is less than or equal to a preset threshold, a connection edge is established, transforming the protein structure into a topological graph containing node features and edge indices. Step 4: Extract pre-trained feature experts 1, and construct a hybrid expert model that includes pre-trained feature experts, physicochemical feature experts, and graph structure experts; embed the sequence features obtained in Step 1 into the model. Input a pre-trained feature expert to extract sequence semantic feature vectors. and category scores ; Step 5: Physicochemical Feature Expert 2 Extraction, extracting the original features from Step 1. The physicochemical eigenvectors of the second step Input physicochemical feature experts, and extract physicochemical environmental feature vectors through gating fusion mechanism and multi-scale convolution. and category scores ; Step 6: Graph Structure Expert 3 Extraction, embedding features from Step 1. The third step, involving a topology graph structure expert, extracts global topology feature vectors through a multi-layer graph neural network. and category scores ; Step 7: Construct a cross-attention router, using the sequence semantic feature vector output in step 4. Using the query vector and the feature vectors of the three experts as key vectors, calculate the dynamic weight of each expert. ; Step 8: Weighted prediction, based on dynamic weights Classification scores of the three experts Weighted summation is performed to obtain the final protein function prediction result.
2. The multi-perspective hybrid expert protein function prediction method based on cross-attention as described in claim 1, characterized in that... The pre-trained feature expert 1 in the fourth step includes a linear projection layer, a channel transformation layer, and a multi-scale one-dimensional convolution module; given the input sequence features After linear projection and channel transformation, the result is obtained The feature extraction process of this expert module can be represented as: ； In the formula, Indicates the kernel size as One-dimensional convolutional layers are used to capture local features of different receptive fields; This indicates an adaptive max-pooling layer. Indicates a splicing operation; This is the output expert feature vector.
3. The multi-perspective hybrid expert protein function prediction method based on cross-attention as described in claim 1, characterized in that... The physicochemical feature expert 2 in the fifth step includes an embedding layer, a linear mapping layer, a gated fusion module, and a convolutional feature extraction module; given the original features Physicochemical characteristics First, the latent variables are obtained through mapping. and The gating fusion process is expressed by the following formula: ； ； Features after fusion The input is further fed into the convolutional feature extraction module, and the process is represented as follows: ； in For the Sigmoid function, For learnable weight matrices, Indicates a splicing operation; Represents the complementary gating coefficients. Indicates the kernel size as One-dimensional convolutional layer This represents the global pooling layer.
4. The multi-perspective hybrid expert protein function prediction method based on cross-attention as described in claim 1, characterized in that... The graph structure expert 3 in step six includes a multi-layer graph convolutional network GCNII path and a graph attention network GAT path; given input node features And graph structure edge index, the first in the GCNII path The node update formula for the layer is as follows: ； In the formula, and For hyperparameters, For the normalized adjacency matrix, The layer weight matrix, The identity matrix; the final output features of the graph structure expert. It is obtained by attention-weighted fusion and pooling of the outputs of the GCNII path and the GAT path.
5. The multi-perspective hybrid expert protein function prediction method based on cross-attention as described in claim 1, characterized in that... The calculation process of the cross-attention router in step seven is as follows: Define query vector Output for pre-trained feature experts Linear projection, defining the key vector Three expert characteristics Stacking; calculating cross-attention weights : ； In the formula, where The dimension is , represents the trust weight of each sample to the three experts, and Batch represents the number of samples processed in a batch; Key vector The feature dimension is used to scale the dot product result to prevent gradient vanishing.
6. The multi-perspective hybrid expert protein function prediction method based on cross-attention as described in claim 1, characterized in that... The weighted prediction in step eight is expressed by the following formula: The classification Logits from the three expert outputs are... According to weight Perform weighted summation: ； Final output Used to calculate loss or the probability of obtaining protein functional class through Sigmoid.