A protein function prediction method based on vector similarity retrieval
By employing a protein function prediction method based on vector similarity retrieval, and utilizing a pre-trained Transformer architecture and fine-tuning of the adaptation layer, the problem of insufficient model flexibility and adaptability in existing technologies is solved, achieving efficient and robust protein function prediction and classification.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NORTHEASTERN UNIV CHINA
- Filing Date
- 2023-02-06
- Publication Date
- 2026-07-21
AI Technical Summary
Existing protein function prediction methods are insufficient in efficiently utilizing existing data, adapting to data changes, and discovering new functional categories. Furthermore, their model structures are not flexible or adaptive enough, making it difficult to perform well under conditions of small samples and imbalanced data.
A protein function prediction method based on vector similarity retrieval is adopted. A pre-trained Transformer architecture is used with an adapter layer inserted. The model is trained using a triplet loss function, and protein function is predicted using vector similarity. In the downstream task, only the parameters of the adapter layer are fine-tuned to avoid full model adjustment.
It achieves efficient and flexible protein function prediction under different tasks, improves the robustness and generalization ability of the model, maintains good performance under small sample and imbalanced data conditions, and has the ability to discover new functional categories.
Smart Images
Figure CN116092579B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of deep learning and protein function technology, and in particular to a protein function prediction method based on vector similarity retrieval. Background Technology
[0002] Proteins are the fundamental building blocks of organisms and the carriers of life; all life activities are carried out through the functions of proteins. Proteins are composed of amino acid sequences, folding to form complex three-dimensional conformations to perform their biological functions. Understanding the relationship between protein sequence, structure, and function is a core issue in protein biology, crucial for understanding disease mechanisms and designing proteins and drugs for medical and bioengineering applications. Theory and practice have proven that the structural and functional information of proteins is hidden within their amino acid sequences, making it possible to predict protein function directly from their amino acid sequences. However, the complexity of the sequence-function relationship continues to challenge our computational modeling capabilities. As of March 2022, the UniProt sequence database contained 280 million protein sequences, but only 560,000 (approximately 0.2%) of these sequences had been manually reviewed and annotated by expert administrators to have clearly defined functions. While the number of protein sequences in public databases is growing explosively, progress in protein functional analysis is relatively slow due to the high cost and time-consuming nature of biological experiments. Therefore, the bioinformatics field urgently needs high-throughput, high-quality protein functional annotation tools.
[0003] Since protein function is a direct result of its amino acid sequence, for the past 25 years, researchers have systematically inferred function using similarity between primary sequences, such as BLAST tools. While generally useful, simple similarity measurements are often insufficient to determine protein function; highly dissimilar protein sequences can sometimes have similar functions, and even a single amino acid change can completely eliminate protein function. Subsequently, more complex statistical models, Profile Hidden Markov Models (pHMMs), were developed to address these challenges. Representative tools include HAMMER and HHblits. While these methods are highly beneficial, they often lack generality. Each pHMM model is typically trained on manually arranged sequences within a given protein family without utilizing information from other protein families or annotations. This results in highly specific Profile Hidden Markov Models, unable to detect functionally related but sequence-dissimilar proteins. Furthermore, these methods require massive computation to run on a large scale, posing a significant challenge for annotating protein databases or metagenomics.
[0004] Thanks to the rapid development of deep learning, especially the groundbreaking progress of data-driven language models in natural language processing, these models and methods have been applied to proteinformatics in recent years. They have the potential to provide general solutions to the challenges of protein function annotation and accelerate our ability to understand and utilize metagenomic data. Representative models include DeepSF (classifying 1195 fold classes using the SCOP dataset), DeepFam (classifying 2892 families using the COG dataset), and SECLAF (classifying hundreds of categories using the SwissProt dataset). However, these deep learning studies are either limited by the impractical or unrepresentative number and size of the datasets, or they cannot be compared with widely used traditional tools and methods. Furthermore, while these models and methods have achieved certain performance improvements, some problems have also arisen. For example: 1) Due to the tedious and complex experimental analysis of protein function, in practical applications of protein function prediction, there are very few functionally annotated protein sequences; some functional categories have only a few dozen, or even just a few, data entries. Furthermore, the distribution of sample sequence data across different functional categories is highly uneven. These issues make it difficult for existing models to extract generalizable representations of different protein categories, leading to overfitting and poor robustness and generalization ability. How to make models more efficient in utilizing existing data to produce reliable results is a pressing issue. 2) Various high-precision biological experiments have continuously improved our understanding of protein functions. Protein functional categories are actually added, deleted, or merged every year. Therefore, how to adapt existing models to constantly changing data and become more robust is a core problem currently facing protein functional classification. 3) The goal of protein function prediction is to annotate protein sequence data functionally. Our higher requirement for models is the ability to discover new functional categories. Existing methods generally add a Softmax function after the model output layer for category classification. This method actually limits the model's ability to explore and discover more. Vector similarity-based retrieval models do not explore the above problems in depth, and their performance is unsatisfactory. Moreover, existing model architectures are not flexible enough, unable to adjust the model structure according to the characteristics of downstream tasks, and lack scalability and adaptability. Summary of the Invention
[0005] The technical problem to be solved by the present invention is to provide a protein function prediction method based on vector similarity retrieval, which addresses the shortcomings of the prior art and enables the prediction of protein function.
[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a protein function prediction method based on vector similarity retrieval, comprising the following steps:
[0007] Step 1: Obtain the protein dataset and, after preprocessing, divide it into training, validation, and test sets according to the specified proportions;
[0008] Step 2: Construct a protein function prediction network model and train the vector representation of proteins;
[0009] The protein function prediction network model is based on a pre-trained Transformer architecture consisting of 33 Transformer blocks, with an application adaptation layer inserted between each Transformer block.
[0010] The input to the protein function prediction network model is the amino acid sequence of the protein, X∈R. l×a Where X is the one-hot encoded representation of the amino acid sequence of the input protein, l is the length of the protein sequence, and a is the dimension of the model training vocabulary; the model training vocabulary includes amino acid types and special characters ' <cls> ',' <pad> ',' <eos> ',' <unk>','.','-','<null_1>',' <mask>The amino acid sequence obtained after the embedding layer is represented as E∈R. l×d Where d is the dimension of each amino acid feature vector, and finally, after passing through 33 Transformer blocks, the vector representation of the amino acid sequence O∈R is obtained. l×d The mean of the obtained vector O is calculated over the amino acid sequence dimension l, and finally the vector representation of the amino acid sequence with dimension d is output.
[0011] Step 3: Pre-train the protein function prediction network model using a contrastive learning method;
[0012] The training sample data input to the protein function prediction network model is divided into three amino acid sequences of proteins: anchor example, positive example, and negative example. The loss function used for model training is triplet loss.
[0013] Step 4: Perform functional prediction on proteins in the test set using vector similarity retrieval;
[0014] Each protein sequence in the test set is represented by the output vector of the trained protein function prediction network model. The vector similarity of each protein sequence with all protein sequences in the training set is calculated, and the sequence with the highest vector similarity is used to label the protein sequence for function. The vector similarity is calculated using Euclidean distance, consistent with the calculation during training.
[0015] Step 5: Perform model transfer on the protein function prediction network model for downstream protein function classification tasks, thereby enabling the classification of different categories of protein functions.
[0016] When performing model transfer for downstream protein function classification tasks, the parameters of the application adaptation layer in the protein function prediction network model are fine-tuned only during the training process to achieve model transfer.
[0017] The beneficial effects of adopting the above technical solution are as follows: The protein function prediction method based on vector similarity retrieval provided by the present invention (1) proposes a lightweight and highly scalable parameter fine-tuning neural network framework (Adapter) based on the protein function prediction network model. Compared with full parameter fine-tuning, it only requires a small training and storage cost to achieve results comparable to full model fine-tuning. Its advantages are very obvious: 1) For different downstream tasks, the overall model parameters can remain unchanged, and only a small number of parameters of the Adapter model need to be trained. The training cost is small and the portability is strong. 2) For continuous learning of different tasks, since only different Adapter layers need to be trained when training different tasks, other model parameters remain unchanged, avoiding forgetting of past tasks when learning new tasks. For the continuous new tasks of protein function prediction, the Adapter parameter fine-tuning scheme proposed in this invention provides a general solution, which is more advantageous than the parameter adjustment of the entire model.
[0018] (2) Due to the various drawbacks of adding a Softmax function after the sequence representation layer for category classification, the method of this invention uses vector similarity retrieval to perform protein function classification tasks. The proposed protein function prediction network model can effectively cluster the semantic representations of proteins with the same function together in the vector space, and the model performance is minimally affected by the number of categories of training data samples in downstream tasks. This solves the problem of model training difficulties caused by the small number of samples and unbalanced category distribution in downstream tasks. At the same time, it enables the model to explore and discover new protein functions, and has stronger robustness and robustness.
[0019] (3) Based on the protein function prediction network model training framework proposed above, in order to enhance the feature representation of protein sequences and improve the performance of protein function classification tasks based on sequence vector representation, the present invention develops a general framework process for protein function classification. Its characteristic is that, based on the protein function prediction network model, the model is first trained on the comprehensive protein family set Pfam database with all parameters, allowing the model to learn the overall characteristics of each protein family. Then, it is applied to downstream classification tasks, with parameter fine-tuning only performed on the Adapter layer for each classification task. Results show that training using this method results in better model generalization and higher reliability, especially demonstrating superior performance on small sample data problems. Attached Figure Description
[0020] Figure 1 A flowchart illustrating a protein function prediction method based on vector similarity retrieval provided in an embodiment of the present invention;
[0021] Figure 2 This is a schematic diagram of the protein function prediction network model structure provided in an embodiment of the present invention;
[0022] Figure 3 A flowchart for function prediction based on vector similarity provided in an embodiment of the present invention;
[0023] Figure 4 The UMAP clustering diagrams of proteins with different functions provided by the protein function prediction model in the embodiments of the present invention are shown, wherein (a) is the characterization obtained by the first layer of the protein function prediction model, and (b) is the characterization obtained by the last layer of the protein function prediction model. Detailed Implementation
[0024] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.
[0025] In this embodiment, a protein function prediction method based on vector similarity retrieval is described, such as... Figure 1 As shown, it includes the following steps:
[0026] Step 1: Protein data is obtained from the Pfam protein family database and preprocessed before being divided into training, validation, and test sets. This embodiment uses a large protein function classification database to train all protein function prediction network model parameters, allowing the protein function prediction network model to extract more effective information from protein sequences, thereby improving the model's performance on downstream protein classification tasks. Pfam is a large protein family database with over 44 million protein sequences and 19,482 protein families (2021 version, PFam23.0). The number of protein families with different sequence counts in Pfam is shown in Table 1. A dataset is constructed using Pfam data. First, different families are divided into training, validation, and test sets in an 8:1:1 ratio. Since 1,422 protein families have fewer than 10 protein sequences, data cannot be partitioned, and this data is placed entirely in the training set. The remaining 18,060 protein families with 44,761,064 protein sequences result in the following training, validation, and test set sizes, as shown in Table 2. Since we aim to cluster protein sequences with similar functions together, we need to shorten the distance between sequences with the same function and widen the distance between sequences with different functions. The model uses a triplet loss function for training. The data fed into the model during training includes anchor examples, positive examples, and negative examples. The training data is constructed by randomly selecting two protein sequences from each of the 19,482 protein families in the training set, using them as the anchor and positive examples respectively, and randomly selecting one protein sequence from other families as the negative example. These three protein sequences constitute one training sample. 100 such training samples are generated for each protein family, resulting in 1,948,200 samples for model training.
[0027] Table 1. Relationship between protein sequences and family numbers in the Pfam database.
[0028] <10 1422 <100 5429 <200 2044 <300 1504 <400 1106 <500 805 ≥500 8509
[0029] Table 2 shows the number of protein sequences in the dataset.
[0030] training set 35,808,851 Validation set 4,476,107 test set 4,476,106 Total 4,476,1064
[0031] Step 2: Construct a protein function prediction network model and train the vector representation of proteins;
[0032] The protein function prediction network model is based on a pre-trained Transformer architecture consisting of 33 Transformer layers. To facilitate transfer learning for downstream tasks, an adapter layer is inserted between each Transformer layer. The model structure is as follows: Figure 2 As shown, the left side is the overall model framework, and the right side is the framework of the Adapter layer inserted between each Transformer block.
[0033] The input to the protein function prediction network model is the amino acid sequence of the protein, X∈R. l×a Where X is the one-hot encoding of the amino acid sequence of the input protein, l is the length of the protein sequence, and a is the dimension of the model training vocabulary (the vocabulary includes amino acid types ('L', 'A', 'G', 'V', 'S', 'E', 'R', 'T', 'I', 'D', 'P', 'K', 'Q', 'N', 'F', 'Y', 'M', 'H', 'W', 'C', 'X', 'B', 'U', 'Z', 'O') and some special characters ('...'). <cls> ',' <pad> ',' <eos> ',' <unk>','.','-','<null_1>',' <mask>The total length is 33). After the embedding layer, the amino acid sequence is represented as E∈R. l×d Where d is the dimension of each amino acid feature vector (the model is set to 1280 dimensions), and finally, after passing through 33 Transformer blocks, the vector representation of the amino acid sequence O∈R is obtained. l ×d The mean of the obtained vector O is calculated over the amino acid sequence dimension l, and finally the vector representation of the amino acid sequence with dimension d (the dimension set by the model is 1280) is output.
[0034] In this embodiment, the specific process of vector representation of proteins using a protein function prediction network model is as follows:
[0035] The protein function prediction network model takes as input the amino acid sequence of a protein, ranging in length from tens to thousands. The model truncates amino acid sequences longer than 1022, using only the first 1022 amino acids. The input amino acid sequence is defined as X = {x1, x2, ... x...}. l }, where l is the length of the amino acid sequence, and the input sequence is processed through the model's embedding layer to obtain the latent feature vector representation E∈R. l×d d represents the dimension of each amino acid feature vector, and finally, positional encoding is added before being fed into the Transformer block. Each Transformer block includes three sub-layers: a multi-head self-attention layer, a feed-forward layer, and an adapter layer for parameter fine-tuning. Each sub-layer has residual connections and layer normalization operations. In the adapter layer, the output vectors from the multi-head self-attention layer and the feed-forward layer are first mapped to a lower dimension, then passed through a ReLU non-linear activation layer, and finally mapped back to the original dimension.
[0036] Step 3: Pre-train the protein function prediction network model using a contrastive learning method;
[0037] The training sample data input to the protein function prediction network model consists of three protein amino acid sequences: anchor examples, positive examples, and negative examples. Each sequence generates a 1280-bit vector representation. The loss function used for model training is the triplet loss, as shown in the formula below:
[0038] Loss=max(d(a,p)-d(a,n))+margin,0)
[0039] Where a refers to the anchor example, p refers to the positive example, n refers to the negative example, d is the Euclidean distance between vectors, and margin is an adjustable threshold (set to 1.0 during model training).
[0040] In this embodiment, when training on the large protein family dataset Pfam, all model parameters are trained to allow the model to learn the overall characteristics of each protein family. Subsequently, for specific downstream tasks, only the parameters of the Adapter layer will be fine-tuned.
[0041] Step 4: Perform functional prediction on proteins in the test set using vector similarity retrieval, such as... Figure 3 As shown;
[0042] Each protein sequence in the test set is represented by the output vector of the trained protein function prediction network model. The vector similarity of each protein sequence is calculated with all protein sequences in the training set, and the sequence with the highest vector similarity is used for functional labeling. Here, the vector similarity is consistent with that used during training, employing Euclidean distance, specifically:
[0043]
[0044] Here, O1 and O2 are two vectors of length d, where d is the dimension of the vector.
[0045] In this embodiment, the vector representation of the test data obtained through the protein function prediction model is displayed using unified manifold approximation and projection (UMAP) clustering. Figure 4 As shown, the sequence representations output by the first Transformer Block are randomly distributed in the vector space, while the vector representations output by the last Transformer Block can perfectly cluster protein sequences with different functions together, thereby achieving protein function prediction based on vector similarity.
[0046] Step 5: Perform model transfer on the protein function prediction network model for downstream protein function classification tasks, thereby enabling the classification of different categories of protein functions.
[0047] A model with extremely large parameters, capable of capturing the functional characteristics of protein sequences, is trained using the Pfam database. When performing specific protein function classification tasks downstream, only parameter fine-tuning of the adapter layer in the model is needed during training. This embodiment uses subcellular localization prediction as an example. The location of a protein within a cell is closely related to its function; only in the appropriate subcellular location can a protein correctly perform its function. The computational prediction of protein localization within cells has always been a hot topic in bioinformatics. Most existing tools are based on protein sequences and machine learning methods. This embodiment uses the classic standard dataset BaCeILo, which includes proteins from animals, fungi, and plants, located in four subcellular compartments: cytoplasm, mitochondria, nucleus, and secretory pathway. Data statistics are shown in Table 3.
[0048] Table 3 Predicted Subcellular Localization Dataset
[0049] Animals_train 302 153 803 632 1890 Animals_test 137 35 363 172 707 Fungi_train 181 177 589 72 1019 Fungi_test 30 11 122 16 179 Plants_train 52 57 60 35 204 Plants_test 6 10 61 6 83
[0050] In the table, cy represents cytoplasm, mi represents mitochondria, nu represents nucleus, and Sp represents secretion pathway.
[0051] The training and testing process during the fine-tuning phase is almost identical to the previous Pfam dataset training and testing process. The only difference is that the training process only fine-tunes the Adapter layer in the model, while other parameters are frozen. The experimental results of the model are shown in Table 4:
[0052] Table 4 Results of the Predictive Subcellular Localization Experiment
[0053] BaCeILo 0.64 0.57 0.69 MultiLoc2-HighRes 0.68 0.53 0.62 YLoc-HighRes 0.74 0.56 0.58 This invention 0.77 0.79 0.71
[0054] The model experiment results show that the protein function prediction method based on vector similarity retrieval of this invention can greatly improve model performance. By applying the general vector representation learned in the large-scale family dataset Pfam to downstream tasks, only a small number of parameters of the Adapter Layer need to be adjusted for different tasks. It is smaller, faster, and more flexible, while maintaining excellent prediction performance. It can be promoted and applied as a paradigm for protein function prediction to more downstream tasks.
[0055] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.< / mask> < / unk> < / eos> < / pad> < / cls> < / mask> < / unk> < / eos> < / pad> < / cls>
Claims
1. A protein function prediction method based on vector similarity retrieval, characterized in that: Includes the following steps: Step 1: Obtain the protein dataset and, after preprocessing, divide it into training, validation, and test sets according to the specified proportions; Step 2: Construct a protein function prediction network model and train the vector representation of proteins; The protein function prediction network model is based on a pre-trained Transformer architecture consisting of 33 Transformer blocks, with an application adaptation layer inserted between each Transformer block. The input to the protein function prediction network model is the amino acid sequence of the protein. ,in, It is a one-hot encoding representation of the amino acid sequence of the input protein. It is the length of the protein sequence. This refers to the dimension of the model's training vocabulary; the amino acid sequence obtained after the embedding layer is represented as... ,in, This represents the dimension of each amino acid feature vector, and finally, after passing through 33 Transformer blocks, a vector representation of the amino acid sequence is obtained. For the obtained vector In terms of amino acid sequence dimension The mean is calculated, and the final output dimension is... Vector representation of the amino acid sequence; Step 3: Pre-train the protein function prediction network model using a contrastive learning method; Step 4: Perform functional prediction on proteins in the test set using vector similarity retrieval; Each protein sequence in the test set is represented by the output vector of the trained protein function prediction network model. The vector similarity of each protein sequence with all protein sequences in the training set is calculated, and the sequence with the highest vector similarity is used to label the protein sequence for function. Step 5: Perform model transfer on the protein function prediction network model for the downstream protein function classification task, thereby achieving the classification of different categories of protein functions.
2. The protein function prediction method based on vector similarity retrieval according to claim 1, characterized in that: The model training vocabulary includes amino acid types and special characters. <cls> ', ' <pad> ', ' <eos> ', ' <unk>','.', '-', '<null_1>', ' <mask> '。< / mask> < / unk> < / eos> < / pad> < / cls> 3. The protein function prediction method based on vector similarity retrieval according to claim 1, characterized in that: The training sample data input to the protein function prediction network model is divided into three amino acid sequences of proteins: anchor example, positive example, and negative example. The loss function used for model training is triplet loss.
4. The protein function prediction method based on vector similarity retrieval according to claim 3, characterized in that: The vector similarity between the protein sequence and all protein sequences in the training set is calculated using Euclidean distance.
5. The protein function prediction method based on vector similarity retrieval according to claim 4, characterized in that: In step 5, when transferring the model of the protein function prediction network model for the downstream protein function classification task, the parameters of the application adaptation layer in the protein function prediction network model are fine-tuned only during the training process to achieve model transfer.