Protein function discrimination and similarity calculation method based on cooperation of protein language model and support vector machine

Abstract

The invention discloses a protein function discrimination and similarity calculation method based on cooperation of a protein language model and a support vector machine, and belongs to the field of bioinformatics and artificial intelligence. The method comprises the following steps: firstly, carrying out standardized pretreatment on a protein sequence, converting the protein sequence into a space partition format, filtering non-standard amino acid characters, and then generating an input tensor and an attention mask tensor through filling and truncation; inputting the tensor into a pre-trained protein language model, extracting the hidden state of the last layer of Transform module, and performing mean pooling to obtain a global feature vector; constructing a data set in combination with the feature vectors and category labels, and performing stratified sampling to divide the data set into a training set and a verification set; training a support vector machine model with a radial basis function kernel by using the training set, optimizing hyper-parameters through grid search, and evaluating performance by using the verification set; and repeating preprocessing and feature extraction on a to-be-detected sequence, inputting a trained model output function category, and quantifying the similarity with known protein by using a decision function value.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of bioinformatics and artificial intelligence, and particularly relates to a deep learning method using a protein language model and a machine learning method without artificial neural networks, for classifying unknown protein sequences and calculating their similarity. BACKGROUND

[0002] In the field of bioinformatics and biotechnology, accurately determining the evolutionary classification of unknown proteins and quantifying the similarity of unknown proteins with known proteins in structure and function are important foundations for analyzing protein function mechanisms.

[0003] Currently, sequence alignment methods represented by BLAST are widely used in protein analysis. BLAST stands for Basic Local Alignment Search Tool. However, these sequence alignment methods still have limitations. On the one hand, this method relies on local sequence similarity for matching, and when protein sequence conservation is low, its recognition ability will be limited, and it is easy to miss detection, and cannot accurately capture the overall correlation information contained in the sequence. On the other hand, the analysis results of this method are easily affected by the parameter settings of E-value threshold, scoring matrix, and gap penalty. If the parameters are not properly selected, the classification results of the same batch of protein sequences may be unstable, making it difficult to ensure the consistency and reliability of the analysis results.

[0004] Compared with the above, artificial intelligence methods based on protein language models have unique advantages in pattern recognition. These methods rely on the Transformer architecture and can effectively analyze the long-range dependencies between amino acid residues while capturing evolutionary conservation patterns in sequences. They are particularly suitable for identifying distantly related homologous proteins with low sequence conservation. In dealing with low-conservation sequences that traditional methods cannot handle, they can achieve better recognition results. Although there are currently protein function prediction patents based on protein language models, existing technical solutions still have limitations. The current mainstream technical framework is mainly limited to two types of homogenization paths. The first type is to introduce specific domain data for fine-tuning based on pre-trained models, and the second type is to use protein language models to extract sequence features and then use artificial neural networks for discrimination.

[0005] The models developed based on the above two types of architectures often have too many parameters, which can cause overfitting problems during the training process. To prevent overfitting, a large-scale dataset is needed for training. For example, the protein sequence classification model developed by Lilhore et al. in 2024 is based on an improved convolutional neural network with attention mechanism and bidirectional long short-term memory unit. During the training process, a total of 14189 data were used, with about 7000 data for each category. Even with such a large dataset, the average accuracy is only close to 90%. Secondly, the large size of the model parameters also means that a large amount of computational resources are consumed during the training process, increasing the training cost. For example, GPT-4 has a parameter size of 1.75 trillion, consumes about 13 trillion tokens during training, and produces carbon emissions of about 20,000 tons. Large-scale resource consumption and high carbon emissions do not meet the development needs of green scientific research.

[0006] Therefore, if the above deep learning methods are directly applied, in the small dataset scenario, i.e., the dataset size is less than or equal to 1000, the model will face a great risk of overfitting, and it is difficult to ensure the accuracy of classification and similarity analysis. In addition, even in the larger dataset scenario, the computational process of this method is also costly, and the high cost will make resource-limited small and medium-sized laboratories hesitant, limiting the widespread application of the technology. At the same time, these existing methods generally lack the ability to quantitatively evaluate the similarity between proteins, and cannot accurately measure the similarity between the test protein and the representative sequence of the known functional category, making it difficult to meet the needs of research on quantitative analysis of similarity. SUMMARY

[0007] The purpose of the present application is to provide a protein function discrimination and similarity calculation method based on the cooperation of protein language model and support vector machine, to solve the problems of current methods in data adaptability, cost control and similarity quantitative evaluation.

[0008] To achieve the above purpose, the present application provides the following technical solutions:

[0009] A protein function discrimination and similarity calculation method based on the cooperation of protein language model and support vector machine, comprising the following steps:

[0010] Standardizing and preprocessing the protein sequence to obtain a standardized protein sequence;

[0011] Generating an input tensor and an attention mask tensor using the standardized protein sequence; inputting the input tensor and the attention mask tensor into a pre-trained protein language model, extracting hidden states and performing mean pooling processing to obtain a global feature vector;

[0012] A dataset is constructed according to the global feature vector and its corresponding class label, and the dataset is divided into a training set and a validation set; a support vector machine model with a specific kernel function is trained using the training set, and the model hyperparameters are optimized in combination with the validation set to obtain a trained support vector machine model;

[0013] The standardization preprocessing, input tensor and attention mask tensor generation, and global feature vector extraction operation are performed on the protein sequence to be tested to obtain a global feature vector of the protein to be tested;

[0014] The global feature vector of the protein to be tested is input into the trained support vector machine model, and the function category of the protein to be tested is output, and based on the decision function value output by the model, a similarity result of the protein to be tested and a known functional protein is obtained.

[0015] In a possible implementation, the step of standardizing the protein sequence includes:

[0016] The protein sequence is converted into a space-separated format to meet the requirements of the tokenizer of the protein language model;

[0017] Non-standard amino acid characters in the protein sequence are filtered, and standard amino acids are retained to obtain the standardized protein sequence.

[0018] In a possible implementation, the step of generating the input tensor and the attention mask tensor using the standardized protein sequence includes:

[0019] A maximum sequence length parameter is set, the standardized protein sequence longer than the parameter is truncated, and the standardized protein sequence shorter than the parameter is padded with special markers at the end to unify the lengths of all sequences;

[0020] An attention mask tensor is generated, the real marker positions corresponding to the standardized protein sequence are marked as 1, and the padding marker positions are marked as 0;

[0021] The processed sequence is mapped to a vocabulary ID tensor by a pre-trained tokenizer, and the vocabulary ID tensor is the input tensor, thereby obtaining the input tensor and the attention mask tensor.

[0022] In a possible implementation, the step of inputting the input tensor and the attention mask tensor into a pre-trained protein language model, extracting a hidden state, and performing mean pooling processing to obtain a global feature vector includes:

[0023] The input tensor and the attention mask tensor are input into a pre-trained protein language model to obtain a hidden state output by a last layer Transformer module of the model;

[0024] performing mean pooling processing on all token embedding vectors in the hidden state along the sequence length dimension to generate a fixed-dimension vector, which is a global feature vector.

[0025] In a possible implementation, constructing a dataset according to the global feature vectors and the corresponding class labels, and dividing the dataset into a training set and a validation set include:

[0026] loading the global feature vectors and the corresponding protein function class labels to combine to form an initial dataset;

[0027] dividing the initial dataset by using a stratified sampling strategy, so that the distribution proportion of each class in the divided training set and the validation set is consistent with that in the initial dataset, to obtain the training set and the validation set.

[0028] In a possible implementation, training a support vector machine model with a specific kernel function by using the training set, and optimizing model hyperparameters by using the validation set to obtain a trained support vector machine model include:

[0029] selecting a support vector machine model with a radial basis function kernel, and inputting the training set into the model for training;

[0030] optimizing the regularization coefficient and the kernel coefficient of the model by using a grid search strategy;

[0031] if there is a class imbalance in the training set, setting the model to assign a higher misclassification penalty weight to a minority class;

[0032] evaluating the performance of the model by using the validation set, and adjusting the hyperparameters until the performance of the model meets a standard to obtain the trained support vector machine model.

[0033] In a possible implementation, inputting the global feature vector of the to-be-tested protein into the trained support vector machine model, and outputting a function class of the to-be-tested protein include:

[0034] inputting the global feature vector of the to-be-tested protein into the trained support vector machine model to obtain a decision function value output by the model;

[0035] judging a function class to which the to-be-tested protein belongs according to the positive or negative of the decision function value, a positive value corresponding to a function class and a negative value corresponding to another function class, to obtain the function class of the to-be-tested protein.

[0036] In a possible implementation, based on the decision function value output by the model, obtaining a similarity result of the to-be-tested protein and a known function protein include:

[0037] Obtaining a decision function value corresponding to the protein to be tested, and a decision function value obtained after processing the known functional protein by the method;

[0038] By comparing the absolute values of the decision function values of the two, the closer the absolute values, the higher the similarity, and the similarity result of the protein to be tested and the known functional protein is obtained.

[0039] In a possible implementation, the step of evaluating the performance of the model by using the verification set comprises:

[0040] The precision, recall, F1 score and support number are used as evaluation indexes to quantitatively evaluate the prediction result of the support vector machine model on the verification set.

[0041] According to the evaluation result, it is judged whether the performance of the model meets the standard, and if not, the hyperparameters are adjusted and trained until the evaluation indexes meet the preset requirements.

[0042] In a possible implementation, the pre-trained protein language model is a protein pre-training model based on the Transformer architecture, which is used to extract context-aware semantic information of the protein sequence.

[0043] Compared with the prior art, the present application has the following advantages:

[0044] ①The present application can effectively avoid processing errors caused by non-standard input data or non-uniform dimensions by standardizing the protein sequence, filtering non-standard amino acid characters, unifying the sequence format, and using padding and truncation strategies to generate fixed-dimension input tensors and attention masks, ensuring that the data input into the pre-trained protein language model is consistent and standardized.

[0045] ②The pre-processed tensor is input into the pre-trained protein language model, the hidden state of the last layer of the Transformer module is extracted, and the global feature vector is obtained by mean pooling, which can more comprehensively capture the overall semantics and evolutionary information of the protein sequence compared with the method of using only the CLS marker feature, and the feature vector is more consistent with the association characteristics of protein function and structure.

[0046] ③The stratified sampling strategy is adopted to divide the data set, so that the distribution ratio of each category of the training set and the validation set is consistent with that of the original data set, and the problems of overfitting or underfitting caused by data distribution deviation can be avoided; the support vector machine with a radial basis function kernel is selected as the classification model, the hyperparameters are optimized by the grid search, and higher misclassification penalty weight is allocated to the minority class when the categories are unbalanced, so that the model can still maintain good classification performance in a small sample scenario, high classification accuracy can be achieved without relying on large data sets, the dependence on large parameter deep learning models is reduced, the calculation resource consumption in the model training process is reduced, and the problem that small and medium laboratories cannot apply related technologies due to high cost is solved.

[0047] ④The decision function value output by the support vector machine is used to realize the combination of unknown protein classification and similarity measurement, which can not only judge the protein function category by the positive and negative of the decision function value, but also measure the similarity between proteins by the absolute value of the decision function value, filling the gap of the lack of quantitative evaluation ability of protein similarity in existing methods. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0049] Figure 1 The flowchart of the protein function discrimination and similarity calculation method in the embodiments of the present application;

[0050] Figure 2 The technical roadmap of the protein function discrimination and similarity calculation method in the embodiments of the present application;

[0051] Figure 3 The technical roadmap of the discrimination of peptidoglycan recognition protein hydrolase activity and the calculation of protein similarity in the embodiments of the present application. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0053] Embodiment:

[0054] It should be noted that the terms "comprising" and "having" and any variations thereof in the embodiments of the present application are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units need not be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0055] Figure 1 The flowchart of the protein function discrimination and similarity calculation method in the embodiments of the present application; see Figure 1 The embodiments of the present application provide a protein function discrimination and similarity calculation method based on a protein language model and a support vector machine, characterized by comprising the following steps:

[0056] Step 101: standardizing and preprocessing the protein sequence to obtain a standardized protein sequence.

[0057] The step of standardizing and preprocessing the protein sequence includes: converting the protein sequence into a space-separated format to adapt to the tokenizer processing requirements of the protein language model; filtering non-standard amino acid characters in the protein sequence to retain standard amino acids to obtain the standardized protein sequence.

[0058] In specific implementation, the input continuous amino acid sequence is converted into a space-separated format, which can adapt to the tokenizer processing requirements of the protein language model, thereby ensuring that each amino acid residue can be identified as an independent semantic unit, so that the tokenizer can accurately capture the information of a single amino acid. While converting the format, non-standard amino acid characters such as the terminator * in the sequence are filtered at the same time. By filtering operation, it is ensured that the sequence input to the model only contains 20 standard amino acids, which can avoid the model encountering undefined lexical labels in subsequent processing, further enhance the standardization of input data, and improve the stability of the model processing process.

[0059] Step 102: generating an input tensor and an attention mask tensor using the standardized protein sequence; inputting the input tensor and the attention mask tensor into a pre-trained protein language model, extracting a hidden state and performing mean pooling processing to obtain a global feature vector.

[0060] The step of generating the input tensor and the attention mask tensor from the standardized protein sequence comprises: setting a maximum sequence length parameter, truncating the standardized protein sequence longer than the parameter, and padding the standardized protein sequence shorter than the parameter by adding special markers at the end to unify the lengths of all sequences; generating an attention mask tensor, marking the real token positions of the standardized protein sequence as 1 and the padding token positions as 0; and mapping the processed sequence to a vocabulary ID tensor by using a pre-trained tokenizer, so as to obtain the input tensor and the attention mask tensor.

[0061] The step of inputting the input tensor and the attention mask tensor into the pre-trained protein language model, extracting the hidden state and performing mean pooling processing to obtain the global feature vector comprises: inputting the input tensor and the attention mask tensor into the pre-trained protein language model to obtain the hidden state output by the last layer of the model; and performing mean pooling processing on the embedding vectors of all tokens in the hidden state along the sequence length dimension to generate a fixed-dimension vector, which is the global feature vector.

[0062] The pre-trained protein language model is a protein pre-training model based on a Transformer architecture and is used to extract context-aware semantic information of a protein sequence.

[0063] In specific implementation, since the pre-trained protein language model has a fixed requirement for the input dimension, a padding and truncation strategy is needed to unify the sequence length. When performing the truncation operation, a maximum sequence length parameter max_length is first set, and for a sequence longer than the parameter, the sequence is truncated in the order from the beginning to the end, and the fragment reflecting the core semantics in the sequence is retained to ensure that the key information of the sequence is retained after truncation. When performing the padding operation, for a sequence shorter than max_length, a special marker PAD is added at the end of the sequence to unify the lengths of all sequences to max_length, thereby meeting the fixed requirement of the model on the input dimension. While performing the padding and truncation operations, an attention mask is generated, and the positions corresponding to the real tokens in the sequence are marked as 1, and the positions corresponding to the padding marker PAD are marked as 0. The attention mask can guide the model to ignore the padding part in the subsequent model calculation process, thereby improving the calculation efficiency and reducing the noise interference caused by the padding part. Then, the tokenized sequence is mapped to a vocabulary ID tensor input_ids and an attention mask tensor attention_mask by using a pre-trained tokenizer. The dimensions of the two tensors are both [1, max_length], thereby realizing the standardization of the model input format and ensuring that the input data can be smoothly read and processed by the model.

[0064] The generated input_ids tensor and attention_mask tensor are input into the pre-trained protein language model. After the model runs, the hidden state output by the last layer of the Transformer module is obtained, which has a dimension of [1, L, D], where L represents the actual length of the sequence, and D represents the dimension of the hidden layer. The hidden state tensor contains deep semantic information of each amino acid in the global sequence context, and can reflect the association between amino acids and the overall characteristics of the sequence. To convert this hidden state into a more suitable feature form for subsequent classification, the mean pooling strategy is used to take the arithmetic mean of all token embedding vectors along the sequence length dimension L. The calculation method corresponds to the mean(dim=1) operation, and finally generates a fixed dimension global feature vector with a dimension of 1, D. Compared with the method of using only the CLS token feature, this mean pooling strategy can integrate the semantic information of all tokens in the sequence, avoid the information bias caused by a single token, and more comprehensively capture the overall sequence information, thereby further improving the robustness of the feature and supporting the improvement of the subsequent classification accuracy.

[0065] Step 103: Construct a data set according to the global feature vector and its corresponding class label, divide the data set into a training set and a validation set; train a support vector machine model with a specific kernel function using the training set, and optimize the model hyperparameters combined with the validation set to obtain a trained support vector machine model.

[0066] Among them, the step of constructing a data set according to the global feature vector and its corresponding class label, and dividing the data set into a training set and a validation set includes: loading the global feature vector and its corresponding protein function class label, and combining to form an initial data set; using stratified sampling strategy to divide the initial data set, so that the distribution ratio of each class in the divided training set and validation set is consistent with that of the initial data set, to obtain the training set and the validation set.

[0067] Among them, the step of training a support vector machine model with a specific kernel function using the training set, and optimizing the model hyperparameters combined with the validation set to obtain a trained support vector machine model includes: selecting a support vector machine model with a radial basis function kernel, inputting the training set into the model for training; using grid search strategy to optimize the regularization coefficient and kernel coefficient of the model; if there is a class imbalance in the training set, set the model to assign higher misclassification penalty weight to the minority class; use the validation set to evaluate the model performance, adjust the hyperparameters until the model performance meets the standard, to obtain the trained support vector machine model.

[0068] Further, the step of evaluating the model performance using the verification set comprises: using precision, recall, F1 score and support number as evaluation indexes to quantitatively evaluate the prediction results of the support vector machine model on the verification set; judging whether the model performance meets the standard according to the evaluation results, and if not, adjusting the hyperparameters and training again until the evaluation indexes meet the preset requirements.

[0069] In specific implementation, the pre-trained feature vectors obtained through the above steps are first loaded, and the category label data corresponding to the feature vectors is also loaded to ensure that each feature vector is associated with the correct category label. To avoid the influence of data distribution deviation on the training effect of the model, a stratified sampling strategy is used to divide the loaded data set, and the data set is divided into a training set and a verification set according to a preset proportion. In the division process, the distribution proportion of the data of each category in the training set and the verification set is strictly controlled to be consistent with that of the original data set. In this way, the problems of overfitting or underfitting of the model caused by uneven data distribution can be effectively avoided, and the generalization ability of the model in actual application is improved, so that the model can better adapt to different data scenarios.

[0070] Step 104: performing the standardization preprocessing, input tensor and attention mask tensor generation, and global feature vector extraction operation on the to-be-tested protein sequence to obtain a to-be-tested protein global feature vector.

[0071] Step 105: inputting the to-be-tested protein global feature vector into the trained support vector machine model to output the function category of the to-be-tested protein, and obtaining the similarity result of the to-be-tested protein and the known function protein based on the decision function value output by the model.

[0072] The step of inputting the to-be-tested protein global feature vector into the trained support vector machine model to output the function category of the to-be-tested protein comprises: inputting the to-be-tested protein global feature vector into the trained support vector machine model to obtain the decision function value output by the model; judging the function category to which the to-be-tested protein belongs according to the positive and negative of the decision function value, a positive value corresponding to one function category and a negative value corresponding to another function category, to obtain the function category of the to-be-tested protein.

[0073] The step of obtaining the similarity result of the to-be-tested protein and the known function protein based on the decision function value output by the model comprises: obtaining the decision function value corresponding to the to-be-tested protein and the decision function value obtained after processing the known function protein by the method; comparing the absolute values of the two decision function values to measure the similarity, the closer the absolute values, the higher the similarity, to obtain the similarity result of the to-be-tested protein and the known function protein.

[0074] In specific implementation, a support vector machine is selected as a classification model, and a radial basis function kernel is combined, which can effectively handle the possible nonlinear classification problem between the feature vector and the category, and the combination of the model and the kernel function is particularly suitable for small sample learning scenarios, and can maintain good classification performance under limited sample quantity. To further optimize the model performance, the key hyperparameters of the model are optimized through a grid search strategy, focusing on optimizing the regularization coefficient C and the kernel coefficient γ, to find a more suitable hyperparameter combination for the current data. If there is a class imbalance in the training data, i.e., the number of samples of different categories differs greatly, the support vector machine can be set to automatically assign a higher misclassification penalty weight to the minority class, so that the model pays more attention to the minority class samples during training, avoids the model being biased towards the majority class due to the difference in sample quantity, and ensures the classification ability of the model for samples of each category. In the model performance evaluation stage, precision, recall, F1 score and support are used as evaluation indexes to comprehensively quantify the classification effect of the model from the category level. These indexes can clearly reflect the performance of the model on different categories, ensuring the interpretability of the evaluation results.

[0075] For the unknown protein sequence to be classified, first, the complete process of the above sequence standardization preprocessing, input tensor generation, and context-aware feature extraction is performed to convert it into a fixed-dimensional feature vector, ensuring that the feature form of the unknown sequence is consistent with the feature form during model training. Then, the feature vector is input into the trained support vector machine model for classification prediction. The model outputs the signed distance of the sample to the decision hyperplane or the decision function value, where the signed distance corresponds to the linear kernel, and the decision function value corresponds to the nonlinear kernel. The class of the unknown sequence is determined according to the positive and negative of the output result. A positive value usually indicates that the sequence belongs to the positive class, and a negative value indicates that it belongs to the negative class. The absolute value of the result reflects the confidence of the model in making the classification judgment. The larger the absolute value, the higher the confidence. At the same time, the distance of the sequence feature vector to the support vector machine decision hyperplane can be used as an index to quantify the similarity between proteins. A positive distance indicates that the unknown protein is more similar to the positive protein, and a negative distance indicates that it is more similar to the negative protein. The absolute value of the distance can also be used to measure the strength of the similarity. The smaller the absolute value, the higher the similarity.

[0076] As a preferred example: the currently published protein language large model includes ESM2, ProtBERT and Ankh. The embodiment of the present application selects ProtBERT as the protein language model used in the study. The model is trained based on UniRef100, which contains about 217 million protein sequences. ProtBERT is a protein pre-training language model based on the Transformer architecture, which can better capture the semantic and evolutionary information in protein sequences and adapt to subsequent feature extraction requirements.

[0077] Figure 2 The technical roadmap of the protein function discrimination and similarity calculation method in the embodiment of the application mainly includes two parts. The first part mainly uses a protein language model to extract a global feature vector. The second part is to construct a protein classification model based on the global feature vector using a support vector machine algorithm, and calculate the similarity between proteins according to the constructed protein classification model. Figure 3 The technical roadmap of the peptide glycan recognition protein hydrolyase activity discrimination and protein similarity calculation in the embodiment of the application. The data for establishing the model is derived from NCBI. The protein language model is a pre-trained protBERT model. The data used to test the model is non-ncbi data. Combined with Figure 2 and Figure 3 A protein function discrimination and similarity calculation method based on a protein language model and a support vector machine, comprising the following steps:

[0078] Step 201: When performing sequence standardization preprocessing, the test data set used in the experiment is a peptide glycan recognition protein sequence, which is derived from the NCBI platform and covers 118 species. According to the chemical properties, these sequences can be divided into two groups. One group is a sequence with hydrolyase activity, a total of 543 sequences, and the other group is a sequence without hydrolyase activity, a total of 625 sequences. To fully verify the model performance, the above data is further divided into two groups for testing. The first group contains all the above sequences, and the second group contains relatively less data, in which the sequences with hydrolyase activity and the sequences without hydrolyase activity are each 375. Take the example sequence [MVSK…WSG] as an example. The sequence length is 198, and only contains standard amino acid characters, without filtering non-standard amino acid characters. Tokenization is performed on the sequence, which is converted into a space-separated form. This form can accurately identify each amino acid residue by the tokenizer of the protein language model. At the same time, max_length is set to 200. Since the length of the example sequence is less than 200, 2 PAD markers need to be added at the end of the sequence, finally forming an input sequence containing 200 tokens, to meet the fixed requirements of the pre-trained protein language model on the input length.

[0079] Step 202: After completing the sequence normalization preprocessing, enter the input tensor generation step. In the generated attention mask tensor, the first 198 positions are marked as 1, because these positions correspond to the actual amino acid residues of the example sequence, and the model needs to pay attention to the information of these positions; the last 2 positions are marked as 0, because these two positions are padding PAD markers, which can guide the model to ignore the padding part during calculation and reduce noise interference. Then the processed sequence is converted into an ID tensor by a tokenizer, for example, amino acid M corresponds to ID 2, amino acid V corresponds to ID 3, and the rest of the amino acids are converted in turn according to the preset mapping rules of the tokenizer. Finally, an ID tensor conforming to the model input format is obtained.

[0080] Step 203: When performing context-aware feature extraction, input the generated input_ids tensor and attention_mask tensor into the pre-trained protein language model ProtBERT to obtain the hidden state last_hidden_state output by the last layer of the Transformer module of the model. The dimension of this hidden state is [1, 198, 768], where 1 represents the number of samples for a single input, 198 represents the actual length of the example sequence, and 768 represents the hidden layer dimension. This tensor completely encodes the semantic information of each amino acid in the global sequence context. To obtain a feature that can reflect the overall characteristics of the sequence, the mean pooling strategy is used to calculate the arithmetic mean of the embedding vectors of all tokens along the sequence length dimension. The calculation process corresponds to the mean(dim=1) operation, and finally a fixed dimension global feature vector with dimensions 1 and 768 is generated. Compared with the method of using only the CLS marker feature, this mean pooling strategy can integrate the semantic information of all amino acids in the sequence, avoid the information bias that may be caused by a single marker, and thus more comprehensively capture the global information of the sequence, further improving the robustness of the feature and the accuracy of the subsequent classification.

[0081] Step 204: Data set construction and model training stage, first load the global feature vector of all samples together with the corresponding category label, where label 0 represents that the sequence has hydrolytic enzyme activity, and label 1 represents that the sequence does not have hydrolytic enzyme activity. In order to avoid the influence of data distribution deviation on the training effect of the model, stratified sampling method is used to divide the loaded data into training set and validation set, and ensure that the distribution ratio of sequences with and without hydrolytic enzyme activity in the training set and the validation set is consistent with the original data set. The SVM model with radial basis function kernel is selected for classification training, and the radial basis function kernel can effectively handle the possible nonlinear relationship between sequence features and categories. Through grid search and 5-fold cross-validation, the model hyperparameters are optimized, where the value of hyperparameter C includes 0.1, 1, 10, 100, and the value of hyperparameter γ includes scale, auto, 0.01, 0.1. The optimal parameters are determined according to the F1 score in the optimization process, and the F1 score can comprehensively reflect the balance of the classification performance of the model. At the same time, the class weight balance option class_weight=balanced is enabled during training, which can automatically assign higher misclassification penalty weight to the class with less sample number, avoid the model training bias to the majority class due to class imbalance, and ensure the classification ability of the model to each category sequence.

[0082] Step 205: After the model training is completed, enter the model evaluation and application step. Evaluate the model on the validation set, calculate the precision, recall, F1 score and support number of each sequence, where the precision can measure the proportion of samples that are actually of a certain category among those predicted to be of that category, the recall can measure the proportion of samples that are actually of a certain category among those predicted to be of that category, the F1 score can comprehensively reflect the balance level of precision and recall, and the support number can reflect the number of samples of each category. Through these indicators, the performance of the model can be comprehensively evaluated. By comparing the discrimination effect and running time of different models on two test data sets, it is found that the method of combining the protein language model with support vector machine can achieve similar discrimination effect as the algorithm based on artificial neural network, and in the case of small data set, i.e. 375 data per category, the operation accuracy and operation time of the method are more obvious, and it is more suitable for the needs of limited resources or small sample research.

[0083] In the function discrimination and similarity calculation of unknown protein sequences, first, the 768-dimensional global feature vector of the unknown protein sequence is extracted according to the above sequence standardization preprocessing, input tensor generation, and context-aware feature extraction process, and then the feature vector is input into the trained SVM model to obtain the decision function value output by the model. The class of the unknown protein sequence is judged according to the positive and negative of the decision function value. A positive value indicates that the sequence has hydrolytic enzyme activity, and a negative value indicates that the sequence does not have hydrolytic enzyme activity. The absolute value of the decision function value reflects the confidence of the model making the classification judgment. The larger the absolute value, the higher the confidence. At the same time, the decision function value can also be used as a measure of protein similarity. For example, if the decision function value of an unknown protein is 5.3 and the decision function value of a known hydrolytic enzyme protein in the database is 5.5, both have positive decision function values and the values are close, indicating that the distance between the unknown protein and the known hydrolytic enzyme protein in the feature vector layer and the SVM decision hyperplane is similar, so the two proteins are considered to be highly similar in chemical properties.

[0084] Table 1 shows the discrimination effect of the model on whether the peptidoglycan recognition protein has hydrolytic activity. Recall rate: how many of the true positive samples are predicted to be positive. Precision: how many of the predicted positive samples are true positive samples. F1 score: harmonic mean of precision and recall. The test data set includes 543 sequences with hydrolytic enzyme activity and 625 sequences without hydrolytic enzyme activity.

[0085] Table 1

[0086]

[0087] Table 2 shows the discrimination effect of the model on whether the peptidoglycan recognition protein has hydrolytic activity. Recall rate: how many of the true positive samples are predicted to be positive. Precision: how many of the predicted positive samples are true positive samples. F1 score: harmonic mean of precision and recall. The test data set includes 375 sequences with hydrolytic enzyme activity and 375 sequences without hydrolytic enzyme activity.

[0088] Table 2

[0089]

[0090] Table 3 shows the operation time consumed by different models in training and predicting the data set including 375 sequences with hydrolytic enzyme activity and 375 sequences without hydrolytic enzyme activity.

[0091] Table 3

[0092]

[0093] Table 4 shows the decision values of different peptidoglycan recognition proteins (only part of the protein sequence decision value results are shown).

[0094] Table 4

[0095]

[0096] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. Furthermore, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples, without contradiction.

[0097] The above embodiments are only for the purpose of illustrating the technical concepts and characteristics of the present application, and the purpose is to enable those of ordinary skill in the art to understand the content of the present application and to implement it, and cannot limit the protection scope of the present application. Any equivalent changes or modifications made according to the essence of the present application should be covered within the protection scope of the present application.

Claims

1. A method for protein function discrimination and similarity calculation based on the collaboration of protein language model and support vector machine, characterized in that, Includes the following steps: The protein sequence is preprocessed to standardize it, resulting in a standardized protein sequence. The standardized protein sequence is used to generate an input tensor and an attention mask tensor; the input tensor and attention mask tensor are input into a pre-trained protein language model, the hidden states are extracted and processed by mean pooling to obtain a global feature vector; A dataset is constructed based on the global feature vectors and their corresponding category labels, and the dataset is divided into a training set and a validation set. A support vector machine model with a specific kernel function is trained using the training set, and the model hyperparameters are optimized using the validation set to obtain the trained support vector machine model. The standardization preprocessing, input tensor and attention mask tensor generation, and global feature vector extraction operations are performed on the protein sequence to be tested to obtain the global feature vector of the protein to be tested. The global feature vector of the protein to be tested is input into the trained support vector machine model, which outputs the functional category of the protein to be tested. At the same time, based on the decision function value output by the model, the similarity result between the protein to be tested and known functional proteins is obtained.

2. The protein function discrimination and similarity calculation method based on the collaboration of protein language model and support vector machine according to claim 1, characterized in that, The steps for standardizing protein sequences include: The protein sequence was converted into a space-separated format to meet the requirements of the protein language model's word segmenter. The non-standard amino acid characters in the protein sequence are filtered out, and the standard amino acids are retained to obtain the standardized protein sequence.

3. The protein function discrimination and similarity calculation method based on the collaboration of protein language model and support vector machine according to claim 1, characterized in that, The steps for generating the input tensor and attention mask tensor using the standardized protein sequence include: Set a maximum sequence length parameter, truncate the standardized protein sequences that exceed this parameter, and add a special marker to the end of the standardized protein sequences that are too short to make all sequences uniform in length; Generate an attention mask tensor, and mark the actual label positions corresponding to the standardized protein sequence as 1 and the filled label positions as 0; The processed sequence is mapped to a vocabulary ID tensor by a pre-trained word segmenter. The vocabulary ID tensor is the input tensor, thereby obtaining the input tensor and the attention mask tensor.

4. The protein function discrimination and similarity calculation method based on the collaboration of protein language model and support vector machine according to claim 1, characterized in that, The steps of inputting the input tensor and attention mask tensor into a pre-trained protein language model, extracting the hidden states, and performing mean pooling to obtain the global feature vector include: Input the input tensor and attention mask tensor into the pre-trained protein language model to obtain the hidden state output by the last Transformer module of the model; The embedding vectors of all labels in the hidden state are arithmetically averaged along the sequence length dimension, and mean pooling is performed to generate a vector of fixed dimension, which is the global feature vector.

5. The protein function discrimination and similarity calculation method based on the collaboration of protein language model and support vector machine according to claim 1, characterized in that, The steps of constructing a dataset based on the global feature vector and its corresponding class labels, and dividing the dataset into a training set and a validation set, include: Load the global feature vector and its corresponding protein functional category labels, and combine them to form the initial dataset; The initial dataset is divided using a stratified sampling strategy so that the distribution ratio of each category in the divided training and validation sets is consistent with that in the initial dataset, thus obtaining the training and validation sets.

6. The protein function discrimination and similarity calculation method based on the collaboration of protein language model and support vector machine according to claim 1, characterized in that, The steps for training a support vector machine model with a specific kernel function using the training set and optimizing the model hyperparameters using the validation set to obtain the trained support vector machine model include: A support vector machine model with radial basis function kernels is selected, and the training set is input into the model for training; The regularization coefficients and kernel coefficients of the model are optimized using a grid search strategy. If there is a class imbalance in the training set, the model is set to assign a higher misclassification penalty weight to the minority class; The model performance is evaluated using the validation set, and the hyperparameters are adjusted until the model performance meets the target, thus obtaining the trained support vector machine model.

7. The protein function discrimination and similarity calculation method based on the collaboration of protein language model and support vector machine according to claim 1, characterized in that, The steps of inputting the global feature vector of the protein to be tested into the trained support vector machine model and outputting the functional category of the protein to be tested include: The global feature vector of the protein to be tested is input into the trained support vector machine model to obtain the decision function value output by the model. The functional category of the protein to be tested is determined by the sign of the decision function value. A positive value corresponds to one functional category, and a negative value corresponds to another functional category, thus obtaining the functional category of the protein to be tested.

8. The protein function discrimination and similarity calculation method based on the collaboration of protein language model and support vector machine according to claim 1, characterized in that, The steps for obtaining the similarity results between the protein to be tested and known functional proteins based on the decision function value output by the model include: Obtain the decision function value corresponding to the protein to be tested, and the decision function value obtained after processing a known functional protein by the method described above; Similarity is measured by comparing the absolute values ​​of the decision function values ​​of the two proteins. The closer the absolute values ​​are, the higher the similarity. This yields the similarity results between the protein to be tested and known functional proteins.

9. The protein function discrimination and similarity calculation method based on the collaboration of protein language model and support vector machine according to claim 6, characterized in that, The steps for evaluating model performance using the validation set include: Precision, recall, F1 score, and number of supports are used as evaluation metrics to quantitatively evaluate the prediction results of the support vector machine model on the validation set. The evaluation results determine whether the model performance meets the standards. If it does not, the hyperparameters are readjusted and the model is trained until the evaluation metrics meet the preset requirements.

10. The protein function discrimination and similarity calculation method based on the collaboration of protein language model and support vector machine according to claim 1, characterized in that, The pre-trained protein language model is a protein pre-training model based on the Transformer architecture, used to extract context-aware semantic information of protein sequences.

Images