Method for identifying phosphorylation modification sites based on large model technology

Abstract

The invention belongs to the field of artificial intelligence algorithm application-biological sequence recognition, and relates to a method for recognizing phosphorylation modification sites based on a large model technology. According to the method, the latest protein large model technology ESM-2 is applied to the phosphorylation modification site recognition direction, the current forefront technology is fully utilized to recognize the phosphorylation modification site infected by SARS-CoV-2, and the recognition accuracy and reliability are remarkably improved. According to the method, the intermediate output of the protein pre-training large model is innovatively extracted for subsequent research. For different tasks, different level output features may be needed, the research provides a clear example, and the effectiveness of model intermediate layer output is displayed. According to the method, transfer learning related technologies are creatively used, so that the problem that classification prediction cannot be effectively carried out due to the fact that tyrosine (Y) data in SARS-CoV-2 infected phosphorylation modification sites are rare is effectively solved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence algorithm application-biological sequence recognition, and relates to a method for identifying phosphorylation modification sites based on large model technology. Background Art

[0002] Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is a highly contagious and pathogenic virus. With the development of bioinformatics and artificial intelligence technologies, more and more computational tools are being used to identify phosphorylation modification sites of SARS-CoV-2 infection. Compared with traditional experimental methods, predictive recognition models based on machine learning and deep learning have higher efficiency and lower cost, and can quickly analyze a large number of protein sequences and identify potential modification sites. However, existing prediction models still face multiple challenges, such as insufficient sample data, the inability of the model to fully learn sequence information, the limited ability of the model to capture the complex dependencies of sequence information, and insufficient recognition accuracy of phosphorylation modification sites.

[0003] Therefore, developing an identification and prediction tool that can deeply learn sequence information, fully capture the complex dependencies of sequence information, and have high accuracy is crucial for gaining a deeper understanding of the molecular mechanisms of SARS-CoV-2 infection and changes in host cell pathways, as well as for rationally repurposing drugs to respond. Summary of the invention

[0004] In order to solve the above problems, the present invention combines the currently popular large model technology and proposes a new method to identify phosphorylation modification sites infected by SARS-CoV-2. This method uses the Transformer-based protein large model ESM-2 and transfer learning technology to fine-tune and extract features of the large model, and combines machine learning (svm, xgboost) for weighted integration to identify phosphorylation modification sites infected by SARS-CoV-2.

[0005] The technical solution of the present invention is as follows:

[0006] The data used in this invention comes from the phosphorylation site dataset of SARS-CoV-2 infected human A549 cells experimentally verified by Stukalov et al. The dataset contains 14,119 phosphorylation sites, which are divided into S / T site and Y site datasets. Since the amount of Y site data is extremely scarce, the large model cannot fully learn its characteristics, so the transfer learning technology is used to make the model's performance on the Y site dataset significantly improved. Finally, a user-friendly Web prediction tool was developed.

[0007] The method for identifying phosphorylation modification sites based on large model technology is as follows:

[0008] Step 1: Dataset Construction

[0009] In order to develop statistical prediction models, it is essential to build reliable training and test datasets. The specific steps are as follows:

[0010] In this invention, a phosphorylation site dataset of SARS-CoV-2 infected human A549 cells experimentally verified by Stukalov et al. was used, which contains a total of 14,119 phosphorylation sites. In order to construct a high-quality benchmark dataset, reduce sequence redundancy and prevent model overfitting, the following steps were taken for data processing:

[0011] First, the 14,119 protein sequences collected were processed using the CD-HIT tool, and sequences with a similarity of more than 30% were removed, ultimately obtaining 10,978 protein sequences. Subsequently, in order to facilitate model training and comparison with other phosphorylation site prediction methods, the processed sequences were truncated into peptides containing serine / threonine (S / T) or tyrosine (Y) in the center. These sequences with phosphorylation sites can be represented by the following structure: S(p) = A -n A -(n-1) …A -2 A -1 PA + 1A +2 …A +(n-1) A +n Here, the highlighted letter P represents the phosphorylation site S / T or Y, while A n The symbol "n" indicates the position of a given amino acid relative to the central residue, where A -n and A +n They represent the nth residue on the left and right side of the phosphorylation site, respectively.

[0012] In order to determine the value of n, the length that can effectively extract sequence information determined by previous researchers (i.e., n=16) was used as a reference, that is, the length of the peptide fragment was truncated to 33. If the S / T or Y residue at the center of the fragment is phosphorylated, it is defined as a positive sample; otherwise, it is a negative sample. In the end, the data set centered on the S / T site contained 10,774 samples, of which both positive and negative samples contained 5,387 sequences; the data set centered on the Y site contained 204 samples, of which positive and negative samples each contained 102 sequences. In order to ensure the reliability and fairness of the results, the data set was strictly divided into training and test sets in a non-overlapping ratio of 8:2. At the same time, in order to balance the number of positive and negative samples in the training data set and to objectively and fairly compare the model performance, this study used a non-redundant negative sample subset randomly selected by Lv et al. to match the number of positive samples.

[0013] Step 2: Feature Extraction

[0014] In order to allow the computer to recognize the data cleaned in the first stage, this stage extracts features from the serine / threonine (S / T) or tyrosine (Y) sequence information.

[0015] 2.1 The steps for extracting the features of the serine / threonine (S / T) site sequence information are as follows:

[0016] 2.1.1: In the present invention, the truncated 33-bit S / T site sequence is input into the ESM-2 model, and the ESM-2 model is fine-tuned to learn more information contained in the S / T site sequence.

[0017] Depending on the number of parameters and hidden layers used in training, ESM-2 has several versions as shown in Table 1.

[0018] Table 1. ESM-2 model parameters for different versions

[0019]

[0020] In the present invention, the ESM-2 model with a 36-layer structure is adopted by comprehensively considering the hardware and software conditions and the scale of the data set.

[0021] Assume that the input sequence is X = [x1, x2, ..., x n ], where x i is the word vector representation of the i-th position in the input sequence. The core calculation steps of the self-attention mechanism include linear transformation, calculation of attention score and weighted summation. First, in order to calculate the attention score, the self-attention mechanism performs a linear transformation on each word vector x of the input. i Perform linear transformation to generate three vectors: query vector (Query), key vector (Key) and value vector (Value). These three vectors are generated by different weight matrices:

[0022] Q=XW Q , K=XW K , V=XW V (1)

[0023] Among them, Q, K, V are query matrix, key matrix and value matrix respectively, and W Q , W K , W V is the learned weight matrix.

[0024] Next, we calculate the attention score. For each position i in the sequence, we need to calculate the dot product between its query vector and the key vectors of all other positions in the sequence to get the attention score. Specifically, the query vector q i With all key vectors k j The dot product can be expressed as:

[0025]

[0026] Among them, d k is the dimension of the key vector, the dot product is divided by This is to avoid the dot product value being too large, which may cause the gradient to disappear or explode.

[0027] The multi-head self-attention mechanism enables the model to simultaneously focus on different features and patterns in the sequence by performing the above self-attention calculations in multiple different subspaces. Each subspace corresponds to a "head", and each head has its own independent linear transformation matrix of queries, keys, and values. Specifically, given the input matrix X, the multi-head self-attention mechanism linearly transforms it into multiple queries, keys, and values ​​of different dimensions, and then performs self-attention operations independently in each subspace, and finally splices the results together. The whole process is divided into multi-head query, key and value calculation, and splicing head structure. The first is multi-head query, key and value calculation, which generates multiple queries, keys, and values ​​through multiple different linear transformations of the input sequence X:

[0028]

[0029] in is the weight matrix of the query, key, and value corresponding to the i-th head.

[0030] Next, the head results are concatenated. After each head completes the self-attention calculation independently, these results are merged to obtain the output of multi-head self-attention:

[0031] MultiHead(Q,K,V)=Concat(head1,...,head h )W O (4)

[0032] Among them, W O is the final linear transformation matrix, h is the number of attention heads, and Concat(·) means concatenating the outputs of multiple heads together.

[0033] 2.1.2: In the present invention, in order to optimize the fine-tuning training process of the ESM-2 model, a linear learning rate scheduler is used. This method helps the model remain stable at different stages of training by gradually adjusting the learning rate, avoiding the instability caused by excessive learning rate in the early stage of training, and also preventing overfitting by reducing the learning rate in the later stage of training. The learning rate is one of the key hyperparameters in deep learning training, which determines the pace of model parameter updates. During the training process, if the learning rate is set too high, the model may oscillate or fail to converge; if the learning rate is too low, the model may converge too slowly or even fall into a local optimal solution. In order to balance the two, a linear attenuation strategy is adopted, that is, during the entire training process, the learning rate decreases linearly from the initial value to near zero. The formula of the linear learning rate scheduler is as follows:

[0034]

[0035] 2.1.3: After obtaining the fine-tuned ESM-2 model, the output of the 33rd layer was selected for subsequent research from multiple perspectives.

[0036] Innovation in model architecture understanding and hierarchical feature mining: The significance of deep hierarchical feature selection. ESM-2 is a deep pre-trained language model. Features closer to the input layer tend to capture local patterns or low-order features of the sequence, while features closer to the output layer are more global and task-specific. Selecting the position of the 33rd layer not only avoids low-order features that are too shallow, but also avoids high-order features in the last layer that may overfit the task, reflecting the ability to accurately mine hierarchical features. The uniqueness of the 33rd layer is that compared to directly using the output of the final layer, the extraction of this intermediate layer can retain the sequence context information while avoiding over-specialization in the original task. Experimental or theoretical analysis can show that the features output by the 33rd layer are more suitable for the phosphorylation site identification task in terms of the biological function or structure of the expression sequence.

[0037] Contribution to biological sequence representation learning: Biological adaptability of deep models: The information contained in different layers in ESM-2 varies to a certain extent, and the 33rd layer may just correspond to a specific local and global information fusion point in the sequence. Through fine-tuning and feature extraction, the adaptability of the output of a specific layer of ESM-2 to biological tasks is verified, which provides new ideas for exploring the biological interpretability of language models. Inspire more sophisticated task design: By analyzing the characteristics of specific layers, it can inspire researchers in other fields to use language models more sophisticatedly, such as designing hierarchical feature fusion or multi-task learning frameworks for the representation capabilities of different layers.

[0038] Innovation in the combination of fine-tuning and feature extraction: The differentiated fine-tuning strategy focuses on the output of a specific level while fine-tuning the ESM-2 model, rather than the traditional global output. This reflects the deep integration of model characteristics and task requirements, and can provide an example of the combination of fine-tuning and feature extraction for subsequent research.

[0039] Balancing computational efficiency and performance, selecting the 33rd layer output instead of extracting features from all layers or completely retraining can significantly reduce computational resource requirements, which meets the actual needs of current model deployment and use.

[0040] 2.2 The steps for extracting tyrosine (Y) sequence information features are as follows:

[0041] 2.2.1: The amount of tyrosine (Y) sequence data is scarce, so the large model cannot fully learn the feature information. However, since both the Y site and the S / T site belong to the recognition category of phosphorylation sites after SARS-CoV-2 infection, transfer learning technology was introduced. The ESM-2 model fine-tuned on the S / T site sequence was placed on the Y site sequence dataset for transfer learning. Based on the feature information learned by the model at the S / T sequence site, the feature information of the Y site sequence was further learned to obtain the best recognition effect of the Y site sequence.

[0042] 2.2.2: Although transfer learning is used to enable the large model to better learn the feature information of the Y site, the number of extracted features is very small because the input Y site data itself is very scarce, so the number of output features is also very scarce, which is not conducive to the subsequent base classifier construction. Therefore, the features of the S / T sequence site are combined with the features of the Y sequence site to finally obtain the features of the Y sequence site.

[0043] Step 3: Classification prediction model construction

[0044] After the data feature extraction was completed, the S / T site sequence feature data and Y site sequence feature data with a dimension of 1280 were obtained. In this stage, in order to accurately identify the phosphorylation modification sites of SARS-CoV-2 infection, a classification prediction model was constructed, which is described as follows:

[0045] 3.1ESM-2for Sequence Classification

[0046] In this study, the SequenceClassification version of the ESM-2 model was used. Compared to the standard ESM-2 model, this version adds a linear layer on the pooled output layer, which is particularly suitable for application scenarios such as GLUE tasks. When loading, the weights and bias parameters of this additional layer are reinitialized, which makes it possible to use ESM-2for SequenceClassification to fine-tune on the dataset. In this way, the model can learn weights and bias parameters that are more suitable for a specific dataset.

[0047] The S / T site dataset is sent to ESM-2for SequenceClassification for fine-tuning training to obtain Model 1. Since the amount of data for the Y site is small, but both the Y site and the S / T site belong to the task of identifying whether phosphorylation is modified, the transfer learning technology can be used to optimize the model performance. Specifically, Model 1 is loaded and further trained on the Y site dataset to learn relevant information of the Y site dataset. Finally, a Model 2 that is more suitable for the Y site is obtained. The pseudo code for model fine-tuning and feature extraction is as follows:

[0048]

[0049]

[0050] 3.2 Machine Learning Model Construction

[0051] Support vector machines (SVM) perform well in high-dimensional space, can effectively process high-dimensional data and have nonlinear classification capabilities. Extreme gradient boosting (XGBoost) performs well in processing large-scale data sets and complex feature relationships, and has excellent generalization ability and high accuracy. Therefore, for the S / T site, the features extracted from the ESM-2 model are sent to SVM and XGBoost for model training respectively, so as to obtain two base classifiers. For the Y site samples with scarce data, the feature data of the S / T site extracted from the ESM-2 model is combined with the feature data of the Y site, and then sent to XGBoost for joint learning to obtain a suitable base classification model. The brief description of SVM and XGBoost is as follows:

[0052] SVM (Support Vector Machine) is an efficient machine learning algorithm widely used in classification and regression tasks, and is particularly good at handling complex classification problems. SVM performs well in handling high-dimensional data sets and classification problems with complex boundaries, especially for small-scale data sets, but has high computational overhead on large-scale data sets. For linear data processing, given a data set D, the equation corresponding to the optimal hyperplane in n-dimensional space is:

[0053] g(x)=w·xb=0 (6)

[0054] The real vector of the sample in space is denoted as x, its category is denoted as y, w, and b is the objective function. Minimum value under the constraint y(w·xb)≥1.

[0055] XGBoost (eXtreme Gradient Boosting) is an efficient machine learning algorithm based on gradient boosting. Its core mechanism is to fit the prediction error of the previous round of models by gradually building decision trees, and use the gradient descent method to minimize the loss function, thereby continuously improving the prediction accuracy of the model. The prediction value of XGBoost can be expressed as:

[0056]

[0057] in is the predicted value of the i-th sample, f k (x i ) is the kth tree for sample x i The prediction results of , K is the total number of trees.

[0058] In order to evaluate the prediction effect of the model in the binary classification task, XGBoost usually uses the logarithmic loss function as its core indicator. This function is specifically used to quantify the degree of deviation between the probability value predicted by the model and the true label. Its mathematical expression is as follows:

[0059] L(y,p)=-ylog(p)-(1-y)log(1-p) (8)

[0060] Here, y is the actual class label (0 or 1) and p is the probability of the model predicting the positive class (class 1).

[0061] Step 4: Weighted integration of ESM-2 and machine learning models

[0062] In this stage, the present invention performs weighted integration based on the ESM-2 for Sequence Classification large model obtained in the third step and the machine learning model to obtain the final classifier. The specific steps are as follows:

[0063] 4.1 In the classification prediction stage of the present invention, an integrated model strategy is adopted to integrate the advantages of different models. Specifically, the deep representation ability of the ESM-2SequenceClassification model on sequence data, the powerful generalization performance of the support vector machine (SVM) in high-dimensional feature space, and the prediction accuracy of the XGBoost model based on the efficient decision tree structure are utilized. By combining the fine-tuned ESM-2SequenceClassification model with the machine learning model (SVM, XGBoost), and integrating the prediction results of the three models by weighted averaging, a more robust and accurate classification prediction result is obtained.

[0064] 4.2 Model Validation Test

[0065] In order to comprehensively evaluate the effectiveness of the deep learning model, accuracy (Acc), sensitivity (Sn), specificity (Sp), MCC and AUC were selected as key performance indicators. The specific descriptions of these indicators are as follows:

[0066]

[0067]

[0068]

[0069]

[0070]

[0071] Among them, TP represents the number of correctly predicted SARS-CoV-2 phosphorylation sites, TN represents the number of correctly identified non-SARS-CoV-2 phosphorylation sites. FP represents the number of incorrectly identified SARS-CoV-2 phosphorylation sites, FN represents the number of incorrectly identified non-SARS-CoV-2, and TPR represents the number of incorrectly identified non-SARS-CoV-2 phosphorylation sites. i Represents the true rate corresponding to the i-th threshold, FPR i represents the false positive rate corresponding to the i-th threshold, and n represents the total number of thresholds.

[0072] Beneficial effects of the present invention:

[0073] 1. The present invention applies the latest protein large model technology ESM-2 to the identification of phosphorylation modification sites, making full use of the most cutting-edge technology to realize the identification of phosphorylation modification sites infected by SARS-CoV-2, significantly improving the accuracy and reliability of identification.

[0074] 2. This paper innovatively extracts the intermediate output of the protein pre-trained large model for subsequent research. Faced with different tasks that may require different levels of output features, this study provides a clear example that demonstrates the effectiveness of the model's intermediate layer output.

[0075] 3. The present invention innovatively uses transfer learning related technologies to effectively solve the problem that tyrosine (Y) data in phosphorylation modification sites of SARS-CoV-2 infection is scarce and thus cannot be effectively classified and predicted.

[0076] 4. By fine-tuning ESM-2 to extract rich feature information to construct a base classifier with significant effects, the recognition and prediction performance of phosphorylation modification sites of SARS-CoV-2 infection was significantly improved by weighted integration of ESM-2for Sequence Classification, SVM, and XGBoost.

[0077] 5. In order to facilitate researchers to use the present invention to improve experimental efficiency, a free and available web site was developed: http: / / www.lmtlips.com. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 It is the overall technical structure flow chart of the present invention.

[0079] Figure 2 The figure is a comparison of the feature extraction effects of different large model methods in the present invention on the S / T site (A) and Y site (B) data sets.

[0080] Figure 3 ROC curves showing the prediction performance of the model in different parts of the independent test datasets at the S / T site (A) and the Y site (B).

[0081] Figure 4 It represents the distribution of positive and negative samples of the S / T and Y site training sets in the original 2D feature space and the 2D feature space after feature extraction.

[0082] Figure 5 Indicates the top ten feature numbers of the machine learning model classifier features in terms of importance for the S / T site (A) and Y site (B). DETAILED DESCRIPTION

[0083] The specific implementation of the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0084] The present invention proposes a method for identifying phosphorylation modification sites based on large model technology, which is used to predict the phosphorylation modification site classification model of SARS-CoV-2 infection. Figure 1 As shown. The model is fine-tuned using the ESM-2 pre-trained model based on the Transformer architecture. By fine-tuning ESM-2 on the S / T site dataset, Model 1 suitable for the S / T site dataset is obtained. Subsequently, in order to overcome the problem of insufficient Y site dataset, a transfer learning strategy was adopted. Specifically, the model was first preliminarily fine-tuned on the S / T dataset, and then further fine-tuned on the Y site dataset to obtain Model 2. In the feature extraction stage, the S / T site and Y site datasets are input into the fine-tuned Model 1 and Model 2 respectively, and features are extracted from the middle layer and saved. In the classification prediction part, the S / T site feature representation extracted by Model 1 is input into the machine learning model for training to build a classification model suitable for the S / T site dataset. Then, the features extracted by Model 1 and Model 2 are jointly input into the machine learning model to train a classification model suitable for the Y site dataset. Finally, the obtained machine learning model is weighted averaged with the fine-tuned ESM-2For Sequence Classification model to obtain the final classification prediction result. In order to evaluate the performance of the model, a five-fold cross validation was used to evaluate and optimize the model, and then an independent test set was used for verification. The evaluation indicators covered accuracy (Acc), sensitivity (Sn), specificity (Sp), Matthews correlation coefficient (MCC) and area under the curve (AUC). The experimental results show that the prediction model LM-TLIPs constructed in this study achieved satisfactory prediction performance on both the S / T and Y datasets, and significantly improved the prediction accuracy of SARS-CoV-2 phosphorylation sites. The experimental results show that the method LMTLIPs proposed in the present invention is significantly superior to existing models such as DeepIPs, PSPred-ALE and Res-GCN in the prediction task of phosphorylation modification sites infected by SARS-CoV-2. In addition, in order to facilitate users to use the prediction model constructed by the present invention, a publicly available Web prediction tool was developed.

[0085] In summary, the SARS-CoV-2 infection phosphorylation modification site prediction model of the present invention solves the problem of not being able to deeply learn feature information due to the scarcity of data through a large protein language model and transfer learning technology, and significantly improves the accuracy of prediction. In addition, the present invention also verifies the superior performance of the model through a comprehensive test of a series of evaluation indicators, ensuring its reliability and effectiveness in practical applications. The present invention also develops a user-friendly Web prediction tool, which makes the prediction model easier to obtain and use, and provides a powerful scientific tool for the fields of biological research and drug development.

[0086] Example 1: Data preprocessing

[0087] In this embodiment, a dataset for the prediction of phosphorylation modification sites for SARS-CoV-2 infection was constructed. First, 14119 sequences were finely intercepted, classified and optimized to screen out 2 different categories of datasets. In order to ensure the quality of the dataset, the CD-HIT program was used to remove redundant samples, thereby improving the purity of the dataset, and finally a dataset centered on the S / T site was obtained containing 10774 samples, of which both positive and negative samples contained 5387 sequences; the dataset centered on the Y site contained 204 samples, of which positive and negative samples each contained 102 sequences.

[0088] Next, we extracted features of the phosphorylation modification site sequence information. This step is crucial because it effectively improves the discrimination of the data, and these feature extractions lay a solid foundation for subsequent model training and evaluation.

[0089] Through this series of data preprocessing and feature extraction steps, high-quality input data can be provided for the phosphorylation modification site prediction model, thereby improving the prediction accuracy and reliability of the model.

[0090] Example 2: Feature extraction

[0091] In this paper, the effects of BERT, RoBERTa, XLNet and the ESM-2 model used in this study on feature extraction are compared. The excellent performance of ESM-2 on S / T and Y sites shows its strong feature expression ability. As a deep learning model based on protein language model, ESM-2 captures complex semantic relationships in protein sequences through large-scale pre-training, which gives it a significant advantage in identifying protein functional regions. Its multi-layer structure design allows the extraction of higher-level features layer by layer, especially the features extracted at the 33rd layer can effectively reflect the biological characteristics of phosphorylation sites. Compared with models designed for natural language such as BERT and RoBERTa, ESM-2 is specially designed for protein sequences, so it shows stronger feature extraction ability in biological tasks. Especially at the S / T site, the five indicators (0.8162, 0.8468, 0.7859, 0.6343, 0.8865, respectively) are significantly better than other models, which shows its strong recognition ability for this dataset. Although BERT and RoBERTa have been adjusted for processing protein sequences, their initial design is mainly for natural language, so they do not perform as well as ESM-2 when dealing with the complexity of protein sequences. Although XLNet performs slightly better than ESM-2 in some indicators (such as the Sn indicator at the Y site), ESM-2 performs significantly better than XLNet in other key indicators, especially MCC (0.5784vs 0.2985). This shows that although XLNet captures certain local information through bidirectional encoding, it is still inferior to ESM-2 in the accuracy of overall feature extraction and protein function prediction. The effects of different feature extraction methods are shown in Figure 2. Figure 2 shown.

[0092] Example 3: Model performance evaluation and comparison

[0093] As shown in Table 2, the model LMTLIPs constructed by the present invention is compared with the existing phosphorylation site prediction models. These models include DeepPSP, DeepIPs, Bert-ST, PSPred-ALE, and Res-GCN, and the comparison results are recorded in Table 2. For the S / T site, the LM-TLIPs model is 0.8309, 0.8443, and 0.6620 in Acc, Sn, and MCC, respectively. It is worth noting that the Sn index of the model is 0.0436, 0.0482, 0.0185, and 0.0519 higher than Bert-ST, DeepIPs, PSPred-ALE, and Res-GCN, respectively. Sn is a key indicator to measure the ability of the model to correctly identify positive samples. It reflects the proportion of all true positive samples that are correctly identified by the model. This indicator is crucial for evaluating the effectiveness of the model in reducing missed detection of positive samples. For the Y site, the model is superior to the currently available models in terms of Acc, Sn, Sp, and MCC indicators. Specifically, compared with the latest Res-GCN model, it achieved advantages of 0.0238, 0.0479 and 0.0504 on Acc, Sn and MCC respectively. This significant improvement further highlights the powerful ability of the large model in parsing complex biological sequence features.

[0094] Table 2. Comparative analysis with existing models on the same independent test set

[0095]

[0096] In order to deeply explore the effectiveness of LM-TLIPs and the impact of its components on the overall performance, detailed ablation experiments were conducted. By removing specific modules in the LM-TLIPs model one by one and comparing its performance with the complete model, it was found that at the S / T site, LM-TLIPs performed well in terms of Acc, Sn and MCC indicators. Although the AUC was not optimal, the higher accuracy and sensitivity showed the high efficiency of the model in classification accuracy and positive sample detection. At the same time, the higher MCC value also reflects the good balance between sensitivity and specificity of the model. In the evaluation of the Y site, LM-TLIPs showed comprehensive advantages in all five evaluation indicators. In order to more intuitively reflect the performance of the model, the ROC curve diagram was drawn, as shown Figure 3 shown.

[0097] Example 4: Data and model interpretability analysis

[0098] In the data analysis part of the present invention, one-hot encoding is used to reveal the feature distribution of the original sequence data set. One-hot encoding is a common method for processing categorical data. It converts each amino acid into a unique binary vector, thereby retaining its position and identity information in the sequence. Although this method is simple, it provides a basic numerical representation for reflecting the overall feature distribution of the original data. Subsequently, the t-SNE dimensionality reduction algorithm is used to reduce the dimensionality of the features after word embedding, reducing the dimension to 2 dimensions, and the results are visualized and analyzed.

[0099] Figure 4 The figure shows the distribution of positive and negative samples of the S / T and Y site training sets in the original 2D feature space and the 2D feature space after feature extraction. The figure shows the sample distribution of the S / T site in the feature space obtained after the data is reduced in dimension using the t-SNE technology. The original feature space distribution A (corresponding to S / T site) and B (corresponding to Y site) based on the input data, and the feature space distribution C (corresponding to S / T site) and D (corresponding to Ysite) after extraction using the ESM-2 method.

[0100] By generating a two-dimensional scatter plot (such as Figure 4 (A) and (B) show that the distribution of all data points is highly overlapping. This indicates that the feature boundaries of the original data are fuzzy and lack clear boundaries. This also shows that although the one-hot encoding can effectively map the sequence to a numerical representation, it fails to reflect the differences between sequences well in the feature space after dimensionality reduction. In order to better capture the complex characteristics of protein sequences, the ESM-2 model is used for feature embedding, and then the t-SNE method is used to reduce the dimensionality of the output 1280-dimensional features. The generated two-dimensional scatter plot is shown in Figure 2. Figure 4 As shown in (C) and (D), it can be clearly seen that the points of two different colors are clearly distributed and show more obvious clustering. This shows that the ESM-2 model can effectively extract feature information from protein sequences, better separate sequences of different categories in the dimensionality reduction space, and provide more discriminative feature representations for downstream classification tasks.

[0101] In the model interpretability analysis part of this invention, the 1280-dimensional features extracted from the model fine-tuned by ESM-2 are input into the machine learning model for binary classification tasks. Figure 5The feature importance analysis shown reveals the top 10 most important features for the model decision. The figure uses F-score to quantify the contribution of each feature in the model, where features with higher F-scores play a more critical role in the decision-making process. It can be observed that features such as f1160, f918, and f1110 have significantly higher F-scores, indicating that they provide stronger information gain in the decision-making process. For two different datasets, many of the top ten features are the same, including f1160, f918, f1110, f234, and f623. These common features have a significant impact on the predictive performance of the model under different data environments, which not only enhances the interpretability of the model, but also provides a valuable reference for feature selection and model optimization.

[0102] Figure 4 The sample distribution of S / T sites in the feature space obtained after dimensionality reduction using t-SNE technology is shown. The original feature space distribution A (corresponding to S / T site) and B (corresponding to Y site) based on the input data, and the feature space distribution C (corresponding to S / T site) and D (corresponding to Y site) extracted using the ESM-2 method.

Claims

1. A method for identifying phosphorylation modification sites based on large model technology, characterized in that: Here are the steps: Step 1: Dataset Construction Construct training and testing datasets; Step 2: Feature Extraction In this stage, feature extraction of data is performed from serine / threonine (S / T) or tyrosine (Y) sequence information; 2.1 The steps for extracting the feature of the serine / threonine (S / T) site sequence information are as follows: 2.1.1: Input the 33-bit truncated S / T site sequence into the ESM-2 model and fine-tune the ESM-2 model to learn more information contained in the S / T site sequence; Suppose the input sequence is X = [x1, x2, ..., x n ], where x i is the word vector representation of the i-th position in the input sequence; The core calculation steps of the self-attention mechanism include linear transformation, calculation of attention score and weighted summation. First, the self-attention mechanism performs a linear transformation on each word vector x of the input. i Perform linear transformation to generate three vectors: query vector (Query), key vector (Key) and value vector (Value); these three vectors are generated by different weight matrices: Q=XW Q , K=XW K , V=XW V (1) Among them, Q, K, V are query matrix, key matrix and value matrix respectively, and W Q , W K , W V is the learned weight matrix; Next, the attention score is calculated; for each position i in the sequence, the dot product between its query vector and the key vectors of all other positions in the sequence needs to be calculated to obtain the attention score; specifically, the query vector q i With all key vectors k j The dot product can be expressed as: Among them, d k is the dimension of the key vector, the dot product is divided by To avoid excessive dot product values, which may lead to gradient vanishing or gradient exploding problems; The multi-head self-attention mechanism performs the above self-attention calculations in multiple different subspaces, allowing the model to simultaneously focus on different features and patterns in the sequence; the first is multi-head query, key, and value calculation, which generates multiple queries, keys, and values ​​through multiple different linear transformations of the input sequence X: in is the weight matrix of query, key and value corresponding to the i-th head; Next, the head results are concatenated; after each head completes the self-attention calculation independently, these results are merged to obtain the output of multi-head self-attention: MultiHead(Q,K,V)=Concat(head1,...,head h )W O (4) Among them, W O is the final linear transformation matrix, h is the number of attention heads, and Concat(·) means concatenating the outputs of multiple heads together; 2.1.2: Use a linear learning rate scheduler; use a linear decay strategy, the learning rate decreases linearly from the initial value to near zero; the formula of the linear learning rate scheduler is as follows: 2.1.3: After obtaining the fine-tuned ESM-2 model, select the output of the 33rd layer for subsequent use; 2.2 The steps for extracting tyrosine (Y) sequence information features are as follows: 2.2.1: Introduce transfer learning technology, put the ESM-2 model fine-tuned on the S / T site sequence on the Y site sequence dataset for transfer learning, and further learn the feature information of the Y site sequence based on the feature information learned by the model on the S / T sequence site to obtain the best recognition effect of the Y site sequence; 2.2.2: Combine the characteristics of the S / T sequence site with the characteristics of the Y sequence site to finally obtain the characteristics of the Y sequence site; Step 3: Classification prediction model construction After the data feature extraction is completed, the S / T site sequence feature data and the Y site sequence feature data with a dimension of 1280 are obtained; in this stage, the classification prediction model is constructed as follows: 3.1ESM-2for Sequence Classification The S / T site dataset was sent to ESM-2 for Sequence Classification for fine-tuning training to obtain Model 1. Model 1 was loaded and further trained on the Y site dataset to learn relevant information of the Y site dataset. Finally, a model 2 that is more suitable for the Y site was obtained; 3.2 Machine Learning Model Construction For the S / T loci, the features extracted from the ESM-2 model were sent to SVM and XGBoost for model training respectively, thereby obtaining two base classifiers. For the Y loci samples with scarce data, the feature data of the S / T loci extracted from the ESM-2 model were combined with the feature data of the Y loci and then sent to XGBoost for joint learning to obtain a suitable base classification model. Step 4: Weighted integration of ESM-2 and machine learning models Based on the ESM-2for Sequence Classification model obtained in the third step, weighted integration is performed with the machine learning model to obtain the final classifier.

2. The method for identifying phosphorylation modification sites based on large model technology according to claim 1, characterized in that the fourth step is specifically performed as follows: 4.1 Utilize the deep representation ability of the ESM-2SequenceClassification model on sequence data, the powerful generalization performance of the support vector machine (SVM) in high-dimensional feature space, and the prediction accuracy of the XGBoost model based on the efficient decision tree structure; by combining the fine-tuned ESM-2SequenceClassification model with the machine learning models (SVM, XGBoost), and integrating the prediction results of the three models by weighted average, a more robust and accurate classification prediction result is obtained; 4.2 Model Validation Test Accuracy (Acc), sensitivity (Sn), specificity (Sp), MCC and AUC were selected as key performance indicators; the specific descriptions of these indicators are as follows: Among them, TP represents the number of correctly predicted SARS-CoV-2 phosphorylation sites, TN represents the number of correctly identified non-SARS-CoV-2 phosphorylation sites; FP represents the number of misidentified SARS-CoV-2 phosphorylation sites, FN represents the number of misidentified non-SARS-CoV-2, and TPR represents the number of misidentified non-SARS-CoV-2 phosphorylation sites. i Represents the true rate corresponding to the i-th threshold, FPR i represents the false positive rate corresponding to the i-th threshold, and n represents the total number of thresholds.

3. The method for identifying phosphorylation modification sites based on large model technology according to claim 1 or 2, characterized in that: The specific operations of step 3.2 are as follows: For linear data processing, given a data set D, the equation corresponding to the optimal hyperplane in n-dimensional space is: g(x)=w·xb=0 (6) The real vector of the sample in space is denoted as x, its category is denoted as y, w, and b is the objective function. Minimum value under the constraint y(w·xb)≥1; The predicted value of XGBoost can be expressed as: in is the predicted value of the i-th sample, f k (x i ) is the kth tree for sample x i The prediction results, K is the total number of trees; In order to evaluate the prediction effect of the model in the binary classification task, XGBoost usually uses the logarithmic loss function as its core indicator; this function is specifically used to quantify the degree of deviation between the probability value predicted by the model and the true label. Its mathematical expression is as follows: L(y,p)=-ylog(p)-(1-y)log(1-p) (8) Here, y is the actual class label (0 or 1) and p is the probability of the model predicting the positive class (class 1).

4. The method for identifying phosphorylation modification sites based on large model technology according to claim 1 or 2, characterized in that: The first step is specifically performed as follows: First, the 14119 protein sequences collected were processed using the CD-HIT tool, and sequences with a similarity of more than 30% were removed, ultimately obtaining 10978 protein sequences. Subsequently, the processed sequences were truncated into peptide segments containing serine / threonine (S / T) or tyrosine (Y) at the center; these peptides were represented by the following structure: S(p) = A -n A -(n-1) …A -2 A -1 PA +1 A +2 …A +(n-1) A +n ; The highlighted letter P represents the phosphorylation site S / T or Y, while A n The symbol "n" indicates the position of a given amino acid relative to the central residue, where A -n and A +n They represent the nth residue on the left and right sides of the phosphorylation site, respectively; The dataset centered on the S / T site contains 10,774 samples, of which both positive and negative samples contain 5,387 sequences; the dataset centered on the Y site contains 204 samples, of which both positive and negative samples contain 102 sequences each; the dataset is strictly divided into training and test sets in a non-overlapping ratio of 8:

2.

5. The method for identifying phosphorylation modification sites based on large model technology according to claim 3, characterized in that: The first step is specifically performed as follows: First, the 14119 protein sequences collected were processed using the CD-HIT tool, and sequences with a similarity of more than 30% were removed, ultimately obtaining 10978 protein sequences. Subsequently, the processed sequences were truncated into peptide segments containing serine / threonine (S / T) or tyrosine (Y) at the center; these peptides were represented by the following structure: S(p) = A -n A -(n-1) …A -2 A -1 PA +1 A +2 …A +(n-1) A +n ; The highlighted letter P represents the phosphorylation site S / T or Y, while A n The symbol "n" indicates the position of a given amino acid relative to the central residue, where A -n and A +n They represent the nth residue on the left and right sides of the phosphorylation site, respectively; The dataset centered on the S / T site contains 10,774 samples, of which both positive and negative samples contain 5,387 sequences; the dataset centered on the Y site contains 204 samples, of which both positive and negative samples contain 102 sequences each; the dataset is strictly divided into training and test sets in a non-overlapping ratio of 8:2.

Images