A method for predicting the function of carbon cycle-related proteins based on an improved Transformer.

Abstract

This invention discloses a method for predicting the function of carbon cycle-related proteins based on an improved Transformer network. The method includes: acquiring and preprocessing the amino acid sequence of the carbon cycle-related protein to be tested, generating a standard amino acid sequence; extracting semantic embedding features from the amino acid sequence and calculating physicochemical property features; fusing the semantic embedding features and physicochemical property features to generate a comprehensive feature representation; inputting the comprehensive feature representation into an improved Transformer network to generate a fixed-length feature vector that captures key global features; calibrating the feature vector distribution using a Gaussian mixture model and variational Bayesian estimation; finally, inputting the feature vector into a fully connected neural network and using a Softmax activation function to accurately predict the protein function into one of five functional categories: carbon fixation, carbon release, organic synthesis, organic degradation, and organic transformation. This invention achieves high-precision function prediction.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and environmental biology, and in particular to a method for predicting the function of carbon cycle-related proteins based on an improved Transformer. Background Technology

[0002] Proteins, as the direct executors of life activities, drive almost all key biochemical processes in the carbon cycle, such as photosynthetic carbon fixation, methane metabolism, and organic matter decomposition. Therefore, accurately identifying key functional proteins in the carbon cycle is of significant scientific importance and practical value for a deeper understanding of climate change and the discovery of new biological resources.

[0003] Currently, the identification of protein function mainly relies on experimental biology methods, such as gene knockout, protein purification, and in vitro enzyme activity assays. While these methods are the "gold standard" for functional confirmation, they are typically time-consuming, expensive, and have extremely low throughput, far from meeting the urgent needs of analyzing massive sequence data in the post-genomic era. In recent years, with the rapid development of deep learning theory, its application in biological sequence analysis has provided a highly promising solution for directly predicting the function of proteins from their sequences. Deep learning-based prediction methods have made significant progress, but when facing functional protein identification tasks related to complex biological processes such as the carbon cycle, the following prominent problems still exist: 1. The function of a protein is determined by its three-dimensional spatial structure. Most sequence-based prediction networks cannot utilize or infer this crucial three-dimensional structural information, resulting in limited biological interpretability and accuracy of their predictions.

[0004] 2. Existing advanced prediction networks generally suffer from a large number of parameters and complex structures, which leads to the consumption of a lot of computing resources and limits the popularization of this type of method.

[0005] 3. The function of a protein is often determined by a few key functional residues (such as active sites and binding sites). Existing networks usually treat all residues in the sequence equally, lacking an effective mechanism to focus on these key sites. They are easily interfered with by information from a large number of non-functional residues, resulting in low recognition accuracy.

[0006] In summary, existing technologies face significant bottlenecks in predicting the function of carbon cycle-related proteins. Therefore, developing a novel prediction method that overcomes these limitations is of great practical value in improving the accuracy and reliability of research in this field. Summary of the Invention

[0007] The purpose of this invention is to provide a carbon cycle-related protein function prediction method based on an improved Transformer, which solves the problem of insufficient prediction accuracy caused by existing methods due to factors such as ignoring three-dimensional structure, complex network structure and difficulty in identifying key sites, and achieves high-precision function prediction.

[0008] To achieve the above objectives, the technical solution provided by this invention is as follows: a method for predicting the function of carbon cycle-related proteins based on an improved Transformer network. This method is based on an improved Transformer network to predict the function of carbon cycle-related proteins. The improved Transformer network improves the position encoding, component structure, and encoder of the original Transformer network. Specifically, the improvement to the position encoding is to replace the original one-dimensional sequence index-based sine and cosine position encoding of the Transformer network with an encoding method based on the three-dimensional spatial relationship of proteins, enabling it to directly learn structural information. The improvement to the component structure is to remove the original decoder part, because the goal of the improved Transformer network is to perform protein function classification prediction, and its core is to extract discriminative features of the sequence rather than generate new sequences. The improvement to the encoder is to add an attention pooling layer after the original encoder. Its function is to dynamically focus on key amino acid residues and generate a more discriminative global protein feature representation through weighted aggregation, thereby improving the accuracy of functional classification. The specific implementation of the carbon cycle-related protein function prediction method includes the following steps: 1) Obtain the amino acid sequence of the carbon cycle-related protein to be tested and preprocess it to generate a standard amino acid sequence to be tested; 2) Extract semantic embedding features of standard amino acid sequences to be tested using a pre-trained sequence coding model, and calculate physicochemical properties from the amino acid sequences; 3) The extracted semantic embedding features and physicochemical property features are fused to generate a comprehensive feature representation; 4) Input the comprehensive feature representation into the pre-trained improved Transformer network to generate a fixed-length feature vector that can capture global key features; 5) The generated feature vectors are calibrated using Gaussian mixture model and variational Bayesian estimation, and then input into a fully connected neural network as a classifier. The protein function is accurately predicted to one of five functional categories through the Softmax activation function: carbon fixation, carbon release, organic synthesis, organic degradation, and organic transformation.

[0009] Further, in step 1), firstly, the obtained amino acid sequences of carbon cycle-related proteins to be tested are verified, and characters in the sequence that do not belong to standard amino acids are removed or replaced; secondly, the length of the verified sequence is checked, and for sequences whose length exceeds a preset maximum length threshold, an intelligent truncation strategy related to functional domain protection is adopted. This intelligent truncation strategy does not simply truncate from the end, but identifies and preferentially retains strongly related amino acid regions responsible for carbon cycle functions in the sequence by comparing with a database containing information on known protein functional domains. Under the premise of ensuring that key functional information is not lost, the sequence is adjusted to meet the computational length requirements of the improved Transformer network; at the same time, in order to address the data imbalance problem that may exist in multi-class tasks, an upsampling technique is used on the sequence to ensure that each functional category has sufficient and balanced samples, avoiding the prediction bias of the subsequent fully connected neural network due to data skew, and finally generating a standard amino acid sequence to be tested.

[0010] Furthermore, in step 2), a pre-trained sequence coding model and a physicochemical feature calculation method are used together for feature extraction. The sequence coding model is the Protein Large Language Model (ESM-3). The standard amino acid sequence to be tested is processed by the Protein Large Language Model (ESM-3) to generate semantic embedding features. ESM-3 can not only capture the contextual relationship of the sequence, but also directly parse high-precision protein spatial structure information from the one-dimensional sequence, providing richer structured input for the network. In parallel, physicochemical property features, including secondary structure, solvent accessibility and physicochemical properties, are calculated from the amino acid sequence to generate features of the same length as the semantic embedding features. The semantic embedding features and physicochemical property features maintain a sequence-level correspondence during the extraction process and output features of the same length that can be directly aligned.

[0011] Furthermore, in step 3), a multimodal feature fusion mechanism is employed for feature fusion. This mechanism receives two types of feature inputs: one is the semantic embedding feature of sequence context extracted by the protein big language model ESM-3; the other is the physicochemical property feature derived from the amino acid sequence. Subsequently, the multimodal feature fusion mechanism aligns and fuses these two features in the dimension corresponding to each amino acid residue to generate an information-enhanced multimodal feature representation, called the comprehensive feature representation. During the fusion process, the contribution ratio of features from different sources is adaptively adjusted through learnable feature weights to highlight structural signals closely related to protein function. This achieves efficient fusion of multi-source features without increasing computational complexity, thereby improving the classification accuracy of the Transformer network for carbon cycle-related protein functions.

[0012] Furthermore, the improved Transformer network is responsible for generating a feature vector that deeply integrates global contextual information of amino acids. Specific improvements include: First, in the encoder part, in order to solve the problem that traditional position encoding cannot characterize the three-dimensional spatial structure of proteins and thus leads to inaccurate identification of functional domains, the original sine and cosine position encoding based on one-dimensional sequence index in the Transformer network is replaced with a position bias encoding mechanism based on the three-dimensional spatial relationship of proteins. This position bias encoding mechanism uses spatial topological information as an inductive bias. Specifically, when the multi-head self-attention module of the encoder calculates the attention score, a bias matrix calculated based on the spatial distance of amino acid residue pairs in the three-dimensional structure is directly injected into the attention score. After adding this bias matrix, the attention calculation will additionally consider the proximity of amino acids in three-dimensional space, thus giving priority to physically close residue pairs. Each feature vector in the comprehensive feature representation corresponds to an original amino acid residue. For any amino acid residue in the standard amino acid sequence to be tested... Let its coordinates in three-dimensional space be... for: ; In the formula, amino acid residues Coordinate components in three-dimensional space; Similarly, for amino acid residues Coordinates in three-dimensional space , is represented as: ; In the formula, amino acid residues Coordinate components in three-dimensional space; Any two amino acid residues and Spatial distance between for: ; Will Convert to attention bias The definition is as follows: ; In the formula, It is a learnable or preset scaling factor; This bias term Attention scores added to the Transformer network: ; In the formula, Represents amino acid residues For amino acid residues The level of attention, Represents amino acid residues The query vector, Represents amino acid residues The key vector, This indicates the dimensions of the query vector and the key vector, and scales the dot product result to avoid the problem of the value being too large or too small; The above-mentioned method of using spatial topological information as an inductive bias enables the network to better retain the conformational information that determines function, thereby improving the ability to identify functional domains formed by long-range interactions and thus improving the prediction accuracy of protein carbon cycle function. Secondly, in terms of the core network architecture, in order to adapt to the requirements of the protein function prediction classification task, the decoder part in the Transformer network was removed and a simplified encoder-only architecture was constructed. This is intended to allow the network to focus on extracting feature representations for discriminative functions from the comprehensive feature representations, rather than generating output sequences as in the standard Transformer network. This significantly reduces the number of network parameters and computational complexity without affecting task performance, and improves inference efficiency. Finally, in the global feature aggregation section, to address the problem of key functional site signals being diluted or lost due to the indiscriminate processing of all residues in traditional pooling methods, the global average pooling layer or max pooling layer after the encoder output is replaced with an attention pooling layer. Specifically, this is implemented by: [The text abruptly ends here, so the translation stops as well.] Dimensions Above the feature matrix, a learnable dimension is introduced. The system generates a category query vector and performs a cross-attention calculation with the entire feature matrix. This calculation generates an importance weight between 0 and 1 for each feature vector in the feature matrix. The final fixed-length feature vector is obtained by weighted summation of all feature vectors in the feature matrix according to this weight. By dynamically focusing on and weighting a few key residues, including catalytic triplet residues, a more discriminative fixed-length feature vector is directly generated, thereby significantly improving the classification accuracy of different functional proteins in the carbon cycle. The specific implementation is as follows: Learnable query vector: Defines a dimension as Learnable vectors This is used to simulate the feature preferences of different functional categories; Cross-attention computing: With the feature matrix output by the encoder Perform attention interactions and calculate weights : ; In the formula, Representation of the characteristic matrix Belongs to the dimension A real vector space, where L is the feature length and D is the feature dimension; The first in the characteristic matrix 1 eigenvector; Weighted aggregation: based on weights The feature sequences are weighted and summed to generate a feature vector that deeply integrates global contextual information of amino acids. : .

[0013] Furthermore, a feature space calibration and classification module based on variational Bayesian inference is constructed. This module does not directly classify the feature vectors, but first performs statistical calibration on the features using the latent distribution structure of the data, and then inputs them into a fully connected neural network as a classifier. The specific execution steps are as follows: a. Constructing the potential distribution space: Treating the feature vectors as observation data, construct a dataset containing... A Gaussian mixture model with 1 component, where Corresponding to the five categories of carbon cycle functions; this Gaussian mixture model no longer uses fixed point estimation parameters, but introduces the prior distribution of parameters through a variational Bayesian framework, and models the mean and precision matrix of each Gaussian component as a probability distribution, thereby capturing cognitive uncertainty in the feature space; b. Posterior calibration of features: In the inference phase, based on the variational evidence lower bound maximization criterion, the posterior responsibility value of the input feature vector relative to each Gaussian component is calculated; this posterior responsibility value is used as a soft gating signal and interacted with the original fixed-length feature vector element by element to generate a calibration feature vector; this step b can automatically suppress noisy features located at the inter-class fuzzy boundary by utilizing statistical distribution information, thereby enhancing the discriminative power of the features; c. Distribution-aware classification: The calibration feature vector obtained in step b is input into the fully connected neural network. Since the input calibration feature vector has incorporated the distribution confidence based on variational Bayes, the fully connected neural network can construct a nonlinear decision boundary that is more robust to noise in a high-dimensional space. Finally, the predicted probabilities of five functional categories—carbon fixation, carbon release, organic synthesis, organic degradation, and organic transformation—are output through the Softmax activation function, thereby accurately predicting the protein function as one of the five functional categories: carbon fixation, carbon release, organic synthesis, organic degradation, and organic transformation.

[0014] Compared with the prior art, the present invention has the following advantages and beneficial effects: 1. This invention utilizes the advanced protein large language model ESM-3 as a sequence encoding model. By pre-training on ultra-large-scale amino acid sequences, ESM-3 can generate high-level semantic embedding features with extremely high information density for each amino acid. These features contain deeper evolutionary information and potential structural information, fundamentally ensuring the quality of data input to the downstream prediction network and laying a solid foundation for achieving high-precision functional prediction.

[0015] 2. This invention creatively proposes a multimodal feature fusion mechanism. This mechanism is not a simple concatenation, but rather achieves deep complementarity between the two modalities by adaptively adjusting the contribution ratio of semantic embedding features and physicochemical property features. This approach overcomes the one-sidedness of existing technologies that rely on single-sequence information, generating a comprehensive feature representation with more prominent structural signals and richer connotations, fundamentally improving the accuracy and robustness of network prediction.

[0016] 3. This invention addresses the significant limitation of existing Transformer networks in utilizing the three-dimensional spatial information of proteins by replacing traditional one-dimensional positional encoding with a position-biased encoding mechanism based on the three-dimensional spatial relationships of proteins. This mechanism directly injects the distance relationships of amino acid residues in three-dimensional space into the attention calculation, enabling the network to prioritize spatially adjacent residue pairs that jointly constitute functional domains, even if they are far apart in sequence. This greatly enhances the network's ability to identify functional domains formed by long-range interactions, solving the core problem of existing technologies' inability to accurately identify functional sites.

[0017] 4. This invention optimizes the network structure by removing the traditional decoder section, making the network simpler and more efficient. This improvement not only reduces computational resources and complexity but also enhances the network's discriminative ability, making it particularly suitable for carbon cycle-related protein function prediction tasks.

[0018] 5. This invention abandons the outdated approach of traditional pooling methods (such as average pooling or max pooling) that indiscriminately treat all residues, thus diluting key signals, and introduces an attention pooling layer. This attention pooling layer performs cross-attention calculations using a learnable query vector and feature matrix, adaptively assigning importance weights to each amino acid residue, thereby dynamically and with high weight aggregating signals from key functional sites such as catalytic triplet. This aggregation method ensures that the most crucial discriminative information is not lost, generating a more discriminative global protein characterization and significantly improving classification accuracy.

[0019] 6. Introducing Variational Bayesian Feature Calibration to Improve Prediction Robustness. This invention constructs a feature space calibration and classification module based on variational Bayesian inference. This module utilizes Gaussian mixture models and variational Bayesian estimation to capture the potential distribution uncertainty of features, and performs soft gating calibration on the features using posterior responsibility values. This mechanism effectively suppresses noisy features at ambiguous inter-class boundaries, enabling subsequent fully connected classifiers to make decisions in a feature space with higher confidence, significantly reducing the risk of misjudgment caused by data uncertainty.

[0020] 7. This invention enables highly efficient and accurate functional prediction of carbon cycle-related proteins, providing researchers with a powerful tool for rapidly screening and annotating functional proteins from massive genomic data. This is of great significance for a deeper understanding of carbon cycle mechanisms, the discovery of novel industrial enzymes, and the development of green biotechnology. Furthermore, the technical solution proposed in this invention also provides valuable new ideas for the accurate analysis and functional prediction of other biological sequences. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating the specific implementation of the method of the present invention. Detailed Implementation

[0022] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0023] This embodiment discloses a method for predicting the function of carbon cycle-related proteins based on an improved Transformer network. This improved Transformer network improves the original Transformer network's position encoding, component structure, and encoder. Specifically, the position encoding is improved by replacing the original one-dimensional sequence index-based sine and cosine position encoding with an encoding method based on the three-dimensional spatial relationships of proteins, enabling it to directly learn structural information. The component structure is improved by removing the original decoder part, as the goal of the improved Transformer network is to perform protein function classification prediction, and its core is to extract discriminative features of the sequence rather than generate new sequences. The encoder is improved by adding an attention pooling layer after the original encoder. This layer dynamically focuses on key amino acid residues, generating a more discriminative global protein feature representation through weighted aggregation, thereby improving the accuracy of functional classification.

[0024] like Figure 1 As shown, the specific implementation of the carbon cycle-related protein function prediction method includes the following steps: 1) Obtain the amino acid sequence of the carbon cycle-related protein to be tested and preprocess it to generate a standard amino acid sequence to be tested, as follows: First, the obtained amino acid sequences of carbon cycle-related proteins to be tested are verified, and characters that do not belong to the standard amino acids in the sequence are removed or replaced. Second, the length of the verified sequences is checked. For sequences whose length exceeds a preset maximum length threshold, an intelligent truncation strategy related to functional domain protection is adopted. This intelligent truncation strategy does not simply truncate from the end, but identifies and prioritizes the retention of strongly related amino acid regions responsible for carbon cycle functions in the sequence by comparing with a database containing information on known protein functional domains. While ensuring that key functional information is not lost, the sequence is adjusted to meet the computational length requirements of the improved Transformer network. At the same time, to address the potential data imbalance problem in multi-class tasks, upsampling technology is used on the sequences to ensure that each functional category has sufficient and balanced samples, avoiding prediction bias caused by data skew in the subsequent fully connected neural network used as a classifier, and finally generating a standard amino acid sequence to be tested.

[0025] 2) Extract semantic embedding features from the standard amino acid sequence to be tested using a pre-trained sequence coding model, and calculate the physicochemical properties from the amino acid sequence, as follows: Feature extraction is performed using a pre-trained sequence encoding model and a physicochemical feature calculation method. The sequence encoding model is the Protein Large Language Model (ESM-3). The standard amino acid sequence to be tested is processed by ESM-3 to generate semantic embedding features. ESM-3 can not only capture the contextual relationships of the sequence, but also directly parse high-precision protein spatial structure information from the one-dimensional sequence, providing richer structured input for the network. In parallel, physicochemical property features, including secondary structure, solvent accessibility, and physicochemical properties, are calculated from the amino acid sequence, generating features of the same length as the semantic embedding features. The semantic embedding features and physicochemical property features maintain a sequence-level correspondence during the extraction process and output features of the same length that can be directly aligned.

[0026] 3) The extracted semantic embedding features and physicochemical property features are fused to generate a comprehensive feature representation, as follows: A multimodal feature fusion mechanism is employed for feature fusion. This mechanism receives two types of feature inputs with different properties: first, semantic embedding features of sequence context extracted by the protein big language model ESM-3; and second, physicochemical property features derived from amino acid sequences. Subsequently, the multimodal feature fusion mechanism aligns and fuses these two features along the dimension corresponding to each amino acid residue, generating an information-enhanced multimodal feature representation, called the comprehensive feature representation. During the fusion process, learnable feature weights adaptively adjust the contribution ratio of features from different sources to highlight structural signals closely related to protein function. This achieves efficient fusion of multi-source features without increasing computational complexity, thereby improving the classification accuracy of the Transformer network for carbon cycle-related protein functions.

[0027] The mathematical expression for the concatenation operation is as follows: ; In the formula, Represents amino acid residues The fusion characteristics Represents semantic embedding features, Indicates physicochemical properties, It is a learnable feature weight used to adaptively adjust the contribution ratio of features from different sources.

[0028] 4) The comprehensive feature representation is input into a pre-trained improved Transformer network to generate a fixed-length feature vector that captures key global features. The improved Transformer network is responsible for generating a feature vector that deeply integrates global contextual information of amino acids. Specific improvements include: First, in the encoder part, in order to solve the problem that traditional position encoding cannot characterize the three-dimensional spatial structure of proteins and thus leads to inaccurate identification of functional domains, the original sine and cosine position encoding based on one-dimensional sequence index in the Transformer network is replaced with a position bias encoding mechanism based on the three-dimensional spatial relationship of proteins. This position bias encoding mechanism uses spatial topological information as an inductive bias. Specifically, when the multi-head self-attention module of the encoder calculates the attention score, a bias matrix calculated based on the spatial distance of amino acid residue pairs in the three-dimensional structure is directly injected into the attention score. After adding this bias matrix, the attention calculation will additionally consider the proximity of amino acids in three-dimensional space, thus giving priority to physically close residue pairs. Each feature vector in the comprehensive feature representation corresponds to an original amino acid residue. For any amino acid residue in the standard amino acid sequence to be tested... Let its coordinates in three-dimensional space be... for: ; In the formula, amino acid residues Coordinate components in three-dimensional space; Similarly, for amino acid residues Coordinates in three-dimensional space , is represented as: ; In the formula, amino acid residues Coordinate components in three-dimensional space; Any two amino acid residues and Spatial distance between for: ; Will Convert to attention bias The definition is as follows: ; In the formula, It is a learnable or preset scaling factor; This bias term Attention scores added to the Transformer network: ; In the formula, Represents amino acid residues For amino acid residues The level of attention, Represents amino acid residues The query vector, Represents amino acid residues The key vector, This indicates the dimensions of the query vector and the key vector, and scales the dot product result to avoid the problem of the value being too large or too small; The above-mentioned method of using spatial topological information as an inductive bias enables the network to better retain the conformational information that determines function, thereby improving the ability to identify functional domains formed by long-range interactions and thus improving the prediction accuracy of protein carbon cycle function. Secondly, in terms of the core network architecture, in order to adapt to the requirements of the protein function prediction classification task, the decoder part in the Transformer network was removed and a simplified encoder-only architecture was constructed. This is intended to allow the network to focus on extracting feature representations for discriminative functions from the comprehensive feature representations, rather than generating output sequences as in the standard Transformer network. This significantly reduces the number of network parameters and computational complexity without affecting task performance, and improves inference efficiency. Finally, in the global feature aggregation section, to address the problem of key functional site signals being diluted or lost due to the indiscriminate processing of all residues in traditional pooling methods, the global average pooling layer or max pooling layer after the encoder output is replaced with an attention pooling layer. Specifically, this is implemented by: [The text abruptly ends here, so the translation stops as well.] Dimensions Above the feature matrix, a learnable dimension is introduced. The system generates a category query vector and performs a cross-attention calculation with the entire feature matrix. This calculation generates an importance weight between 0 and 1 for each feature vector in the feature matrix. The final fixed-length feature vector is obtained by weighted summation of all feature vectors in the feature matrix according to this weight. By dynamically focusing on and weighting a few key residues, including catalytic triplet residues, a more discriminative fixed-length feature vector is directly generated, thereby significantly improving the classification accuracy of different functional proteins in the carbon cycle. The specific implementation is as follows: Learnable query vector: Defines a dimension as Learnable vectors This is used to simulate the feature preferences of different functional categories; Cross-attention computing: With the feature matrix output by the encoder Perform attention interactions and calculate weights : ; In the formula, Representation of the characteristic matrix Belongs to the dimension A real vector space, where L is the feature length and D is the feature dimension; The first in the characteristic matrix 1 eigenvector; Weighted aggregation: based on weights The feature sequences are weighted and summed to generate a feature vector that deeply integrates global contextual information of amino acids. : ; 5) The generated feature vectors are calibrated using a Gaussian mixture model and variational Bayesian estimation, and then input into a fully connected neural network used as a classifier. The Softmax activation function is then used to accurately predict the protein function into one of five functional categories: carbon fixation, carbon release, organic synthesis, organic degradation, and organic transformation, as detailed below: First, a feature space calibration and classification module based on variational Bayesian inference is constructed. This module does not directly classify the feature vectors, but first performs statistical calibration on the features using the latent distribution structure of the data, and then inputs them into a fully connected neural network. The specific execution steps are as follows: a. Constructing the potential distribution space: Treating the feature vectors as observation data, construct a dataset containing... A Gaussian mixture model with 1 component, where Corresponding to the five categories of carbon cycle functions; this Gaussian mixture model no longer uses fixed point estimation parameters, but introduces the prior distribution of parameters through a variational Bayesian framework, and models the mean and precision matrix of each Gaussian component as a probability distribution, thereby capturing cognitive uncertainty in the feature space; b. Posterior calibration of features: In the inference phase, based on the variational evidence lower bound maximization criterion, the posterior responsibility value of the input feature vector relative to each Gaussian component is calculated; this posterior responsibility value is used as a soft gating signal and interacted with the original fixed-length feature vector element by element to generate a calibration feature vector; this step b can automatically suppress noisy features located at the inter-class fuzzy boundary by utilizing statistical distribution information, thereby enhancing the discriminative power of the features; c. Distribution-aware classification: The calibration feature vector obtained in step b is input into the fully connected neural network. Since the input calibration feature vector has incorporated the distribution confidence based on variational Bayes, the fully connected neural network can construct a nonlinear decision boundary that is more robust to noise in a high-dimensional space. Finally, the predicted probabilities of five functional categories—carbon fixation, carbon release, organic synthesis, organic degradation, and organic transformation—are output through the Softmax activation function, thereby accurately predicting the protein function as one of the five functional categories: carbon fixation, carbon release, organic synthesis, organic degradation, and organic transformation.

[0029] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for predicting the function of carbon cycle-related proteins based on an improved Transformer, characterized in that, This method is based on an improved Transformer network for functional prediction of carbon cycle-related proteins. The improved Transformer network modifies the original Transformer network's position encoding, component structure, and encoder. Specifically, the position encoding is improved by replacing the original one-dimensional sequence index-based sine and cosine position encoding with an encoding method based on the three-dimensional spatial relationships of proteins, enabling it to directly learn structural information. The component structure is improved by removing the original decoder part, as the goal of the improved Transformer network is to perform protein function classification prediction, focusing on extracting discriminative features of the sequence rather than generating new sequences. The encoder is improved by adding an attention pooling layer after the original encoder. This layer dynamically focuses on key amino acid residues, generating a more discriminative global protein feature representation through weighted aggregation, thereby improving functional classification accuracy. The specific implementation of the carbon cycle-related protein function prediction method includes the following steps: 1) Obtain the amino acid sequence of the carbon cycle-related protein to be tested and preprocess it to generate a standard amino acid sequence to be tested; 2) Extract semantic embedding features of standard amino acid sequences to be tested using a pre-trained sequence coding model, and calculate physicochemical properties from the amino acid sequences; 3) The extracted semantic embedding features and physicochemical property features are fused to generate a comprehensive feature representation; 4) The comprehensive feature representation is input into a pre-trained improved Transformer network to generate a fixed-length feature vector that captures key global features. The improved Transformer network is responsible for generating a feature vector that deeply integrates global contextual information of amino acids. Specific improvements include: First, in the encoder part, in order to solve the problem that traditional position encoding cannot characterize the three-dimensional spatial structure of proteins and thus leads to inaccurate identification of functional domains, the original sine and cosine position encoding based on one-dimensional sequence index in the Transformer network is replaced with a position bias encoding mechanism based on the three-dimensional spatial relationship of proteins. This position bias encoding mechanism uses spatial topological information as an inductive bias. Specifically, when the multi-head self-attention module of the encoder calculates the attention score, a bias matrix calculated based on the spatial distance of amino acid residue pairs in the three-dimensional structure is directly injected into the attention score. After adding this bias matrix, the attention calculation will additionally consider the proximity of amino acids in three-dimensional space, thus giving priority to physically close residue pairs. Each feature vector in the comprehensive feature representation corresponds to an original amino acid residue. For any amino acid residue in the standard amino acid sequence to be tested... Let its coordinates in three-dimensional space be... for: ； In the formula, amino acid residues Coordinate components in three-dimensional space; Similarly, for amino acid residues Coordinates in three-dimensional space , represented as: ； In the formula, amino acid residues Coordinate components in three-dimensional space; Any two amino acid residues and Spatial distance between for: ； Will Convert to attention bias The definition is as follows: ； In the formula, It is a learnable or preset scaling factor; This bias term Attention scores added to the Transformer network: ； In the formula, Represents amino acid residues For amino acid residues The level of attention, Represents amino acid residues The query vector, Represents amino acid residues The key vector, This indicates the dimensions of the query vector and the key vector, and scales the dot product result to avoid the problem of the value being too large or too small; The above-mentioned method of using spatial topological information as an inductive bias enables the network to better retain the conformational information that determines function, thereby improving the ability to identify functional domains formed by long-range interactions and thus improving the prediction accuracy of protein carbon cycle function. Secondly, in terms of the core network architecture, in order to adapt to the requirements of the protein function prediction classification task, the decoder part in the Transformer network was removed and a simplified encoder-only architecture was constructed. This is intended to allow the network to focus on extracting feature representations for discriminative functions from the comprehensive feature representations, rather than generating output sequences as in the standard Transformer network. This significantly reduces the number of network parameters and computational complexity without affecting task performance, and improves inference efficiency. Finally, in the global feature aggregation section, to address the problem of key functional site signals being diluted or lost due to the indiscriminate processing of all residues in traditional pooling methods, the global average pooling layer or max pooling layer after the encoder output is replaced with an attention pooling layer. Specifically, this is implemented by: [The text abruptly ends here, so the translation stops as well.] Dimensions Above the feature matrix, a learnable dimension is introduced. The system generates a category query vector and performs a cross-attention calculation with the entire feature matrix. This calculation generates an importance weight between 0 and 1 for each feature vector in the feature matrix. The final fixed-length feature vector is obtained by weighted summation of all feature vectors in the feature matrix according to this weight. By dynamically focusing on and weighting a few key residues, including catalytic triplet residues, a more discriminative fixed-length feature vector is directly generated, thereby significantly improving the classification accuracy of different functional proteins in the carbon cycle. The specific implementation is as follows: Learnable query vector: Defines a dimension as Learnable vectors This is used to simulate the feature preferences of different functional categories; Cross-attention computing: With the feature matrix output by the encoder Perform attention interactions and calculate weights : ； In the formula, Representation of the characteristic matrix Belongs to the dimension A real vector space, where L is the feature length and D is the feature dimension; The first in the characteristic matrix 1 eigenvector; Weighted aggregation: based on weights The feature sequences are weighted and summed to generate a feature vector that deeply integrates global contextual information of amino acids. : ； 5) The generated feature vectors are calibrated using Gaussian mixture model and variational Bayesian estimation, and then input into a fully connected neural network as a classifier. The protein function is predicted to be one of five functional categories through the Softmax activation function: carbon fixation, carbon release, organic synthesis, organic degradation, and organic transformation.

2. The method for predicting the function of carbon cycle-related proteins based on the improved Transformer according to claim 1, characterized in that, In step 1), firstly, the obtained amino acid sequences of carbon cycle-related proteins to be tested are verified, and characters that do not belong to the standard amino acids in the sequence are removed or replaced. Secondly, the length of the verified sequence is checked. For sequences whose length exceeds a preset maximum length threshold, an intelligent truncation strategy related to functional domain protection is adopted. This intelligent truncation strategy does not simply truncate from the end, but identifies and preferentially retains strongly related amino acid regions responsible for carbon cycle functions in the sequence by comparing with a database containing information on known protein functional domains. While ensuring that key functional information is not lost, the sequence is adjusted to meet the computational length requirements of the improved Transformer network. At the same time, to address the potential data imbalance problem in multi-class tasks, an upsampling technique is used on the sequence to ensure that each functional category has sufficient and balanced samples, avoiding prediction bias caused by data skew in the subsequent fully connected neural network, and finally generating a standard amino acid sequence to be tested.

3. The method for predicting the function of carbon cycle-related proteins based on the improved Transformer according to claim 2, characterized in that, In step 2), a pre-trained sequence coding model and a physicochemical feature calculation method are used together to extract features. The sequence coding model is the Protein Large Language Model (ESM-3). The standard amino acid sequence to be tested is processed by the Protein Large Language Model (ESM-3) to generate semantic embedding features. ESM-3 can not only capture the contextual relationship of the sequence, but also directly parse high-precision protein spatial structure information from the one-dimensional sequence, providing richer structured input for the network. In parallel, physicochemical property features, including secondary structure, solvent accessibility and physicochemical properties, are calculated from the amino acid sequence to generate features of the same length as the semantic embedding features. The semantic embedding features and physicochemical property features maintain a sequence-level correspondence during the extraction process and output features of the same length that can be directly aligned.

4. The method for predicting the function of carbon cycle-related proteins based on the improved Transformer according to claim 3, characterized in that, In step 3), a multimodal feature fusion mechanism is used for feature fusion. This multimodal feature fusion mechanism receives two different types of feature inputs: one is the semantic embedding features of sequence context relations extracted by the protein big language model ESM-3; Secondly, it uses physicochemical properties derived from amino acid sequences. Subsequently, this multimodal feature fusion mechanism aligns and fuses these two types of features in the dimension corresponding to each amino acid residue, generating an information-enhanced multimodal feature representation, called a comprehensive feature representation. During the fusion process, the contribution ratio of features from different sources is adaptively adjusted through learnable feature weights to highlight structural signals closely related to protein function. This achieves efficient fusion of multi-source features without increasing computational complexity, thereby improving the classification accuracy of the Transformer network for carbon cycle-related protein functions.

5. The method for predicting the function of carbon cycle-related proteins based on the improved Transformer according to claim 4, characterized in that, A feature space calibration and classification module based on variational Bayesian inference is constructed. This module does not directly classify feature vectors, but first performs statistical calibration on the features using the latent distribution structure of the data, and then inputs them into a fully connected neural network as a classifier. The specific execution steps are as follows: a. Constructing the potential distribution space: Treating the feature vectors as observation data, construct a dataset containing... A Gaussian mixture model with 1 component, where Corresponding to the five categories of carbon cycle functions; this Gaussian mixture model no longer uses fixed point estimation parameters, but introduces the prior distribution of parameters through a variational Bayesian framework, and models the mean and precision matrix of each Gaussian component as a probability distribution, thereby capturing cognitive uncertainty in the feature space; b. Posterior calibration of features: In the inference phase, based on the variational evidence lower bound maximization criterion, the posterior responsibility value of the input feature vector relative to each Gaussian component is calculated; this posterior responsibility value is used as a soft gating signal and interacted with the original fixed-length feature vector element by element to generate a calibration feature vector; this step b can automatically suppress noisy features located at the inter-class fuzzy boundary by utilizing statistical distribution information, thereby enhancing the discriminative power of the features; c. Distribution-aware classification: The calibration feature vector obtained in step b is input into the fully connected neural network. Since the input calibration feature vector has incorporated the distribution confidence based on variational Bayes, the fully connected neural network can construct a nonlinear decision boundary that is more robust to noise in a high-dimensional space. Finally, the predicted probabilities of five functional categories—carbon fixation, carbon release, organic synthesis, organic degradation, and organic transformation—are output through the Softmax activation function, thereby predicting the protein function as one of the five functional categories: carbon fixation, carbon release, organic synthesis, organic degradation, and organic transformation.

Images