Protein sequence joint coding method combining protein language model and crosslinking mass spectrum constraint
By transforming cross-linked mass spectrometry data into a sparse adjacency matrix and injecting it into the attention mechanism of a protein language model, the spatial constraint problem of the protein language model failing to fully integrate cross-linked mass spectrometry data is solved, achieving efficient sequence-structure joint characterization and improving the accuracy and computational efficiency of three-dimensional conformation characterization.
Patent Information
- Application Number
- CN202511705527.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-20
- Publication Date
- 2026-02-24
AI Technical Summary
Existing protein language models fail to fully integrate the spatial constraints of cross-linked mass spectrometry data during modeling, resulting in insufficient sequence-structure associations, making it difficult to achieve end-to-end joint representation learning, and also incurring high computational costs and highly invasive model architectures.
Cross-linked mass spectrometry data is transformed into a sparse adjacency matrix and injected into the attention mechanism of the protein language model of the Transformer architecture through bias, explicitly introducing spatial proximity information. A hybrid attention mechanism is used for information fusion, and the forward propagation interface is modified to seamlessly integrate cross-linked mass spectrometry constraints.
It improves the accuracy of protein three-dimensional conformation characterization, reduces computational overhead, supports rapid reasoning, has physical interpretability, is suitable for structure prediction and functional analysis, lowers the threshold for technology implementation, and provides a reliable feature foundation.
Smart Images
Figure CN121565254A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a method for joint encoding of protein sequences by combining a protein language model with cross-linked mass spectrometry constraints. Background Technology
[0002] Proteins, as fundamental molecules of life, possess sequence and spatial structure information of great significance for life science research and drug development. With the widespread adoption of high-throughput sequencing technology, public databases have accumulated massive amounts of protein sequences, providing a sufficient data foundation for deep learning methods. Protein Language Models (PLMs), through self-supervised training on large-scale sequences, can map one-dimensional sequences to high-dimensional vector representations, thereby significantly improving structure prediction performance. However, existing PLMs primarily rely on pure sequence information for modeling and have not yet fully integrated experimentally obtained spatial constraints, limiting their ability to characterize the true distance relationships between residues and macroscopic folding topology.
[0003] Cross-linking mass spectrometry (XL-MS) can capture spatial proximity information of residue pairs within proteins and in protein-protein interactions at the molecular level, providing sparse yet reliable experimental evidence for sequence-structure association. How to incorporate XL-MS data into protein language models in a computationally efficient and physically interpretable manner to enhance the ability of sequence embeddings to characterize three-dimensional conformations and functional interfaces has become a cutting-edge research topic in the field of protein artificial intelligence characterization, and also a pressing technical challenge.
[0004] Despite the rich spatial structure information contained in XL-MS data, effectively integrating it into the deep representation framework of protein language models in a computationally efficient, physically interpretable, and model-compatible manner has become a cutting-edge hot topic and an urgent technical challenge in the field of protein artificial intelligence. Existing methods mostly rely on post-processing splicing or independent optimization strategies, making it difficult to achieve end-to-end joint representation learning. These methods suffer from insufficient information fusion, high computational overhead, and strong model architecture intrusion. Therefore, there is an urgent need to develop a joint encoding method that can seamlessly integrate with mainstream protein language model architectures, explicitly injecting experimental space constraints while maintaining the model's original semantic modeling capabilities. This would improve the accuracy and biological interpretability of protein sequence representations of real three-dimensional conformations. Summary of the Invention
[0005] The present invention provides a protein sequence co-encoding method that combines protein language modeling and cross-linked mass spectrometry constraints to solve the problems existing in the prior art.
[0006] The technical solutions adopted in this invention are as follows:
[0007] A method for joint encoding of protein sequences combining protein language models and cross-linked mass spectrometry constraints includes the following steps:
[0008] S1: Obtain the amino acid sequence of the target protein and the cross-linking mass spectrometry experimental data containing cross-linking residue pairs corresponding to the target protein;
[0009] S2: Analyze the cross-linking mass spectrometry experimental data, extract cross-linking residue pairs, and construct a sparse adjacency matrix;
[0010] S3: Encode the amino acid sequence into a token sequence;
[0011] S4: Input the token sequence and the sparse adjacency matrix into the protein language model based on the Transformer architecture. The forward propagation interface of the protein language model is updated with an xlms_bias parameter to receive the sparse adjacency matrix. The protein language model contains at least one Transformer module, and each Transformer module contains a hybrid attention mechanism for calculating attention logits.
[0012] S5: In the hybrid attention mechanism, the sparse adjacency matrix is added to the attention logits as a bias term to obtain the corrected attention logits;
[0013] S6: The protein language model performs the addition operation layer by layer through a multi-layer Transformer module, and outputs a protein sequence jointly encoded by fused sequence semantics and cross-linked mass spectrometry spatial constraints.
[0014] Furthermore, the sparse adjacency matrix constructed in S2 is a binary matrix, where the element value at the corresponding position of cross-linked residue pairs is 1, and the element value at the corresponding position of non-cross-linked residue pairs is 0. The matrix construction formula is as follows:
[0015] ,
[0016] In the formula, i and j represent the index positions of the i-th and j-th residues in the protein sequence, respectively. This represents the element value at the corresponding position in the sparse adjacency matrix.
[0017] Furthermore, the binary matrix is multiplied by the weighting coefficients. The weighted adjacency matrix is obtained, where For adjustable hyperparameters greater than 0, the formula for constructing the weighted matrix is:
[0018] ,
[0019] In the formula, This represents the element value at the corresponding position in the weighted adjacency matrix. Used to adjust the intensity of the effect of crosslinking constraints on attention distribution.
[0020] Furthermore, the hybrid attention mechanism includes local attention computation and global linear attention computation, which respectively generate corresponding attention logits, wherein the local attention computation is as follows:
[0021] ,
[0022] Global linear attention is calculated as follows:
[0023] ,
[0024] In the formula, These are the local attention query vector and the key vector, respectively. Here, V represents the global linear attention query vector and key vector, respectively; V represents the value vector; b represents the trainable bias term; G represents the number of blocks; RoPE represents rotational position encoding; and ReLU represents the linear rectified activation function.
[0025] Furthermore, both the local attention calculation and the global linear attention calculation employ rotational position encoding (RoPE) to encode the query vector and the key vector.
[0026] Furthermore, in S5, the sparse adjacency matrix is reshaped according to the block structure of the hybrid attention mechanism, and then added to the attention logits as a bias term. The attention calculation formula is as follows:
[0027] ,
[0028] In the formula, Q, K, and V are the query matrix, key matrix, and value matrix, respectively. Let be the dimension of the key vector. The bias matrix is the reshaped bias matrix of the sparse adjacency matrix.
[0029] Furthermore, the protein language model includes 12 Transformer modules, and the sparse adjacency matrix is used as a bias term in the hybrid attention mechanism of each Transformer module.
[0030] Furthermore, in S3, a batch converter is used to encode the amino acid sequence into a token sequence, and the encoding formula is as follows:
[0031] ,
[0032] In the formula, S is the protein amino acid sequence, T is the mapped token sequence, and batchConverter is the batch converter function.
[0033] Furthermore, the protein sequences are jointly encoded for protein structure prediction or protein-protein interaction prediction tasks.
[0034] The present invention has the following beneficial effects:
[0035] (1) By converting cross-linked mass spectrometry data into a sparse adjacency matrix and injecting an attention mechanism in the form of bias, the spatial proximity information measured by experiments is explicitly introduced on the basis of maintaining the original semantic modeling ability of the protein language model, so that the sequence representation can more accurately reflect the real spatial relationship between residues and improve the model's accuracy in representing the three-dimensional conformation of proteins.
[0036] (2) Information fusion is achieved by adding sparse matrices and bias terms. There is no need to make deep modifications to the main architecture of the model or add an independent training stage. The additional computational overhead is low, and it supports fast inference on proteomic scale data. It is suitable for high-throughput bioinformatics scenarios such as structure prediction and functional analysis.
[0037] (3) This method achieves constraint injection by modifying the forward propagation interface, which can be seamlessly integrated into mainstream protein language models based on the Transformer architecture (such as ESM, ProtTrans, etc.) without retraining the basic model, thus reducing the threshold for technology implementation.
[0038] (4) Attention weights can be directly traced back to experimental crosslinking sites, making the model decision-making process physically interpretable. The generated embedding vectors simultaneously carry sequence evolution information and spatial structure priors, providing a more reliable feature basis for downstream structural biology research.
[0039] (5) The joint encoding vector output by this method can be directly used for tasks such as protein structure prediction, protein-protein interaction identification, and functional site analysis, providing a general encoding paradigm for experimental data-driven protein artificial intelligence research and enriching the toolbox in the intersection of structural proteomics and deep learning. Attached Figure Description
[0040] Figure 1 This is a flowchart of the invention.
[0041] Figure 2 This is a schematic diagram of the protein language model network structure.
[0042] Figure 3 This is a schematic diagram of the attention mechanism for cross-linked mass spectrometry-assisted bias injection in a protein language model.
[0043] Figure 4 This is a schematic diagram illustrating the construction principle of the cross-linked mass spectrometry adjacency matrix. Detailed Implementation
[0044] The invention will now be further described with reference to the accompanying drawings.
[0045] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to the accompanying drawings.
[0046] Overall framework as Figure 1 As shown, this invention includes six main steps: protein sequence processing, cross-linked mass spectrometry (XL-MS) data parsing and adjacency matrix construction, design of the backbone structure of the protein language model, modification of the attention mechanism, and output and application of high-dimensional embedding vectors.
[0047] Figure 2 The core network structure of the protein language model described in this invention is shown, including an embedding layer, a mixed-chunk attention mechanism (MCA), and a multi-layer Transformer module.
[0048] Figure 3 The specific process of cross-linked mass spectrometry-assisted bias injection into the attention mechanism is demonstrated.
[0049] Figure 4 The principle of constructing the adjacency matrix of cross-linked mass spectrometry is demonstrated, including key steps such as the extraction of cross-linked pairs, normalization, and sparse matrix generation.
[0050] The technical solution of this invention mainly includes the following steps:
[0051] Step 1: Acquisition and analysis of protein sequence and cross-linking mass spectrometry data.
[0052] This invention obtains protein sequence data from the UniRef50 database. The amino acid sequences are stored in the form of one-dimensional strings of varying lengths and contain 20 standard amino acids.
[0053] The cross-linked mass spectrometry data were obtained from ProteomeXChange JPOST. For the raw cross-linked mass spectrometry data, a unified parsing function `parse_xlms()` was designed to convert various data structures into a unified list of residue pair indices.
[0054] ,
[0055] Outlier filtering and confidence level normalization are performed. The standard normalization formula is as follows:
[0056] ,
[0057] in, The first in the raw data of cross-linked mass spectrometry and the Crosslinking confidence value of residue pairs and These are the minimum and maximum cross-link confidence values in the dataset, respectively.
[0058] Step 2: Encode the amino acid sequence into a token sequence.
[0059] The protein amino acid sequence is input into the token encoder of the protein language model.
[0060] The batch converter (batchConverter()) is used to normalize the sequence into a recognizable discrete token sequence:
[0061] ,
[0062] in, This is the amino acid sequence of a protein. This represents the mapped token sequence. L is the sequence length. Each A dictionary index representing amino acids ensures consistent input dimensions.
[0063] Step 3: Generation of cross-linked mass spectrometry constrained adjacency matrix.
[0064] Based on the crosslinked residue pair information obtained in step 1, a sparse adjacency matrix A is generated. The position (i,j) of a crosslinked residue pair is assigned a value of 1, and the remaining positions are assigned 0. The matrix construction formula is as follows:
[0065] ,
[0066] Introducing a weighting mechanism for crosslinking strength, and introducing a coefficient This is used to adjust for the influence of spatial priors on the attention distribution, resulting in a weighted adjacency matrix:
[0067] ,
[0068] The adjacency matrix Subsequently, it serves as a biased information input attention mechanism to control the explicit information flow between specific residues.
[0069] Step 4: Modify the model structure and expand the interface.
[0070] like Figure 2 As shown, the protein language model is based on the Transformer architecture and contains multi-layer encoder blocks. Each block contains positional encoding, LayerNorm, feedforward network, and hybrid attention mechanism.
[0071] This invention maintains the original architecture while modifying the adjacency matrix. The xlms_bias parameter is injected into the forward interface as an additional input parameter to implement the following calculation process:
[0072] ,
[0073] in, The final high-dimensional representation of each token, This represents the hidden layer dimension. This design ensures that structural priors of cross-linked mass spectrometry can be injected during both the training and inference phases.
[0074] Step 5: Injection of cross-linked mass spectrometry confinement into the attention mechanism.
[0075] like Figure 3 As shown, in the Mixed Chunk Attention (MCA) module inside the protein language model, the adjacency matrix xlms_bias is injected as a bias term into the calculation process of the attention mechanism.
[0076] Attention is calculated as follows:
[0077] ,
[0078] Add xlms_bias to correct:
[0079] ,
[0080] In the formula, , Let be the dimension of the key vector. The bias matrix reconstructed from the adjacency matrix.
[0081] The specific calculation formula for the Mixed chunk attention module used in the protein language model is as follows:
[0082] Local attention calculation:
[0083] ,
[0084] ,
[0085] ,
[0086] Global Linear Attention Calculation:
[0087] ,
[0088] ,
[0089] Finally, merge local and global attention:
[0090] ,
[0091] LayerNorm represents the layer normalization operator, which is used to improve the stability of feature distribution and promote model convergence.
[0092] Linear represents a linear transformation (fully connected layer), used for linear mapping and projection of features;
[0093] SetScale represents mapping the input tensor blocks to the query / key spaces of different attention heads;
[0094] RoPE stands for Rotation Position Encoding, and ReLU stands for Rectified Linear Activation Function, which enhances the nonlinear expressive power of the model.
[0095] This represents element-wise multiplication, enabling feature interaction and information fusion;
[0096] These are trainable mapping parameters used for the final feature projection;
[0097] B represents the trainable bias term, which adjusts the bias effect in attention calculation;
[0098] G represents the number of groups, where the sequence is divided into G groups for attention calculation.
[0099] , The outputs are for local and global attention blocks, respectively.
[0100] Step 6: Output of high-dimensional embedding vector and implementation of joint representation.
[0101] like Figure 1 As shown in the output, the model ultimately outputs a high-dimensional embedding vector. The vector output process can be represented as:
[0102] ,
[0103] Among them, EmbeddingLayer maps the token sequence to the initial embedding vector, PLMModule represents the repeatedly stacked encoding modules (N=12), and ProjectionLayer projects the final representation to the desired target dimension.
[0104] This embedding vector can be further used for downstream bioinformatics tasks such as protein structure prediction, significantly improving the model's generalization ability and reliability in structural biology.
[0105] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements without departing from the principle of the present invention, and these improvements should also be considered within the scope of protection of the present invention.
Claims
1. A method for joint encoding of protein sequences combining protein language models and cross-linked mass spectrometry constraints, characterized in that: Includes the following steps: S1: Obtain the amino acid sequence of the target protein and the cross-linking mass spectrometry experimental data containing cross-linking residue pairs corresponding to the target protein; S2: Analyze the cross-linking mass spectrometry experimental data, extract cross-linking residue pairs, and construct a sparse adjacency matrix; S3: Encode the amino acid sequence into a token sequence; S4: Input the token sequence and the sparse adjacency matrix into the protein language model based on the Transformer architecture. The forward propagation interface of the protein language model is updated with an xlms_bias parameter to receive the sparse adjacency matrix. The protein language model contains at least one Transformer module, and each Transformer module contains a hybrid attention mechanism for calculating attention logits. S5: In the hybrid attention mechanism, the sparse adjacency matrix is added to the attention logits as a bias term to obtain the corrected attention logits; S6: The protein language model performs the addition operation layer by layer through a multi-layer Transformer module, and outputs a protein sequence jointly encoded by fused sequence semantics and cross-linked mass spectrometry spatial constraints.
2. The protein sequence co-encoding method combining protein language model and cross-linked mass spectrometry constraint as described in claim 1, characterized in that: The sparse adjacency matrix constructed in S2 is a binary matrix, where the element value at the corresponding position of cross-linked residue pairs is 1, and the element value at the corresponding position of non-cross-linked residue pairs is 0. The matrix construction formula is as follows: ， In the formula, i and j represent the index positions of the i-th and j-th residues in the protein sequence, respectively. This represents the element value at the corresponding position in the sparse adjacency matrix.
3. The protein sequence co-encoding method combining protein language model and cross-linked mass spectrometry constraint as described in claim 2, characterized in that: Multiply the binary matrix by the weighting coefficients. The weighted adjacency matrix is obtained, where For adjustable hyperparameters greater than 0, the formula for constructing the weighted matrix is: ， In the formula, This represents the element value at the corresponding position in the weighted adjacency matrix. Used to adjust the intensity of the effect of crosslinking constraints on attention distribution.
4. The protein sequence joint encoding method combining protein language model and cross-linked mass spectrometry constraint as described in claim 1, characterized in that: The hybrid attention mechanism includes local attention computation and global linear attention computation, which respectively generate corresponding attention logits. The local attention computation is as follows: ， Global linear attention is calculated as follows: ， In the formula, These are the local attention query vector and the key vector, respectively. Here, V represents the global linear attention query vector and key vector, respectively; V represents the value vector; b represents the trainable bias term; G represents the number of blocks; RoPE represents rotational position encoding; and ReLU represents the linear rectified activation function.
5. The protein sequence co-encoding method combining protein language model and cross-linked mass spectrometry constraint as described in claim 4, characterized in that: Both the local attention calculation and the global linear attention calculation use rotational position encoding (RoPE) to encode the query vector and the key vector.
6. The protein sequence joint encoding method combining protein language model and cross-linked mass spectrometry constraint as described in claim 1, characterized in that: In S5, the sparse adjacency matrix is reshaped according to the block structure of the hybrid attention mechanism, and then added to the attention logits as a bias term. The attention calculation formula is as follows: ， In the formula, Q, K, and V are the query matrix, key matrix, and value matrix, respectively. Let be the dimension of the key vector. The bias matrix is the reshaped bias matrix of the sparse adjacency matrix.
7. The protein sequence joint encoding method combining protein language model and cross-linked mass spectrometry constraint as described in claim 1, characterized in that: The protein language model comprises 12 Transformer modules, and the sparse adjacency matrix is used as a bias term in the hybrid attention mechanism of each Transformer module.
8. The protein sequence joint encoding method combining protein language model and cross-linked mass spectrometry constraint as described in claim 1, characterized in that: In S3, a batch converter is used to encode the amino acid sequence into a token sequence. The encoding formula is as follows: ， In the formula, S is the protein amino acid sequence, T is the mapped token sequence, and batchConverter is the batch converter function.
9. The protein sequence joint encoding method combining protein language model and cross-linked mass spectrometry constraint as described in claim 1, characterized in that: The protein sequences are jointly encoded for protein structure prediction or protein-protein interaction prediction tasks.