Protein language model pre-training and protein sequence processing methods and related products

Abstract

The present disclosure provides a protein language model pre-training method and related products. In one embodiment of the protein language model pre-training method, a sample protein sequence and co-evolution pairs in the sample protein sequence are masked to obtain a sample mask token sequence. Then, the sample mask token sequence is embedded and encoded based on a protein language model to obtain an encoded feature sequence. The encoded feature sequence is decoded based on a preset sequence feature decoder to obtain a decoded protein sequence. The protein language model is optimized based on the difference between the decoded protein sequence and the sample protein sequence and each co-evolution pair. That is, by masking the co-evolution pairs in the protein sequence during pre-training, the understanding and prediction ability of the protein language model for the protein sequence can be optimized, and the accuracy of downstream protein sequence / structure related tasks based on the protein language model can be improved.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of bioinformatics analysis technology, specifically to protein language model pre-training and protein sequence processing methods and related products. Background Technology

[0002] As a cutting-edge field in computational biology, protein structure prediction has undergone a paradigm shift in recent years, driven by deep learning technology. Currently, there are two main methods for protein structure prediction:

[0003] 1. Traditional co-evolutionary methods (represented by AlphaFold2):

[0004] This method constructs a Markov random field model based on multiple sequence alignment (MSA) and infers contact maps through evolutionary correlations between residues. Although it achieves atomic-level accuracy in protein prediction with abundant homologous sequences, its performance significantly degrades in single-sequence prediction, and its computational complexity increases exponentially.

[0005] 2. Large Language Model Approach (represented by ESM):

[0006] This method, typified by the ESM series models (ESM-1b / ESM2 / ESMFold), employs a Transformer architecture to achieve end-to-end sequence-to-structure mapping. It uses a Protein Language Model (PLM) to learn evolutionary information from protein sequences, replacing the MSA module in the AlphaFold2 framework. While this achieves some speed improvement, structure prediction models based solely on PLM still cannot surpass AlphaFold2. This indicates that current PLM models still have room for improvement in capturing and modeling protein evolutionary information. Summary of the Invention

[0007] Embodiments of this disclosure provide methods, apparatus, electronic devices, storage media, and computer program products for protein language model pre-training and protein sequence processing.

[0008] In a first aspect, embodiments of this disclosure provide a protein language model pre-training method, the method comprising:

[0009] The sample protein sequence and the co-evolutionary pairs in the above sample protein sequences are masked to obtain the sample masked label sequence;

[0010] Based on the protein language model, the above sample mask label sequence is embedded and feature-encoded to obtain the encoded feature sequence.

[0011] Based on a preset sequence feature decoder, the above encoded feature sequence is decoded to obtain the decoded protein sequence;

[0012] Based on the differences between the decoded protein sequence and the sample protein sequence and each of the co-evolutionary pairs, the protein language model is optimized.

[0013] In some optional embodiments, the above-mentioned masking of the sample protein sequence and the co-evolutionary pairs in the sample protein sequence to obtain a sample masked marker sequence includes:

[0014] Based on the co-evolutionary pair position information of the above sample protein sequences, at least one mask position is determined for masking the above sample protein sequences.

[0015] Based on at least one of the above mask positions, the above sample protein sequence is masked to obtain the first mask marker sequence;

[0016] Based on each co-evolutionary pair in the above sample protein sequences, a causal learning marker sequence is generated, which includes each of the above co-evolutionary pairs.

[0017] The above causal learning marker sequence is masked to obtain a second mask marker sequence, wherein the second mask marker sequence includes at least one mask co-evolutionary pair corresponding to the co-evolutionary pair in the above sample protein sequence, and the mask co-evolutionary pair includes a preset causal mask marker and an amino acid residue marker in the corresponding co-evolutionary pair;

[0018] By concatenating the first mask marker sequence and the second mask marker sequence, the above sample mask marker sequence is obtained.

[0019] In some optional implementations, determining at least one mask position for masking the sample protein sequence based on the co-evolutionary pair position information of the sample protein sequence includes:

[0020] The positions of each co-evolutionary pair in the protein sequences of the above samples were determined as mask positions;

[0021] The number of masks to be used should be determined based on the preset masking ratio and the length of the protein sequence of the sample.

[0022] The difference between the number of masking positions and the number of special masking positions is determined as the number of ordinary masking positions, where the number of special masking positions is the number of positions of each co-evolutionary pair in the protein sequence of the sample.

[0023] Among the protein sequences in the above samples that were not identified as mask locations, the number of locations that were previously identified as ordinary mask locations were selected and determined as mask locations.

[0024] In some optional implementations, the above-described masking process of the causal learning label sequence to obtain a second masked label sequence includes:

[0025] The co-evolutionary pairs with a preset positive mask ratio in each co-evolutionary pair in the above causal learning label sequence are determined as positive co-evolutionary pairs;

[0026] The co-evolutionary pairs in the above causal learning marker sequence, excluding the above positive co-evolutionary pairs, are identified as negative co-evolutionary pairs.

[0027] For each of the above positive co-evolutionary pairs, the amino acid residue marker that appears first in the positive co-evolutionary pair is replaced with the preset causal mask marker to generate the mask co-evolutionary pair corresponding to the positive co-evolutionary pair;

[0028] For each of the above reverse co-evolutionary pairs, the amino acid residue marker that appears later in the reverse co-evolutionary pair is replaced with the preset causal mask marker to generate the mask co-evolutionary pair corresponding to the reverse co-evolutionary pair.

[0029] Based on the mask co-evolution pairs corresponding to each co-evolution pair in the aforementioned causal learning label sequence, the aforementioned second mask label sequence is generated.

[0030] In some optional implementations, the protein language model described above includes an embedding representation model and a sequence feature encoding model, wherein the sequence feature encoding model includes stacked multi-head attention layers; and

[0031] Based on the protein language model, the above sample masked label sequence is embedded and feature-encoded to obtain the encoded feature sequence, including:

[0032] The above-mentioned embedding representation model is used to embed the above sample mask label sequence to obtain the input feature sequence;

[0033] The above input feature sequence is input into the above sequence feature encoding model to obtain the above encoded feature sequence.

[0034] In some optional implementations, the above-mentioned input feature sequence is input into the above-mentioned sequence feature encoding model to obtain the above-mentioned encoded feature sequence, including:

[0035] The above input feature sequence is input into the stacked multi-head attention layer to obtain the query matrix, key matrix, and value matrix;

[0036] The key matrix and the query matrix are processed to obtain the original attention score matrix, which is used to characterize the original attention between any two tags in the sample mask tag sequence.

[0037] The attention mask matrix is ​​applied to the original attention score matrix to obtain the masked attention score matrix. The attention mask matrix is ​​used to define whether attention calculation is performed between two markers in the sample masked marker sequence. Specifically: mutual attention calculation is performed between any two markers in the first masked marker sequence; no attention calculation is performed on any marker in the first masked marker sequence for any marker in the second masked marker sequence; the amino acid residue marker in the mask co-evolution pair in the second masked marker sequence performs attention calculation on all positions in the first masked marker sequence except those corresponding to the amino acid residue marker in the mask co-evolution pair, and performs attention calculation on the position of the amino acid residue marker itself in the mask co-evolution pair in the second masked marker sequence; the preset mask marker in the mask co-evolution pair in the second masked marker sequence performs attention calculation on all positions in the first masked marker sequence except those corresponding to the co-evolution pair, and performs attention calculation on the position of the mask co-evolution pair in the second masked marker sequence.

[0038] Based on the above-mentioned masked attention score matrix and the above-mentioned value matrix, the above-mentioned encoded feature sequence is determined.

[0039] Secondly, embodiments of this disclosure provide a protein sequence processing method, the method comprising:

[0040] Obtain the protein sequence to be processed;

[0041] The protein sequence to be processed is input into the protein language model to obtain the encoded feature sequence corresponding to the protein sequence to be processed. The protein language model is pre-trained by the method described in any of the implementations in the first aspect.

[0042] Thirdly, embodiments of this disclosure provide a protein language model pre-training apparatus, the apparatus comprising:

[0043] The masking module is configured to mask the sample protein sequence and the co-evolutionary pairs in the sample protein sequence to obtain the sample masked label sequence.

[0044] The embedding representation and encoding module is configured to perform embedding representation and feature encoding on the above sample mask label sequence based on the protein language model to obtain the encoded feature sequence.

[0045] The decoding module is configured to decode the above-mentioned encoded feature sequence based on a preset sequence feature decoder to obtain the decoded protein sequence;

[0046] The optimization module is configured to optimize the protein language model based on the differences between the decoded protein sequence and the sample protein sequence and each of the co-evolutionary pairs.

[0047] In some alternative implementations, the masking module includes:

[0048] The mask position determination unit is configured to determine at least one mask position for masking the protein sequence based on the co-evolutionary pair position information of the protein sequence.

[0049] The first masking unit is configured to perform masking processing on the sample protein sequence according to at least one of the above masking positions to obtain a first masking marker sequence;

[0050] The causal learning sequence generation unit is configured to generate a causal learning marker sequence based on each co-evolutionary pair in the above-mentioned sample protein sequence, wherein the causal learning marker sequence includes each of the above-mentioned co-evolutionary pairs;

[0051] The second masking unit is configured to mask the aforementioned causal learning marker sequence to obtain a second masking marker sequence, wherein the second masking marker sequence includes at least one mask co-evolutionary pair corresponding to the co-evolutionary pair in the aforementioned sample protein sequence, and the mask co-evolutionary pair includes a preset causal masking marker and an amino acid residue marker in the corresponding co-evolutionary pair.

[0052] The splicing unit is configured to splice the first mask mark sequence and the second mask mark sequence to obtain the sample mask mark sequence.

[0053] In some optional implementations, the mask location determination unit is further configured to:

[0054] The positions of each co-evolutionary pair in the protein sequences of the above samples were determined as mask positions;

[0055] The number of masks to be used should be determined based on the preset masking ratio and the length of the protein sequence of the sample.

[0056] The difference between the number of masking positions and the number of special masking positions is determined as the number of ordinary masking positions, where the number of special masking positions is the number of positions of each co-evolutionary pair in the protein sequence of the sample.

[0057] Among the protein sequences in the above samples that were not identified as mask locations, the number of locations that were previously identified as ordinary mask locations were selected and determined as mask locations.

[0058] In some alternative implementations, the second mask unit is further configured as follows:

[0059] The co-evolutionary pairs with a preset positive mask ratio in each co-evolutionary pair in the above causal learning label sequence are determined as positive co-evolutionary pairs;

[0060] The co-evolutionary pairs in the above causal learning marker sequence, excluding the above positive co-evolutionary pairs, are identified as negative co-evolutionary pairs.

[0061] For each of the above positive co-evolutionary pairs, the amino acid residue marker that appears first in the positive co-evolutionary pair is replaced with the preset causal mask marker to generate the mask co-evolutionary pair corresponding to the positive co-evolutionary pair;

[0062] For each of the above reverse co-evolutionary pairs, the amino acid residue marker that appears later in the reverse co-evolutionary pair is replaced with the preset causal mask marker to generate the mask co-evolutionary pair corresponding to the reverse co-evolutionary pair.

[0063] Based on the mask co-evolution pairs corresponding to each co-evolution pair in the aforementioned causal learning label sequence, the aforementioned second mask label sequence is generated.

[0064] In some optional implementations, the protein language model described above includes an embedding representation model and a sequence feature encoding model, wherein the sequence feature encoding model includes stacked multi-head attention layers; and

[0065] The aforementioned embedding representation and encoding modules include:

[0066] The embedding representation unit is configured to embed the sample mask label sequence using the above-mentioned embedding representation model to obtain the input feature sequence;

[0067] The encoding unit is configured to input the above-mentioned input feature sequence into the above-mentioned sequence feature encoding model to obtain the above-mentioned encoded feature sequence.

[0068] In some alternative implementations, the above-mentioned encoding unit is further configured as follows:

[0069] The above input feature sequence is input into the stacked multi-head attention layer to obtain the query matrix, key matrix, and value matrix;

[0070] The key matrix and the query matrix are processed to obtain the original attention score matrix, which is used to characterize the original attention between any two tags in the sample mask tag sequence.

[0071] The attention mask matrix is ​​applied to the original attention score matrix to obtain the masked attention score matrix. The attention mask matrix is ​​used to define whether attention calculation is performed between two markers in the sample masked marker sequence. Specifically: mutual attention calculation is performed between any two markers in the first masked marker sequence; no attention calculation is performed on any marker in the first masked marker sequence for any marker in the second masked marker sequence; the amino acid residue marker in the mask co-evolution pair in the second masked marker sequence performs attention calculation on all positions in the first masked marker sequence except those corresponding to the amino acid residue marker in the mask co-evolution pair, and performs attention calculation on the position of the amino acid residue marker itself in the mask co-evolution pair in the second masked marker sequence; the preset mask marker in the mask co-evolution pair in the second masked marker sequence performs attention calculation on all positions in the first masked marker sequence except those corresponding to the co-evolution pair, and performs attention calculation on the position of the mask co-evolution pair in the second masked marker sequence.

[0072] Based on the above-mentioned masked attention score matrix and the above-mentioned value matrix, the above-mentioned encoded feature sequence is determined.

[0073] Fourthly, embodiments of this disclosure provide a protein sequence processing apparatus, the apparatus comprising:

[0074] The acquisition module is configured to acquire the protein sequence to be processed.

[0075] The processing module is configured to input the protein sequence to be processed into a protein language model to obtain a coding feature sequence corresponding to the protein sequence to be processed, wherein the protein language model is pre-trained by a method described in any of the implementations of the first aspect.

[0076] Fifthly, embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, which, when executed by the one or more processors, cause the one or more processors to implement the method described in any of the implementations of the first and / or second aspects.

[0077] In a sixth aspect, embodiments of the present disclosure provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by one or more processors, implements the method as described in any of the implementations of the first and / or second aspects.

[0078] In a seventh aspect, embodiments of this disclosure provide a computer program product, including a computer program / instructions that, when executed by a processor, implement the method described in either the first aspect or the second aspect.

[0079] To address the issue that existing protein language models cannot simultaneously balance computational efficiency and prediction accuracy when processing protein sequences, embodiments of this disclosure provide a protein language model pre-training and protein sequence processing method, apparatus, electronic device, storage medium, and computer program product. This method involves masking a sample protein sequence and its co-evolutionary pairs to obtain a sample masked sequence; then, based on the protein language model, embedding and feature encoding are performed on the sample masked sequence to obtain an encoded feature sequence; next, based on a pre-defined sequence feature decoder, the encoded feature sequence is decoded to obtain a decoded protein sequence; finally, the protein language model is optimized based on the differences between the decoded protein sequence and the sample protein sequence and each co-evolutionary pair. This method can achieve, but is not limited to, the following technical effects:

[0080] First, while co-evolutionary pairs are highly correlated at the structural and functional levels of biomolecules, traditional protein language models are relatively easy to understand conserved and variable regions in protein sequences. Conserved regions can be learned by masking local properties, and variable regions can be learned from various sequences in the training data. However, co-evolutionary regions are difficult to learn because there is often a long-range dependency between the two amino acid residues in a co-evolutionary region. Therefore, protein language models need to learn long-range dependencies in sequences, which is difficult for traditional protein language models to learn. The embodiments of this disclosure, however, mask co-evolutionary pairs in protein sequences, allowing protein language models to learn the structural information, or causal information, of these co-evolutionary pairs. This means that a change in one amino acid residue in a co-evolutionary pair leads to a corresponding change in another amino acid residue in the same pair, ultimately optimizing the protein language model's ability to understand and predict protein sequences.

[0081] Second, since multiple sequence alignment is not used, the computational cost is relatively low compared to multiple sequence alignment.

[0082] Third, performing downstream protein sequence analysis tasks, such as sequence-related or structure-related tasks, on a protein language model pre-trained by masking co-evolutionary pairs in protein sequences can improve the accuracy of downstream protein sequence analysis tasks. Attached Figure Description

[0083] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are for illustrative purposes only and are not intended to limit the invention. In the drawings:

[0084] Figure 1 This is an exemplary system architecture diagram to which one embodiment of this disclosure can be applied;

[0085] Figure 2A This is a flowchart of an embodiment of a protein language model pre-training method according to the present disclosure;

[0086] Figure 2B This is an exploded flowchart of one embodiment of step 201 of the present disclosure;

[0087] Figure 2C This is an exploded flowchart of one embodiment of step 2011 according to the present disclosure;

[0088] Figure 2D This is an exploded flowchart of one embodiment of step 2014 according to the present disclosure;

[0089] Figure 2E This is an exploded flowchart of one embodiment of step 202 of the present disclosure;

[0090] Figure 2F This is an exploded flowchart of one embodiment of step 2022 according to the present disclosure;

[0091] Figure 3 This is a specific example of a sample protein sequence according to the present disclosure, and a specific example of a first masking sequence, a second masking sequence, and a sample masking sequence generated based on the sample protein sequence;

[0092] Figure 4 This is a specific example of a sample mask label sequence and a corresponding attention mask matrix according to this disclosure;

[0093] Figure 5 This is a flowchart of one embodiment of the protein sequence processing method according to the present disclosure;

[0094] Figure 6 This is a comparative diagram of a specific embodiment of a protein sequence-related task according to the present disclosure;

[0095] Figure 7 This is a comparative diagram of a specific embodiment of a protein structure-related task according to the present disclosure;

[0096] Figure 8 A schematic diagram of the structure of an embodiment of the protein language model pre-training device according to the present disclosure;

[0097] Figure 9 A schematic diagram of a structure of an embodiment of the protein sequence processing apparatus according to the present disclosure;

[0098] Figure 10 A schematic diagram of the structure of a computer system suitable for implementing embodiments of the present disclosure. Detailed Implementation

[0099] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0100] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0101] Figure 1 An exemplary system architecture 100 is shown, in which embodiments of the protein language model pre-training and protein sequence processing methods, apparatuses, electronic devices and storage media of the present disclosure can be applied.

[0102] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0103] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as protein language model pre-training applications and protein sequence processing applications.

[0104] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with information input devices (e.g., keyboard, mouse, touchscreen, microphone, camera, etc.) and information output devices (e.g., display screen, speaker, etc.), including but not limited to smartphones, tablets, e-book readers, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed on the terminal devices listed above. They can be implemented as multiple software programs or software modules (e.g., used to provide model pre-training services) or as a single software program or software module. No specific limitations are imposed here.

[0105] In some cases, the protein language model pre-training and protein sequence processing methods provided in this disclosure can be executed by terminal devices 101, 102, and 103, and correspondingly, the protein language model pre-training and protein sequence processing apparatus can be set in terminal devices 101, 102, and 103. In this case, the system architecture 100 may not include server 105.

[0106] In some cases, the protein language model pre-training and protein sequence processing methods provided in this disclosure can be jointly executed by terminal devices 101, 102, and 103 and server 105. For example, the step of "masking the sample protein sequence and co-evolutionary pairs in the sample protein sequence to obtain the sample mask label sequence" can be executed by terminal devices 101, 102, and 103, and the steps of "embedding representation and feature encoding of the sample mask label sequence based on the protein language model to obtain the encoded feature sequence" can be executed by server 105. This disclosure does not limit this. Accordingly, the protein language model pre-training device and the protein sequence processing device can also be respectively set in terminal devices 101, 102, and 103 and server 105.

[0107] In some cases, the protein language model pre-training and protein sequence processing methods provided in this disclosure can be executed by server 105. Accordingly, the protein language model pre-training and protein sequence processing apparatus can also be set in server 105. In this case, system architecture 100 may not include terminal devices 101, 102, and 103.

[0108] It should be noted that server 105 can be either hardware or software. When server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 105 is software, it can be implemented as multiple software programs or software modules (for example, used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0109] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0110] Continue to refer to Figure 2A The diagram illustrates a flow 200 of an embodiment of a protein language model pre-training method according to the present disclosure, which includes the following steps:

[0111] Step 201: Mask the sample protein sequence and the co-evolutionary pairs in the sample protein sequence to obtain the sample masked label sequence.

[0112] Here, the protein sequence of the sample can be obtained through methods including but not limited to the following:

[0113] 1. Direct sequencing methods: such as Edman degradation method, mass spectrometry, etc.

[0114] II. Gene Sequence Derivation Method: For example, DNA or RNA is first extracted from a biological sample, and the target gene fragment is amplified using techniques such as PCR. Then, gene sequencing is performed to obtain the nucleotide sequence of the gene. Based on the genetic code table, the nucleotide sequence of the gene is translated into the amino acid sequence of the protein.

[0115] III. Database Retrieval Method: If a protein is known to be similar to a known protein in a database, its sequence information can be obtained through database retrieval. Input a portion of the protein's sequence or characteristic information into a protein database, such as Swiss-Prot or PDB. Sequence alignment algorithms will then be used to find matching known protein sequences.

[0116] A sample protein sequence refers to a sequence obtained directly through the above-mentioned methods, which is formed by sequentially arranging amino acid residues as tokens.

[0117] Co-evolutionary pairs in a sample protein sequence can be determined by methods including, but not limited to, the following:

[0118] I. Extracting co-evolutionary pairs based on MSA data:

[0119] 1. Data Preparation: First, high-quality MSA data needs to be obtained. Protein sequence data can be downloaded from public databases such as UniProt and PDB, and the sequences can be aligned using multiple sequence alignment tools (such as ClustalW, MAFFT, etc.) to obtain MSA results.

[0120] 2. Site Selection: Since not all sites in a protein sequence are relevant to co-evolutionary analysis, it is necessary to select biologically significant sites for analysis. These typically include conserved sites, functional sites, or sites located in key regions of the protein structure. The selection of these sites can be based on existing literature reports, protein structural information, or functional annotations.

[0121] 3. Calculating the correlation between sites: A commonly used method is the mutual information (MI)-based calculation. Mutual information measures the degree of information transfer between two sites; a higher value indicates a stronger correlation. Specifically, for each pair of sites in the MSA, the frequencies of their amino acid residues in different sequences are counted, and then the mutual information is calculated using the formula. Besides mutual information, other methods such as direct coupling analysis (DCA) and covariance analysis can also be used to calculate the correlation between sites.

[0122] 4. Setting a Threshold: Determine an appropriate threshold to identify which site pairs are co-evolving pairs. The threshold setting usually needs to be adjusted based on the specific dataset and analysis objectives. Generally, the optimal threshold can be determined by comparing the number and quality of co-evolving pairs obtained at different thresholds, combined with biological knowledge. For example, thresholds used in published studies can be referenced, or a threshold that can distinguish between true co-evolving pairs and random noise can be determined through analysis of randomly shuffled sequences.

[0123] 5. Result Screening and Validation: Based on a set threshold, site pairs with correlations higher than the threshold are selected as co-evolutionary pairs. To ensure the reliability of the results, the obtained co-evolutionary pairs can be further validated. For example, it can be checked whether the co-evolutionary pairs are close to each other in the three-dimensional structure of proteins, or whether they are consistent with known protein-protein functional relationships. In addition, they can be compared with other experimental data (such as protein-protein interaction data, mutation experimental data, etc.) to verify the biological significance of the co-evolutionary pairs.

[0124] II. Co-evolutionary pairs extracted from structured data:

[0125] 1. Distance Calculation: Based on coordinate data in structural data (e.g., AFDB (AlphaFold Database)), protein sequences are analyzed to calculate the Cβ atomic spacing between any two amino acid residues. The Cβ atomic spacing refers to the spatial distance (in Å) between the Cβ atoms (β carbon atoms) of any two amino acid residues in the three-dimensional structure of a protein. The Cβ atom is the first atom of the amino acid's side chain (except for glycine, which has only one H atom in its side chain and therefore no Cβ). This distance reflects the proximity of two amino acids in three-dimensional space and can be used to analyze interactions between residues (such as hydrophobic interactions, disulfide bonds, etc.) or structural stability.

[0126] 2. Amino acid residue pairs with a Cβ interatomic spacing smaller than a preset distance threshold (e.g., 6 Å) are identified as co-evolutionary pairs.

[0127] The co-evolutionary pairs identified using the methods described above are highly correlated at the biomolecular structure and function levels.

[0128] For example, please refer to Figure 3 , Figure 3 The image shows an example of the correspondence between amino acid residue markers and their corresponding positions in a sample protein sequence, specifically “HAAEAISASQ…”. The Cβ atomic distance between “H” at position 1 and “A” at position 5 is 4.7 Å, and the Cβ atomic distance between “E” at position 4 and “S” at position 9 is 5.4 Å. Since both Cβ atomic distances are less than 6 Å, “H” at position 1 and “A” at position 5 form a co-evolutionary pair (e.g.,…). Figure 3 (As shown in purple), "E" at position 4 and "S" at position 9 form another co-evolutionary pair (as shown in purple). Figure 3 (The blue markings shown in the image).

[0129] Here, various implementation methods can be used to mask the sample protein sequence and its co-evolutionary pairs to obtain a sample masked label sequence. The sample masked label sequence includes at least a traditional masked label sequence (or Mask Language Model, MLM) sequence for masking the sample protein sequence and a causal masked label sequence (or Causal Language Model, CLM) sequence for masking the co-evolutionary pairs in the sample protein sequence. Here, the traditional masked label sequence and the causal masked label sequence can be independent parts; or the causal masked label sequence can be a part of the traditional masked label sequence; or the traditional masked label sequence can be a part of the causal masked label sequence. The traditional masked label sequence obtained by masking the sample protein sequence enables the protein language model to understand and predict protein sequences. The causal masked label sequence obtained by masking the co-evolutionary pairs in the sample protein sequence enhances the protein language model's ability to understand and predict the structural information of co-evolutionary pairs in protein sequences.

[0130] As an example, causal masking can be performed on co-evolutionary pairs in the sample protein sequence. Causal masking for each co-evolutionary pair can involve leaving one amino acid residue marker unchanged while masking the other, allowing subsequent prediction of the masked amino acid residue marker using the unmasked one.

[0131] Then, random masking is applied to the amino acid residue marker sequences obtained after causal masking, except for co-evolutionary pairs. Specifically, the number of amino acid residue markers to be randomly masked can be determined by using the ratio of the total number of amino acid residue markers obtained through causal masking and random masking to the sample protein sequence as a preset masking ratio (e.g., 15%).

[0132] Finally, the label sequence after causal masking and random masking can be determined as the sample mask label sequence.

[0133] The sample masking sequence obtained by masking in the above manner has the same length as the sample protein sequence. The part of the sample protein sequence that performs causal masking on co-evolutionary pairs can be considered as the causal masking sequence. The entire sample masking sequence can be considered as the traditional masking sequence. That is, the causal masking sequence is a part of the traditional masking sequence.

[0134] In some alternative implementations, step 201 may include, for example: Figure 2BSteps 2011 to 2015 are shown below:

[0135] Step 2011: Based on the co-evolutionary pair position information of the sample protein sequence, determine at least one mask position for masking the sample protein sequence.

[0136] Here, various implementation methods can be used to determine at least one mask position for masking the sample protein sequence based on the co-evolutionary pair position information. The determined at least one mask position can include the location of at least one co-evolutionary pair in the sample protein sequence. Optionally, statistical analysis shows that co-evolutionary pairs account for approximately 3% of the total amino acid pairs in a protein, while the mask ratio for pre-training protein language models is generally much higher than 3%. Therefore, the determined at least one mask position can include the locations of all co-evolutionary pairs in the sample protein sequence. Other positions can be selected using different specific methods depending on the actual situation.

[0137] Optionally, step 2011 may include, for example: Figure 2C The following steps 20111 to 20114 are shown:

[0138] Step 20111: Determine the positions of each co-evolutionary pair in the sample protein sequence as mask positions.

[0139] Step 20112: Determine the number of masks to be applied based on the preset mask ratio and the length of the sample protein sequence.

[0140] Here, we assume the preset mask ratio is... The length of the sample protein sequence is , The value is a positive integer. The number of masks can be calculated using the following formula:

[0141]

[0142] in, This is the number of masks that should be calculated.

[0143] As an example, the preset mask ratio is It can be 15%.

[0144] Length of sample protein sequence It can refer to the number of markers (i.e., tokens) included in the protein sequence of a sample.

[0145] Step 20113: Subtract the number of special mask positions from the number of masks required to determine the number of ordinary masks.

[0146] Here, using the formula from step 20112, step 20113 can be expressed by the following formula:

[0147] = –

[0148] Among them, the number of special mask positions This represents the number of positions of each co-evolutionary pair in the sample protein sequence.

[0149] This represents the number of ordinary masks obtained from the calculation.

[0150] Combining steps 20112 and 20113, the formula for calculating the number of ordinary masks can be expressed as follows:

[0151]

[0152] Step 20114: Select the number of ordinary mask positions from the positions in the sample protein sequence that have not been identified as mask positions, and determine them as mask positions.

[0153] After step 20111, the sample protein sequence already contains a number of special mask positions. One marker is identified as a mask location. In step 20114, a number of ordinary mask locations can be selected from the protein sequence in the sample that were not identified as mask locations in step 20111. The position is determined as the mask position.

[0154] As an example, a number of ordinary masks can be randomly and uniformly sampled from locations in the protein sequence that have not been identified as mask positions. The position is determined as the mask position.

[0155] Through the optional implementation methods of steps 20111 to 20114 described above, it is possible to determine the positions in the sample protein sequence with a preset mask ratio as mask positions, and the positions of all co-evolutionary pairs in the sample protein sequence are also determined as mask positions. Furthermore, when performing masking processing on the sample protein sequence subsequently, it is possible to mask all co-evolutionary pairs in the sample protein sequence, and to mask the markers in the sample protein sequence at a total preset mask ratio.

[0156] For example, please refer to Figure 3 The positions of the two co-evolutionary pairs in the sample protein sequence "HAAEAISASQ..."—namely, positions 1, 4, 5, and 9—were determined as mask positions. Figure 3In the diagram, the location of the first co-evolutionary pair (positions 1 and 5, marked in purple) is indicated by S1, and the location of the second co-evolutionary pair (positions 4 and 9, marked in blue) is indicated by S2. Proteins in the sample that are not co-evolutionary pairs but are identified as mask positions are indicated by M (marked in red).

[0157] Step 2012: Mask the sample protein sequence according to at least one mask position to obtain the first masked marker sequence.

[0158] Here, various implementation methods can be used to mask each mask position in the sample protein sequence to obtain the first masked label sequence.

[0159] As an example, random masking can be applied to each mask position in the sample protein sequence to obtain the first masked label sequence.

[0160] Specifically, the random mask can be performed as follows:

[0161] For each mask position determined in step 2011, there is an 80% probability that it is replaced with a preset mask marker. <m>There is a 10% probability that the label will be replaced with a random label, and a 10% probability that the original label will remain unchanged.

[0162] Here, the random label can be any random amino acid residue label.

[0163] Here, employing the aforementioned random masking strategy can prevent the protein language model from over-relying on preset mask markers. <m>"Because protein language models do not use pre-defined mask markers during inference." <m>Additionally, it can improve the robustness of protein language models to noise and erroneous inputs.

[0164] Preferably, using 15% as the preset mask ratio can balance the difficulty of the task. Too high a mask ratio will lead to insufficient contextual information, while too low a mask ratio will reduce training efficiency.

[0165] Understandably, here, the length of the first mask marker sequence can be the same as the length of the sample protein sequence, i.e. .

[0166] Step 2013: Generate causal learning marker sequences based on each co-evolutionary pair in the sample protein sequence.

[0167] Here, various implementation methods can be used to generate causal learning marker sequences based on each co-evolutionary pair in the sample protein sequence. The generated causal learning marker sequences can include each co-evolutionary pair, and according to specified rules, it can be determined which two markers in the causal learning marker sequences form a co-evolutionary pair.

[0168] For example, the co-evolutionary pairs can first be sorted according to the position of the first amino acid residue in each co-evolutionary pair in the sample protein sequence; then, the co-evolutionary pairs can be spliced ​​together according to their sorting to form a causal learning marker sequence. The order of the two amino acid residues in each spliced ​​co-evolutionary pair is the same as their order in the sample protein sequence.

[0169] As an example, for such Figure 3 The two co-evolutionary pairs in the sample protein sequence "HAAEAISASQ..." are ordered according to the position of their first amino acid residue markers, resulting in the first co-evolutionary pair ("H" at position 1 and "A" at position 5) and the second co-evolutionary pair ("E" at position 4 and "S" at position 9), arranged sequentially. This is because the first amino acid residue marker in the first co-evolutionary pair (i.e., position 1) is earlier than the first amino acid residue marker in the second co-evolutionary pair (i.e., position 4). Concatenating the first and second co-evolutionary pairs yields the causal learning marker sequence "HAES".

[0170] As another example, corresponding markers representing the start and end of each co-evolutionary pair can be added to both sides to facilitate the differentiation of different co-evolutionary pairs. Following the order of the co-evolutionary pairs, the pairs with added start and end markers are then concatenated to form a causal learning marker sequence. Specifically, such as... Figure 3 As shown, by splicing the co-evolutionary pair "HA" and "ES", the causal learning label sequence can be obtained. HA , ES ",in, Used to indicate the beginning of a co-evolutionary pair, while Used to indicate the end of a co-evolutionary pair.

[0171] Step 2014: Mask the causal learning label sequence to obtain the second masked label sequence.

[0172] Here, after masking the causal learning marker sequence, the resulting second mask marker sequence includes at least one mask co-evolutionary pair corresponding to a co-evolutionary pair in the sample protein sequence. The mask co-evolutionary pair includes a predefined causal mask marker and one amino acid residue marker from the corresponding co-evolutionary pair. For example, the mask co-evolutionary pair corresponding to the co-evolutionary pair "HA" could be "H". <m>"or" <m>A". Among them, " <m>" is a preset causal mask marker.

[0173] Here, it is assumed that the sample protein sequence contains a total of One co-evolutionary pair, It is a positive integer. The causal learning label sequence includes the above. The length of the co-evolutionary pairs and the causal learning label sequence ( (positive integer) greater than or equal to The length of the second mask label sequence can be the same as the length of the causal learning label sequence, that is, the length of the second mask label sequence can also be... .

[0174] Here, the second mask marker sequence needs to include the sequence that corresponds to the protein sequence in the sample. The mask co-evolution pairs corresponding to each co-evolution pair. Here, Less than or equal to Positive integers.

[0175] when equal At this time, it is equivalent to performing causal masking on each co-evolutionary pair in the causal learning label sequence, that is, masking and replacing one amino acid residue label in the co-evolutionary pair with a preset causal mask label, while the other amino acid residue label remains unchanged.

[0176] when Less than At that time, it is equivalent to learning the causal label sequence in One co-evolutionary pair was causally masked, while for the rest ( - If there are 10 co-evolutionary pairs, no causal masking is performed, such as no masking or masking both amino acid residues in the co-evolutionary pair.

[0177] Because the second mask marker sequence includes Co-evolutionary pairs of masks enable protein language models to predict the label of one amino acid residue from the label of another amino acid residue in the co-evolutionary pair, thus allowing the protein language model to learn the structural information of the co-evolutionary pair.

[0178] For example, masking the causal learning label sequence "HAES" yields a second masked label sequence "HS1ES2". Another example is... Figure 3 As shown, for causal learning of the labeled sequence " HA , ES "Perform masking to obtain the second mask marker sequence." HS1 , ES2 Here, only S1 and S2 are used to distinguish the masked positions of different co-evolutionary pairs. In practice, the masking label at this position can be various masking labels. For example, it can be a preset causal masking label. <m>".

[0179] Optionally, step 2014 may include, for example: Figure 2D The following steps 20141 to 20145 are shown:

[0180] Step 20141: Determine the co-evolutionary pairs with a preset positive mask ratio in each co-evolutionary pair in the causal learning label sequence as positive co-evolutionary pairs.

[0181] Here, various implementation methods can be used to determine the co-evolutionary pairs in the causal learning label sequence that have a preset positive mask ratio as positive co-evolutionary pairs. For example, a random uniform sampling method can be used.

[0182] Step 20142: Determine the remaining co-evolutionary pairs in each co-evolutionary pair in the causal learning marker sequence, excluding the forward co-evolutionary pairs, as reverse co-evolutionary pairs.

[0183] Step 20143: For each positive co-evolutionary pair, replace the amino acid residue marker that appears first in the positive co-evolutionary pair with a preset causal mask marker to generate the mask co-evolutionary pair corresponding to the positive co-evolutionary pair.

[0184] For example, for the positive co-evolutionary pair "HA", the corresponding mask co-evolutionary pair is " <m>A". Among them, " <m>" is a preset causal mask marker.

[0185] Step 20144: For each reverse co-evolution pair, replace the amino acid residue marker that appears later in the reverse co-evolution pair with a preset causal mask marker to generate the mask co-evolution pair corresponding to that reverse co-evolution pair.

[0186] For example, for the reverse co-evolutionary pair "HA", the corresponding mask co-evolutionary pair is "H". <m>".

[0187] Following steps 20141 to 20144, in the co-evolution of the causal learning marker sequences with a preset positive masking ratio, the amino acid residues that appear earlier in the sequence are masked, while those that appear later are not. For the remaining co-evolution pairs, the amino acid residues that appear later in the sequence are masked, while those that appear earlier are not.

[0188] Optionally, the forward masking ratio can be preset to 50%. That is, in half of the co-evolutionary pairs in the causal learning sequence, the amino acid residues that appear earlier in the sequence are masked, while those that appear later are not. In the other half of the co-evolutionary pairs, the amino acid residues that appear later in the sequence are masked, while those that appear earlier are not. In this way, the protein language model can learn to predict one amino acid residue from another based on any one amino acid residue in the co-evolutionary pair.

[0189] Step 20145: Generate a second masked label sequence based on the masked co-evolutionary pairs corresponding to each co-evolutionary pair in the causal learning label sequence.

[0190] Here, the second masked label sequence can be obtained by replacing each co-evolutionary pair in the causal learning label sequence with the corresponding masked co-evolutionary pair.

[0191] For example, in the causal learning label sequence "HAES", the mask co-evolutionary pair corresponding to the co-evolutionary pair "HA" is "HS1", and the mask co-evolutionary pair corresponding to the co-evolutionary pair "ES" is "ES2". Then, "HA" and "ES" in the causal learning label sequence "HA ES" can be replaced with "HS1" and "ES2" respectively, thus obtaining the second mask label sequence "HS1ES2". Another example is... Figure 3 As shown, the causal learning label sequence " HA , ES "The masked co-evolutionary pair corresponding to the co-evolutionary pair 'HA' is 'HS1', and the masked co-evolutionary pair corresponding to the co-evolutionary pair 'ES' is 'ES2'. Therefore, the causal learning label sequence can be replaced with 'HS1' and 'ES2' respectively." HA , ES The "HA" and "ES" in the sequence are used to obtain the second mask marker sequence. HS1 , ES2 ".

[0192] Step 2015: Concatenate the first mask marker sequence and the second mask marker sequence to obtain the sample mask marker sequence.

[0193] For example, such as Figure 3 As shown, the first mask marker sequence "S1MMS2S1ISAS2M…" and the second mask marker sequence " are concatenated. HS1 , ES2 The sample mask label sequence "S1MMS2S1ISAS2M…" was obtained. HS1 , ES2 ".

[0194] The sample masking marker sequence obtained by splicing according to the optional implementation of steps 2011 to 2015 above includes a first masking marker sequence that masks all co-evolutionary pairs in the sample protein sequence and a second masking marker sequence that identifies one amino acid residue of the co-evolutionary pair in the causal learning marker sequence.

[0195] Step 202: Based on the protein language model, the sample mask label sequence is embedded and encoded to obtain the encoded feature sequence.

[0196] Here, the protein language model can be any language model used for embedding representation and encoding of protein sequences.

[0197] By inputting the sample mask label sequence into the protein language model, the sample mask label sequence can be embedded and encoded to obtain the encoded feature sequence.

[0198] Specifically, protein language models can include embedding representation models and sequence feature encoding models.

[0199] Here, the embedding representation model can be any model that is currently known or will be developed in the future to convert tokens into vectors or tensors. As examples, the embedding representation model can include, but is not limited to, the following models: One-Hot Encoding, Word2Vec, GloVe (Global Vectors for Word Representation), FastText, etc.

[0200] Optionally, step 202 may include, for example: Figure 2E The following steps 2021 and 2022 are shown:

[0201] Step 2021: The sample mask label sequence is embedded using an embedding representation model to obtain the input feature sequence.

[0202] Specifically, the sample mask label sequence can be input into the embedding representation model to obtain the input feature sequence. The input features in the input feature sequence correspond one-to-one with the labels in the sample mask label sequence and are in the same order. The input features in the input feature sequence are the embedding representations of the labels in the sample mask label sequence.

[0203] Here, when inputting the sample mask label sequence into the embedding representation model, the position information of each label in the sample protein sequence can also be input simultaneously. In this way, the embedding representation model can perform a token embedding representation based on the label itself and a position embedding representation based on the position information of the label in the sample protein sequence. Finally, the token embedding representation and the position embedding representation of each label are merged to obtain the embedding representation of the corresponding label.

[0204] As an example, when using such Figure 2C When obtaining the sample mask label sequence in steps 20111 to 20114, as shown, you can refer to, for example, [the following text is incomplete and requires further context]. Figure 3 The sample masking sequence includes a first masking sequence "S1MMS2S1ISAS2M…" and a second masking sequence " H S1 , ES2 ".

[0205] When inputting the sample masked label sequence into the embedding representation model, the position information of each label in the first masked label sequence ("1,2,3,4,5,6,7,8,9,10…") and the position information of each label in the second masked label sequence ("1,5,4,9") can be input simultaneously. This allows the embedding representation model to perform positional embedding representation for each label's position information. It should be noted that, here, the position information of each label in the second masked label sequence represents the position of that label within the sample protein sequence.

[0206] Step 2022: Input the input feature sequence into the sequence feature coding model to obtain the coded feature sequence.

[0207] Here, various models suitable for encoding feature vector sequences can be used as sequence feature encoding models. For example, the encoder in Transformer can be used.

[0208] Optionally, the encoded feature sequence can have the same length as the sample mask label sequence, that is, the corresponding encoded features are obtained by encoding the embedding representation of each label in the sample mask label sequence.

[0209] Specifically, you can refer to Figure 3 The sample mask label sequence is "S1MMS2S1ISAS2M…"+" H S1 , ES2 "After inputting the protein language model CoE-PLM for embedding representation and feature encoding, the sample mask label sequence "S1MMS2S1ISAS2M…"+" can be output. HS1 , ES2 The encoded feature sequence is obtained by arranging the encoded features corresponding to each tag in order. For example, the encoded features can be feature vectors or feature tensors.

[0210] Optionally, the sequence feature encoding model may include stacked multi-head attention layers, and step 2022 may include, for example, Figure 2F The following steps 20221 to 20224 are shown:

[0211] Step 20221: Input the input feature sequence into the stacked multi-head attention layer to obtain the query matrix, key matrix and value matrix.

[0212] Here, the query matrix, key matrix, and value matrix are represented by Q, K, and V, respectively.

[0213] Step 20222: Process the key matrix and query matrix to obtain the original attention score matrix.

[0214] Here, the original attention score matrix is ​​used to characterize the original attention between any two labels in the sample mask label sequence.

[0215] Specifically, the original attention score matrix can be calculated using the following formula.

[0216]

[0217] in, and All The matrix, where, The length of the sample mask sequence is marked. , The length of the first mask marker sequence, The length of the second mask marker sequence, It is also the length of the input feature sequence. This represents the dimension of each feature in the input feature sequence. for The matrix, and then Too The matrix, The element value in the i-th row and j-th column represents the original attention of the i-th label to the j-th label in the sample mask label sequence. Here, i and j are both positive integers. That is, when the model generates the output at the i-th target position, the original attention (weight) of the model to the input at the j-th source position.

[0218] Step 20223: Apply the attention mask matrix to the original attention score matrix to obtain the masked attention score matrix.

[0219] Here, if a bidirectional attention mechanism is adopted, that is, using the original attention score matrix... The final attention score matrix and value matrix are processed to obtain the encoded feature sequence. In subsequent step 203, the encoded feature sequence is decoded based on a preset sequence feature decoder to obtain the decoded protein sequence. During the prediction of the i-th marker in the sample mask marker sequence, it is equivalent to paying attention to all markers in the sample mask marker sequence, i.e., using all contextual information for prediction. However, this will lead to a situation in the subsequent optimization of the protein language model in step 204. When calculating the loss function, the probabilities between two positions forming a co-evolutionary pair or a mask co-evolutionary pair in the sample mask marker sequence are not independent, even though there is a causal relationship between these two positions, where a change in one position causes a change in the other. Therefore, it is necessary to determine the attention mask matrix according to the generation method of the sample marker sequence, and then apply the attention mask matrix to the original attention score matrix to obtain the masked attention score matrix. The purpose is to define whether attention calculation is performed between two markers in the sample mask marker sequence through the attention mask matrix, so that in the subsequent calculation of the loss function in step 204, a causal relationship is established between the two positions of the co-evolutionary pair or the mask co-evolutionary pair, i.e., the probability of their synchronous changes is related.

[0220] Specifically, assuming the attention mask matrix is The attention score matrix after masking can be calculated using the following formula. :

[0221]

[0222] here, This represents the Hadamard Product operation, also known as the element-wise Multiplication operation.

[0223] Attention Mask Matrix Used to act on the original attention score matrix This defines whether attention calculation is performed between two tags in a sample mask tag sequence.

[0224] Specifically, an attention mask matrix can be generated based on the positions of amino acid residues and preset mask marks in the first mask mark sequence, the second mask mark sequence, and the mask co-evolution pairs in the second mask mark sequence. So that the attention mask matrix Used to act on the original attention score matrix The generated attention score matrix after masking It can achieve the following four objectives:

[0225] 1. Perform mutual attention calculation between any two tags in the first mask tag sequence.

[0226] 2. No attention calculation is performed on any tag in the first mask tag sequence for any tag in the second mask tag sequence.

[0227] As can be seen from Objectives 1 and 2 above, for each marker in the first mask marker sequence, the number of markers in the sample mask marker sequence that perform attention calculations with that marker is equal to the length of the first mask marker sequence. .

[0228] 3. Attention calculation is performed on the amino acid residue markers in the mask co-evolution pair in the second mask marker sequence for positions in the first mask marker sequence other than those corresponding to the amino acid residue markers in the mask co-evolution pair, and attention calculation is performed on the positions of the amino acid residue markers in the mask co-evolution pair in the second mask marker sequence.

[0229] As can be seen from objective 3 above, for each amino acid residue in the second masked label sequence, the number of labels in the first masked label sequence that perform attention calculations with that label is the length of the first masked label sequence minus 1, i.e. The number of tags in the second masked tag sequence that perform attention calculations with this tag is 1. Therefore, the total number of tags in the sample masked tag sequence that perform attention calculations with this tag is equal to the length of the first masked tag sequence. .

[0230] 4. In the mask co-evolution pair in the second mask label sequence, attention calculation is performed on the preset mask label pairs in the first mask label sequence except for the position of the co-evolution pair corresponding to the mask co-evolution pair, and attention calculation is performed on the position of the mask co-evolution pair in the second mask label sequence.

[0231] As can be seen from objective 4 above, for each preset mask mark in the second mask mark sequence, the number of marks in the first mask mark sequence that perform attention calculations with that mark is the length of the first mask mark sequence minus 2, that is... The number of tags in the second masked tag sequence that perform attention calculations with this tag is 2. Therefore, the total number of tags in the sample masked tag sequence that perform attention calculations with this tag is equal to the length of the first masked tag sequence. .

[0232] In summary, for each label in the sample mask label sequence, the number of labels for which attention is performed is equal to the length of the first mask label sequence. In this way, it can be ensured that when predicting each tag in the sample mask tag sequence, the pre-defined sequence feature decoder only needs to focus on a fixed number of tags in the sample mask tag sequence, i.e. There are 100 tags, and the fixed number of tags mentioned above includes the tag itself.

[0233] The following is for reference. Figure 4 The above four points will be explained. Figure 4 An example of a sample masking sequence and a corresponding attention mask matrix according to this disclosure is shown. Figure 4 As shown, the first mask marker sequence is "H <m> <m> R <m> E <m> <m> <m>"K" corresponds to Figure 4 Rows 1 to 10 and columns 1 to 10. The second mask marker sequence is "T". <m> R <m> E <m>",correspond Figure 4 Rows 11 to 16 and columns 11 to 16. Among them, " <m>"This is a preset mask marker. There are three co-evolutionary pairs in the first and second mask marker sequences, ordered from left to right:"

[0234] The second position in the first mask marker sequence <m>And the 8th position <m>As the first co-evolutionary pair, corresponding Figure 4 Rows 2 and 8, and columns 2 and 8, are marked in blue; the first co-evolutionary pair corresponding to this first mask co-evolutionary pair in the second mask marker sequence is "T". <m>",correspond Figure 4 Rows 11 and 12, and columns 11 and 12, are marked in blue;

[0235] The third position in the first mask marker sequence <m>And the 7th position <m>As the second co-evolutionary pair, corresponding Figure 4 Rows 3 and 7, and columns 3 and 7, are marked in purple; the second co-evolutionary pair corresponding to this second mask co-evolutionary pair in the second mask marking sequence is "R". <m>",correspond Figure 4 Rows 13 and 14, and columns 13 and 14, are marked in purple;

[0236] The fifth position in the first mask marker sequence <m>And the 9th position <m>This is the third co-evolutionary pair, corresponding to Figure 4 Rows 5 and 9, and columns 5 and 9, are marked in orange; the third co-evolutionary pair corresponding to this third mask co-evolutionary pair in the second mask marking sequence is "E". <m>",correspond Figure 4 Rows 15 and 16, and columns 15 and 16, are marked in orange.

[0237] if Figure 4 The cell in the i-th row and j-th column contains " ", indicating the attention mask matrix The element in the i-th row and j-th column It is 0 or a very small value. In the attention mask matrix... Compared with the original attention score matrix When performing element-wise multiplication and The element in the i-th row and j-th column Multiplication yields the attention score matrix after masking. The element in the i-th row and j-th column ,because It is 0 or a very small value. exist The value is 0 or a minimum under the influence of the attention mask matrix, indicating that... Under the influence of the sample mask label sequence, the attention of the i-th label to the j-th label is... There is no relationship; that is, the i-th marker in the sample mask marker sequence does not perform attention calculation on the j-th marker. Furthermore, in the subsequent step 203, during the process of decoding the encoded feature sequence based on the preset sequence feature decoder to obtain the decoded protein sequence, when predicting the i-th marker in the sample mask marker sequence by decoding the encoded feature sequence, the preset sequence feature decoder does not pay attention to the j-th marker in the sample mask marker sequence, or in other words, it cannot see the j-th marker.

[0238] Conversely, if Figure 4 There is no " in the i-th row and j-th column of the middle cell. ", indicating the attention mask matrix The element in the i-th row and j-th column A value of 1 or very close to 1. In the attention mask matrix... Compared with the original attention score matrix When performing element-wise multiplication and The element in the i-th row and j-th column Multiplication yields the attention score matrix after masking. The element in the i-th row and j-th column ,because A value of 1 or very close to 1. Will equal to Or very close ,and Used to characterize the attention of the i-th label to the j-th label in the sample mask label sequence, thus It can also be used to characterize the attention of the i-th label to the j-th label in a sample mask label sequence, i.e., in the attention mask matrix. Under the influence of [the function], the i-th marker in the sample mask marker sequence performs attention calculation on the j-th marker. Furthermore, in subsequent step 203, during the decoding of the encoded feature sequence based on the preset sequence feature decoder to obtain the decoded protein sequence, in the process of predicting the i-th marker in the sample mask marker sequence by decoding the encoded feature sequence, the preset sequence feature decoder focuses on the j-th marker in the sample mask marker sequence, or in other words, it can see the j-th marker, and the degree of attention the preset sequence feature decoder pays to the j-th marker in the sample mask marker sequence is [the degree of attention]. .

[0239] The following is for reference. Figure 4 Let's explain the attention mask matrix in detail. The above four objectives have been achieved:

[0240] 1. Perform mutual attention calculation between any two tags in the first mask tag sequence.

[0241] 2. No attention calculation is performed on any tag in the first mask tag sequence for any tag in the second mask tag sequence.

[0242] Regarding the two objectives mentioned above, specifically... Figure 4 It can be seen that for rows 1 to 10 (corresponding to the first mask mark sequence), none of the squares in columns 1 to 10 (corresponding to the first mask mark sequence) contain "". That is, mutual attention is calculated between any two tags in the first mask tag sequence; while the squares in columns 11 to 16 (corresponding to the second mask tag sequence) contain " "This means that no attention calculation is performed on any tag in the first mask tag sequence for any tag in the second mask tag sequence."

[0243] That is, the encoded features corresponding to rows 1 to 10 are subsequently decoded using a pre-defined sequence feature decoder. This means that when generating the decoded protein sequence corresponding to the first masked label sequence, any label in the first masked label sequence can see all labels in the first masked label sequence. In other words, when predicting a label at a certain position in the first masked label sequence, all positions in the first masked label sequence (including itself, forward, and backward) can be considered simultaneously. Through this bidirectional self-attention mechanism, the pre-trained protein language model can more comprehensively understand the contextual relationships of each amino acid residue label in the protein sequence (e.g., understanding long-distance dependencies), improving the protein language model's ability to comprehensively capture global and local information about the protein sequence. However, when predicting a label at a certain position in the first masked label sequence, it is not necessary to consider any label in the second masked label sequence. This is because the second masked label sequence does not carry the relevant contextual information from the first masked label sequence, therefore, it is not necessary to consider any label in the second masked label sequence. When predicting the label at a certain position in the first mask label sequence, the first mask label sequence can already take into account all the context information and also carries the initial causal information (i.e., co-evolutionary pair information), so there is no need to provide causal information from the second mask label sequence. Therefore, there is no need to take into account any label in the second mask label sequence.

[0244] In other words, for each tag in the first mask tag sequence, the number of tags in the sample mask tag sequence that perform attention calculations with that tag is equal to the length of the first mask tag sequence. .

[0245] 3. Attention calculation is performed on the amino acid residue markers in the mask co-evolution pair in the second mask marker sequence for positions in the first mask marker sequence other than those corresponding to the amino acid residue markers in the mask co-evolution pair, and attention calculation is performed on the positions of the amino acid residue markers in the mask co-evolution pair in the second mask marker sequence.

[0246] Specifically, please refer to Figure 4 ,like Figure 4 As shown:

[0247] The first mask co-evolution pair "T" in the second mask marker sequence <m>( Figure 4 (marked in blue) Figure 4 Rows 11 and 12, and corresponding columns 11 and 12. From Figure 4 As can be seen from this, the first mask co-evolution pair "T" <m>The amino acid residues in the text are marked as "T" in row 11 and column 11.

[0248] For the first mask co-evolution pair "T" in the second mask tag sequence <m>The amino acid residue marker "T" in the sequence means that, for row 11, in the squares of columns 1 to 10 (corresponding to the first mask marker sequence), only the square in column 2 (corresponding to the amino acid residue marker "T" itself in the first mask marker sequence) contains "T". (That is, attention calculation is not performed), and there are no "" in the other squares. (i.e., perform attention calculation).

[0249] In the squares of columns 11 to 16 of row 11 (corresponding to the second masking sequence), only the square in column 11 (corresponding to the amino acid residue marker "T" in the second masking sequence) does not contain "T". (That is, to perform attention calculations), all other squares contain " (That is, attention calculation is not performed).

[0250] That is, when the encoded features corresponding to row 11 are subsequently decoded using a preset sequence feature decoder, the positions other than column 2 in columns 1 to 10, as well as column 11, can be seen. Essentially, to predict the marker corresponding to row 11, attention needs to be paid to positions other than the amino acid residue marker in the first mask marker sequence. This means that attention also needs to be paid to the amino acid residue itself, but not in the first mask marker sequence; rather, attention is paid to the amino acid residue itself in the second mask marker sequence. Therefore, for each amino acid residue marker in the second mask marker sequence, the number of markers in the first mask marker sequence that undergo attention calculation with that marker is the length of the first mask marker sequence minus 1. The number of tags in the second masked tag sequence that perform attention calculations with this tag is 1, therefore the total number of tags in the sample masked tag sequence that perform attention calculations with this amino acid residue tag is still the length of the first masked tag sequence. .

[0251] about Figure 4 The second mask co-evolution pair "R" <m>The amino acid residue marker "R" and the third mask co-evolution pair "E" in the text. <m>The amino acid residue marker "E" can be referenced above regarding the first mask co-evolution pair "T". <m>The amino acid residue marker "T" in "" will not be elaborated upon here.

[0252] 4. In the mask co-evolution pair in the second mask label sequence, attention calculation is performed on the preset mask label pairs in the first mask label sequence except for the position of the co-evolution pair corresponding to the mask co-evolution pair, and attention calculation is performed on the position of the mask co-evolution pair in the second mask label sequence.

[0253] Specifically, please refer to Figure 4 ,like Figure 4 As shown:

[0254] The first mask co-evolution pair "T" in the second mask marker sequence <m>( Figure 4 (marked in blue), corresponding to Figure 4 Rows 11 and 12, and corresponding columns 11 and 12. From Figure 4 As can be seen from this, the first mask co-evolution pair "T" <m>The preset mask in the text is marked as "12th row and 12th column". <m>".

[0255] For the mask co-evolution pair "T" in the second mask tag sequence <m>Preset mask marker in "" <m>(i.e., blue) <m>", meaning that for row 12, in the squares of columns 1 to 10 (corresponding to the first mask marker sequence), only the first co-evolutionary pairs corresponding to the first mask co-evolutionary pairs in columns 2 and 8 (corresponding to the first mask marker sequence, the two blue " <m>) There are squares in the grid (That is, attention calculation is not performed), and there are no "" in the other squares. (i.e., perform attention calculation).

[0256] In the 12th row, columns 11 to 16 (corresponding to the second mask marker sequence), only the 11th and 12th columns (corresponding to the first mask co-evolution pair in the second mask marker sequence) do not contain "". (That is, to perform attention calculations), the remaining squares contain " (That is, attention calculation is not performed).

[0257] That is, when the encoded features corresponding to row 12 are subsequently decoded using the preset sequence feature decoder, the positions in columns 1 to 10, excluding columns 2 and 8, and columns 11 and 12, can be seen. This is equivalent to needing to focus on positions in the first masked label sequence other than the position corresponding to the co-evolutionary pair of the preset masked label in the masked co-evolutionary pair. Similarly, it's necessary to focus on the masked co-evolutionary pair itself in the second masked label sequence, but not the corresponding co-evolutionary pair in the first masked label sequence. Instead, focus on the two positions of the masked co-evolutionary pair in the second masked label sequence. Therefore, for each preset masked label in the second masked label sequence, the number of labels in the first masked label sequence that undergo attention calculation with that label is the length of the first masked label sequence minus 2. The number of tags in the second mask tag sequence that perform attention calculations with the preset mask tag is 2, therefore the total number of tags in the sample mask tag sequence that perform attention calculations with the preset mask tag is still equal to the length of the first mask tag sequence. .

[0258] The attention score matrix obtained using the above method after masking Too Given a matrix, for the i-th row of that matrix... Among them, there are The element value and The values ​​at the corresponding positions in the [data] are the same or very close, while the rest [data] The element values ​​are 0 or very close to 0. This allows the protein language model to learn contextual dependencies similar to a bidirectional attention mechanism, as well as causal relationships between two markers in a co-evolutionary pair, thereby enhancing the protein language model's understanding of protein sequence structures.

[0259] Step 20224: Determine the encoded feature sequence based on the post-masked attention score matrix and value matrix.

[0260] Specifically, the encoded feature sequence can be calculated using the following formula:

[0261]

[0262] in, This is the attention score matrix after masking. For value matrices, The calculated encoded feature sequence is denoted as . , and All The matrix.

[0263] By employing the optional implementation methods of steps 20221 to 20224 described above, an autoregressive mechanism is introduced by improving the bidirectional attention mechanism of stacked multi-head attention layers. The autoregressive strategy is designed according to the generation method of the sample mask label sequence, i.e., autoregression is achieved by designing a corresponding mask attention matrix. This allows the protein language model to learn co-evolutionary structural information. Co-evolutionary relationships are crucial for protein folding, function, and dynamic processes. Therefore, by enabling the protein language model to learn co-evolutionary information, it can learn the structural information of proteins, providing a model understanding foundation for subsequent protein structure prediction tasks.

[0264] Step 203: Based on the preset sequence feature decoder, decode the encoded feature sequence to obtain the decoded protein sequence.

[0265] Specifically, the encoded feature sequence can be input into a preset sequence feature decoder to obtain the decoded protein sequence.

[0266] Here, a predefined sequence feature decoder is used to characterize the correspondence between feature sequences and protein sequences. The predefined sequence feature decoder may include linear transformation layers and / or nonlinear transformation layers. For example, the predefined sequence feature decoder may include a sequentially connected fully connected network and an activation function layer. Inputting the encoded feature sequence into the fully connected network yields P values ​​corresponding to each token in the sample mask token sequence, where P is the number of different amino acid residue tokens that may appear in the protein sequence. Normalizing these P values ​​yields P probability values, each corresponding to the probability of belonging to one of the P amino acid residue tokens. The amino acid residue token with the highest probability value is then used as the decoded amino acid residue token corresponding to that token in the sample mask token sequence. Finally, a decoded protein sequence can be generated using the decoded amino acid residue tokens corresponding to each token in the sample mask token sequence. That is, the decoded protein sequence is formed by sequentially arranging the decoded amino acid residue tokens, and the sequence length of the decoded protein sequence can be the same as the sequence length of the sample mask token sequence.

[0267] Step 204: Optimize the protein language model based on the differences between the decoded protein sequence and the sample protein sequence and each co-evolutionary pair.

[0268] Here, various implementation methods can be adopted to optimize the protein language model by minimizing the difference between the decoded protein sequence and the original unmasked label sequence corresponding to the sample masked label sequence. This allows for the adjustment of the protein language model's parameters based on the differences between the decoded protein sequence, the sample protein sequence, and each co-evolutionary pair.

[0269] As an example, step 204 can be performed as follows:

[0270] The first step is to calculate the loss function: calculate the loss function between the decoded protein sequence and the original label sequence before it was masked, corresponding to the sample masked label sequence.

[0271] Here, various known or future loss function calculation methods can be used, such as cross-entropy or mean squared error, etc.

[0272] The second step is to perform backpropagation:

[0273] First, gradient calculation is performed: using the chain rule, the gradient of the loss function with respect to each parameter is calculated layer by layer from the output layer of the protein language model.

[0274] Next, gradient propagation is performed: the gradient is propagated from the output layer to each layer, and the contribution of each parameter to the loss is identified.

[0275] The third step is to perform parameter updates (Optimization):

[0276] Updating parameters based on gradients using optimization algorithms is a common approach, including:

[0277] Gradient descent: such as batch gradient descent (BGD), stochastic gradient descent (SGD), mini-batch gradient descent (Mini-batch GD), etc.

[0278] Other improved optimization algorithms include: Momentum, Adam (Adaptive Moment Estimation), etc.

[0279] Step 4, hyperparameter adjustment:

[0280] The update step size is controlled by a pre-set learning rate η. Too large a rate leads to oscillations, while too small a rate results in slow convergence. A learning rate scheduler (such as ReduceLROnPlateau or cosine annealing) can be used.

[0281] Setting the batch size affects the accuracy of gradient estimation and memory usage.

[0282] Overfitting can be prevented by using regularization methods such as L2 regularization (weight decay) and Dropout.

[0283] Step 5, iterative training:

[0284] Repeat the following steps until the model converges:

[0285] 1. Forward propagation → Calculate loss.

[0286] 2. Backpropagation → Calculate gradient.

[0287] 3. The optimizer updates parameters.

[0288] 4. (Optional) Validation set evaluation, early stopping to prevent overfitting.

[0289] It should be noted that steps 201 to 205 above only show the process of pre-training the protein language model based on a single sample protein sequence. In practice, the protein language model can be pre-trained multiple times. Each pre-training can use at least one sample protein sequence to execute steps 201 to 203. In step 204, the difference between the corresponding decoded protein sequence of each sample protein sequence and the original label sequence before masking of the sample mask label sequence is calculated, and the model parameters of the protein language model are adjusted based on the sum of the calculated differences.

[0290] It should be noted that in step 204, in addition to adjusting the model parameters of the protein language model, the model parameters of the preset sequence feature decoder can also be adjusted simultaneously.

[0291] The protein language model pre-training method provided in the above embodiments of this disclosure obtains a sample masked sequence by masking the sample protein sequence and co-evolutionary pairs within the sample protein sequence; then, based on the protein language model, the sample masked sequence is embedded and feature-encoded to obtain an encoded feature sequence; next, the encoded feature sequence is decoded using a preset sequence feature decoder to obtain a decoded protein sequence; finally, the protein language model is optimized based on the differences between the decoded protein sequence and the sample protein sequence and each co-evolutionary pair. This method can achieve, but is not limited to, the following technical effects:

[0292] First, while co-evolutionary pairs are highly correlated at the structural and functional levels of biomolecules, traditional protein language models are relatively easy to understand conserved and variable regions in protein sequences. Conserved regions can be learned by masking local properties, and variable regions can be learned from various sequences in the training data. However, co-evolutionary regions are difficult to learn because there is often a long-range dependency between the two amino acid residues in a co-evolutionary region. Therefore, protein language models need to learn long-range dependencies in sequences, which is difficult for traditional protein language models to learn. The embodiments of this disclosure, however, mask co-evolutionary pairs in protein sequences, allowing protein language models to learn the structural information, or causal information, of these co-evolutionary pairs. This means that a change in one amino acid residue in a co-evolutionary pair leads to a corresponding change in another amino acid residue in the same pair, ultimately optimizing the protein language model's ability to understand and predict protein sequences.

[0293] Second, since multiple sequence alignment is not used, the computational cost is relatively low compared to multiple sequence alignment.

[0294] Third, performing downstream protein sequence analysis tasks, such as sequence-related or structure-related tasks, on a protein language model pre-trained by masking co-evolutionary pairs in protein sequences can improve the accuracy of downstream protein sequence analysis tasks.

[0295] The following is for reference. Figure 5 The diagram illustrates a flow 500 of an embodiment of a protein sequence processing method according to the present disclosure. This protein sequence processing method includes the following steps:

[0296] Step 501: Obtain the protein sequence to be processed.

[0297] Here, the protein sequence to be processed can also be obtained by the same or similar method as described in step 201.

[0298] Step 502: Input the protein sequence to be processed into the protein language model to obtain the coding feature sequence corresponding to the protein sequence to be processed.

[0299] Here, the protein language model can be achieved through methods such as... Figure 2A The protein language model is pre-trained using the methods described in the illustrated embodiments and their alternative implementations.

[0300] In the aforementioned protein sequence encoding pre-trained model, masking co-evolutionary pairs in protein sequences allows the protein language model to learn the structural information of these pairs, thereby optimizing its ability to understand and predict protein sequences. Furthermore, since multiple sequence alignment is not used, the computational cost is relatively low compared to multiple sequence alignment.

[0301] Furthermore, based on the aforementioned protein sequence language model, decoders for other protein sequence analysis tasks can be concatenated to perform downstream protein sequence analysis tasks. Because the aforementioned protein sequence encoding pre-trained model has been specifically pre-trained for the characteristics of protein sequences, it can improve the accuracy and computational speed of downstream protein sequence analysis tasks. For example, downstream protein analysis tasks can be protein sequence-related tasks or structure-related tasks.

[0302] The protein sequence processing method provided in the above embodiments of this disclosure improves the accuracy and computational speed of protein sequence understanding by employing a protein language model pre-trained on structural information based on co-evolution of protein sequences and by not involving multiple sequence alignments.

[0303] Example 1:

[0304] Please refer to Figure 6 The applicant will Figure 2A The illustrated embodiment shows a comparison between the protein language model pre-training method disclosed therein and BERT as a baseline model on a protein sequence-related task.

[0305] Among them, 150_Coe PLM corresponds to Figure 2A The illustrated embodiment discloses a protein language model pre-training method, with 150_bert_baseline_30 corresponding to BERT as the baseline model. Both models have a parameter size of 150MB.

[0306] Figure 6 In the graph, the horizontal axis represents the number of model iterations during pre-training; the vertical axis represents the Spearman Rank Correlation coefficient between the two models pre-trained according to the corresponding number of iterations on ProteinGym (ProteinGym is a benchmark platform for evaluating protein machine learning models, encompassing various tasks). Here, the Spearman Rank Correlation coefficient is a nonparametric statistic used to measure the monotonicity (without linear assumptions) between the model-predicted mutant ranking and the experimentally measured true ranking, ranging from -1 (representing a perfect negative correlation) to 1 (representing a perfect positive correlation). In ProteinGym, the core of the Spearman task is to test whether protein language models can relatively rank the functional impact of mutants as experimentally, which is crucial for guiding protein engineering or understanding evolution.

[0307] In the Spearman task, the input data consists of protein sequence mutants (such as single-point mutations, insertions, deletions, etc.). The output data consists of predicted functional effects of the mutants relative to the wild type (such as experimental measurements of enzyme activity, stability, fluorescence intensity, etc.). The evaluation method is to compare the correlation between the mutant ranking predicted by the model and the actual ranking in the experiment (Spearman coefficient).

[0308] Therefore, the Spearman task directly relies on the relationship between sequence variations (i.e., changes in the amino acid sequence) and function, rather than the three-dimensional structure of the protein. In other words, the Spearman coefficient reflects the model's performance on sequence-related tasks.

[0309] from Figure 6 It can be seen that when the number of iterations is less than 60,000, the Spearman rank correlation coefficient of 150_Coe PLM is slightly higher than that of 150_bert_baseline_30. However, when the number of iterations is 70,000, 80,000, and 100,000, the Spearman rank correlation coefficient of 150_Coe PLM is slightly lower than that of 150_bert_baseline_30.

[0310] It is evident that, with the same number of model parameters and a smaller computational load (less than 100,000 model iterations), the protein language model trained using the protein language model pre-training method provided in this embodiment can achieve an accuracy comparable to BERT on protein sequence-related tasks.

[0311] Example 2:

[0312] Please refer to Figure 7 The applicant will Figure 2A The illustrated embodiment shows a comparison between the protein language model pre-training method disclosed therein and BERT as a baseline model on protein structure-related tasks.

[0313] Among them, 150_Coe PLM corresponds to Figure 2A The illustrated embodiment discloses a protein language model pre-training method, with 150_bert_baseline_30 corresponding to BERT as the baseline model. Both models have a parameter size of 150MB.

[0314] Figure 7 In the figure, the horizontal axis represents the number of model iterations during pre-training; the vertical axis represents the Top-L / 5 Precision of the two models pre-trained according to the corresponding number of iterations in Casp14 (Critical Assessment of Protein Structure Prediction).

[0315] Top-L / 5 Precision is one of the important metrics for evaluating the accuracy of protein structure prediction models (including protein language models) in predicting contact maps or distances between residues.

[0316] In the task of predicting contact maps, the objective is to evaluate the spatial proximity (i.e., whether or not the protein residue pairs are in contact) predicted by the model. The input data is the amino acid sequence of the protein. The output data is the distance or contact probability between the predicted residue pairs (usually converted into a binary contact map). In protein structure prediction, accurate contact / distance prediction can significantly improve the accuracy of 3D structure modeling.

[0317] The calculation method for Top-L / 5 Precision is as follows:

[0318] Assume the length of the protein sequence is L.

[0319] Top-L / 5 refers to selecting the L / 5 pairs with the highest confidence from all residue pairs predicted by the model (i.e., a standard subset of the number of long-range residue pairs).

[0320] Top-L / 5 Precision refers to the proportion of selected Top-L / 5 residue pairs that actually contact in the real structure (i.e., the distance is ≤ a certain threshold, such as 6 Å).

[0321] In CASP14, Top-L / 5 Precision is a core metric for evaluating the reliability of protein language models in "high-confidence predictions" of residue contact predictions. It measures the model's ability to identify key interactions by standardizing the selection of Top-L / 5 long-range residue pairs, providing important constraints for subsequent 3D structure modeling. The performance of protein language models in this task reflects their ability to directly infer structural information from sequences.

[0322] from Figure 7 It can be seen that when the number of iterations is less than 100,000, the Top-L / 5 Precision of 150_Coe PLM is significantly higher than that of 150_bert_baseline_30.

[0323] It is evident that, with the same number of model parameters and a smaller computational load (less than 100,000 model iterations), the protein language model trained using the protein language model pre-training method provided in this embodiment significantly outperforms BERT on protein structure-related tasks.

[0324] As can be seen from Examples 1 and 2, the protein language model trained by the protein language model pre-training method provided in this disclosure achieves or exceeds the performance of BERT in both protein sequence-related tasks and structure-related tasks.

[0325] Further reference Figure 8 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a protein language model pre-training device, which is similar to... Figure 2A Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0326] like Figure 8 As shown, the protein language model pre-training device 800 of this embodiment includes: a masking module 801, an embedding representation and encoding module 802, a decoding module 803, and an optimization module 804. The masking module 801 is configured to mask a sample protein sequence and co-evolutionary pairs within the sample protein sequence to obtain a sample masked label sequence. The embedding representation and encoding module 802 is configured to perform embedding representation and feature encoding on the sample masked label sequence based on the protein language model to obtain an encoded feature sequence. The decoding module 803 is configured to decode the encoded feature sequence based on a preset sequence feature decoder to obtain a decoded protein sequence. The optimization module 804 is configured to optimize the protein language model based on the differences between the decoded protein sequence and the sample protein sequence and each of the co-evolutionary pairs.

[0327] In this embodiment, the specific processing of the masking module 801, the embedding representation and encoding module 802, the decoding module 803, and the optimization module 804 of the protein language model pre-training device 800, and the resulting technical effects, can be found in reference to [the documentation / reference]. Figure 2A The relevant descriptions of steps 201, 202, 203 and 204 in the corresponding embodiments will not be repeated here.

[0328] In some optional embodiments, the masking module 801 may include: a mask position determination unit 8011, configured to determine at least one mask position for masking the sample protein sequence based on the co-evolutionary pair position information of the sample protein sequence; a first masking unit 8012, configured to mask the sample protein sequence according to the at least one mask position to obtain a first mask marker sequence; a causal learning sequence generation unit 8013, configured to generate a causal learning marker sequence based on each co-evolutionary pair in the sample protein sequence, the causal learning marker sequence including each of the co-evolutionary pairs; a second masking unit 8014, configured to mask the causal learning marker sequence to obtain a second mask marker sequence, wherein the second mask marker sequence includes at least one mask co-evolutionary pair corresponding to the co-evolutionary pair in the sample protein sequence, the mask co-evolutionary pair including a preset causal mask marker and an amino acid residue marker in the corresponding co-evolutionary pair; and a splicing unit 8015, configured to splice the first mask marker sequence and the second mask marker sequence to obtain the sample mask marker sequence.

[0329] In some optional embodiments, the mask position determination unit 8011 may be further configured to: determine the positions of each co-evolutionary pair in the sample protein sequence as mask positions; determine the number of positions to be masked according to a preset mask ratio and the length of the sample protein sequence; determine the number of ordinary mask positions by subtracting the number of special mask positions from the number of positions to be masked, wherein the number of special mask positions is the number of positions of each co-evolutionary pair in the sample protein sequence; and select the number of ordinary mask positions from the positions in the sample protein sequence that have not been identified as mask positions and determine them as mask positions.

[0330] In some optional embodiments, the second masking unit 8014 may be further configured to: determine the co-evolutionary pairs with a preset forward masking ratio in each co-evolutionary pair in the causal learning marker sequence as forward co-evolutionary pairs; determine the remaining co-evolutionary pairs in each co-evolutionary pair in the causal learning marker sequence other than each of the aforementioned forward co-evolutionary pairs as reverse co-evolutionary pairs; for each of the aforementioned forward co-evolutionary pairs, replace the first amino acid residue marker in the forward co-evolutionary pair with the preset causal masking marker to generate a mask co-evolutionary pair corresponding to the forward co-evolutionary pair; for each of the aforementioned reverse co-evolutionary pairs, replace the last amino acid residue marker in the reverse co-evolutionary pair with the preset causal masking marker to generate a mask co-evolutionary pair corresponding to the reverse co-evolutionary pair; and generate the second masking marker sequence based on the mask co-evolutionary pairs corresponding to each of the aforementioned co-evolutionary pairs in the causal learning marker sequence.

[0331] In some optional implementations, the protein language model described above may include an embedding representation model and a sequence feature encoding model, wherein the sequence feature encoding model may include stacked multi-head attention layers; and

[0332] The aforementioned embedding representation and encoding module 802 may include: an embedding representation unit 8021, configured to embed the aforementioned sample mask label sequence using the aforementioned embedding representation model to obtain an input feature sequence; and an encoding unit 8022, configured to input the aforementioned input feature sequence into the aforementioned sequence feature encoding model to obtain the aforementioned encoded feature sequence.

[0333] In some optional implementations, the encoding unit 8022 may be further configured to: input the input feature sequence into the stacked multi-head attention layer to obtain a query matrix, a key matrix, and a value matrix; process the key matrix and the query matrix to obtain an original attention score matrix, which is used to characterize the original attention between any two tags in the sample masked tag sequence; apply the attention mask matrix to the original attention score matrix to obtain a masked attention score matrix, wherein the attention mask matrix is ​​applied to the original attention score matrix to define whether attention calculation is performed between two tags in the sample masked tag sequence, specifically: mutual attention calculation is performed between any two tags in the first masked tag sequence; any tag in the first masked tag sequence... Attention calculation is not performed on any of the markers in the second masking sequence; attention calculation is performed on the amino acid residue markers in the mask co-evolution pairs in the second masking sequence for all positions in the first masking sequence except those corresponding to the amino acid residue markers in the mask co-evolution pairs, and attention calculation is performed on the position of the amino acid residue marker itself in the mask co-evolution pairs in the second masking sequence; attention calculation is performed on the preset masking markers in the mask co-evolution pairs in the second masking sequence for all positions in the first masking sequence except those corresponding to the co-evolution pairs, and attention calculation is performed on the position of the mask co-evolution pairs in the second masking sequence; based on the post-masking attention score matrix and the value matrix, the encoded feature sequence is determined.

[0334] It should be noted that the implementation details and technical effects of each module in the protein language model pre-training device provided in the embodiments of this disclosure can be referred to the descriptions of other embodiments in this disclosure, and will not be repeated here.

[0335] The following is for reference. Figure 9 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a protein sequence processing apparatus, which is similar to... Figure 5 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0336] like Figure 9 As shown, the protein sequence processing apparatus 900 of this embodiment includes an acquisition module 901 and a processing module 902. The acquisition module 901 is configured to acquire a protein sequence to be processed; the processing module 902 is configured to input the protein sequence to be processed into a protein language model to obtain a coding feature sequence corresponding to the protein sequence to be processed, wherein the protein language model is pre-trained using a method described in any implementation of the first aspect.

[0337] In this embodiment, the specific processing of the acquisition module 901 and the processing module 902 of the protein sequence processing device 900 and the resulting technical effects can be referred to respectively. Figure 5 The relevant descriptions of steps 501 and 502 in the corresponding embodiments will not be repeated here.

[0338] It should be noted that the implementation details and technical effects of each module in the protein sequence processing device provided in the embodiments of this disclosure can be referred to the descriptions of other embodiments in this disclosure, and will not be repeated here.

[0339] The following is for reference. Figure 10 It shows a schematic diagram of the structure of a computer system 1000 suitable for implementing the electronic device of the present disclosure. Figure 10 The computer system 1000 shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0340] like Figure 10 As shown, the computer system 1000 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1008 into a random access memory (RAM) 1003. The RAM 1003 also stores various programs and data required for the operation of the computer system 1000. The processing device 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0341] Typically, the following devices can be connected to the I / O interface 1005: input devices 1006 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, etc.; output devices 1007 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1008 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the computer system 1000 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 10 A computer system 1000 with various electronic devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0342] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1009, or installed from storage device 1008, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of embodiments of this disclosure.

[0343] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0344] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0345] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the following functions: Figure 2A The embodiments shown and their alternative implementations illustrate protein language model pre-training methods and / or, as described above. Figure 5 The examples and their alternative implementations illustrate protein sequence processing methods.

[0346] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Python, Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0347] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0348] The modules described in the embodiments of this disclosure can be implemented in software or in hardware. The names of modules do not necessarily limit the module itself; for example, a masking module can also be described as "a module that performs masking processing on a sample protein sequence and co-evolutionary pairs within the sample protein sequence to obtain a sample masked label sequence."

[0349] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.< / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m>

Claims

1. A protein language model pre-training method, comprising: determining positions of each co-evolution pair in a sample protein sequence as mask positions; determining a number of maskable positions according to a preset mask ratio and a length of the sample protein sequence; determining a number of normal masks as a difference between the number of maskable positions and a number of special mask positions, wherein the number of special mask positions is a number of positions of each co-evolution pair in the sample protein sequence; selecting a number of normal mask positions from positions in the sample protein sequence that are not determined as mask positions, wherein the number of normal mask positions includes positions of each co-evolution pair in the sample protein sequence; masking the sample protein sequence according to the number of normal mask positions to obtain a first mask token sequence; generating a causal learning token sequence based on each co-evolution pair in the sample protein sequence, wherein the causal learning token sequence includes each co-evolution pair; masking the causal learning token sequence to obtain a second mask token sequence, wherein the second mask token sequence includes at least one mask co-evolution pair corresponding to a co-evolution pair in the sample protein sequence, and the mask co-evolution pair includes a preset causal mask token and an amino acid residue token in the corresponding co-evolution pair; concatenating the first mask token sequence and the second mask token sequence to obtain a sample mask token sequence; embedding and feature encoding the sample mask token sequence based on a protein language model to obtain an encoded feature sequence; decoding the encoded feature sequence based on a preset sequence feature decoder to obtain a decoded protein sequence; optimizing the protein language model based on differences between the decoded protein sequence and the sample protein sequence and each co-evolution pair.

2. The method of claim 1, wherein, The masking the causal learning token sequence to obtain a second mask token sequence comprises: determining co-evolution pairs with a preset forward mask ratio in the causal learning token sequence as forward co-evolution pairs; determining the remaining co-evolution pairs in the causal learning token sequence except for the forward co-evolution pairs as reverse co-evolution pairs; for each forward co-evolution pair, replacing an amino acid residue token in front of the order in the forward co-evolution pair with the preset causal mask token to generate a mask co-evolution pair corresponding to the forward co-evolution pair; for each reverse co-evolution pair, replacing an amino acid residue token behind the order in the reverse co-evolution pair with the preset causal mask token to generate a mask co-evolution pair corresponding to the reverse co-evolution pair; generating the second mask token sequence based on the mask co-evolution pairs corresponding to each co-evolution pair in the causal learning token sequence.

3. The method of claim 1, wherein, The protein language model includes an embedding representation model and a sequence feature encoding model, and the sequence feature encoding model includes a stacked multi-head attention layer. and The embedding and feature encoding the sample mask token sequence based on the protein language model to obtain an encoded feature sequence comprises: The sample mask token sequence is embedded by the embedding representation model to obtain an input feature sequence; The input feature sequence is input into the sequence feature encoding model to obtain the encoded feature sequence.

4. The method of claim 3, wherein, The input feature sequence is input into the sequence feature encoding model to obtain the encoded feature sequence. The input feature sequence is input into the stacked multi-head attention layer to obtain a query matrix, a key matrix and a value matrix; The key matrix and the query matrix are processed to obtain an original attention score matrix, which is used to represent the original attention between any two tokens in the sample mask token sequence; An attention mask matrix is applied to the original attention score matrix to obtain a masked attention score matrix, wherein the attention mask matrix is used to act on the original attention score matrix to define whether attention calculation is performed between two tokens in the sample mask token sequence, specifically: mutual attention calculation is performed between any two tokens in the first mask token sequence; no attention calculation is performed between any token in the first mask token sequence and any token in the second mask token sequence; the amino acid residue tokens in the mask co-evolution pair in the second mask token sequence perform attention calculation on other positions in the first mask token sequence except the positions corresponding to the amino acid residue tokens in the mask co-evolution pair; the preset mask tokens in the mask co-evolution pair in the second mask token sequence perform attention calculation on the positions in the first mask token sequence except the positions corresponding to the co-evolution pair in the mask co-evolution pair, and perform attention calculation on the position of the amino acid residue token in the mask co-evolution pair in the second mask token sequence itself; The encoded feature sequence is determined based on the masked attention score matrix and the value matrix.

5. A protein sequence processing method, comprising: obtaining a protein sequence to be processed; inputting the protein sequence to be processed into a protein language model to obtain an encoded feature sequence corresponding to the protein sequence to be processed, wherein the protein language model is pre-trained by the method of any one of claims 1-4.

6. A protein language model pre-training device, comprising: a mask module configured to determine the positions of each co-evolution pair in a sample protein sequence as mask positions; determine the number of mask positions according to a preset mask proportion and the length of the sample protein sequence; subtract the difference between the number of special mask positions and the number of mask positions from the number of mask positions to determine the number of ordinary mask positions, wherein the number of special mask positions is the number of positions of each co-evolution pair in the sample protein sequence; select the number of ordinary mask positions from the positions in the sample protein sequence that are not confirmed as mask positions to determine the mask positions, wherein the number of ordinary mask positions includes the positions of each co-evolution pair in the sample protein sequence. According to the number of common mask positions, the sample protein sequence is masked to obtain a first mask label sequence; Based on each co-evolution pair in the sample protein sequence, a causal learning label sequence is generated, and the causal learning label sequence includes each co-evolution pair; The causal learning label sequence is masked to obtain a second mask label sequence, wherein the second mask label sequence includes at least one mask co-evolution pair corresponding to the co-evolution pair in the sample protein sequence, and the mask co-evolution pair includes a preset causal mask label and an amino acid residue label in the corresponding co-evolution pair; The first mask label sequence and the second mask label sequence are spliced to obtain a sample mask label sequence; An embedding and encoding module is configured to perform embedding representation and feature encoding on the sample mask label sequence based on a protein language model to obtain an encoded feature sequence; A decoding module is configured to decode the encoded feature sequence based on a preset sequence feature decoder to obtain a decoded protein sequence; An optimization module is configured to optimize the protein language model based on the difference between the decoded protein sequence and the sample protein sequence and each co-evolution pair.

7. A protein sequence processing device, comprising: An acquisition module configured to acquire a protein sequence to be processed; A processing module configured to input the protein sequence to be processed into a protein language model to obtain an encoded feature sequence corresponding to the protein sequence to be processed, wherein the protein language model is obtained by pre-training according to any one of the methods of claims 1-4.

8. An electronic device, comprising: One or more processors; A storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-4 and / or the method of claim 5.

9. A computer readable storage medium having stored thereon a computer program, wherein, The computer program is executed by one or more processors to implement the method of any one of claims 1-4 and / or the method of claim 5.

10. A computer program product comprising computer programs / instructions that, when executed by a processor, implement the method of any one of claims 1-4 and / or the method of claim 5.

Images