Protein language model pre-training and protein sequence processing method and related products

Abstract

The invention provides a protein language model pre-training and protein sequence processing method and related products. According to one specific embodiment of the protein language model pre-training method, a sample protein sequence and a co-evolution pair in the sample protein sequence are subjected to mask processing, and a sample mask marking sequence is obtained; based on a protein language model, performing embedding representation and feature coding on the sample mask marking sequence to obtain a coding feature sequence; based on a preset sequence feature decoder, decoding the coding feature sequence to obtain a decoded protein sequence; a protein language model is optimized based on differences between the decoded protein sequence and the sample protein sequence and each co-evolution pair. Namely, by masking the co-evolution pairs in the protein sequence in the pre-training process, the understanding and prediction ability of the protein language model to the protein sequence can be optimized, and the accuracy of downstream protein sequence / structure related tasks based on the protein language model can be improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of bioinformatics analysis technology, and particularly to protein language model pre-training and protein sequence processing methods and related products. Background Art

[0002] As a frontier field of computational biology, protein structure prediction has undergone a paradigm shift in recent years driven by deep learning technology. Currently, there are mainly two methods for protein structure prediction: 1. Traditional co-evolution method (represented by AlphaFold2): This method constructs a Markov random field model based on multiple sequence alignment (MSA), and infers the contact map through the evolutionary correlation between residues. Although this method achieves atomic-level accuracy in the prediction of proteins with rich homologous sequences, its performance significantly degrades in single-sequence prediction. In addition, the computational complexity of this method grows exponentially.

[0003] 2. Large language model method (represented by ESM): This method is typically represented by the ESM series of models (ESM-1b / ESM2 / ESMFold), and uses the Transformer architecture to achieve an end-to-end mapping from sequence to structure. By learning the biological evolution knowledge in the protein sequence through a protein language model (PLM) to replace the MSA module in the AlphaFold2 framework, although a certain improvement in speed is obtained, the structure prediction model based solely on PLM still cannot surpass AlphaFold2. This indicates that there is still room for improvement in the current PLM in capturing and modeling protein evolution information. Summary of the Invention

[0004] Embodiments of the present disclosure propose a protein language model pre-training and protein sequence processing method, apparatus, electronic device, storage medium, and computer program product.

[0005] In a first aspect, embodiments of the present disclosure provide a protein language model pre-training method, the method comprising: Performing a masking process on a sample protein sequence and co-evolution pairs in the sample protein sequence to obtain a sample masked token sequence; Based on a protein language model, performing an embedding representation and feature encoding on the sample masked token sequence to obtain an encoded feature sequence; Based on a preset sequence feature decoder, decoding the encoded feature sequence to obtain a decoded protein sequence; Based on the differences between the decoded protein sequence and the above sample protein sequence and each of the above co-evolution pairs, the above protein language model is optimized.

[0006] In some alternative embodiments, masking the sample protein sequence and co-evolution pairs in the sample protein sequence to obtain a sample masked token sequence includes: Based on the co-evolution pair position information of the above sample protein sequence, determining at least one masking position for masking the above sample protein sequence; According to the above at least one masking position, masking the above sample protein sequence to obtain a first masked token sequence; Based on each co-evolution pair in the above sample protein sequence, generating a causal learning token sequence, where the causal learning token sequence includes each of the above co-evolution pairs; Masking the above causal learning token sequence to obtain a second masked token sequence, where the second masked token sequence includes at least one masked co-evolution pair corresponding to the co-evolution pair in the above sample protein sequence, and the masked co-evolution pair includes a preset causal mask token and one amino acid residue token in the corresponding co-evolution pair; Concatenating the above first masked token sequence and the above second masked token sequence to obtain the above sample masked token sequence.

[0007] In some alternative embodiments, the above determining at least one masking position for masking the above sample protein sequence based on the co-evolution pair position information of the above sample protein sequence includes: Determining the positions where each co-evolution pair in the above sample protein sequence is located as masking positions; According to a preset masking ratio and the length of the above sample protein sequence, determining the number of positions to be masked; Determining the difference between the number of positions to be masked minus the number of special masking positions as the number of ordinary masking positions, where the number of special masking positions is the number of positions where each co-evolution pair in the above sample protein sequence is located; Selecting the number of ordinary masking positions from the positions in the above sample protein sequence that are not confirmed as masking positions and determining them as masking positions.

[0008] In some alternative embodiments, the above masking the above causal learning token sequence to obtain a second masked token sequence includes: Determining co-evolution pairs with a preset forward masking ratio in each co-evolution pair in the above causal learning token sequence as forward co-evolution pairs; Determining the remaining co-evolution pairs in each co-evolution pair in the above causal learning token sequence except each of the above forward masked co-evolution pairs as reverse co-evolution pairs; For each of the above positive co-evolution pairs, replace the amino acid residue label with a higher rank in the positive co-evolution pair with the above preset causal mask label to generate a masked co-evolution pair corresponding to the positive co-evolution pair; For each of the above negative co-evolution pairs, replace the amino acid residue label with a lower rank in the negative co-evolution pair with the above preset causal mask label to generate a masked co-evolution pair corresponding to the negative co-evolution pair; Based on the masked co-evolution pairs corresponding to the co-evolution pairs in the above causal learning label sequence, generate the above second masked label sequence.

[0009] In some alternative embodiments, the above protein language model includes an embedding representation model and a sequence feature encoding model, and the above sequence feature encoding model includes a stacked multi-head attention layer; and Based on the above protein language model, perform embedding representation and feature encoding on the above sample masked label sequence to obtain an encoded feature sequence, including: Perform embedding representation on the above sample masked label sequence through the above embedding representation model to obtain an input feature sequence; Input the above input feature sequence into the above sequence feature encoding model to obtain the above encoded feature sequence.

[0010] In some alternative embodiments, the above inputting the above input feature sequence into the above sequence feature encoding model to obtain the above encoded feature sequence includes: Input the above input feature sequence into the above stacked multi-head attention layer to obtain a query matrix, a key matrix, and a value matrix; Process the above key matrix and the above query matrix to obtain an original attention score matrix, and the above original attention score matrix is used to represent the original attention between any two labels in the above sample masked label sequence; Apply the attention mask matrix to the above-mentioned original attention score matrix to obtain a masked attention score matrix. The attention mask matrix is used to act on the original attention score matrix to define whether attention calculation is performed between two tokens in the sample mask token sequence. Specifically: mutual attention calculation is performed between any two tokens in the first mask token sequence; no attention calculation is performed between any token in the first mask token sequence and any token in the second mask token sequence; the amino acid residue markers in the co-evolutionary pair in the second mask token sequence perform attention calculation on other positions in the first mask token sequence except for the positions corresponding to the amino acid residue markers in the co-evolutionary pair, and perform attention calculation on the position where the amino acid residue marker in the co-evolutionary pair in the second mask token sequence itself is located; the preset mask marker in the co-evolutionary pair in the second mask token sequence performs attention calculation on other positions in the first mask token sequence except for the positions corresponding to the co-evolutionary pairs corresponding to the co-evolutionary pair, and performs attention calculation on the position where the co-evolutionary pair in the second mask token sequence is located. Determine the above-mentioned encoded feature sequence based on the above-mentioned masked attention score matrix and the above-mentioned value matrix.

[0011] In a second aspect, an embodiment of the present disclosure provides a protein sequence processing method, which includes: Obtain a protein sequence to be processed; Input the above-mentioned protein sequence to be processed into a protein language model to obtain an encoded feature sequence corresponding to the above-mentioned protein sequence to be processed, where the above-mentioned protein language model is pre-trained in advance by the method described in any implementation manner of the first aspect.

[0012] In a third aspect, an embodiment of the present disclosure provides a protein language model pre-training device, which includes: A masking module configured to perform masking processing on a sample protein sequence and co-evolutionary pairs in the above-mentioned sample protein sequence to obtain a sample mask token sequence; An embedding representation and encoding module configured to perform embedding representation and feature encoding on the above-mentioned sample mask token sequence based on a protein language model to obtain an encoded feature sequence; A decoding module configured to decode the above-mentioned encoded feature sequence based on a preset sequence feature decoder to obtain a decoded protein sequence; An optimization module configured to optimize the above-mentioned protein language model based on the differences between the above-mentioned decoded protein sequence and the above-mentioned sample protein sequence and each of the above-mentioned co-evolutionary pairs.

[0013] In some alternative implementation manners, the above-mentioned masking module includes: A mask position determination unit, configured to determine at least one mask position for masking the sample protein sequence based on the co-evolution pair position information of the above sample protein sequence; A first masking unit, configured to perform masking on the sample protein sequence according to the at least one mask position to obtain a first masked tag sequence; A causal learning sequence generation unit, configured to generate a causal learning tag sequence based on each co-evolution pair in the sample protein sequence, where the causal learning tag sequence includes each of the above co-evolution pairs; A second masking unit, configured to perform masking on the causal learning tag sequence to obtain a second masked tag sequence, where the second masked tag sequence includes at least one masked co-evolution pair corresponding to the co-evolution pair in the sample protein sequence, and the masked co-evolution pair includes a preset causal mask tag and an amino acid residue tag in the corresponding co-evolution pair; A splicing unit, configured to splice the first masked tag sequence and the second masked tag sequence to obtain the sample masked tag sequence.

[0014] In some alternative embodiments, the mask position determination unit is further configured to: Determine the positions where each co-evolution pair in the sample protein sequence is located as mask positions; Determine the number of positions to be masked according to a preset mask ratio and the length of the sample protein sequence; Determine the difference between the number of positions to be masked minus the number of special mask positions as the number of ordinary mask positions, where the number of special mask positions is the number of positions where each co-evolution pair in the sample protein sequence is located; Among the positions in the sample protein sequence that are not confirmed as mask positions, select the number of ordinary mask positions and determine them as mask positions.

[0015] In some alternative embodiments, the second masking unit is further configured to: Determine the co-evolution pairs with a preset forward mask ratio among each co-evolution pair in the causal learning tag sequence as forward co-evolution pairs; Determine the remaining co-evolution pairs among each co-evolution pair in the causal learning tag sequence except each of the above forward co-evolution pairs as reverse co-evolution pairs; For each of the above forward co-evolution pairs, replace the amino acid residue tag with a higher ranking in the forward co-evolution pair with the preset causal mask tag to generate a masked co-evolution pair corresponding to the forward co-evolution pair; For each of the above reverse co-evolution pairs, replace the amino acid residue label with a lower ranking in the reverse co-evolution pair with the above preset causal mask label to generate a masked co-evolution pair corresponding to the reverse co-evolution pair; Generate the above second masked label sequence based on the masked co-evolution pairs corresponding to the co-evolution pairs in the above causal learning label sequence.

[0016] In some alternative embodiments, the above protein language model includes an embedding representation model and a sequence feature encoding model, and the above sequence feature encoding model includes a stacked multi-head attention layer; and The above embedding representation and encoding module includes: An embedding representation unit configured to perform an embedding representation on the above sample masked label sequence through the above embedding representation model to obtain an input feature sequence; An encoding unit configured to input the above input feature sequence into the above sequence feature encoding model to obtain the above encoded feature sequence.

[0017] In some alternative embodiments, the above encoding unit is further configured to: Input the above input feature sequence into the above stacked multi-head attention layer to obtain a query matrix, a key matrix, and a value matrix; Process the above key matrix and the above query matrix to obtain an original attention score matrix, and the above original attention score matrix is used to characterize the original attention between any two labels in the above sample masked label sequence; Apply an attention mask matrix to the above original attention score matrix to obtain a masked attention score matrix, where the above attention mask matrix is used to act on the above original attention score matrix to define whether attention calculation is performed between two labels in the above sample masked label sequence. Specifically: mutual attention calculation is performed between any two labels in the above first masked label sequence; no attention calculation is performed between any label in the above first masked label sequence and any label in the above second masked label sequence; the amino acid residue labels in the masked co-evolution pairs in the above second masked label sequence perform attention calculation on other positions in the above first masked label sequence except for the positions corresponding to the amino acid residue labels in the masked co-evolution pairs, and perform attention calculation on the positions where the amino acid residue labels in the masked co-evolution pairs in the above second masked label sequence are located; the preset mask labels in the masked co-evolution pairs in the above second masked label sequence perform attention calculation on other positions in the above first masked label sequence except for the positions corresponding to the co-evolution pairs corresponding to the masked co-evolution pairs, and perform attention calculation on the positions where the masked co-evolution pairs in the above second masked label sequence are located; Determine the above encoded feature sequence based on the above masked attention score matrix and the above value matrix.

[0018] Fourthly, embodiments of the present disclosure provide a protein sequence processing device, which includes: An acquisition module, configured to acquire a protein sequence to be processed; A processing module, configured to input the protein sequence to be processed into a protein language model to obtain an encoded feature sequence corresponding to the protein sequence to be processed, where the protein language model is pre-trained by the method described in any implementation manner of the first aspect.

[0019] Fifthly, embodiments of the present disclosure provide an electronic device, including: one or more processors; a storage device storing one or more programs thereon, and when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the method described in any implementation manner of the first aspect and / or the second aspect.

[0020] Sixthly, embodiments of the present disclosure provide a computer-readable storage medium storing a computer program thereon, where the computer program, when executed by one or more processors, implements the method described in any implementation manner of the first aspect and / or the second aspect.

[0021] Seventhly, embodiments of the present disclosure provide a computer program product including computer programs / instructions, where the computer programs / instructions, when executed by a processor, implement the method described in any implementation manner of the first aspect and / or the second aspect.

[0022] To solve the problem that the existing protein language model cannot balance computational efficiency and prediction accuracy when processing protein sequences, the protein language model pre-training and protein sequence processing method, device, electronic device, storage medium, and computer program product provided by the embodiments of the present disclosure perform mask processing on the sample protein sequence and the co-evolution pairs in the sample protein sequence to obtain a sample mask token sequence; then, based on the protein language model, perform embedding representation and feature encoding on the sample mask token sequence to obtain an encoded feature sequence; then, based on a preset sequence feature decoder, decode the encoded feature sequence to obtain a decoded protein sequence; finally, optimize the protein language model based on the differences between the decoded protein sequence and the sample protein sequence and each co-evolution pair. This method can achieve technical effects including but not limited to: First, due to the high correlation of co-evolution pairs at the level of biomolecular structure and function, traditional protein language models are relatively easy to understand the conserved regions (Conservation) and variable regions (Variation) in protein sequences. Among them, local properties can be learned through masked language models to understand conserved regions. And variable regions can be learned through various different sequences in the training data. However, co-evolution regions are difficult to learn because there are often long-range dependencies between two amino acid residues in co-evolution regions. Therefore, protein language models need to learn long-range dependencies in sequences, which is difficult for traditional protein language models. In the above embodiments of the present disclosure, by masking co-evolution pairs in protein sequences, protein language models can learn the co-evolution pair structure information, or causal information, in protein sequences, that is, because of the change of one amino acid residue in the co-evolution pair, the corresponding change of the other amino acid residue in the co-evolution pair results, and finally optimize the understanding and prediction ability of protein language models for protein sequences.

[0023] Second, since multiple sequence alignment is not used, the computational complexity is relatively low compared to multiple sequence alignment.

[0024] Third, based on the protein language model pre-trained by masking co-evolution pairs in protein sequences, performing downstream protein sequence analysis tasks, such as sequence-related or structure-related tasks, can improve the accuracy of downstream protein sequence analysis tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Other features, objects, and advantages of the present disclosure will become more apparent by reading the detailed description of non-limiting embodiments with reference to the following drawings. The drawings are only for the purpose of illustrating the specific embodiments and are not considered as a limitation of the present invention. In the drawings: Figure 1 is an exemplary system architecture diagram to which an embodiment of the present disclosure can be applied; Figure 2A is a flowchart of an embodiment of a method for pre-training a protein language model according to the present disclosure; Figure 2B is a decomposed flowchart of an embodiment of step 201 according to the present disclosure; Figure 2C is a decomposed flowchart of an embodiment of step 2011 according to the present disclosure; Figure 2D is a decomposed flowchart of an embodiment of step 2014 according to the present disclosure; Figure 2E is a decomposed flowchart of an embodiment of step 202 according to the present disclosure; Figure 2F is a decomposition flowchart of an embodiment of step 2022 according to the present disclosure; Figure 3 is a specific example of a sample protein sequence according to the present disclosure, and specific examples of a first masked token sequence, a second masked token sequence, and a sample masked token sequence generated based on the sample protein sequence; Figure 4 is a specific example of a sample masked token sequence and a corresponding attention mask matrix according to the present disclosure; Figure 5 is a flowchart of an embodiment of a protein sequence processing method according to the present disclosure; Figure 6 is a comparison diagram of a specific embodiment of a protein sequence related task according to the present disclosure; Figure 7 is a comparison diagram of a specific embodiment of a protein structure related task according to the present disclosure; Figure 8 Schematic structural diagram of an embodiment of a protein language model pre-training device according to the present disclosure; Figure 9 Schematic structural diagram of an embodiment of a protein sequence processing device according to the present disclosure; Figure 10 Schematic structural diagram of a computer system of an electronic device suitable for implementing an embodiment of the present disclosure. Detailed Description of Specific Embodiments

[0026] The present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only for explaining the related invention and not for limiting the invention. It should be noted that, for the sake of description, only parts related to the relevant invention are shown in the drawings.

[0027] It should be noted that, without conflict, the embodiments in the present disclosure and the features in the embodiments can be combined with each other. The present disclosure will be described in detail below with reference to the drawings and embodiments.

[0028] Figure 1 Exemplary system architecture 100 showing embodiments of a protein language model pre-training and protein sequence processing method, apparatus, electronic device, and storage medium to which the present disclosure can be applied.

[0029] As Figure 1As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0030] Users can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the terminal devices 101, 102, 103, such as protein language model pre-training applications, protein sequence processing applications, etc.

[0031] The terminal devices 101, 102, 103 can be either hardware or software. When the terminal devices 101, 102, 103 are hardware, they can be various electronic devices with information input devices (such as keyboards, mice, touchscreens, microphones, cameras, etc.) and information output devices (such as displays, speakers, etc.), including but not limited to smartphones, tablets, e-book readers, laptop computers, and desktop computers, etc. When the terminal devices 101, 102, 103 are software, they can be installed in the above-listed terminal devices. It can be implemented as multiple software or software modules (such as those used to provide model pre-training services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0032] In some cases, the protein language model pre-training and protein sequence processing methods provided by the present disclosure can be executed by the terminal devices 101, 102, 103. Correspondingly, the protein language model pre-training and protein sequence processing devices can be set in the terminal devices 101, 102, 103. In this case, the system architecture 100 may not include the server 105 either.

[0033] In some cases, the protein language model pre-training and protein sequence processing methods provided by the present disclosure can be jointly executed by the terminal devices 101, 102, 103 and the server 105. For example, the step of "masking the sample protein sequence and the co-evolutionary pairs in the sample protein sequence to obtain a sample masked token sequence" can be executed by the terminal devices 101, 102, 103, and steps such as "performing embedding representation and feature encoding on the sample masked token sequence based on the protein language model to obtain an encoded feature sequence" can be executed by the server 105. The present disclosure does not make a limitation in this regard. Correspondingly, the protein language model pre-training device and the protein sequence processing device can also be respectively set in the terminal devices 101, 102, 103 and the server 105.

[0034] In some cases, the protein language model pre-training and protein sequence processing methods provided by the present disclosure can be executed by the server 105. Correspondingly, the protein language model pre-training and protein sequence processing device can also be set in the server 105. At this time, the system architecture 100 may not include the terminal devices 101, 102, and 103 either.

[0035] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers or as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (e.g., for providing distributed services) or as a single software or software module. No specific limitation is made here.

[0036] It should be understood that Figure 1 the numbers of terminal devices, networks, and servers in

[0037] Continue to refer to Figure 2A , which shows a flow 200 of an embodiment of the protein language model pre-training method according to the present disclosure. The protein language model pre-training method includes the following steps: Step 201, perform masking processing on the sample protein sequence and the co-evolutionary pairs in the sample protein sequence to obtain a sample masked token sequence.

[0038] Here, the sample protein sequence can be obtained by including but not limited to the following methods: I. Direct sequencing method: For example, Edman degradation method, mass spectrometry method, etc.

[0039] II. Gene sequence derivation method: For example, first extract DNA or RNA from a biological sample, amplify the target gene fragment through techniques such as PCR, then perform gene sequencing to obtain the nucleotide sequence of the gene, and then translate the nucleotide sequence of the gene into the amino acid sequence of the protein according to the genetic codon table.

[0040] III. Database retrieval method: If it is already known that a certain protein has similarity with a known protein in the database, its sequence information can be obtained through database retrieval. Input part of the sequence or characteristic information of the protein to be queried into a protein database, such as Swiss - Prot, PDB, etc., and find the matching known protein sequence through a sequence alignment algorithm.

[0041] The sample protein sequence refers to the sequence directly obtained through the above various channels and arranged in order by amino acid residues marked as tokens (Tokens).

[0042] Co-evolution pairs in a sample protein sequence can be determined by methods including but not limited to the following: I. Extracting co-evolution pairs based on MSA data: 1. Data preparation: First, high-quality MSA data needs to be obtained. Protein sequence data can be downloaded from public databases such as UniProt, PDB, etc., and the sequences can be aligned using multiple sequence alignment tools (such as ClustalW, MAFFT, etc.) to obtain the MSA results.

[0043] 2. Selecting sites: Since not all sites in a protein sequence are relevant to co-evolution analysis, sites with biological significance need to be selected for analysis. Usually, conserved sites, functional sites, or sites located in key regions of the protein structure can be selected. The selection of these sites can be based on existing literature reports, protein structure information, or functional annotations, etc.

[0044] 3. Calculating the correlation between sites: Commonly used methods include those based on mutual information (MI). Mutual information can measure the degree of information transfer between two sites, and the higher the value, the stronger the correlation between the two sites. Specifically, when calculating, for each pair of sites in the MSA, the frequencies of amino acid residues that appear in different sequences are counted, and then calculated according to the formula for mutual information. In addition to mutual information, other methods such as direct coupling analysis (DCA), covariation analysis, etc. can also be used to calculate the correlation between sites.

[0045] 4. Setting a threshold: Determine a suitable threshold to judge which site pairs are co-evolution pairs. The setting of the threshold usually needs to be adjusted according to the specific dataset and analysis purpose. Generally, the optimal threshold can be determined by comparing the number and quality of co-evolution pairs obtained under different thresholds and combining biological knowledge. For example, the thresholds used in published related studies can be referred to, or a threshold that can distinguish true co-evolution pairs from random noise can be determined through the analysis of randomly shuffled sequences.

[0046] 5. Result screening and verification: According to the set threshold, site pairs with a correlation higher than the threshold are selected as co-evolution pairs. To ensure the reliability of the results, the obtained co-evolution pairs can be further verified. For example, it can be checked whether the co-evolution pairs are close to each other in the three-dimensional structure of the protein, or whether they are consistent with known protein functional relationships. In addition, they can also be compared with other experimental data (such as protein-protein interaction data, mutation experiment data, etc.) to verify the biological significance of the co-evolution pairs.

[0047] II. Extracting co-evolution pairs based on structural data: 1. Calculate distance: Based on the coordinate data in the structural data (e.g., AFDB (AlphaFold Database)), analyze the protein sequence and calculate the Cβ atom distance between any two amino acid residues. The Cβ atom distance refers to the spatial distance (in Å, angstrom) between the Cβ atoms (β-carbon atoms) of any two amino acid residues in the three-dimensional structure of the protein. The Cβ atom is the first atom of the amino acid side chain (except for glycine, whose side chain has only one H atom and no Cβ). This distance reflects the proximity of two amino acids in three-dimensional space and can be used to analyze the interactions between residues (such as hydrophobic interactions, disulfide bonds, etc.) or structural stability.

[0048] 2. Determine co-evolutionary pairs: Identify amino acid residue pairs with a Cβ atom distance less than a preset distance threshold (e.g., 6 Å) as co-evolutionary pairs.

[0049] The co-evolutionary pairs determined by the above various methods are highly correlated at the levels of biomolecular structure and function.

[0050] As an example, please refer to Figure 3 , Figure 3 which shows an example of the correspondence between the labels of each amino acid residue and its corresponding position in a sample protein sequence. The sample protein sequence is specifically "HAAEAISASQ...". Among them, the Cβ atom distance between "H" at the 1st position and "A" at the 5th position is 4.7 Å, and the Cβ atom distance between "E" at the 4th position and "S" at the 9th position is 5.4 Å. Both of the above Cβ atom distances are less than 6 Å. Therefore, "H" at the 1st position and "A" at the 5th position form a co-evolutionary pair (as shown by the purple label in Figure 3 ), and "E" at the 4th position and "S" at the 9th position form another co-evolutionary pair (as shown by the blue label in Figure 3 ).

[0051] Here, various implementation methods can be adopted to mask the sample protein sequence and the co-evolution pairs in the sample protein sequence to obtain a sample masked token sequence. The sample masked token sequence at least includes a traditional masked token sequence (or Mask Language Model (MLM) sequence) for masking the sample protein sequence and a causal masked token sequence (or Causal Language Model (CLM) sequence) for masking the co-evolution pairs in the sample protein sequence. Here, the traditional masked token sequence and the causal masked token sequence can be independent different parts; or the causal masked token sequence can also be a part of the traditional masked token sequence; or the traditional masked token sequence can also be a part of the causal masked token sequence. The traditional masked token sequence obtained by masking the sample protein sequence can realize the protein language model's understanding and prediction ability of the protein sequence. And the causal masked token sequence obtained by masking the co-evolution pairs in the sample protein sequence can enhance the protein language model's understanding and prediction ability of the co-evolution pair structure information in the protein sequence.

[0052] As an example, the co-evolution pairs in the sample protein sequence can be causally masked first. Among them, for each co-evolution pair, causal masking can be performed by keeping one amino acid residue label in the co-evolution pair unchanged and masking the other amino acid residue label, so as to predict the masked other amino acid residue label through the unmasked amino acid residue label subsequently.

[0053] Then, random masking is performed on other positions in the amino acid residue label sequence obtained after the above causal masking except for the co-evolution pairs. Specifically, the number of amino acid residue labels to be randomly masked can be determined based on the ratio of the total number of amino acid residue labels subjected to causal masking and random masking to the sample protein sequence as a preset masking ratio (for example, 15%).

[0054] Finally, the above-mentioned token sequence after causal masking and random masking can be determined as the sample masked token sequence.

[0055] The sample masked token sequence obtained by masking in the above manner has the same length as the sample protein sequence. The part that causally masks the co-evolution pairs in the sample protein sequence can be regarded as the causal masked token sequence, and the whole sample masked token sequence can be regarded as the traditional masked token sequence, that is, the causal masked token sequence is a part of the traditional masked token sequence.

[0056] In some alternative embodiments, step 201 may include, for example Figure 2BSteps 2011 to 2015 shown below: In step 2011, based on the co-evolutionary pair position information of the sample protein sequence, at least one masking position for masking the sample protein sequence is determined.

[0057] Here, various implementation methods can be adopted to determine at least one masking position for masking the sample protein sequence based on the co-evolutionary pair position information of the sample protein sequence. Among them, at least one masking position determined may include the positions of at least one pair of co-evolutionary pairs in the sample protein sequence. Optionally, according to statistical analysis, co-evolutionary pairs account for about 3% of the total amino acid pairs in proteins, and generally the masking ratio for pre-training a protein language model is much higher than 3%. Therefore, at least one masking position determined may include the positions of all co-evolutionary pairs in the sample protein sequence. For other positions, different specific selection methods can be adopted according to the actual situation.

[0058] Optionally, step 2011 may include the following steps 20111 to 20114 as shown in Figure 2C below: In step 20111, the positions of each co-evolutionary pair in the sample protein sequence are determined as masking positions.

[0059] In step 20112, according to the preset masking ratio and the length of the sample protein sequence, the number of positions to be masked is determined.

[0060] Here, assuming the preset masking ratio is , the length of the sample protein sequence is , is a positive integer. The number of positions to be masked can be calculated using the following formula:

[0061] where is the calculated number of positions to be masked.

[0062] As an example, the preset masking ratio can be 15%.

[0063] The length of the sample protein sequence may refer to the number of tokens included in the sample protein sequence.

[0064] In step 20113, the difference obtained by subtracting the number of special masking positions from the number of positions to be masked is determined as the number of ordinary masking positions.

[0065] Here, following the formula in step 20112, step 20113 can be expressed by the following formula:

[0066] Among them, the number of special mask positions is the number of positions where each co-evolution pair is located in the sample protein sequence.

[0067] is the calculated number of ordinary masks.

[0068] Combining step 20112 and step 20113, the calculation formula for the number of ordinary masks can be expressed by the following formula:

[0069] Step 20114, among the positions in the sample protein sequence that have not been confirmed as mask positions, select the number of ordinary mask positions and determine them as mask positions.

[0070] After step 20111, there are already the number of special mask positions in the sample protein sequence, and a number of markers have been determined as mask positions. In step 20114, it can be in the sample protein sequence, among the positions that have not been confirmed as mask positions in step 20111, select the number of ordinary mask positions and determine them as mask positions.

[0071] As an example, in the positions in the sample protein sequence that have not been confirmed as mask positions, randomly and uniformly sample the number of ordinary mask positions and determine them as mask positions.

[0072] Through the optional implementation manners of the above steps 20111 to 20114, it is possible to determine the positions with a preset mask ratio in the sample protein sequence as mask positions, and all the positions where the co-evolution pairs are located in the sample protein sequence are also determined as mask positions. Furthermore, when performing mask processing on the sample protein sequence subsequently, it is possible to mask all the co-evolution pairs in the sample protein sequence and, in total, mask the markers with the preset mask ratio in the sample protein sequence.

[0073] As an example, please refer to Figure 3 , and determine the positions where the two co-evolution pairs are located in the sample protein sequence "HAAEAISASQ...", that is, the 1st, 4th, 5th, and 9th positions, as mask positions. In Figure 3 , the positions where the first co-evolution pair is located (the 1st position and the 5th position, purple markers) are marked as S1, and the positions where the second co-evolution pair is located (the 4th position and the 9th position, blue markers) are marked as S2. For those in the sample protein that are not co-evolution pairs but are determined as mask positions, they are marked as M (red markers).

[0074] Step 2012, mask the sample protein sequence according to at least one masking position to obtain a first masked token sequence.

[0075] Here, various implementation methods can be adopted to mask each masking position in the sample protein sequence to obtain the first masked token sequence.

[0076] As an example, each masking position in the sample protein sequence can be randomly masked to obtain the first masked token sequence.

[0077] Specifically, the random masking can be performed as follows: For each masking position determined in step 2011, there is an 80% probability of replacing it with a preset masked token " <m>", with a 10% probability, it is replaced by a random token, and with a 10% probability, the original token remains unchanged.

[0078] Here, the random token can be a randomly arbitrary amino acid residue token.

[0079] Here, by adopting the above random masking strategy, it is possible to prevent the protein language model from relying too much on the preset mask tokens <m>”, because no predefined masked tokens appear during the inference of protein language models <m>”. Additionally, the robustness of the protein language model against noise and incorrect inputs can also be enhanced.

[0080] Preferably, using 15% as the preset masking ratio can balance the task difficulty. An overly high masking rate will result in insufficient context information, while an overly low rate will lead to a decrease in training efficiency.

[0081] It can be understood that here, the length of the first masked token sequence can be the same as the length of the sample protein sequence, that is .

[0082] Step 2013: Generate a causal learning token sequence based on each co-evolution pair in the sample protein sequence.

[0083] Here, various implementation methods can be adopted to generate a causal learning token sequence based on each co-evolution pair in the sample protein sequence. The generated causal learning token sequence can include each co-evolution pair, and according to the specified rules, it can be determined which two tokens in the causal learning token sequence form a co-evolution pair.

[0084] For example, first, each co-evolution pair can be sorted according to the position of the first amino acid residue token with a relatively earlier position in each co-evolution pair in the sample protein sequence; then, according to the sorting of each co-evolution pair, each co-evolution pair is concatenated to form a causal learning token sequence. Among them, the order of the two amino acid residue identifiers in each concatenated co-evolution pair is the same as the order in the sample protein sequence.

[0085] As an example, for the two co-evolution pairs in the sample protein sequence such as Figure 3 shown as "HAAEAISASQ……”, sorting according to the position of the first amino acid residue token with a relatively earlier position in each co-evolution pair, the first co-evolution pair (including "H” at the 1st position and "A” at the 5th position) and the second co-evolution pair (including "E” at the 4th position and "S” at the 9th position) can be obtained in the order of precedence. At this time, since the position of the first amino acid residue token with a relatively earlier position in the first co-evolution pair (i.e., the 1st position) is earlier than the position of the first amino acid residue token with a relatively earlier position in the second co-evolution pair (i.e., the 4th position). Then, the first co-evolution pair and the second co-evolution pair are concatenated to obtain the causal learning token sequence "HAES”.

[0086] As another example, corresponding tokens for representing the start and end of the co-evolution pair can also be added on both sides of each co-evolution pair to facilitate distinguishing different co-evolution pairs, and according to the sorting of each co-evolution pair, the co-evolution pairs after adding the start and end tokens are concatenated to form a causal learning token sequence. Specifically, as Figure 3 shown, concatenating the co-evolution pairs "HA” and "ES”, the causal learning token sequence " HA , ES ”, where is used to indicate the start of a co-evolution pair, and is used to indicate the end of the co-evolutionary pair.

[0087] Step 2014, mask the causal learning tag sequence to obtain a second masked tag sequence.

[0088] Here, after masking the causal learning tag sequence, the obtained second masked tag sequence includes at least one masked co-evolutionary pair corresponding to the co-evolutionary pair in the sample protein sequence. Among them, the masked co-evolutionary pair includes a preset causal mask tag and an amino acid residue tag in the corresponding co-evolutionary pair. For example, for the co-evolutionary pair "HA", the corresponding masked co-evolutionary pair can be "H <m>"or" <m>"A". Among them, " <m>” is a preset causal mask token.

[0089] Here, it is assumed that there are a total of co-evolutionary pairs in the sample protein sequence, where is a positive integer. The causal learning token sequence includes the above co-evolutionary pairs. The length of the causal learning token sequence ( is a positive integer) is greater than or equal to . The length of the second mask token sequence can be the same as the length of the causal learning token sequence, that is, the length of the second mask token sequence can also be .

[0090] Here, the second mask token sequence needs to include mask co-evolutionary pairs corresponding to the co-evolutionary pairs in the sample protein sequence. Here, is a positive integer less than or equal to .

[0091] When is equal to , it is equivalent to performing causal masking on each co-evolutionary pair in the causal learning token sequence, that is, masking and replacing one amino acid residue token in the co-evolutionary pair with the preset causal mask token, and keeping the other amino acid residue token unchanged.

[0092] When is less than , it is equivalent to performing causal masking on the co-evolutionary pairs in the causal learning token sequence, and not performing causal masking on the remaining ( - ) co-evolutionary pairs, such as not masking or masking both amino acid residue tokens in the co-evolutionary pair.

[0093] Since the second mask token sequence includes mask co-evolutionary pairs, it can enable the protein language model to have the ability to predict one amino acid residue token from another amino acid residue token in the co-evolutionary pair, that is, enable the protein language model to learn the structural information of the co-evolutionary pair.

[0094] For example, masking the causal learning token sequence "HAES" gives the second mask token sequence "HS1ES2". Another example, as Figure 3 shown, masking the causal learning token sequence " HA , ES " gives the second mask token sequence " HS1 , ES2 ". Here, only S1 and S2 are used to distinguish different co-evolutionary pair masked positions, and in practice, the mask tokens at this position can be various mask tokens. For example, it can be a preset causal mask token" <m>”.

[0095] Optionally, step 2014 may include the following steps 20141 to 20145 as Figure 2D shown: Step 20141: Determine the co-evolution pairs with a preset positive mask ratio in each co-evolution pair in the causal learning label sequence as positive co-evolution pairs.

[0096] Here, various implementation manners may be adopted to determine the co-evolution pairs with a preset positive mask ratio in each co-evolution pair in the causal learning label sequence as positive co-evolution pairs. For example, a method of random uniform sampling may be adopted.

[0097] Step 20142: Determine the remaining co-evolution pairs in each co-evolution pair in the causal learning label sequence except the positive co-evolution pairs as negative co-evolution pairs.

[0098] Step 20143: For each positive co-evolution pair, replace the amino acid residue label with a higher rank in the positive co-evolution pair with a preset causal mask label to generate a masked co-evolution pair corresponding to the positive co-evolution pair.

[0099] For example, for the positive co-evolution pair "HA", the masked co-evolution pair corresponding to the positive co-evolution pair is " <m>"A". Among them, " <m>” is a preset causal mask token.

[0100] Step 20144, for each reverse co-evolution pair, replace the amino acid residue label with a lower ranking in the reverse co-evolution pair with a preset causal mask token to generate a masked co-evolution pair corresponding to the reverse co-evolution pair.

[0101] For example, for the reverse co-evolution pair "HA", the masked co-evolution pair corresponding to this reverse co-evolution pair is "H <m>”.

[0102] After masking the amino acid residue markers ranked ahead in the co - evolution of the preset positive masking ratio in the causal learning marker sequence with steps 20141 to 20144, the amino acid residue markers ranked behind are not masked. For the remaining co - evolution pairs, the amino acid residue markers ranked behind are masked while the amino acid residue markers ranked ahead are not masked.

[0103] Optionally, the preset positive masking ratio here can be 50%. That is, in half of the co - evolution pairs in the causal learning sequence, the amino acid residue markers ranked ahead are masked and the amino acid residue markers ranked behind are not masked. For the other half of the co - evolution pairs, the amino acid residue markers ranked behind are masked and the amino acid residue markers ranked ahead are not masked. In this way, the protein language model can evenly learn to predict one amino acid residue marker based on any one amino acid residue marker in the co - evolution pair.

[0104] Step 20145, generate a second masked marker sequence based on the masked co - evolution pairs corresponding to each co - evolution pair in the causal learning marker sequence.

[0105] Here, each co - evolution pair in the causal learning marker sequence can be replaced with the corresponding masked co - evolution pair to obtain the second masked marker sequence.

[0106] For example, for the causal learning marker sequence "HAES", the masked co - evolution pair corresponding to the co - evolution pair "HA" is "HS1", and the masked co - evolution pair corresponding to the co - evolution pair "ES" is "ES2". Then, "HA" and "ES" in the causal learning marker sequence "HA ES" can be replaced with "HS1" and "ES2" respectively, so as to obtain the second masked marker sequence "HS1ES2". Another example is as Figure 3 shown, for the causal learning marker sequence " HA , ES ", the masked co - evolution pair corresponding to the co - evolution pair "HA" is "HS1", and the masked co - evolution pair corresponding to the co - evolution pair "ES" is "ES2". Then, "HA" and "ES" in the causal learning marker sequence " HA , ES " can be replaced with "HS1" and "ES2" respectively, so as to obtain the second masked marker sequence " HS1 , ES2 ".

[0107] Step 2015, splice the first masked marker sequence and the second masked marker sequence to obtain a sample masked marker sequence.

[0108] For example, as Figure 3 As shown, splice the first masked token sequence "S1MMS2S1ISAS2M…" and the second masked token sequence " HS1 , ES2 ” to obtain the sample masked token sequence "S1MMS2S1ISAS2M… HS1 , ES2 ”.

[0109] In the sample masked token sequence spliced according to the alternative embodiments of steps 2011 to 2015 above, it includes the first masked token sequence that masks all co-evolutionary pairs in the sample protein sequence and the second masked token sequence that masks one amino acid residue identifier in the co-evolutionary pairs of the causal learning token sequence.

[0110] Step 202, based on the protein language model, perform embedding representation and feature encoding on the sample masked token sequence to obtain an encoded feature sequence.

[0111] Here, the protein language model can be various language models for performing embedding representation and encoding on protein sequences.

[0112] Input the sample masked token sequence into the protein language model, which can perform embedding representation and feature encoding on the sample masked token sequence and obtain an encoded feature sequence.

[0113] Specifically, the protein language model can include an embedding representation model and a sequence feature encoding model.

[0114] Here, the embedding representation model can be various models that are currently known or will be developed in the future to convert tokens into vectors or tensors. As an example, the embedding representation model can include, but is not limited to, the following models: One-Hot Encoding, Word2Vec, GloVe (Global Vectors for Word Representation), FastText, etc.

[0115] Optionally, step 202 may include the following steps 2021 and 2022 as Figure 2E shown: Step 2021, perform embedding representation on the sample masked token sequence through the embedding representation model to obtain an input feature sequence.

[0116] Specifically, the sample masked token sequence can be input into the embedding representation model to obtain an input feature sequence. The input features in the input feature sequence correspond one-to-one and are in the same order as the tokens in the sample masked token sequence, and the input features in the input feature sequence are the embedding representations of the tokens in the sample masked token sequence.

[0117] Here, when inputting the sample mask token sequence into the embedding representation model, the position information corresponding to each token in the sample mask token sequence in the sample protein sequence can also be input synchronously. In this way, the embedding representation model can perform token embedding representation (i.e., Token embedding) for each token in the sample mask token sequence based on the token itself, and perform position embedding representation (i.e., position embedding) based on the position information corresponding to the token in the sample protein sequence. Finally, the token embedding representation and the position embedding representation of each token are combined to obtain the embedding representation of the corresponding token.

[0118] As an example, when obtaining the sample mask token sequence by adopting steps 20111 to 20114 as shown in Figure 2C For example, it can be referred to Figure 3 , the sample mask token sequence includes the first mask token sequence "S1MMS2S1ISAS2M…” and the second mask token sequence " H S1 , ES2 ”.

[0119] When inputting the sample mask token sequence into the embedding representation model, the position information "1,2,3,4,5,6,7,8,9,10…” of each token in the first mask token sequence and the position information "1,5,4, 9” of each token in the second mask token sequence can be input synchronously, so that the embedding representation model can perform position embedding representation on the position information of each token. It should be noted that here, the position information of each token in the second mask token sequence is the position corresponding to the token in the sample protein sequence.

[0120] Step 2022, input the input feature sequence into the sequence feature encoding model to obtain the encoded feature sequence.

[0121] Here, various models suitable for encoding the feature vector sequence can be adopted as the sequence feature encoding model. For example, the encoder in Transformer can be adopted, etc.

[0122] Optionally, the encoded feature sequence can be the same length as the sample mask token sequence, that is, the embedding representation of each token in the sample mask token sequence is encoded to obtain the corresponding encoded feature.

[0123] Specifically, it can be referred to Figure 3 , after inputting the sample mask token sequence "S1MMS2S1ISAS2M…” + " H S1 , ES2 ” into the protein language model CoE-PLM for embedding representation and feature encoding, the sample mask token sequence "S1MMS2S1ISAS2M…” + " HS1 , ES2 The encoded feature sequence obtained by arranging the encoded features corresponding to each token in in order. For example, the encoded feature can be a feature vector or a feature tensor.

[0124] Optionally, the sequence feature encoding model may include a stacked multi-head attention layer, and step 2022 may include the following steps 20221 to 20224 as shown in Figure 2F : Step 20221: Input the input feature sequence into the stacked multi-head attention layer to obtain a query matrix, a key matrix, and a value matrix.

[0125] Here, the query matrix, the key matrix, and the value matrix are represented by Q, K, and V respectively.

[0126] Step 20222: Process the key matrix and the query matrix to obtain an original attention score matrix.

[0127] Here, the original attention score matrix is used to represent the original attention between any two tokens in the sample mask token sequence.

[0128] Specifically, the original attention score matrix can be calculated according to the following formula.

[0129]

[0130] Among them, and are both matrices, where is the length of the sample mask token sequence, , is the length of the first mask token sequence, is the length of the second mask token sequence, is also the length of the input feature sequence, is the dimension of each feature in the input feature sequence. is matrix, and then is also matrix, The element value in the i-th row and j-th column of is used to represent the original attention of the i-th token to the j-th token in the sample mask token sequence. Here, both i and j are positive integers. That is, when the model generates the output at the i-th target position, the original attention degree (weight) of the model to the input at the j-th source position.

[0131] Step 20223: Apply the attention mask matrix to the original attention score matrix to obtain a masked attention score matrix.

[0132] Here, if a bidirectional attention mechanism is adopted, that is, processing is performed between the original attention score matrix as the final attention score matrix and the value matrix to obtain the encoded feature sequence, then in the subsequent step 203, based on a preset sequence feature decoder, when decoding the encoded feature sequence to obtain the decoded protein sequence, during the process of predicting the i-th token in the sample mask token sequence, it is equivalent to needing to pay attention to all tokens in the sample mask token sequence, that is, using all context information for prediction. However, this will lead to the fact that in the subsequent process of optimizing the protein language model in step 204, when calculating the loss function, the probabilities between two positions forming a co-evolution pair or a masked co-evolution pair in the sample mask token sequence are independent. In fact, there is a causal relationship between the above two positions, and it is the change of one position that causes the change of the other position. Therefore, it is necessary to determine the attention mask matrix according to the generation method of the sample token sequence, and then apply the attention mask matrix to the original attention score matrix to obtain the masked attention score matrix. The purpose is to define whether attention calculation is performed between two tokens in the sample mask token sequence through the attention mask matrix, so as to establish a causal relationship between two positions forming a co-evolution pair or a masked co-evolution pair in the subsequent process of calculating the loss function in step 204, that is, the probabilities of their synchronous changes are related.

[0133] Specifically, assuming the attention mask matrix is , the masked attention score matrix can be calculated through the following formula:

[0134] Here, represents the Hadamard Product operation, or the Element-wise Multiplication operation.

[0135] The attention mask matrix is used to act on the original attention score matrix to define whether attention calculation is performed between two tokens in the sample mask token sequence.

[0136] Specifically, the attention mask matrix can be generated according to the first mask token sequence, the second mask token sequence in the sample mask token sequence, and the amino acid residue tokens and the preset mask token positions in the masked co-evolution pairs in the second mask token sequence, so that the attention mask matrix is used to act on the original attention score matrix to generate the masked attention score matrix The following four objectives can be achieved: 1. Perform mutual attention calculation between any two markers in the first masked marker sequence.

[0137] 2. Do not perform attention calculation between any marker in the first masked marker sequence and any marker in the second masked marker sequence.

[0138] From the above objectives 1 and 2, it can be seen that for each marker in the first masked marker sequence, the number of markers in the sample masked marker sequence that perform attention calculation with this marker is the length of the first masked marker sequence .

[0139] 3. The amino acid residue markers in the co-evolved pair of masks in the second masked marker sequence perform attention calculation on other positions in the first masked marker sequence except for the positions corresponding to the amino acid residue markers in this co-evolved pair of masks, and perform attention calculation on the position where the amino acid residue markers in this co-evolved pair of masks are located in the second masked marker sequence.

[0140] From the above objective 3, it can be seen that for each amino acid residue marker in the second masked marker sequence, the number of markers in the first masked marker sequence that perform attention calculation with this marker is the length of the first masked marker sequence minus 1, that is , and the number of markers in the second masked marker sequence that perform attention calculation with this marker is 1. Furthermore, the total number of markers in the sample masked marker sequence that perform attention calculation with this marker is the length of the first masked marker sequence .

[0141] 4. The preset masked markers in the co-evolved pair of masks in the second masked marker sequence perform attention calculation on other positions in the first masked marker sequence except for the positions corresponding to the co-evolved pairs corresponding to this co-evolved pair of masks, and perform attention calculation on the position where this co-evolved pair of masks is located in the second masked marker sequence.

[0142] From the above objective 4, it can be seen that for each preset masked marker in the second masked marker sequence, the number of markers in the first masked marker sequence that perform attention calculation with this marker is the length of the first masked marker sequence minus 2, that is , and the number of markers in the second masked marker sequence that perform attention calculation with this marker is 2. Furthermore, the total number of markers in the sample masked marker sequence that perform attention calculation with this marker is the length of the first masked marker sequence .

[0143] In summary, it can be seen that for each marker in the sample masked marker sequence, the number of markers that perform attention calculation with this marker is the length of the first masked marker sequence In this way, it can be ensured that when predicting each token in the sample mask token sequence, the preset sequence feature decoder only needs to focus on a fixed number of tokens in the sample mask token sequence, that is, tokens, and the fixed number of tokens includes the token itself.

[0144] The following is an illustration of the above four points with reference to Figure 4 as follows. Figure 4 FIG. shows an example of a sample mask token sequence and a corresponding attention mask matrix according to the present disclosure. As Figure 4 shown, the first mask token sequence is "H <m> <m> R <m> E <m> <m> <m>"K", corresponding to Figure 4 from the 1st row to the 10th row, and from the 1st column to the 10th column in. The second mask token sequence is "T" <m> R <m> E <m>”, corresponding to Figure 4 lines 11 to 16, and columns 11 to 16 in. Among them, " <m>” is a preset mask token. There are three co-evolution pairs in total in the first mask token sequence and the second mask token sequence in the left-to-right order, which are respectively: The one at the second position in the first mask token sequence <m>and at the 8th position <m>is the first co-evolution pair, corresponding to Figure 4 lines 2 and 8, and columns 2 and 8 in Figure 4 , marked in blue; the first masked co-evolution pair corresponding to this first co-evolution pair in the second masked marker sequence is "T <m>”, corresponding to Figure 4 lines 11 and 12, and columns 11 and 12 in, marked in blue; the one in the 3rd position in the first mask marking sequence <m>and the <m>is the second co-evolution pair, corresponding to Figure 4 lines 3 and 7, and columns 3 and 7 in <m>”, corresponding to Figure 4 lines 13 and 14, and columns 13 and 14 in, marked in purple; the one in the 5th position in the first mask marking sequence <m>and the <m>is the third co-evolution pair, corresponding to Figure 4 lines 5 and 9, and columns 5 and 9 in Figure 4 , marked in orange; the third masked co-evolution pair corresponding to this third co-evolution pair in the second masked marker sequence is "E <m>”, corresponding to Figure 4 lines 15 and 16, and columns 15 and 16, marked in orange.

[0145] If Figure 4 the grid at the i-th row and j-th column in has " it indicates that the element at the i-th row and j-th column in the attention mask matrix is 0 or a very small value. When performing an element-wise product between the attention mask matrix and the original attention score matrix the product of and the element at the i-th row and j-th column in can obtain the masked attention score matrix the element at the i-th row and j-th column in Since is 0 or a very small value, under the action of it is 0 or a very small value, indicating that under the action of the attention mask matrix the attention of the i-th token to the j-th token in the sample mask token sequence has no relation with That is, the i-th token in the sample mask token sequence does not calculate the attention to the j-th token. Furthermore, in the subsequent step 203, when decoding the encoded feature sequence based on the preset sequence feature decoder to obtain the decoded protein sequence, during the process of predicting the i-th token in the sample mask token sequence by decoding the encoded feature sequence, the preset sequence feature decoder does not pay attention to the j-th token in the sample mask token sequence, or in other words, cannot see the j-th token.

[0146] Conversely, if Figure 4 the grid at the i-th row and j-th column in has no " it indicates that the element at the i-th row and j-th column in the attention mask matrix is 1 or a value very close to 1. When performing an element-wise product between the attention mask matrix and the original attention score matrix the product of and the element at the i-th row and j-th column in can obtain the masked attention score matrix the element at the i-th row and j-th column in Since is 1 or a value very close to 1, will be equal to or very close to while Used to characterize the attention of the i-th token to the j-th token in the sample mask token sequence, so that It can also be used to characterize the attention of the i-th token to the j-th token in the sample mask token sequence, that is, under the action of the attention mask matrix The i-th token in the sample mask token sequence calculates the attention to the j-th token. Furthermore, in the subsequent step 203, when decoding the encoded feature sequence based on the preset sequence feature decoder to obtain the decoded protein sequence, when predicting the i-th token in the sample mask token sequence by decoding the encoded feature sequence, the preset sequence feature decoder pays attention to the j-th token in the sample mask token sequence, or it can be said that it can see the j-th token, and the degree of attention of the preset sequence feature decoder to the j-th token in the sample mask token sequence is .

[0147] The following specifically explains the above four purposes achieved by the attention mask matrix Figure 4 with reference to : 1. Mutual attention calculation is performed between any two tokens in the first mask token sequence.

[0148] 2. Any token in the first mask token sequence does not calculate the attention to any token in the second mask token sequence.

[0149] Regarding the above two purposes, specifically for Figure 4 , it can be seen that for the 1st to 10th rows (corresponding to the first mask token sequence), among them, there is no " " in the squares from the 1st column to the 10th column (corresponding to the first mask token sequence), that is, mutual attention calculation is performed between any two tokens in the first mask token sequence; while there is " " in the squares from the 11th column to the 16th column (corresponding to the second mask token sequence), that is, any token in the first mask token sequence does not calculate the attention to any token in the second mask token sequence.

[0150] That is, the encoding features corresponding to the 1st to 10th rows are subsequently decoded by the preset sequence feature decoder, that is, when generating the decoded protein sequence corresponding to the first mask tag sequence, any tag in the first mask tag sequence can see all tags in the first mask tag sequence, that is, when predicting the tag of a certain position in the first mask tag sequence, all positions in the first mask tag sequence (including itself, forward and backward) can be paid attention to at the same time, that is, through the bidirectional self-attention mechanism, the pre-trained protein language model can more comprehensively understand the contextual relationship of each amino acid residue tag in the protein sequence (for example, understanding long-distance dependence), and improve the protein language model's ability to capture the global and local information of the protein sequence. When predicting the tag of a certain position in the first mask tag sequence, there is no need to pay attention to any tag in the second mask tag sequence, because the second mask tag sequence does not carry the relevant contextual information in the first mask tag sequence, and therefore, there is no need to pay attention to any tag in the second mask tag sequence. When predicting the mark of a certain position in the first mask mark sequence, the first mask mark sequence can already pay attention to all the context information and also carry the initial causal information (i.e., co-evolution pair information), so there is no need for the second mask mark sequence to provide causal information. Therefore, there is no need to pay attention to any mark in the second mask mark sequence.

[0151] That is, for each token in the first mask token sequence, the number of tokens in the sample mask token sequence that perform attention calculations with the token is the length of the first mask token sequence .

[0152] 3. The amino acid residue marks in the masked co-evolutionary pair in the second masked marker sequence perform attention calculation on the positions other than the positions corresponding to the amino acid residue marks in the masked co-evolutionary pair in the first masked marker sequence, and perform attention calculation on the positions of the amino acid residue marks in the masked co-evolutionary pair themselves in the second masked marker sequence.

[0153] For more information, see Figure 4 ,like Figure 4 As shown: The second mask marks the sequence of the first mask co-evolved pairs "T <m>” ( Figure 4 in blue markings) corresponds to Figure 4 lines 11 and 12 in, and columns 11 and 12 corresponding thereto. As can be seen from Figure 4 the first mask coevolution pair "T <m>The amino acid residues in "" are labeled as "T" in the 11th row and the 11th column.

[0154] For the first masked co-evolution pair "T" in the second masked marker sequence <m>The amino acid residue in "" is marked with "T". That is, for the 11th row, in the squares from the 1st column to the 10th column (corresponding to the first masked token sequence), there is only in the 2nd column (corresponding to the position of this amino acid residue marked with "T" itself in the first masked token sequence) square " " (that is, no attention calculation is performed), and there is no " " in the remaining squares (that is, attention calculation is performed).

[0155] In the squares from the 11th column to the 16th column of the 11th row (corresponding to the second masked token sequence), there is no " " only in the 11th column (corresponding to the position of this amino acid residue marked with "T" itself in the second masked token sequence), and there is " " in all other squares (that is, no attention calculation is performed).

[0156] That is, when subsequently decoding the encoded features corresponding to the 11th row through the preset sequence feature decoder, other positions except the 2nd column in the 1st column to the 10th column can be seen, and the 11th column can be seen. It is equivalent to that in order to predict the token corresponding to the 11th row, other positions in the first masked token sequence except this amino acid residue mark need to be concerned. It can be understood that this amino acid residue itself also needs to be concerned, but not about this amino acid residue itself in the first masked token sequence, but about this amino acid residue itself in the second masked token sequence. Thus, for each amino acid residue mark in the second masked token sequence, the number of marks in the first masked token sequence that perform attention calculation with this mark is the length of the first masked token sequence minus 1, that is , and the number of marks in the second masked token sequence that perform attention calculation with this mark is 1. Furthermore, the total number of marks in the sample masked token sequence that perform attention calculation with this amino acid residue mark is still the length of the first masked token sequence .

[0157] Regarding Figure 4 the second co-evolution pair "R" in <m>The amino acid residues in "" are labeled "R", and the third masked co-evolution pair "E" <m>The amino acid residue is labeled "E", and reference can be made to the above-mentioned first masked co-evolution pair "T" <m>The amino acid residues in "" are labeled "T", which will not be elaborated here.

[0158] 4. The preset mask marker pair in the mask co-evolution pair of the second mask marker sequence calculates the attention for other positions in the first mask marker sequence except for the position where the co-evolution pair corresponding to the mask co-evolution pair is located, and calculates the attention for the position where the mask co-evolution pair is located in the second mask marker sequence.

[0159] Specifically, please refer to Figure 4 , as Figure 4 shown: The first mask co-evolution pair "T" in the second mask marker sequence <m>” ( Figure 4 in blue markings), corresponding to Figure 4 lines 11 and 12 in, and corresponding to columns 11 and 12. As can be seen from Figure 4 the first co-evolved mask pair "T <m>The preset mask in "" is marked as the one at the 12th row and 12th column <m>”.

[0160] For the co-evolved pair "T" in the second masked token sequence <m>" preset mask token in" <m>” (i.e., blue" <m>”), that is, for the 12th row, in the squares of columns 1 to 10 (corresponding to the first masked token sequence), only in columns 2 and 8 (corresponding to the first co-evolution pair corresponding to the first masked co-evolution pair in the first masked token sequence, the two blue " <m>”)There is a " in the grid ” (i.e., no attention calculation is performed), and there is no " in the remaining grids ” (i.e., attention calculation is performed).

[0161] In the grids from the 11th column to the 16th column of the 12th row (corresponding to the second mask token sequence), there is no " only in the grids of the 11th column and the 12th column (corresponding to the first mask co - evolution pair in the second mask token sequence) ” (i.e., attention calculation is performed), and there is a " in the remaining grids ” (i.e., no attention calculation is performed).

[0162] That is, when subsequently decoding the encoded features corresponding to the 12th row through the preset sequence feature decoder, it can be seen that in the positions from the 1st column to the 10th column except for the 2nd column and the 8th column, and it can also be seen the 11th column and the 12th column. It is equivalent to that in order to predict the token corresponding to the 12th row, attention needs to be paid to the positions in the first mask token sequence except for the positions corresponding to the co - evolution pairs where the co - evolution pairs corresponding to the preset mask token are located. It can be understood that attention also needs to be paid to the mask co - evolution pair itself where the preset mask token is located in the second mask token sequence, but it is not necessary to pay attention to the co - evolution pairs corresponding to the mask co - evolution pair where the preset mask token is located in the first mask token sequence, but to pay attention to the two positions where the mask co - evolution pair where the preset mask token is located in the second mask token sequence. Thus, for each preset mask token in the second mask token sequence, the number of tokens in the first mask token sequence that perform attention calculation with this token is the length of the first mask token sequence minus 2, that is , and the number of tokens in the second mask token sequence that perform attention calculation with this token is 2. Furthermore, the total number of tokens in the sample mask token sequence that perform attention calculation with this preset mask token is still the length of the first mask token sequence .

[0163] The post - mask attention score matrix obtained by the above method is also a matrix. For the column where the i - th row of this matrix is located, there are element values that are the same as or extremely close to the values in the corresponding positions of , and the remaining element values are 0 or extremely close to 0. In this way, the protein language model can learn context - dependence similar to the bidirectional attention mechanism, as well as learn the causal relationship between the two tokens in the co - evolution pair, thereby enhancing the protein language model's understanding of the protein sequence structure.

[0164] Step 20224: Determine the encoded feature sequence based on the masked attention score matrix and the value matrix.

[0165] Specifically, the encoded feature sequence can be calculated by the following formula:

[0166] where is the masked attention score matrix, is the value matrix, is the calculated encoded feature sequence. 、 and are all matrices.

[0167] By adopting the optional implementation manners of the above Steps 20221 to 20224, the bidirectional attention mechanism of the improved stacked multi-head attention layer is used to introduce the autoregressive mechanism, and the autoregressive strategy is designed accordingly according to the generation manner of the sample mask token sequence, that is, the autoregressive is realized by designing the corresponding masked attention matrix, so that the protein language model can learn the co-evolution pair structure information. The co-evolution relationship is crucial for the folding, function and dynamic process of proteins. Therefore, by enabling the protein language model to learn the co-evolution information, the protein language model can learn the structure information of proteins, providing a model understanding basis for the subsequent protein structure prediction task.

[0168] Step 203: Decode the encoded feature sequence based on a preset sequence feature decoder to obtain a decoded protein sequence.

[0169] Specifically, the encoded feature sequence can be input into the preset sequence feature decoder to obtain the decoded protein sequence.

[0170] Here, a preset sequence feature decoder is used to represent the correspondence between the feature sequence and the protein sequence. The preset sequence feature decoder may include a linear transformation layer and / or a non-linear transformation layer. For example, the preset sequence feature decoder may include a fully connected network and an activation function layer connected in sequence. Inputting the encoded feature sequence into the fully connected network can obtain P values corresponding to each token in the sample mask token sequence, where P is the number of different amino acid residue tokens that may appear in the protein sequence. Then, normalizing the above P values can obtain P probability values, respectively corresponding to the probability values of belonging to each amino acid residue token among the P amino acid residue tokens, and taking the amino acid residue token with the largest corresponding probability value as the decoded amino acid residue token corresponding to the corresponding token in the sample mask token sequence. Finally, a decoded protein sequence can be generated using the decoded amino acid residue tokens corresponding to each token in the sample mask token sequence. That is, the decoded protein sequence is formed by arranging the decoded amino acid residue tokens in order, and the sequence length of the decoded protein sequence can be the same as the sequence length of the sample mask token sequence.

[0171] Step 204, optimize the protein language model based on the differences between the decoded protein sequence, the sample protein sequence, and each co-evolution pair.

[0172] Here, various implementation methods can be adopted, with the goal of minimizing the difference between the decoded protein sequence and the original token sequence before the corresponding sample mask token sequence is masked, to adjust the model parameters of the protein language model, so as to optimize the protein language model based on the differences between the decoded protein sequence, the sample protein sequence, and each co-evolution pair.

[0173] As an example, Step 204 can be carried out as follows: The first step, calculate the loss function: Calculate the loss function between the decoded protein sequence and the original token sequence before the corresponding sample mask token sequence is masked.

[0174] Here, various known or future-developed loss function calculation methods can be adopted, such as cross-entropy or mean squared error, etc.

[0175] The second step, perform backpropagation: First, perform gradient calculation: Using the chain rule, calculate the gradient of the loss function with respect to each parameter layer by layer in reverse from the output layer of the protein language model.

[0176] Secondly, perform gradient transfer: Transfer the gradient from the output layer to each layer, identifying the contribution of each parameter to the loss.

[0177] The third step, perform parameter update (Optimization): Update parameters according to the gradient using optimization algorithms. Common methods include: Gradient Descent: such as Batch Gradient Descent (BGD), Stochastic Gradient Descent (SGD), Mini-batch Gradient Descent (Mini-batch GD), etc.

[0178] Other improved optimization algorithms, such as: Momentum, Adam (Adaptive Moment Estimation), etc.

[0179] Fourth step, hyperparameter tuning: Control the update step size through the pre-set learning rate η. If it is too large, it will cause oscillations; if it is too small, the convergence will be slow. A learning rate scheduler (such as ReduceLROnPlateau, cosine annealing) can be used.

[0180] Affect the accuracy of gradient estimation and memory occupancy by setting the batch size.

[0181] Prevent overfitting through regularization methods such as L2 regularization (weight decay), Dropout.

[0182] Fifth step, iterative training: Repeat the following steps until the model converges: 1. Forward propagation → Calculate the loss.

[0183] 2. Backward propagation → Calculate the gradient.

[0184] 3. The optimizer updates the parameters.

[0185] 4. (Optional) Evaluate on the validation set, and early stopping to prevent overfitting.

[0186] It should be noted that the above steps 201 to 205 only show the process of pre-training a protein language model based on a single sample protein sequence. In practice, the protein language model can be pre-trained multiple times. Each time for pre-training, at least one sample protein sequence can be used to execute steps 201 to 203, and in step 204, calculate the difference between the corresponding decoded protein sequence of each sample protein sequence and the original unmasked marker sequence corresponding to the sample mask marker sequence, and adjust the model parameters of the protein language model based on the sum of the calculated differences.

[0187] It should be noted that in step 204, in addition to adjusting the model parameters of the protein language model, the model parameters of the preset sequence feature decoder can also be adjusted synchronously.

[0188] The protein language model pre-training method provided by the above embodiments of the present disclosure obtains a sample masked token sequence by masking a sample protein sequence and co-evolution pairs in the sample protein sequence; then, based on the protein language model, performs embedding representation and feature encoding on the sample masked token sequence to obtain an encoded feature sequence; then, based on a preset sequence feature decoder, decodes the encoded feature sequence to obtain a decoded protein sequence; finally, optimizes the protein language model based on the differences between the decoded protein sequence and the sample protein sequence and each co-evolution pair. This method can achieve the following technical effects including but not limited to: First, since co-evolution pairs have a high degree of correlation at the biomolecular structure and function levels, traditional protein language models are relatively easy to understand the conserved regions (Conservation) and variable regions (Variation) in protein sequences. Among them, the conserved regions can be learned through the masked language model to learn local properties. And the variable regions can be learned through various different sequences in the training data. However, co-evolution regions are difficult to learn because there are often long-range dependencies between two amino acid residues in co-evolution regions. Therefore, protein language models need to learn long-range dependencies in sequences, which is difficult for traditional protein language models. The above embodiments of the present disclosure can let the protein language model learn the co-evolution pair structure information, or causal information, in the protein sequence by masking the co-evolution pairs in the protein sequence, that is, because of the change of one amino acid residue in the co-evolution pair, the corresponding change of the other amino acid residue in the co-evolution pair results, and finally optimize the protein language model's understanding and prediction ability of protein sequences.

[0189] Second, since multiple sequence alignment is not used, the computational cost is relatively low compared to multiple sequence alignment.

[0190] Third, performing downstream protein sequence analysis tasks, such as sequence-related or structure-related tasks, based on the protein language model pre-trained by masking co-evolution pairs in protein sequences can improve the accuracy of downstream protein sequence analysis tasks.

[0191] Next, refer to Figure 5 , which shows a flowchart 500 of an embodiment of the protein sequence processing method according to the present disclosure. The protein sequence processing method includes the following steps: Step 501, obtain a protein sequence to be processed.

[0192] Here, the protein sequence to be processed can also be obtained by the same or similar method as described in step 201.

[0193] Step 502: Input the protein sequence to be processed into the protein language model to obtain the encoded feature sequence corresponding to the protein sequence to be processed.

[0194] Here, the protein language model can be pre-trained in advance by the protein language model pre-training method described in the embodiments and their optional embodiments as shown in Figure 2A and its optional implementation manners.

[0195] In the above protein sequence encoding pre-training model, by masking the co-evolutionary pairs in the protein sequence, the protein language model can learn the co-evolutionary pair structure information in the protein sequence, thereby optimizing the protein language model's understanding and prediction capabilities of the protein sequence. In addition, since multiple sequence alignment is not used, the computational complexity is relatively low compared to multiple sequence alignment.

[0196] Furthermore, based on the above protein sequence language model, a decoder for other protein sequence analysis tasks can be spliced to perform downstream protein sequence analysis tasks. Since the above protein sequence encoding pre-training model has been specifically pre-trained according to the characteristics of the protein sequence, the accuracy and computational speed of downstream protein sequence analysis tasks can be improved. For example, the downstream protein analysis task can be a protein sequence-related task or a structure-related task.

[0197] The protein sequence processing method provided by the above embodiments of the present disclosure can improve the accuracy and computational speed of protein sequence understanding because it uses a protein language model pre-trained based on the co-evolutionary pair structure information in the protein sequence and does not involve multiple sequence alignment.

[0198] Example 1: Please refer to Figure 6 . The applicant compares the protein language model pre-training method disclosed in the embodiments shown in Figure 2A with BERT as a baseline model in protein sequence-related tasks.

[0199] Among them, the embodiments of the present application correspond to Figure 2A the protein language model pre-training method disclosed in the embodiments shown in, and the baseline model is BERT. The parameter sizes of both models are 150MB.

[0200] Figure 6 Among them, the horizontal axis is: the number of model iterations during the pre-training process; the vertical axis is: the Spearman Rank Correlation coefficient of two models pre-trained according to the corresponding number of iterations on Proteingym (ProteinGym is a benchmark platform for evaluating protein machine learning models, including various tasks). Here, the Spearman Rank Correlation coefficient is a non-parametric statistic used to measure the monotonicity (without linear hypothesis) between the mutant rankings predicted by the model and the true rankings measured experimentally, and its value range is from -1 (used to represent a perfect negative correlation) to 1 (used to represent a perfect positive correlation). In ProteinGym, the core of the Spearman task is to test whether the protein language model can relatively rank the functional effects of mutants like experiments, which is crucial for guiding protein engineering or understanding evolution.

[0201] In the Spearman task, the input data is: sequence mutants of proteins (such as single-point mutations, insertions, deletions, etc.). The output data is: the functional effects of the predicted mutants relative to the wild type (such as experimentally measured values like enzyme activity, stability, fluorescence intensity, etc.). And the evaluation method is to compare the correlation (Spearman coefficient) between the mutant rankings predicted by the model and the true experimental rankings.

[0202] Therefore, in the Spearman task, what is directly relied on is the relationship between sequence variation (i.e., changes in the amino acid sequence) and function, rather than the three-dimensional structure of the protein. That is, the performance of the model in sequence-related tasks can be reflected by the Spearman coefficient.

[0203] From Figure 6 It can be seen that when the number of iterations is less than 60,000 times, the Spearman Rank Correlation coefficient of the embodiment of the present application is slightly higher than that of the baseline model, while when the number of iterations is 70,000 times, 80,000 times, and 100,000 times, the Spearman Rank Correlation coefficient of the embodiment of the present application is slightly lower than that of the baseline model.

[0204] It can be seen that under the condition of the same order of magnitude of model parameters and relatively small computational cost (the number of model iterations is less than 100,000 times), the protein language model trained by using the protein language model pre-training method provided by the embodiment of the present disclosure can be comparable to BERT in terms of accuracy in protein sequence-related tasks.

[0205] Example 2: Please refer to Figure 7 . The applicant will Figure 2A Comparison chart of the protein language model pre-training method disclosed in the illustrated embodiment and BERT as a baseline model in protein structure-related tasks.

[0206] Among them, the embodiment of the present application corresponds to Figure 2A the protein language model pre-training method disclosed in the illustrated embodiment, and the baseline model is BERT. The parameter sizes of both models are 150MB.

[0207] Figure 7 In it, the horizontal axis is: the number of model iterations during pre-training; the vertical axis is: the Top-L / 5 Precision of the two models pre-trained according to the corresponding number of iterations in Casp14 (Critical Assessment of Protein Structure Prediction, the 14th Critical Assessment of Protein Structure Prediction).

[0208] Top-L / 5 Precision is one of the important indicators for evaluating the accuracy of protein structure prediction models (including protein language models) when predicting contact maps or distances between residues.

[0209] In the task of predicting contact maps, the task objective is to evaluate the spatial proximity of the protein residue pairs predicted by the model (such as whether they are in contact). The input data is: the amino acid sequence of the protein. The output data is: the distance or contact probability between the residue pairs predicted by the model (usually converted into a binary contact map). In protein structure prediction, accurate contact / distance prediction can significantly improve the accuracy of three-dimensional structure modeling.

[0210] The calculation method of Top-L / 5 Precision is as follows: Assume that the length of the protein sequence is L.

[0211] Top-L / 5 refers to selecting the L / 5 pairs with the highest confidence from all the residue pairs predicted by the model (i.e., a standard subset of the number of long-range residue pairs).

[0212] Top-L / 5 Precision refers to the proportion of these selected Top-L / 5 residue pairs that are actually in contact in the true structure (i.e., the distance ≤ a certain threshold, such as 6Å).

[0213] In CASP14, Top-L / 5 Precision is the core metric for evaluating the reliability of "high-confidence predictions" of protein language models in residue contact prediction. It measures the model's ability to identify key interactions by standardizing the selection of Top-L / 5 long-range residue pairs, providing important constraints for subsequent three-dimensional structure modeling. The performance of the protein language model in this task reflects its ability to directly infer structural information from the sequence.

[0214] From Figure 7 It can be seen that when the number of iterations is less than 100,000, the Top-L / 5 Precision of the embodiments of this application is significantly higher than that of the baseline model.

[0215] It can be seen that with the same order of magnitude of model parameters and a relatively small computational cost (less than 100,000 model iterations), the protein language model trained by the protein language model pre-training method provided in the embodiments of this disclosure significantly outperforms BERT in protein structure-related tasks.

[0216] Combining Example 1 and Example 2, it can be known that the protein language model trained by the protein language model pre-training method provided in the embodiments of this disclosure meets or exceeds BERT in both protein sequence-related tasks and structure-related tasks.

[0217] In some alternative embodiments, further referring to Figure 8 , as an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a protein language model pre-training device. This device embodiment corresponds to the method embodiment shown in Figure 2A , and this device can be specifically applied to various electronic devices.

[0218] As Figure 8 shown, the protein language model pre-training device 800 of this embodiment includes: a masking module 801, an embedding representation and encoding module 802, a decoding module 803, and an optimization module 804. Among them, the masking module 801 is configured to perform masking processing on the sample protein sequence and the co-evolution pairs in the above sample protein sequence to obtain a sample masked token sequence; the embedding representation and encoding module 802 is configured to perform embedding representation and feature encoding on the above sample masked token sequence based on the protein language model to obtain an encoded feature sequence; the decoding module 803 is configured to decode the above encoded feature sequence based on a preset sequence feature decoder to obtain a decoded protein sequence; the optimization module 804 is configured to optimize the above protein language model based on the differences between the above decoded protein sequence and the above sample protein sequence and each of the above co-evolution pairs.

[0219] In this embodiment, for the specific processing of the masking module 801, the embedding representation and encoding module 802, the decoding module 803, and the optimization module 804 of the protein language model pre-training device 800 and the technical effects brought thereby, reference can be respectively made to Figure 2A the relevant descriptions of steps 201, 202, 203, and 204 in the corresponding embodiments, which will not be elaborated here.

[0220] In some optional implementation manners, the above-mentioned masking module 801 may include: a masking position determination unit 8011, configured to determine at least one masking position for masking the above-mentioned sample protein sequence based on the co-evolution pair position information of the above-mentioned sample protein sequence; a first masking unit 8012, configured to perform masking processing on the above-mentioned sample protein sequence according to the above-mentioned at least one masking position to obtain a first masked token sequence; a causal learning sequence generation unit 8013, configured to generate a causal learning token sequence based on each co-evolution pair in the above-mentioned sample protein sequence, where the above-mentioned causal learning token sequence includes each of the above-mentioned co-evolution pairs; a second masking unit 8014, configured to perform masking processing on the above-mentioned causal learning token sequence to obtain a second masked token sequence, where the above-mentioned second masked token sequence includes at least one masked co-evolution pair corresponding to the co-evolution pair in the above-mentioned sample protein sequence, and the masked co-evolution pair includes a preset causal masking token and an amino acid residue token in the corresponding co-evolution pair; a splicing unit 8015, configured to splice the above-mentioned first masked token sequence and the above-mentioned second masked token sequence to obtain the above-mentioned sample masked token sequence.

[0221] In some optional implementation manners, the above-mentioned masking position determination unit 8011 may be further configured to: determine the positions where each co-evolution pair in the above-mentioned sample protein sequence is located as masking positions; determine the number of positions to be masked according to a preset masking ratio and the length of the above-mentioned sample protein sequence; determine the difference between the number of positions to be masked and the number of special masking positions as the number of ordinary masking positions, where the number of special masking positions is the number of positions where each co-evolution pair in the above-mentioned sample protein sequence is located; select the above-mentioned number of ordinary masking positions from the positions in the above-mentioned sample protein sequence that are not determined as masking positions and determine them as masking positions.

[0222] In some alternative embodiments, the second masking unit 8014 may be further configured to: determine co-evolution pairs with a preset positive masking ratio in each co-evolution pair in the causal learning token sequence as positive co-evolution pairs; determine the remaining co-evolution pairs in each co-evolution pair in the causal learning token sequence other than the positive co-evolution pairs as negative co-evolution pairs; for each of the positive co-evolution pairs, replace the amino acid residue token with a lower order in the positive co-evolution pair with the preset causal masking token to generate a masked co-evolution pair corresponding to the positive co-evolution pair; for each of the negative co-evolution pairs, replace the amino acid residue token with a higher order in the negative co-evolution pair with the preset causal masking token to generate a co-evolution pair corresponding to step 20142 for the negative co-evolution pair; generate the second masked token sequence based on the masked co-evolution pairs corresponding to the co-evolution pairs in the causal learning token sequence.

[0223] In some alternative embodiments, the protein language model may include an embedding representation model and a sequence feature encoding model, and the sequence feature encoding model may include a stacked multi-head attention layer; and The embedding representation and encoding module 802 may include: an embedding representation unit 8021 configured to perform an embedding representation on the sample masked token sequence through the embedding representation model to obtain an input feature sequence; an encoding unit 8022 configured to input the input feature sequence into the sequence feature encoding model to obtain the encoded feature sequence.

[0224] In some alternative embodiments, the above-mentioned encoding unit 8022 may be further configured to: input the above-mentioned input feature sequence into the above-mentioned stacked multi-head attention layer to obtain a query matrix, a key matrix, and a value matrix; process the above-mentioned key matrix and the above-mentioned query matrix to obtain an original attention score matrix, where the original attention score matrix is used to represent the original attention between any two tokens in the above-mentioned sample mask token sequence; apply an attention mask matrix to the above-mentioned original attention score matrix to obtain a masked attention score matrix, where the attention mask matrix is used to act on the above-mentioned original attention score matrix to define whether attention calculation is performed between two tokens in the above-mentioned sample mask token sequence. Specifically: mutual attention calculation is performed between any two tokens in the above-mentioned first mask token sequence; no attention calculation is performed between any token in the above-mentioned first mask token sequence and any token in the above-mentioned second mask token sequence; the amino acid residue tokens in the co-evolutionary pair in the above-mentioned second mask token sequence perform attention calculation on other positions in the above-mentioned first mask token sequence except for the position corresponding to the amino acid residue token in the co-evolutionary pair, and perform attention calculation on the position where the amino acid residue token in the co-evolutionary pair in the above-mentioned second mask token sequence itself is located; the preset mask tokens in the co-evolutionary pair in the above-mentioned second mask token sequence perform attention calculation on other positions in the above-mentioned first mask token sequence except for the position corresponding to the co-evolutionary pair corresponding to the co-evolutionary pair, and perform attention calculation on the position corresponding to the co-evolutionary pair in the above-mentioned second mask token sequence; based on the above-mentioned masked attention score matrix and the above-mentioned value matrix, determine the above-mentioned encoded feature sequence.

[0225] It should be noted that the implementation details and technical effects of each module in the protein language model pre-training device provided in the embodiments of the present disclosure can refer to the descriptions of other embodiments in the present disclosure, and will not be elaborated here.

[0226] Next, referring to Figure 9 , as an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a protein sequence processing device. This device embodiment corresponds to the method embodiment shown in Figure 5 , and this device can be specifically applied to various electronic devices.

[0227] As shown in Figure 9 , the protein sequence processing device 900 in this embodiment includes: an acquisition module 901 and a processing module 902. Among them, the acquisition module 901 is configured to acquire a protein sequence to be processed; the processing module 902 is configured to input the above-mentioned protein sequence to be processed into a protein language model to obtain an encoded feature sequence corresponding to the above-mentioned protein sequence to be processed, where the above-mentioned protein language model is pre-trained by the method described in any implementation manner of the first aspect.

[0228] In this embodiment, for the specific processing of the acquisition module 901 and the processing module 902 of the protein sequence processing device 900 and the technical effects brought by them, reference can be made respectively to Figure 5 the relevant descriptions of step 501 and step 502 in the corresponding embodiment, which will not be elaborated here.

[0229] It should be noted that for the implementation details and technical effects of each module in the protein sequence processing device provided in the embodiments of the present disclosure, reference can be made to the descriptions of other embodiments in the present disclosure, which will not be elaborated here.

[0230] Next, with reference to Figure 10 , which shows a schematic structural diagram of a computer system 1000 suitable for implementing the electronic device of the present disclosure. Figure 10 The shown computer system 1000 is merely an example and should not impose any limitation on the functions and usage scope of the embodiments of the present disclosure.

[0231] As Figure 10 shown, the computer system 1000 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 1001, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage device 1008 into the random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the computer system 1000 are also stored. The processing device 1001, the ROM 1002, and the RAM 1003 are connected to each other through a bus 1004. The input / output (I / O) interface 1005 is also connected to the bus 1004.

[0232] Generally, the following devices may be connected to the I / O interface 1005: an input device 1006 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, etc.; an output device 1007 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1008 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the computer system 1000 to communicate with other devices wirelessly or wiredly to exchange data. Although Figure 10 the computer system 1000 of the electronic device with various devices is shown, it should be understood that it is not required to implement or have all the shown devices. More or fewer devices can be alternatively implemented or had.

[0233] In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, an embodiment of the present disclosure includes a computer program product that includes a computer program carried on a computer-readable medium, and the computer program includes program code for performing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network via the communication device 1009, or installed from the storage device 1008, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-described functions defined in the methods of the embodiments of the present disclosure are performed.

[0234] It should be noted that the above-mentioned computer-readable medium in the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, and the computer-readable signal medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0235] The above-mentioned computer-readable medium can be included in the above-mentioned electronic device; or it can exist separately without being assembled into the electronic device.

[0236] The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the electronic device, the electronic device is caused to implement as Figure 2A The protein language model pre-training method shown in the illustrated embodiments and their alternative embodiments and / or the protein sequence processing method shown in the illustrated embodiments and their alternative embodiments. Figure 5 The protein sequence processing method shown in the illustrated embodiments and their alternative embodiments.

[0237] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Python, Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partially on the user's computer, execute as a stand-alone software package, execute partially on the user's computer and partially on a remote computer, or execute entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., by connecting through the Internet using an Internet service provider).

[0238] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a portion of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks may occur in a different order than noted in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system for performing the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0239] The modules described in the embodiments of the present disclosure may be implemented in software or in hardware. Among them, the name of the module does not constitute a limitation on the module itself in some cases. For example, the masking module may also be described as "a module that performs masking processing on the sample protein sequence and the co-evolution pairs in the sample protein sequence to obtain the sample masked token sequence".

[0240] The above description is only a preferred embodiment of the present disclosure and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosure concept. For example, the technical solutions formed by mutually replacing the above features with the technical features (but not limited to) disclosed in the present disclosure that have similar functions.< / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m> < / m>

Claims

1. A method for pre-training a protein language model, comprising: Mask the sample protein sequence and the co-evolutionary pairs in the sample protein sequence to obtain a sample masked token sequence; Based on a protein language model, perform embedding representation and feature encoding on the sample masked token sequence to obtain an encoded feature sequence; Based on a preset sequence feature decoder, decode the encoded feature sequence to obtain a decoded protein sequence; Optimize the protein language model based on the differences between the decoded protein sequence, the sample protein sequence, and each of the co-evolutionary pairs.

2. The method according to claim 1, wherein, The masking of the sample protein sequence and the co-evolutionary pairs in the sample protein sequence to obtain a sample masked token sequence includes: Based on the co-evolutionary pair position information of the sample protein sequence, determine at least one masking position for masking the sample protein sequence; According to the at least one masking position, mask the sample protein sequence to obtain a first masked token sequence; Based on each co-evolutionary pair in the sample protein sequence, generate a causal learning token sequence, where the causal learning token sequence includes each co-evolutionary pair; Mask the causal learning token sequence to obtain a second masked token sequence, where the second masked token sequence includes at least one masked co-evolutionary pair corresponding to the co-evolutionary pairs in the sample protein sequence, and the masked co-evolutionary pair includes a preset causal mask token and one amino acid residue token in the corresponding co-evolutionary pair; Concatenate the first masked token sequence and the second masked token sequence to obtain the sample masked token sequence.

3. The method according to claim 2, wherein, The determining of at least one masking position for masking the sample protein sequence based on the co-evolutionary pair position information of the sample protein sequence includes: Determine the positions where each co-evolutionary pair in the sample protein sequence is located as masking positions; According to a preset masking ratio and the length of the sample protein sequence, determine the number of positions to be masked; Determine the difference between the number of positions to be masked and the number of special masking positions as the number of ordinary masking positions, where the number of special masking positions is the number of positions where each co-evolutionary pair in the sample protein sequence is located; Among the positions in the sample protein sequence that have not been determined as masking positions, select the number of ordinary masking positions and determine them as masking positions.

4. The method according to claim 3, wherein, The masking of the causal learning token sequence to obtain a second masked token sequence includes: Determine the co-evolutionary pairs with a preset forward masking ratio in each co-evolutionary pair in the causal learning token sequence as forward co-evolutionary pairs; Determine the remaining co-evolutionary pairs in each co-evolutionary pair in the causal learning token sequence except for each of the forward co-evolutionary pairs as reverse co-evolutionary pairs; For each forward co-evolutionary pair, replace the amino acid residue token with a higher rank in the forward co-evolutionary pair with the preset causal mask token to generate a masked co-evolutionary pair corresponding to the forward co-evolutionary pair; For each reverse co-evolutionary pair, replace the amino acid residue token with a lower rank in the reverse co-evolutionary pair with the preset causal mask token to generate a masked co-evolutionary pair corresponding to the reverse co-evolutionary pair; Generate the second masked token sequence based on the masked co-evolution pairs corresponding to the co-evolution pairs in the causal learning token sequence.

5. The method according to claim 2, wherein, The protein language model includes an embedding representation model and a sequence feature encoding model, and the sequence feature encoding model includes a stacked multi-head attention layer; And Based on the protein language model, performing embedding representation and feature encoding on the sample masked token sequence to obtain an encoded feature sequence, including: Performing embedding representation on the sample masked token sequence through the embedding representation model to obtain an input feature sequence; Inputting the input feature sequence into the sequence feature encoding model to obtain the encoded feature sequence.

6. The method according to claim 5, wherein, The inputting the input feature sequence into the sequence feature encoding model to obtain the encoded feature sequence includes: Inputting the input feature sequence into the stacked multi-head attention layer to obtain a query matrix, a key matrix, and a value matrix; Processing the key matrix and the query matrix to obtain an original attention score matrix, where the original attention score matrix is used to represent the original attention between any two tokens in the sample masked token sequence; Applying an attention mask matrix to the original attention score matrix to obtain a masked attention score matrix, where the attention mask matrix is used to act on the original attention score matrix to define whether attention calculation is performed between two tokens in the sample masked token sequence. Specifically: mutual attention calculation is performed between any two tokens in the first masked token sequence; no attention calculation is performed between any token in the first masked token sequence and any token in the second masked token sequence; amino acid residue tokens in the masked co-evolution pairs in the second masked token sequence perform attention calculation on other positions in the first masked token sequence except for the positions corresponding to the amino acid residue tokens in the masked co-evolution pairs, and perform attention calculation on the positions where the amino acid residue tokens in the masked co-evolution pairs in the second masked token sequence are located; preset masked tokens in the masked co-evolution pairs in the second masked token sequence perform attention calculation on other positions in the first masked token sequence except for the positions corresponding to the co-evolution pairs corresponding to the masked co-evolution pairs, and perform attention calculation on the positions where the masked co-evolution pairs in the second masked token sequence are located; Determine the encoded feature sequence based on the masked attention score matrix and the value matrix.

7. A method for processing a protein sequence, comprising: Obtain a protein sequence to be processed; Input the protein sequence to be processed into the protein language model to obtain an encoded feature sequence corresponding to the protein sequence to be processed, where the protein language model is pre-trained by the method according to any one of claims 1-6.

8. A protein language model pre-training device, comprising: A masking module configured to perform masking processing on a sample protein sequence and co-evolution pairs in the sample protein sequence to obtain a sample masked token sequence; An embedding representation and encoding module configured to perform embedding representation and feature encoding on the sample masked token sequence based on the protein language model to obtain an encoded feature sequence; A decoding module, configured to decode the encoded feature sequence based on a preset sequence feature decoder to obtain a decoded protein sequence; An optimization module, configured to optimize the protein language model based on the differences between the decoded protein sequence, the sample protein sequence, and each of the co-evolution pairs; 9. A protein sequence processing device, comprising: An acquisition module, configured to acquire a protein sequence to be processed; A processing module, configured to input the protein sequence to be processed into a protein language model to obtain an encoded feature sequence corresponding to the protein sequence to be processed, wherein the protein language model is pre-trained by the method according to any one of claims 1-6; 10. An electronic device, comprising: One or more processors; A storage device having stored thereon one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the method according to any one of claims 1-6 and / or the method according to claim 7; 11. A computer-readable storage medium, having a computer program stored thereon, wherein, When the computer program is executed by one or more processors, the method according to any one of claims 1-6 and / or the method according to claim 7 is implemented.

12. A computer program product, comprising computer programs / instructions, which when executed by a processor implement the method as described in any one of claims 1-6 and / or the method as described in claim 7.

Images