Protein language model pre-training and protein mutation method and related products

Abstract

The invention provides a protein language model pre-training and protein mutation method and related products. According to one specific embodiment of the protein language model pre-training method, a sample protein data set is obtained; generating a multi-sequence alignment probability distribution sequence corresponding to the sample protein data according to the amino acid residue probability distribution of multiple sequences in each sample protein data alignment at each site; the sample protein sequence in each sample protein data and the corresponding multi-sequence comparison probability distribution sequence and structure sequence are sequentially connected in series in the forward direction or the reverse direction, and a multi-modal sequence corresponding to the corresponding sample protein data is generated; and finally, performing autoregression pre-training on the protein language model based on the multi-modal sequence corresponding to each sample protein data to obtain a pre-trained protein language model. Namely, the prediction performance of the model is improved by introducing a multi-sequence comparison probability distribution sequence as an independent intermediate reasoning mode and thinking chains in two directions.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of bio-information analysis, and in particular to a protein language model pre-training and protein mutation method and related products. BACKGROUND

[0002] Protein language models (PLMs) learn the potential distribution of protein evolution rules, structural features and functional information from massive amounts of unlabeled amino acid residue sequences by drawing on natural language processing (NLP) techniques. Currently, protein language models have developed from having only one modality of amino acid residue sequences to multi-modal protein language models. Multi-modal protein language models significantly enhance the expression ability of the model by fusing protein sequences, multiple sequence alignment (MSA), three-dimensional structures, biological function annotations and other modal information. SUMMARY

[0003] Embodiments of the present disclosure provide a protein language model pre-training and protein mutation method, device, electronic equipment, storage medium and computer program product.

[0004] In a first aspect, embodiments of the present disclosure provide a protein language model pre-training method, comprising:

[0005] obtaining a set of sample protein data, wherein the sample protein data comprises a sample protein sequence, and corresponding multiple sequence alignment and structure sequence;

[0006] For each sample protein data, performing the following multi-modal sequence generation operation: generating a multiple sequence alignment probability distribution sequence corresponding to the sample protein data according to the amino acid residue probability distribution of the multiple sequence alignment at each site in the sample protein data; concatenating the sample protein sequence in the sample protein data, and the corresponding multiple sequence alignment probability distribution sequence and structure sequence in sequence in a forward or reverse direction, to generate a multi-modal sequence corresponding to the sample protein data.

[0007] Based on the multi-modal sequence corresponding to each of the sample protein data, performing autoregressive pre-training on a protein language model to obtain a pre-trained protein language model.

[0008] In some optional embodiments, the generating a multiple sequence alignment probability distribution sequence corresponding to the sample protein data according to the amino acid residue probability distribution of the multiple sequence alignment at each site in the sample protein data comprises:

[0009] For each site of the multiple sequence alignment in the sample protein data, the following site amino acid residue probability distribution determination operation is performed: for each preset amino acid residue identifier in the preset amino acid residue identifier set, the number of sequences in the multiple sequence alignment in the sample protein data whose amino acid residue identifier at the site is the preset amino acid residue identifier is divided by the total number of sequences in the multiple sequence alignment in the sample protein data, and the ratio is determined as the probability that the multiple sequence alignment in the sample protein data is the preset amino acid residue identifier at the site; and according to the probabilities that the multiple sequence alignment in the sample protein data is each of the preset amino acid residue identifiers at the site, an amino acid residue probability distribution of the multiple sequence alignment in the sample protein data at the site is generated.

[0010] In the order of the sites of the multiple sequence alignment in the sample protein data, based on the amino acid residue probability distributions of the multiple sequence alignment in the sample protein data at the corresponding sites, a multiple sequence alignment probability distribution sequence corresponding to the sample protein data is generated.

[0011] In some optional embodiments, the method further includes:

[0012] obtaining a protein wet experiment data set, wherein the protein wet experiment data includes a wild protein sequence and a corresponding multiple sequence alignment probability distribution sequence, a structure sequence, and a mutant protein wet experiment data set, and the mutant protein wet experiment data includes a mutant protein sequence and a corresponding wet experiment score;

[0013] For the protein wet experiment data in the protein wet experiment data set, the following prior amino acid residue influence degree distribution sequence generation operation is performed: based on the wet experiment scores of each mutant protein sequence in the protein wet experiment data, a preset number of mutant protein sequences are selected from each mutant protein sequence in the protein wet experiment data to generate a prior mutant protein sequence set; for each position in the prior amino acid residue influence degree distribution sequence corresponding to the protein wet experiment data, based on the sum of the wet experiment scores of each mutant protein sequence in each of the prior mutant protein sequences that has a corresponding amino acid residue identifier at the position, a prior amino acid residue influence degree distribution of the position is determined, wherein the prior amino acid residue influence degree distribution sequence corresponding to the protein wet experiment data is the same as the sequence length of the wild protein sequence.

[0014] In some optional embodiments, the method further includes:

[0015] For each protein wet experiment data in the protein wet experiment data set, the following preferred optimization multi-modal sequence pair generation operation is performed: in each of the mutant protein sequences of the protein wet experiment data, except for the other mutant protein sequences of each of the prior mutant protein sequences, a preferred mutant protein sequence and a substandard protein sequence are selected, wherein the wet experiment score of the preferred mutant protein sequence is greater than the wet experiment score of the substandard protein sequence; the structure sequence, the multiple sequence alignment probability distribution sequence, the wild protein sequence, and the corresponding prior amino acid residue influence degree distribution sequence of the protein wet experiment data are sequentially concatenated with the preferred mutant protein sequence to obtain a corresponding preferred multi-modal sequence; the structure sequence, the multiple sequence alignment probability distribution sequence, the wild protein sequence, and the corresponding prior amino acid residue influence degree distribution sequence of the protein wet experiment data are sequentially concatenated with the substandard mutant protein sequence to obtain a corresponding substandard multi-modal sequence; and the preferred multi-modal sequence and the substandard multi-modal sequence of the protein wet experiment data are taken as the preferred optimization multi-modal sequence pair of the protein wet experiment data.

[0016] Based on each of the preferred optimization multi-modal sequence pairs, the pre-trained protein language model is directly preferred optimized.

[0017] In some optional embodiments, the method further comprises:

[0018] For each protein wet experiment data in the protein wet experiment data set, the structure sequence, the multiple sequence alignment probability distribution sequence, the wild protein sequence, and the corresponding prior amino acid residue influence degree distribution sequence of the protein wet experiment data are sequentially concatenated to obtain a corresponding near-end strategy optimization multi-modal sequence.

[0019] Based on each of the near-end strategy optimization multi-modal sequences, the pre-trained protein language model is near-end strategy optimized.

[0020] In a second aspect, embodiments of the present disclosure provide a protein mutation method, which comprises:

[0021] obtaining a to-be-optimized protein sequence and corresponding multiple sequence alignment and structure sequence;

[0022] generating a corresponding multiple sequence alignment probability distribution sequence according to the amino acid residue probability distribution of the multiple sequence alignment of the to-be-optimized protein sequence at each site;

[0023] concatenating the structure sequence, the multiple sequence alignment probability distribution sequence, and the to-be-optimized protein sequence corresponding to the to-be-optimized protein sequence in sequence to generate a to-be-optimized protein multi-modal sequence corresponding to the to-be-optimized protein sequence;

[0024] inputting the to-be-optimized protein multi-modal sequence into the pre-trained protein language model to obtain a first-round predicted mutant protein amino acid residue probability distribution sequence, wherein the pre-trained protein language model is pre-trained by the method described in any one of the implementations of the first aspect.

[0025] In some optional implementations, the method further comprises:

[0026] obtaining a first-round mutant protein sequence set, wherein the first-round mutant protein sequence set is obtained based on first-round mutation design on the to-be-optimized protein sequence based on the first-round predicted mutant protein amino acid residue probability distribution sequence;

[0027] obtaining wet experiment scores of wet experiments on each of the first-round mutant protein sequences;

[0028] For each position in the first-round prior amino acid residue influence degree distribution sequence corresponding to the to-be-optimized protein sequence, determining a prior amino acid residue influence degree distribution of the first-round prior amino acid residue influence degree distribution sequence corresponding to the to-be-optimized protein sequence at the position based on a sum of wet experiment scores of each first-round prior mutant protein sequence in which the position is a corresponding amino acid residue identifier in each of the first-round prior mutant protein sequences, wherein the first-round prior amino acid residue influence degree distribution sequence corresponding to the to-be-optimized protein sequence has a same sequence length as the to-be-optimized protein sequence;

[0029] concatenating, in sequence, a structure sequence of the to-be-optimized protein sequence, a multi-sequence alignment probability distribution sequence, the to-be-optimized protein sequence, and a corresponding first-round amino acid residue prior probability distribution sequence to obtain a first-round to-be-optimized multi-modal sequence;

[0030] taking the first round as a current round, based on the current round to-be-optimized multi-modal sequence and the pre-trained protein language model that is supervised fine-tuned and directly preferred optimized or proximal strategy optimized, performing next-round mutation design on the to-be-optimized protein sequence to obtain a next-round mutant protein sequence set of the current round.

[0031] In some optional implementations, the method further comprises:

[0032] obtaining wet experiment scores of wet experiments on each of the next-round mutant protein sequences of the current round, and updating the current round to a next round of the current round;

[0033] The following next round of protein mutation design operations are performed: in response to determining that the next round of protein mutation design needs to be performed according to the wet experiment scoring of each of the current round of mutant protein sequences, for each position in the current round of prior amino acid residue influence degree distribution sequence corresponding to the protein sequence to be optimized, the prior amino acid residue influence degree distribution of the current round of prior amino acid residue influence degree distribution sequence corresponding to the protein sequence to be optimized at the position is determined based on the sum of the wet experiment scores of each of the current round of prior mutant protein sequences in which the position is identified by the corresponding amino acid residue in each of the current round of prior mutant protein sequences, wherein the current round of prior amino acid residue influence degree distribution sequence corresponding to the protein sequence to be optimized has the same sequence length as the protein sequence to be optimized; concatenating the current round of prior amino acid residue influence degree distribution sequence to the tail of the last round of protein sequence to be optimized in the current round to obtain a current round of protein sequence to be optimized; based on the current round of protein sequence to be optimized and the pre-trained protein language model optimized by supervised fine-tuning and direct preference optimization or by proximal strategy optimization, the next round of mutation design is performed on the protein sequence to be optimized to obtain a set of next round of mutant protein sequences in the current round; wet experiment scores of wet experiments on each of the next round of mutant protein sequences in the current round are obtained, and the current round is updated to the next round of the current round, and the next round of protein mutation design operations is continued to be performed.

[0034] In a third aspect, embodiments of the present disclosure provide a protein language model pre-training apparatus, which comprises:

[0035] A sample data acquisition module configured to acquire a set of sample protein data, wherein the sample protein data comprises a sample protein sequence, and corresponding multiple sequence alignment and structure sequence;

[0036] A multi-modal sequence generation module configured to, for each sample protein data, perform the following multi-modal sequence generation operation: generating a multiple sequence alignment probability distribution sequence corresponding to the sample protein data according to the amino acid residue probability distribution of each position in the multiple sequence alignment in the sample protein data; concatenating the sample protein sequence in the sample protein data, and the corresponding multiple sequence alignment probability distribution sequence and structure sequence in sequence in a forward or reverse direction to generate a multi-modal sequence corresponding to the sample protein data;

[0037] A pre-training module configured to perform autoregressive pre-training on a protein language model based on the multi-modal sequence corresponding to each of the sample protein data to obtain a pre-trained protein language model.

[0038] In some optional embodiments, the generating, according to the amino acid residue probability distribution of each position in the multiple sequence alignment in the sample protein data, a multiple sequence alignment probability distribution sequence corresponding to the sample protein data comprises:

[0039] For each position in the multiple sequence alignment in the sample protein data, the following position amino acid residue probability distribution determination operation is performed: for each preset amino acid residue identifier in a preset amino acid residue identifier set, a ratio of a sequence number of an amino acid residue identifier at the position in the multiple sequence alignment in the sample protein data to the total number of sequences in the multiple sequence alignment in the sample protein data is determined as a probability of the multiple sequence alignment in the sample protein data being the preset amino acid residue identifier at the position; and according to the probabilities of the multiple sequence alignment in the sample protein data being each of the preset amino acid residue identifiers at the position, an amino acid residue probability distribution of the multiple sequence alignment in the sample protein data at the position is generated.

[0040] According to the order of each position in the multiple sequence alignment in the sample protein data, a multiple sequence alignment probability distribution sequence corresponding to the sample protein data is generated based on the amino acid residue probability distribution of the multiple sequence alignment in the sample protein data at the corresponding position.

[0041] In some optional embodiments, the apparatus further comprises:

[0042] A wet experiment data acquisition module configured to acquire a set of protein wet experiment data, wherein the protein wet experiment data comprises a wild protein sequence and a corresponding multiple sequence alignment probability distribution sequence, a structure sequence, and a set of mutant protein wet experiment data, and the mutant protein wet experiment data comprises a mutant protein sequence and a corresponding wet experiment score.

[0043] A prior influence degree generation module configured to, for protein wet experiment data in the set of protein wet experiment data, perform the following prior amino acid residue influence degree distribution sequence generation operation: based on wet experiment scores of each mutant protein sequence in the protein wet experiment data, select a preset number of mutant protein sequences from each mutant protein sequence in the protein wet experiment data to generate a set of prior mutant protein sequences; and for each position in a prior amino acid residue influence degree distribution sequence corresponding to the protein wet experiment data, determine a prior amino acid residue influence degree distribution of the position based on a sum of wet experiment scores of each mutant protein sequence in each of the prior mutant protein sequences having a corresponding amino acid residue identifier at the position, wherein the prior amino acid residue influence degree distribution sequence corresponding to the protein wet experiment data is the same length as the wild protein sequence.

[0044] In some optional embodiments, the apparatus further comprises:

[0045] The preference pair generation module is configured to perform the following preference optimization multi-modal sequence pair generation operation on the protein wet experiment data in the protein wet experiment data set: selecting a preferred mutant protein sequence and a substandard protein sequence from among the mutant protein sequences other than the prior mutant protein sequences in each of the mutant protein sequences of the protein wet experiment data, wherein the wet experiment score of the preferred mutant protein sequence is greater than the wet experiment score of the substandard protein sequence; sequentially concatenating the structure sequence, the multi-sequence alignment probability distribution sequence, the wild protein sequence, and the corresponding prior amino acid residue influence degree distribution sequence in the protein wet experiment data and the preferred mutant protein sequence to obtain a corresponding preferred multi-modal sequence; sequentially concatenating the structure sequence, the multi-sequence alignment probability distribution sequence, the wild protein sequence, and the corresponding prior amino acid residue influence degree distribution sequence in the protein wet experiment data and the substandard mutant protein sequence to obtain a corresponding substandard multi-modal sequence; and taking the preferred multi-modal sequence and the substandard multi-modal sequence of the protein wet experiment data as the preference optimization multi-modal sequence pair of the protein wet experiment data.

[0046] The direct preference optimization module is configured to perform direct preference optimization on the pre-trained protein language model based on each of the preference optimization multi-modal sequence pairs.

[0047] In some optional embodiments, the apparatus further comprises:

[0048] The proximal strategy optimization multi-modal sequence concatenation module is configured to sequentially concatenate the structure sequence, the multi-sequence alignment probability distribution sequence, the wild protein sequence, and the corresponding prior amino acid residue influence degree distribution sequence in the protein wet experiment data to obtain a corresponding proximal strategy optimization multi-modal sequence for the protein wet experiment data in the protein wet experiment data set.

[0049] The proximal strategy optimization module is configured to perform proximal strategy optimization on the pre-trained protein language model based on each of the proximal strategy optimization multi-modal sequences.

[0050] In a fourth aspect, embodiments of the present disclosure provide a protein mutation apparatus, which comprises:

[0051] The data to be optimized acquisition module is configured to acquire a protein sequence to be optimized and corresponding multi-sequence alignment and structure sequence.

[0052] The multi-sequence alignment probability distribution sequence generation module is configured to generate a corresponding multi-sequence alignment probability distribution sequence according to the amino acid residue probability distribution of the multi-sequence alignment of the protein sequence to be optimized at each site.

[0053] a multi-modal serial module configured to serially concatenate the structure sequence corresponding to the protein sequence to be optimized, the multiple sequence alignment probability distribution sequence, and the protein sequence to be optimized, to generate a multi-modal sequence to be optimized corresponding to the protein sequence to be optimized;

[0054] a first round prediction module configured to input the multi-modal sequence to be optimized into the pre-trained protein language model to obtain a first round predicted mutant protein amino acid residue probability distribution sequence, wherein the pre-trained protein language model is pre-trained by the method described in any one of the implementations of the first aspect.

[0055] In some optional implementations, the apparatus further comprises:

[0056] a first round mutant sequence acquisition module configured to acquire a first round mutant protein sequence set, wherein the first round mutant protein sequence set is obtained based on first round mutation design of the protein sequence to be optimized based on the first round predicted mutant protein amino acid residue probability distribution sequence;

[0057] a wet experiment score acquisition module configured to acquire wet experiment scores of wet experiments performed on each of the first round mutant protein sequences;

[0058] a first round prior influence degree generation module configured to, for each position in a first round prior amino acid residue influence degree distribution sequence corresponding to the protein sequence to be optimized, determine a prior amino acid residue influence degree distribution of the first round prior amino acid residue influence degree distribution sequence corresponding to the protein sequence to be optimized at the position based on a sum of wet experiment scores of each first round prior mutant protein sequence in which the position is a corresponding amino acid residue identifier in each of the first round prior mutant protein sequences, wherein the first round prior amino acid residue influence degree distribution sequence corresponding to the protein sequence to be optimized has the same sequence length as the protein sequence to be optimized;

[0059] a first round multi-modal sequence to be optimized serial module configured to serially concatenate a structure sequence of the protein sequence to be optimized, a multiple sequence alignment probability distribution sequence, the protein sequence to be optimized, and a corresponding first round amino acid residue prior probability distribution sequence to obtain a first round multi-modal sequence to be optimized;

[0060] a first round mutation module configured to take the first round as a current round, based on the multi-modal sequence to be optimized of the current round and the pre-trained protein language model after supervised fine-tuning and direct preference optimization or after proximal strategy optimization, to perform next round mutation design on the protein sequence to be optimized to obtain a next round mutant protein sequence set of the current round.

[0061] In some optional embodiments, the apparatus further comprises:

[0062] The first-round mutation wet experiment score acquisition module is configured to acquire wet experiment scores of wet experiments performed on each next-round mutant protein sequence of the current round, and update the current round to a next round of the current round.

[0063] The next-round protein mutation design module is configured to perform the following next-round protein mutation design operation: in response to a determination that next-round protein mutation design needs to be performed according to the wet experiment scores of the wet experiments performed on each current-round mutant protein sequence, for each position in the current-round prior amino acid residue influence degree distribution sequence corresponding to the protein sequence to be optimized, determining a prior amino acid residue influence degree distribution of the current-round prior amino acid residue influence degree distribution sequence corresponding to the protein sequence to be optimized at the position based on a sum of the wet experiment scores of each current-round prior mutant protein sequence in which the position is occupied by the corresponding amino acid residue identifier, wherein the current-round prior amino acid residue influence degree distribution sequence corresponding to the protein sequence to be optimized has the same length as the protein sequence to be optimized; concatenating the current-round prior amino acid residue influence degree distribution sequence to the tail of the previous-round protein sequence to be optimized of the current round to obtain a current-round protein sequence to be optimized; performing next-round mutation design on the protein sequence to be optimized based on the current-round protein sequence to be optimized and the pre-trained protein language model that has been supervised and fine-tuned and directly preferred or optimized by a proximal strategy, to obtain a set of next-round mutant protein sequences of the current round; acquiring wet experiment scores of wet experiments performed on each next-round mutant protein sequence of the current round, and updating the current round to a next round of the current round, and continuing to perform the next-round protein mutation design operation.

[0064] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, which, when executed by the one or more processors, cause the one or more processors to implement the method described in any implementation manner of the first aspect and / or the second aspect.

[0065] In a sixth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by one or more processors, implements the method described in any implementation manner of the first aspect and / or the second aspect.

[0066] In a seventh aspect, an embodiment of the present disclosure provides a computer program product comprising computer programs / instructions, which, when executed by a processor, implement the method described in any implementation manner of the first aspect and / or the second aspect.

[0067] Most of the existing multi-modal protein language models use end-to-end generation logic, only pay attention to the mapping of sequences and structures at the input and output layers, and mix the data of multiple modalities through modal parallel or modal mixing, which is difficult to reflect the reasoning process between different modalities and ignores the enhancement effect of the intermediate process on the reasoning quality.

[0068] For example, ESM-3 directly concatenates or linearly adds the embedding vectors of the Sequence, Structure and Function modalities at the input layer to form a mixed input. The features of each modality are forcibly mixed before entering the Transformer layer, and the attention mechanism cannot distinguish "which modality's features are currently being calculated" and cannot independently model each modality's features and interactions at the attention mechanism level.

[0069] In addition, the existing multi-modal protein language models do not introduce MSA as an independent modality, and lack of intermediate reasoning links, so they cannot utilize the reasoning relationship between modalities.

[0070] To solve the problem of missing reasoning process in the existing multi-modal protein language model, the protein language model pre-training and protein mutation method, device, electronic equipment, storage medium and computer program product provided by the embodiments of the present disclosure are provided. The method comprises the following steps: obtaining a sample protein data set, wherein the sample protein data comprises a sample protein sequence and corresponding multiple sequence alignment and structure sequence; for each sample protein data, performing the following multi-modal sequence generation operation: generating a multiple sequence alignment probability distribution sequence corresponding to the sample protein data according to the amino acid residue probability distribution of the multiple sequence alignment at each site in the sample protein data; concatenating the sample protein sequence in the sample protein data and the corresponding multiple sequence alignment probability distribution sequence and structure sequence in sequence in the forward or reverse direction to generate a multi-modal sequence corresponding to the sample protein data; and finally, based on the multi-modal sequence corresponding to each sample protein data, performing autoregressive pre-training on the protein language model to obtain a pre-trained protein language model. The method can achieve the following technical effects:

[0071] First, by calculating the multiple sequence alignment probability distribution sequence from the multiple sequence alignment, the problem of not being able to use MSA as an independent modality in the existing multi-modal protein language model is solved.

[0072] Second, using the multiple sequence alignment probability distribution sequence as an intermediate modality to concatenate the protein sequence and protein structure can enable the multi-modal protein language model to have two thought chains and improve the model prediction performance.

[0073] The forward thinking chain corresponds to the structure prediction task, that is, the sample protein sequence in the sample protein data, the corresponding multiple sequence alignment probability distribution sequence and the structure sequence are sequentially connected in the forward direction to generate a multi-modal sequence corresponding to the sample protein data, and the protein language model is pre-trained through the forward multi-modal sequence. The process of predicting the structure of the protein sequence through the MSA intermediate thinking process can make the structure prediction result more accurate.

[0074] The reverse thinking chain corresponds to the reverse folding task, that is, the sample protein sequence in the sample protein data, the corresponding multiple sequence alignment probability distribution sequence and the structure sequence are sequentially connected in the reverse direction to generate a reverse multi-modal sequence corresponding to the sample protein data, that is, the reverse multi-modal sequence includes the structure sequence, the multiple sequence alignment probability distribution sequence and the sample protein sequence in the sample protein data sequentially connected, and the protein language model is pre-trained through the reverse multi-modal sequence. The process of predicting the structure of the protein structure reverse folding through the MSA intermediate thinking process can make the protein design effect better. BRIEF DESCRIPTION OF DRAWINGS

[0075] Other features, objects, and advantages of the disclosure will become more apparent from the following detailed description of non-limiting embodiments made with reference to the accompanying drawings. The drawings are for the purpose of illustrating specific embodiments and are not to be considered limiting of the application. In the drawings:

[0076] Figure 1 is an exemplary system architecture diagram to which one embodiment of the present disclosure can be applied;

[0077] Figure 2A is a flowchart of one embodiment of the protein language model pre-training method according to the present disclosure;

[0078] Figure 2B is a decomposition flowchart of one embodiment of step 202 according to the present disclosure;

[0079] Figure 2C is a decomposition flowchart of one embodiment of step 2021 according to the present disclosure;

[0080] Figure 2D is a flowchart of one embodiment of the site amino acid residue probability distribution determination operation according to the present disclosure;

[0081] Figure 2E is a flowchart of one embodiment of the prior amino acid residue influence degree distribution sequence generation operation according to the present disclosure;

[0082] Figure 2Fis a decomposition flowchart according to one embodiment of step 2051 of the present disclosure;

[0083] Figure 2G is a decomposition flowchart according to one embodiment of step 2052 of the present disclosure;

[0084] Figure 2H is a decomposition flowchart according to one embodiment of step 209 of the present disclosure;

[0085] Figure 3 is a comparison diagram showing the representation of MSA in a structural class model and the multi-sequence alignment probability distribution sequence in the present disclosure;

[0086] Figure 4 is a diagram of one embodiment of a forward multi-modal sequence according to the present disclosure;

[0087] Figure 5 is a diagram of one embodiment of a reverse multi-modal sequence according to the present disclosure;

[0088] Figure 6 is a diagram of one embodiment of a prior mutant protein sequence set, its corresponding wet experiment score, normalized wet experiment score, and prior amino acid residue prior influence degree distribution sequence according to the present disclosure;

[0089] Figure 7A is a flowchart of one embodiment of a protein mutation method according to the present disclosure;

[0090] Figure 7B is a decomposition flowchart according to one embodiment of step 711 of the present disclosure;

[0091] Figure 7C is a diagram showing one embodiment of a to-be-optimized multi-modal sequence input to the current round of a pre-trained protein language model;

[0092] Figure 8 is a structural diagram of one embodiment of a protein language model pre-training device according to the present disclosure;

[0093] Figure 9 is a structural diagram of one embodiment of a protein mutation device according to the present disclosure;

[0094] Figure 10 is a structural diagram of a computer system of an electronic device suitable for implementing embodiments of the present disclosure. DETAILED DESCRIPTION

[0095] The present disclosure will be further described below in conjunction with the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related application, and not to limit the application. In addition, it should be noted that, for the sake of description, only the parts related to the application are shown in the drawings.

[0096] It should be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict. The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0097] Figure 1 An exemplary system architecture 100 of the embodiments of the protein language model pre-training and protein mutation method, device, electronic equipment and storage medium to which the present disclosure can be applied is shown.

[0098] As shown in Figure 1 The system architecture 100 can include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is a medium for providing a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0099] A user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as protein language model pre-training applications, protein mutation applications, etc.

[0100] The terminal devices 101, 102, 103 can be hardware or software. When the terminal devices 101, 102, 103 are hardware, they can be various electronic devices with information input devices (for example, keyboard, mouse, touch screen, microphone, camera, etc.) and information output devices (for example, display screen, speaker, etc.), including but not limited to smart phones, tablet computers, e-book readers, laptop computers and desktop computers, etc. When the terminal devices 101, 102, 103 are software, they can be installed in the above-mentioned terminal devices. They can be implemented as multiple software or software modules (for example, to provide model pre-training services), or as a single software or software module. No specific limitation is made herein.

[0101] In some cases, the protein language model pre-training and protein mutation method provided by the present disclosure can be executed by the terminal devices 101, 102, 103, and accordingly, the protein language model pre-training and protein mutation device can be arranged in the terminal devices 101, 102, 103. At this time, the system architecture 100 can also not include the server 105.

[0102] In some cases, the protein language model pre-training and protein mutation method provided by the present disclosure can be jointly performed by the terminal devices 101, 102, 103 and the server 105. For example, the step of "obtaining a sample protein data set" can be performed by the terminal devices 101, 102, 103, and the steps of "performing autoregressive pre-training on the protein language model based on the multi-modal sequences corresponding to each sample protein data to obtain a pre-trained protein language model" and the like can be performed by the server 105. The present disclosure does not limit this. Accordingly, the protein language model pre-training apparatus and the protein mutation apparatus can also be respectively arranged in the terminal devices 101, 102, 103 and the server 105.

[0103] In some cases, the protein language model pre-training and protein mutation method provided by the present disclosure can be performed by the server 105, and accordingly, the protein language model pre-training and protein mutation apparatus can also be arranged in the server 105. At this time, the system architecture 100 can also not include the terminal devices 101, 102, 103.

[0104] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (for example, to provide distributed services), or as a single software or software module. This is not specifically limited here.

[0105] It should be understood that Figure 1 The number of terminal devices, networks, and servers in

[0106] Continuing to refer to Figure 2A FIG. 200 shows a flow 200 of one embodiment of a protein language model pre-training method according to the present disclosure, which includes the following steps:

[0107] Step 201, obtaining a sample protein data set.

[0108] Here, the sample protein data can include sample protein sequences and corresponding multiple sequence alignments and structure sequences.

[0109] Here, the sample protein sequence can be an amino acid residue identifier sequence arranged in order by amino acid residue identifiers.

[0110] Here, the sample protein sequence can be obtained by methods including but not limited to direct sequencing, gene sequence deduction, database retrieval, etc.

[0111] The Multiple Sequence Alignment (MSA) corresponding to the sample protein sequence can be obtained, for example, by the following method:

[0112] First, input the sample protein sequence and make sure that the sample protein amino acid residue identifier sequence is a standard amino acid sequence (e.g., FASTA format).

[0113] Then, search for homologous sequences of the sample protein sequence using a sequence search tool. The sequence search tool can be, for example, Hhblits, Jackhmmer, MMseqs2, NCBIBLAST, HMMER (Pfam), InterProScan, etc.

[0114] Next, redundant sequences among the searched homologous sequences can be filtered out.

[0115] Finally, a Multiple Sequence Alignment (MSA) can be generated. For example, HHblits or the like can be used to directly output A3M format (lightweight MSA with insertion / deletion information). Then, other formats such as FASTA, CLUSTAL, etc. can be converted.

[0116] It can be understood that, assuming the length of the sample protein sequence is L, the Multiple Sequence Alignment can be composed of N homologous amino acid residue identifier sequences of length L from the sample protein sequence. For ease of understanding, a sample protein sequence and a corresponding Multiple Sequence Alignment composed of five homologous sequences are given below:

[0117] Sample protein sequence: MALWMRLLPLLALLALWGPDPAAAF

[0118] First homologous sequence: MALWMRLLPLLAL---WGPDPAAAF

[0119] Second homologous sequence: MALWMRLLPLLALLALWGPDP----

[0120] Third homologous sequence: MALWMRLLPLLAL--LWGPDPAAAV

[0121] Fourth homologous sequence: MALWMRLLPLL--LALWGPDPAAAE

[0122] Fifth homologous sequence: MALWMRLLPLLALLALW----AAAF

[0123] Here, the length of the structure sequence is also L, and the structure sequence is arranged by the Cα atomic coordinates (or all-atom coordinates) of each amino acid residue in the sample protein sequence. Among them, the Cα atomic coordinates of the amino acid residue can be obtained, for example, by the following method:

[0124] Obtained from the experimental structure database. For example, Protein Data Bank (PDB), website: https: / / www.rcsb.org / . Or from the cryo-electron microscopy database (EMDB), website: https: / / www.ebi.ac.uk / emdb / Or, it can be obtained by computational prediction method: for example, by using AlphaFold2 prediction.

[0125] Alternatively, the all-atom (37) three-dimensional coordinates of each amino acid residue in the structure sequence can be expanded and then converted into one-dimensional data by a pre-trained structure label quantification model, and then the structure sequence can also be a 1*L matrix.

[0126] Step 202, for each sample protein data, performing a multi-modal sequence generation operation.

[0127] Here, the multi-modal sequence generation operation can include the following steps 2021 and 2022 as shown in Figure 2B

[0128] Step 2021, generating a multiple sequence alignment probability distribution sequence corresponding to the sample protein data according to the amino acid residue probability distribution of the multiple sequence alignment at each site in the sample protein data.

[0129] Here, the multiple sequence alignment probability distribution sequence corresponding to the sample protein data is arranged by the amino acid residue probability distribution of the multiple sequence alignment at each site according to the position of the corresponding site.

[0130] Continue to use the assumption in step 201, the multiple sequence alignment is composed of N homologous amino acid residue identification sequences with a length of L. The multiple sequence alignment probability distribution sequence is arranged by the amino acid residue probability distribution of each site from 1 to the Lth site according to the position of the corresponding site. For a specific site, the amino acid residue probability distribution of the site is composed of the probability of each amino acid residue at the site in the multiple sequence alignment.

[0131] Specifically, step 2021 can include the following steps 20211 and 20212 as shown in Figure 2C

[0132] ​​Step 20211, for each site of the multiple sequence alignment in the sample protein data, a site amino acid residue probability distribution determination operation is performed. The site amino acid residue probability distribution determination operation can include the following steps 202111 and 202112 as shown in 2D:

[0133] Step 202111, for each preset amino acid residue identifier in the preset amino acid residue identifier set, the number of sequences in the multiple sequence alignment in the sample protein data with the amino acid residue identifier at the site being the preset amino acid residue identifier is divided by the total number of sequences in the multiple sequence alignment in the sample protein data, and the ratio is determined as the probability of the multiple sequence alignment in the sample protein data at the site being the preset amino acid residue identifier.

[0134] Here, the preset amino acid residue identifier set can include a natural amino acid residue identifier for indicating a natural amino acid and a preset gap identifier. The preset gap identifier is used to represent an alignment gap of a sequence due to insertion or deletion (indel) in the alignment process. Alternatively, the preset amino acid residue identifier set can include 20 natural amino acid residue identifiers and a preset gap identifier. That is, the preset amino acid residue identifier set can include 21 preset amino acid residue identifiers. In this way, assuming that the multiple sequence alignment in the sample protein data includes N amino acid residue identifier sequences, each with a length of L, i.e. L sites. Then for the i^(i is a positive integer between 1 and L) site of the above multiple sequence alignment, in step 20111, for the j^(j is a positive integer between 1 and 21) preset amino acid residue identifier of the 21 preset amino acid residue identifiers, the number of sequences in the N amino acid residue identifier sequences of the multiple sequence alignment with the j^(j is a positive integer between 1 and 21) preset amino acid residue identifier at the i^(i is a positive integer between 1 and L) site is divided by N, and the ratio Pi,j is calculated as the probability of the multiple sequence alignment in the sample protein data at the i^(i is a positive integer between 1 and L) site being the j^(j is a positive integer between 1 and 21) preset amino acid residue identifier.

[0135] Step 202112, according to the probabilities of the multiple sequence alignment in the sample protein data at the site being each preset amino acid residue identifier, an amino acid residue probability distribution of the multiple sequence alignment in the sample protein data at the site is generated.

[0136] Using the assumptions from step 202111, the probability distribution of amino acid residues at the i-th site of the multiple sequence alignment in the sample protein data can be generated by considering the probability of each of the 21 preset amino acid residue identifiers in the multiple sequence alignment at the i-th site. The probability distribution of amino acid residues at the i-th site of the multiple sequence alignment in the sample protein data includes 21 probability values, and the sum of the above 21 probability values ​​is 1.

[0137] For example, 21 preset amino acid residue identifiers can be arranged in a specified order, and the probability of each preset amino acid residue identifier at the i-th site of the multiple sequence alignment in the protein sample data can be arranged accordingly, generating the amino acid residue probability distribution at the i-th site of the multiple sequence alignment in the protein sample data. As an example, the amino acid residue probability distribution at the i-th site of the multiple sequence alignment can be represented by a one-dimensional array of length 21.

[0138] Step 20212: Based on the order of the multiple sequence alignments in the protein sample data, and the probability distribution of amino acid residues at the corresponding sites in the multiple sequence alignments of the protein sample data, generate the multiple sequence alignment probability distribution sequence corresponding to the protein sample data.

[0139] As an example, a 21-row, L-column matrix (i.e., a 21*L matrix) can be used to represent the multiple sequence alignment probability distribution sequence corresponding to the protein sample data. Here, the i-th column corresponds to the i-th site among the L sites in the multiple sequence alignment, and the j-th row corresponds to the j-th preset amino acid residue identifier among the 21 preset amino acid residue identifiers. The element in the j-th row and i-th column of the matrix is ​​Pi,j, which represents the probability that the multiple sequence alignment in the protein sample data corresponds to the j-th preset amino acid residue identifier at the i-th site.

[0140] After step 2021, the multiple sequence alignment probability distribution sequence corresponding to the protein data of the sample can be obtained. The data dimension of the multiple sequence alignment probability distribution sequence is only related to the length of the multiple sequence alignment and is not related to the number of sequences in the multiple sequence alignment.

[0141] Please refer to the details. Figure 3 , Figure 3 The diagram shows a comparison between the representation of MSA in a structure-based model (i.e., a model for predicting protein structure based on protein sequence) and the multiple sequence alignment probability distribution sequence in this disclosure. Figure 3 The left side of the middle image shows the structure prediction method using MSA in Alphafold. From... Figure 3As can be seen from the left middle figure, in Alphafold2, the MSA is usually input as a juxtaposed sequence. First, each amino acid in the N sequences of length L included in the MSA is directly mapped to a corresponding integer index (Token), that is, an N*L integer matrix can be obtained. Then, the N*L integer matrix is converted to a fixed dimension (such as 512 sequences x 1024 columns) through padding or truncation. The above conversion process increases the computing power or loses information. Since the number of sequences included in the MSA of different proteins is different, the MSA of different proteins needs to be preprocessed independently, which is difficult to realize batch processing for acceleration.

[0142] In addition, the information of the MSA in the structure-based model is raw: first, the MSA often comes from alignment tools (such as HHblits), which contains a large amount of unprocessed evolutionary noise (no processed information can be extracted). Second, in the MSA with insufficient clustering, highly similar sequences are repeated (such as 90% of the same sequence has a high proportion), which leads to sequence redundancy and repeated information.

[0143] In summary, the MSA representation in Alphafold is not suitable for pre-training the protein language model as an independent modality.

[0144] As Figure 3 As shown in the right middle figure, the multiple sequence alignment probability distribution sequence in the present disclosure uses a 21*L matrix to represent the MSA, that is, the dimension of the multiple sequence alignment probability distribution sequence is fixed at 21 and the length is fixed at L, regardless of the number of sequences in the MSA. The multiple sequence alignment probability distribution sequence is pre-trained as an independent modality for the protein language model, which is independent of the number of homologous amino acid residue sequences in the multiple sequence alignment, and is more suitable for pre-training the protein language model as an independent modality, and technically solves the possibility of MSA modality as an independent modality.

[0145] In addition, using the multiple sequence alignment probability distribution sequence as an independent modality can avoid information redundancy compared to the representation of the MSA in the structure-based model.

[0146] In addition, using the multiple sequence alignment probability distribution sequence (21*L probability matrix) as an independent modality, since the amino acid residue probability in the multiple sequence alignment amino acid residue probability distribution is a decimal number, the MSA (N*L integer matrix) can be efficiently concentrated into a continuous representation, which is more suitable as an intermediate "thinking" modality.

[0147] Finally, since the values in the multiple sequence alignment probability distribution sequence are continuous values rather than integer indexes, inputting the continuous values as an independent modality to the protein language model can endow the model with the ability to understand the continuous value matrix, laying a foundation for subsequent implementation of continuous value-driven optimization tasks by the protein language model. For example, subsequent implementation of the protein optimization task by using wet experiment scoring.

[0148] In step 2022, the sample protein sequence in the sample protein data and the corresponding multiple sequence alignment probability distribution sequence and structure sequence are sequentially concatenated in the forward direction or the reverse direction to generate a multi-modal sequence corresponding to the sample protein data.

[0149] Here, the sample protein sequence in the sample protein data and the corresponding multiple sequence alignment probability distribution sequence and structure sequence can be sequentially concatenated in the forward direction to generate a forward multi-modal sequence corresponding to the sample protein data.

[0150] Specifically, the forward multi-modal sequence corresponding to the sample protein data can be represented as follows:

[0151] Sample protein sequence → multiple sequence alignment probability distribution sequence → structure sequence.

[0152] Alternatively, the sample protein sequence in the sample protein data and the corresponding multiple sequence alignment probability distribution sequence and structure sequence can be sequentially concatenated in the reverse direction to generate a reverse multi-modal sequence corresponding to the sample protein data.

[0153] Specifically, the reverse multi-modal sequence corresponding to the sample protein data can be represented as follows:

[0154] Structure sequence → multiple sequence alignment probability distribution sequence → sample protein sequence.

[0155] Optionally, when step 202 is performed, a preset proportion of the sample protein data in the sample protein data set can be concatenated in the forward direction, and the rest can be concatenated in the reverse direction, to ensure that the subsequent protein language model can learn the inference process in both directions. As an example, the preset proportion can be 50%, so that the protein language model can learn the inference process in both directions evenly.

[0156] In step 203, the protein language model is autoregressively pre-trained based on the multi-modal sequence corresponding to each sample protein data to obtain a pre-trained protein language model.

[0157] Here, the protein language model can be various language models for embedding representation, encoding and decoding of protein sequences.

[0158] Optionally, the protein language model can include an embedding representation model and at least one layer of a Transformer.

[0159] In particular, as shown in Figure 4 , Figure 4 In particular, as shown in Figure 4 In particular, as shown in Figure 4 In particular, as shown in Figure 4 In particular, as shown in <sor>AA Sequence <eor> <som>MSA Profile <eom> <sos>StructureTokens <eos>". wherein <sor>and <eor>representing the sample protein sequence (corresponding Figure 4 start and end of the "AA Sequence”, <som>and <eom>respectively, represent the sequence of multiple sequence alignment probability distributions (corresponding Figure 4 MSA Profile” start and end, <sos>and <eos> Figure 4

[0160] <sor>AA Sequence <eor> <som>MSA Profile <eom> <sos>Structure Tokens <eos>"Inputting a protein language model can allow for forward multi-modal sequences" <sor>AA Sequence <eor> <som>MSA Profile <eom> <sos>Structure Tokens <eos>"AA Sequence") start of the sample protein sequence (corresponding Figure 4 "AA Sequence”) start of the sample protein sequence (corresponding <sor>.

[0161] After obtaining the forward multi-modal decoding sequence of the protein language model, the forward multi-modal decoding sequence can be aligned with the input forward multi-modal sequence, and then the loss function between the aligned forward multi-modal decoding sequence and the input forward multi-modal sequence is calculated, and the parameter of the protein language model is optimized based on the calculated loss function.

[0162] It can be understood that, by using the forward multi-modal sequence shown in Figure 4 , autoregressive pre-training is performed on the protein language model, so that the protein language model has a forward thinking chain, and the structure prediction task can be realized, that is, based on the protein sequence, through the multi-sequence alignment probability distribution sequence as an intermediate thinking process, the protein structure is predicted.

[0163] Please refer to Figure 5 , Figure 5 , the sample protein sequences in the sample protein data (corresponding to "AA Sequence" in Figure 5 ) and the corresponding multi-sequence alignment probability distribution sequences (corresponding to "MSA Profile" in Figure 5 ) and structure sequences (corresponding to "Structure Tokens" in Figure 5 ) are generated in reverse order, and the reverse multi-modal sequence corresponding to the sample protein data is generated <sos>Structure Tokens <eos> <som>MSA Profile <eom> <sor>AA Sequence <eor>". wherein <sor> 、 <eor> 、 <som> 、 <eom> 、 <sos>and <eos>have the same meaning as described above for Figure 4 and are not repeated here.

[0164] The above reverse multimodal sequence <sos>Structure Tokens <eos> <som>MSA Profile <eom> <sor>AA Sequence <eor>"Inputting a protein language model can allow for reverse multi-modal sequences" <sos>StructureTokens <eos> <som>MSA Profile <eom> <sor>AA Sequence <eor>"Structure Tokens") start of the inputted sequence. The embedding representation, feature encoding and decoding, and the reverse multimodal decoding sequence can be understood as follows. It is understood that the reverse multimodal decoding sequence is shifted one position forward (or left) relative to the inputted reverse multimodal sequence, i.e. less tokens are used to represent the structure sequence (corresponding to Figure 5 the "Structure Tokens”) start of the inputted sequence. The embedding representation, feature encoding and decoding, and the reverse multimodal decoding sequence can be understood as follows. It is understood that the reverse multimodal decoding sequence is shifted one position forward (or left) relative to the inputted reverse multimodal sequence, i.e. less tokens are used to represent the structure sequence (corresponding to <sos>.

[0165] After obtaining the reverse multi-modal decoding sequence of the protein language model, the reverse multi-modal decoding sequence can be aligned with the input reverse multi-modal sequence, and then a loss function between the aligned reverse multi-modal decoding sequence and the input reverse multi-modal sequence is calculated, and the parameter of the protein language model is optimized based on the calculated loss function.

[0166] It can be understood that, by using the forward multi-modal sequence as shown in Figure 5 , autoregressive pre-training of the protein language model can make the protein language model have a reverse thinking chain, and can realize the protein design task, i.e., based on the structure sequence, through the multi-sequence alignment probability distribution sequence as an intermediate thinking process, the mutation design of the protein is realized.

[0167] After at least one round of parameter optimization, a pre-trained protein language model can be obtained.

[0168] It should be noted that, during the entire process of autoregressive pre-training of the protein language model based on the multi-modal sequence corresponding to each sample protein data, the sample protein sequence, the multi-sequence alignment probability distribution sequence and the structure sequence all exist as independent modalities, i.e., during the entire process from input to output, including embedding representation, encoding and decoding, the three modalities all exist independently, and there is no operation such as adding the embedding representation of one modality to the embedding representation of another modality, or using the embedding representation of a modality as a bias of attention in the transformer architecture (such as in Alphafold2). That is, the final model also outputs three independent modalities.

[0169] Optionally, when the attention mechanism is performed inside the protein language model, attention calculation is performed inside each modality and between any two modalities, but the modalities still exist independently, and there is no fusion between any two modalities.

[0170] In some optional embodiments, the multi-modal sequence generation operation can further include the following step 2023 as shown in Figure 2B :

[0171] Step 2023: generating a rotation position encoding sequence based on the position of each sample protein marker in the sample protein sequence in the sample protein data, and concatenating the generated rotation position encoding sequence for three times in succession to obtain a multi-modal position encoding sequence corresponding to the sample protein data.

[0172] For example, when the sample protein sequence has a length of L, the rotation position encoding sequence of each of the three modes in the multi-modal position encoding sequence is one-to-one corresponding, and the rotation position encoding of each mode is respectively a sequence of 0, 1, …, (L-1) corresponding rotation position encoding sequences. The multi-modal position encoding sequence corresponding to the sample protein data can be obtained by concatenating the rotation position encoding sequence of a mode for three times in succession.

[0173] Correspondingly, step 203 can be executed as follows: based on the multi-modal sequence and the multi-modal position encoding sequence corresponding to each sample protein data, the protein language model is self- recurrently pre-trained to obtain a pre-trained protein language model.

[0174] That is, in the process of inputting the multi-modal sequence (forward multi-modal sequence or reverse multi-modal sequence) into the protein language model for embedding representation, the rotation position encoding sequence corresponding to each independent mode can also be input synchronously. That is, the token sequence and the position sequence of each independent mode are input into the protein language model as independent modes.

[0175] After obtaining the pre-trained protein language model through step 203, in order to enable the pre-trained protein language model to design mutations for the wild protein amino acid residue identification sequence based on the experimental results of wet experiments, it is also necessary to supervise the fine-tuning and / or preference optimization of the pre-trained protein language model, therefore in some optional embodiments, the above protein language model pre-training method 200 can further execute the following steps 204 and 205 after executing step 203 to generate a prior amino acid residue influence degree distribution sequence corresponding to each protein wet experiment data in the protein wet experiment data set, and then based on the generated prior amino acid residue influence degree distribution sequence, the pre-trained protein language model is supervised and / or preference optimized according to actual conditions:

[0176] Step 204: obtaining a protein wet experiment data set.

[0177] Here, the protein wet experiment data can include a wild protein sequence and corresponding multiple sequence alignment probability distribution sequence, structure sequence, and a mutant protein wet experiment data set. Among them, the mutant protein wet experiment data can include a mutant protein sequence and a corresponding wet experiment score.

[0178] Here, the obtaining method of the wild protein sequence and the corresponding multiple sequence alignment probability distribution sequence and structure sequence can refer to the related description in steps 201 and 202, which will not be repeated here.

[0179] Here, at least one mutant protein sequence is obtained by pre-designing at least one mutation for the wild protein sequence in the protein wet experiment data, and a corresponding wet experiment is performed on each corresponding mutant protein sequence and a wet experiment score is obtained.

[0180] The mutant protein wet experiment data set can be obtained, for example, by the following approaches: public database, literature and supplementary data, high-throughput experiment data set, etc.

[0181] At step 205, for the protein wet experiment data in the protein wet experiment data set, a prior amino acid residue influence degree distribution sequence generation operation is performed.

[0182] Here, the prior amino acid residue influence degree distribution sequence generation operation can include the following steps 2051 and 2052 as shown in Figure 2E

[0183] At step 2051, based on the wet experiment scores of the mutant protein sequences in the protein wet experiment data, a preset number of mutant protein sequences are selected from the mutant protein sequences in the protein wet experiment data, and a prior mutant protein sequence set is generated.

[0184] Since the number of mutant protein sequences in the protein wet experiment data is huge, not all mutant protein sequences are suitable for supervised fine-tuning and / or preference optimization of the pre-trained protein language model, and therefore a part of the mutant protein sequences need to be selected as prior knowledge for supervised fine-tuning and / or preference optimization of the pre-trained protein language model. For this purpose, various implementation manners can be adopted, based on the wet experiment scores of the mutant protein sequences in the protein wet experiment data, a preset number of mutant protein sequences are selected from the mutant protein sequences in the protein wet experiment data, and a prior mutant protein sequence set is generated.

[0185] As an example, the top preset number of mutant protein sequences with the highest wet experiment scores can be selected from the mutant protein sequences in the protein wet experiment data, and a prior mutant protein sequence set is generated.

[0186] As another example, in order to more simulate the process of protein mutation and wet experiment in actual production process, step 2051 can also include the following steps 20511 to 20514 as shown in Figure 2F

[0187] At step 20511, the structure sequence, the multiple sequence alignment probability distribution sequence and the wild protein sequence in the protein wet experiment data are sequentially concatenated to obtain a multi-modal sequence corresponding to the protein wet experiment data.

[0188] ​​Step 20512: Input the multimodal sequence corresponding to the protein wet experimental data into the pre-trained protein language model to obtain the predicted amino acid residue probability distribution at each position in the predicted amino acid residue probability distribution sequence.

[0189] The multimodal sequence generated in step 20511 concatenates the structural sequence, multiple sequence alignment probability distribution sequence, and wild-type protein sequence from the wet experimental data of the protein. This corresponds to the backfolding task, i.e., designing protein mutations through structure. Therefore, by inputting the multimodal sequence generated in step 20511 into the pre-trained protein language model, the predicted amino acid residue probability distribution at each position in the predicted amino acid residue probability distribution sequence can be obtained.

[0190] Here, assuming the length of the wild-type protein sequence is L, the length of the predicted amino acid residue probability distribution sequence is also L. That is, the predicted amino acid residue probability distribution sequence is composed of L predicted amino acid residue probability distributions arranged sequentially from position 1 to position L. Each position's predicted amino acid residue probability distribution includes the predicted probability of each preset amino acid residue identifier in the preset amino acid residue identifier set. Assuming the preset amino acid residue identifier set has 20 natural amino acid identifiers and 1 preset alignment gap identifier, then step 20512 yields the predicted probability value of each of the 21 preset amino acid identifiers at each position from position 1 to position L in the predicted amino acid residue probability distribution sequence of length L. This is the predicted amino acid residue probability distribution at each position from position 1 to position L in the predicted amino acid residue probability distribution sequence.

[0191] In practice, the 21-row, L-column logits matrix output by the Transformer in the protein language model can be normalized by softmax and used as the predicted amino acid residue probability distribution for each position in the predicted amino acid residue probability distribution sequence.

[0192] Step 20513: For each mutant protein sequence in the wet experimental data of the protein, determine the predicted probability corresponding to the mutant protein sequence based on the predicted amino acid residue probability distribution sequence.

[0193] Specifically, assuming the length of the wild-type protein sequence is L, the lengths of each mutant protein sequence corresponding to the wild-type protein sequence and the predicted amino acid residue probability distribution sequence are all L. The probability that the i-th site (i is a positive integer between 1 and L) in the predicted amino acid residue probability distribution sequence is the j-th preset amino acid residue identifier is Pi,j, where j is a positive integer between 1 and the total number of preset amino acid residue identifiers J (e.g., 21).

[0194] For a specific mutant protein sequence, the amino acid residue at the i-th position in the mutant protein sequence is denoted as A i The predicted probability P corresponding to the mutant protein sequence can be calculated according to the following formula mutant :

[0195]

[0196] That is, the predicted probability corresponding to the mutant protein sequence is the product of the probability of each position in the predicted amino acid residue probability distribution sequence being the amino acid residue identification of the corresponding position in the mutant protein sequence.

[0197] Step 20514, selecting the top pre-set number of mutant protein sequences with the highest corresponding predicted probability from each mutant protein sequence in the protein wet experiment data to generate a set of prior mutant protein sequences.

[0198] After step 20513, the predicted probability of each mutant protein sequence in the protein wet experiment data can be obtained, and the top pre-set number of mutant protein sequences with the highest corresponding predicted probability are selected from each mutant protein sequence in the protein wet experiment data. These mutant protein sequences have a higher probability predicted by the pre-trained protein language model, but the actual wet experiment scores of these mutant protein sequences may be higher or lower, which is uncertain. This is also more in line with the actual production process, because in the actual process of designing mutations based on wild protein sequences, it is also uncertain whether the wet experiment score of the mutant protein sequence obtained by mutation design will be higher than that of the wild protein sequence. However, at least the pre-trained protein language model considers these mutant protein sequences to be better, which are suitable for subsequent supervised fine-tuning and / or preference optimization of the pre-trained protein language model (i.e., simulating the wet experiment of these prior mutant protein sequences in the actual production process and obtaining the wet experiment score), that is, the set of prior mutant protein sequences can be generated using these mutant protein sequences with higher predicted probability.

[0199] As an example, assume that the protein wet experiment data includes 3000 mutant protein sequences, and 50 mutant protein sequences with the highest predicted probability are selected to generate a set of prior mutant protein sequences.

[0200] Step 2052, for each position in the prior amino acid residue influence degree distribution sequence corresponding to the protein wet experiment data, based on the sum of the wet experiment scores of each mutant protein sequence in which the position is identified by the corresponding amino acid residue in each prior mutant protein sequence, determine the prior amino acid residue influence degree distribution of the position.

[0201] Here, the length of the prior amino acid residue influence degree distribution sequence corresponding to the protein wet experiment data is the same as the length of the wild protein sequence.

[0202] Here, assuming that the length of the wild protein sequence is L, and the length of each prior mutant protein sequence is also L, the length of the prior amino acid residue influence degree distribution sequence corresponding to the protein wet experiment data is also L, that is, the prior amino acid residue influence degree distribution sequence is composed of L prior amino acid residue influence degree distributions from the first position to the Lth position. Among them, the prior amino acid residue influence degree distribution of each position includes the influence degree of each preset amino acid residue identifier in the preset amino acid residue identifier set. Assuming that the preset amino acid residue identifier set has 20 natural amino acid identifiers and 1 preset alignment vacancy identifier, then after step 2052, the influence degree of each amino acid identifier in the 21 preset amino acid identifiers from the first position to the Lth position in the prior amino acid residue influence degree distribution sequence with a length of L can be obtained, that is, the prior amino acid residue influence degree distribution of each position from the first position to the Lth position in the prior amino acid residue influence degree distribution sequence.

[0203] Specifically, step 2052 can include the following steps 20521 and 20522 as shown in Figure 2G

[0204] Step 20521, normalizing the wet experiment scores of each prior mutant protein sequence to obtain the corresponding normalized wet experiment scores.

[0205] Here, various normalization methods can be used to normalize the wet experiment scores of each prior mutant protein sequence determined in step 2051 to obtain the normalized wet experiment scores of the corresponding prior mutant protein sequence.

[0206] Step 20522, for each position in the prior amino acid residue influence degree distribution sequence corresponding to the protein wet experiment data, based on the sum of the normalized wet experiment scores of each mutant protein sequence in each prior mutant protein sequence with the corresponding amino acid residue identifier at the position, determining the prior amino acid residue influence degree distribution of the prior amino acid residue influence degree distribution sequence corresponding to the protein wet experiment data at the position.

[0207] For ease of understanding, please refer to Figure 6 , Figure 6 ​The first column from the left is the set of prior mutant protein sequences, assuming that the set of prior mutant protein sequences includes M prior mutant protein sequences with length L, the second column is the wet experiment score corresponding to each prior mutant protein sequence, and the third column is the normalized wet experiment score corresponding to each prior mutant protein sequence. For example, the set of prior mutant protein sequences can include a wild protein sequence, and the wet experiment score of each prior mutant protein sequence can be the wet experiment score relative to the wild protein sequence. If the wet experiment score of a prior mutant protein sequence is higher than the wet experiment score of the wild protein sequence (e.g., the second prior mutant protein sequence), it is displayed in green, and if the wet experiment score of a prior mutant protein sequence is lower than the wet experiment score of the wild protein sequence (e.g., the second prior mutant protein sequence), it is displayed in red (e.g., the third and fourth prior mutant protein sequences). Figure 6 The white prior mutant protein sequence in the first row is the wild protein sequence, and the wet experiment score of each prior mutant protein sequence can be the wet experiment score relative to the wild protein sequence. If the wet experiment score of a prior mutant protein sequence is higher than the wet experiment score of the wild protein sequence (e.g., the second prior mutant protein sequence), it is displayed in green, and if the wet experiment score of a prior mutant protein sequence is lower than the wet experiment score of the wild protein sequence (e.g., the second prior mutant protein sequence), it is displayed in red (e.g., the third and fourth prior mutant protein sequences). Figure 6 The white prior mutant protein sequence in the first row is the wild protein sequence, and the wet experiment score of each prior mutant protein sequence can be the wet experiment score relative to the wild protein sequence. If the wet experiment score of a prior mutant protein sequence is higher than the wet experiment score of the wild protein sequence (e.g., the second prior mutant protein sequence), it is displayed in green, and if the wet experiment score of a prior mutant protein sequence is lower than the wet experiment score of the wild protein sequence (e.g., the second prior mutant protein sequence), it is displayed in red (e.g., the third and fourth prior mutant protein sequences). Figure 6 The white prior mutant protein sequence in the first row is the wild protein sequence, and the wet experiment score of each prior mutant protein sequence can be the wet experiment score relative to the wild protein sequence. If the wet experiment score of a prior mutant protein sequence is higher than the wet experiment score of the wild protein sequence (e.g., the second prior mutant protein sequence), it is displayed in green, and if the wet experiment score of a prior mutant protein sequence is lower than the wet experiment score of the wild protein sequence (e.g., the second prior mutant protein sequence), it is displayed in red (e.g., the third and fourth prior mutant protein sequences).

[0208] Assuming that there are J preset amino acid residue identifiers in the preset amino acid residue identifier (e.g., J can be 21). Here, for each position (e.g., the i-th position, i is a positive integer between 1 and L) in the prior amino acid residue influence degree distribution sequence, for each preset amino acid residue identifier (e.g., the j-th preset amino acid residue identifier, j is a positive integer between 1 and J) in the set of preset amino acid residue identifiers, the sum of the normalized wet experiment scores of the prior mutant protein sequences in the M prior mutant protein sequences at the position (e.g., the i-th position) with the preset amino acid residue identifier (e.g., the j-th preset amino acid residue identifier) can be determined as the prior amino acid residue influence degree of the prior amino acid residue influence degree distribution sequence at the position (e.g., the i-th position) with the preset amino acid residue identifier (e.g., the j-th preset amino acid residue identifier). The prior amino acid residue influence degrees of the prior amino acid residue influence degree distribution sequence at the position (e.g., the i-th position) with each preset amino acid residue identifier (e.g., J preset amino acid residue identifiers) constitute the prior amino acid residue influence degree distribution at the position of the prior amino acid residue influence degree distribution sequence corresponding to the protein wet experiment data, i.e., the prior amino acid residue influence degree distribution at the position of the prior amino acid residue influence degree distribution sequence includes J prior amino acid residue influence degrees. The prior amino acid residue influence degree distribution sequence corresponding to the protein wet experiment data has J*L elements, which can be represented as a J-row L-column matrix, and when J is 21, it can be represented as a 21*L matrix.

[0209] Here, the prior amino acid residue influence degree distribution sequence at the position (e.g., the ith position) is the prior amino acid residue influence degree of the preset amino acid residue identifier (e.g., the jth preset amino acid residue identifier), which can be used to represent the wet experiment score according to each prior protein sequence. In the subsequent process of designing mutations of the wild protein sequence to generate a new mutant protein sequence, the influence degree (or influence weight) of the new mutant protein sequence at the position (e.g., the ith position) of the preset amino acid residue identifier (e.g., the jth preset amino acid residue identifier) can be obtained.

[0210] After steps 204 and 205, the set of prior mutant protein sequences and the prior amino acid residue influence degree distribution sequence corresponding to the protein wet experiment data can be obtained.

[0211] In some optional embodiments, after step 205 is performed, steps 206 to 208 can be performed.

[0212] Step 206: Among the mutant protein sequences in the protein wet experiment data other than the prior mutant protein sequences, a mutant protein sequence corresponding to a wet experiment score greater than that of the wild protein sequence in the protein wet experiment data is selected as a target mutant protein sequence.

[0213] Here, continuing with the above example, for example, the protein wet experiment data includes 3000 mutant protein sequences, of which 50 are prior mutant protein sequences. Here, one of the 2950 (3000-50) mutant protein sequences other than the prior mutant protein sequences can be selected as a target mutant protein sequence, and the wet experiment score of the target mutant protein sequence is greater than that of the wild protein sequence in the protein wet experiment data. That is, the target mutant protein sequence performs better than the wild protein sequence in the wet experiment score item, and can be used as a target for mutation of the wild protein sequence.

[0214] Step 207: The structure sequence, the multiple sequence alignment probability distribution sequence, the wild protein sequence, and the corresponding prior amino acid residue influence degree distribution sequence and the target mutant protein sequence in the protein wet experiment data are sequentially concatenated to obtain a corresponding supervised fine-tuning multi-modal sequence.

[0215] Here, i.e., the formed supervised fine-tuning multi-modal sequence includes five modalities, wherein: the structure sequence in the protein wet experiment data, the wild protein sequence and the target mutant protein sequence can each be a 1*L matrix, and the multiple sequence alignment probability distribution sequence and the prior amino acid residue influence degree distribution sequence can each be a J*L matrix, J may be 21, for example. Since the lengths of the five modalities are each L, and the dimensions of the multiple sequence alignment probability distribution sequence and the prior amino acid residue influence degree distribution sequence are the same, the pre-trained protein language model can easily understand the above five modalities.

[0216] In step 208, the pre-trained protein language model is fine-tuned based on each supervised fine-tuning multi-modal sequence.

[0217] It should be noted that the supervised fine-tuning multi-modal sequence in step 207 is obtained by concatenating the structure sequence in the protein wet experiment data, the multiple sequence alignment probability distribution sequence, the wild protein sequence, and the corresponding prior amino acid residue influence degree distribution sequence and the target mutant protein sequence. Using the above supervised fine-tuning multi-modal sequence to fine-tune the pre-trained protein language model is equivalent to giving the pre-trained protein model some prior knowledge, including: (1) the structure sequence; (2) the information of natural evolution, i.e., the multiple sequence alignment probability distribution sequence; (3) the wild protein sequence; (4) the information of directed evolution, i.e., the prior amino acid residue influence degree distribution sequence determined according to the mutation design and wet experiment scoring of the wild protein sequence. In this way, the pre-trained protein language model can further generate mutant protein sequences based on the above prior knowledge and try to learn and approach the target mutant protein sequence.

[0218] Specifically, step 208 can be performed as follows: for the supervised fine-tuning multi-modal sequence, the pre-trained protein language model is fine-tuned with the supervised fine-tuning multi-modal sequence as input data and the target mutant protein sequence in the supervised fine-tuning multi-modal sequence as supervised data.

[0219] It can be understood that, here, in addition to inputting the supervised fine-tuning multi-modal sequence into the pre-trained protein language model, the supervised fine-tuning multi-modal position encoding sequence corresponding to the above supervised fine-tuning multi-modal sequence can also be inputted synchronously.

[0220] Here, since the supervised fine-tuning multi-modal sequence includes five independent modalities, the corresponding supervised fine-tuning multi-modal position encoding sequence can be a supervised fine-tuning multi-modal position encoding sequence obtained by concatenating the rotation position encoding sequence of one modality five times in succession, wherein the rotation position encoding sequence of each modality is one-to-one corresponding, and the rotation position encoding of each modality is respectively the sequence of sequence numbers: 0, 1, …, (L-1) corresponding to the rotation position encoding sequence.

[0221] The steps 206 to 208 can enable the pre-trained protein language model to learn to generate better (i.e., higher wet-lab score) mutant sequences, and can enable the pre-trained protein language model to adapt to the input of four concatenated independent modalities and five concatenated independent modalities.

[0222] In some optional embodiments, the following steps 209 and 210 can be performed after the step 205 is performed or after the step 208 is performed:

[0223] The step 209, for the protein wet-lab data in the protein wet-lab data set, performs a preference-optimized multi-modality sequence pair generation operation.

[0224] Here, the preference-optimized multi-modality sequence pair generation operation can include the following steps 2091 to 2094 as shown in FIG. 21: Figure 2H

[0225] The step 2091, selects a preferred mutant protein sequence and a less-preferred mutant protein sequence from among the other mutant protein sequences in the protein wet-lab data, except for the prior mutant protein sequence.

[0226] Here, the wet-lab score of the preferred mutant protein sequence is greater than the wet-lab score of the less-preferred mutant protein sequence.

[0227] The step 2092, concatenates the structure sequence, the multiple sequence alignment probability distribution sequence, the wild protein sequence, and the corresponding prior amino acid residue impact degree distribution sequence and the preferred mutant protein sequence in the protein wet-lab data in sequence to obtain a corresponding preferred multi-modality sequence.

[0228] Here, the preferred multi-modality sequence includes five modalities, wherein: the structure sequence, the wild protein sequence, and the preferred mutant protein sequence in the protein wet-lab data can each be a 1*L matrix, and the multiple sequence alignment probability distribution sequence and the prior amino acid residue impact degree distribution sequence can each be a J*L matrix, J can be 21 for example. Since the lengths of the five modalities are each L, and the multiple sequence alignment probability distribution sequence and the prior amino acid residue impact degree distribution sequence have the same dimension, the pre-trained protein language model can easily understand the above five modalities.

[0229] The step 2093, concatenates the structure sequence, the multiple sequence alignment probability distribution sequence, the wild protein sequence, and the corresponding prior amino acid residue impact degree distribution sequence and the less-preferred mutant protein sequence in the protein wet-lab data in sequence to obtain a corresponding less-preferred multi-modality sequence.

[0230] ​Similarly, the suboptimal multi-modal sequence includes five modalities, wherein: the structure sequence in the protein wet experiment data, the wild protein sequence, and the suboptimal mutant protein sequence can each be a 1*L matrix, and the multiple sequence alignment probability distribution sequence and the prior amino acid residue influence degree distribution sequence can each be a J*L matrix, J can be 21, for example. Since the lengths of the five modalities are each L, and the multiple sequence alignment probability distribution sequence and the prior amino acid residue influence degree distribution sequence have the same dimension, the pre-trained protein language model can easily understand the above five modalities.

[0231] Step 2094, taking the optimal multi-modal sequence and the suboptimal multi-modal sequence of the protein wet experiment data as a preferred optimization multi-modal sequence pair of the protein wet experiment data.

[0232] Step 210, based on each preferred optimization multi-modal sequence pair, directly performing preference optimization on the pre-trained protein language model.

[0233] Here, the pre-trained protein language model can be a protein language model pre-trained through steps 201 to 203, or a protein language model supervised fine-tuned through steps 204 to 206.

[0234] Specifically, step 210 can be performed as follows:

[0235] First, for the preferred optimization multi-modal sequence pair, the optimal multi-modal sequence and the suboptimal multi-modal sequence in the preferred optimization multi-modal sequence pair are respectively input into the pre-trained protein language model to obtain a first predicted amino acid residue probability distribution sequence and a second predicted amino acid residue probability distribution sequence.

[0236] Here, assuming that the length of the wild protein sequence is L, then the length of the first predicted amino acid residue probability distribution sequence and the second predicted amino acid residue probability distribution sequence is also L, that is, the first predicted amino acid residue probability distribution sequence and the second predicted amino acid residue probability distribution sequence are each arranged in order from a total of L predicted amino acid residue probability distributions from the 1st position to the Lth position. The predicted amino acid residue probability distribution at each position includes the predicted probability of each preset amino acid residue identifier in the preset amino acid residue identifier set. Assuming that the preset amino acid residue identifier set has 20 natural amino acid identifiers and 1 preset alignment vacancy identifier, then the first predicted amino acid residue probability distribution sequence and the second predicted amino acid residue probability distribution sequence can each include a predicted probability value of each amino acid identifier in the 21 preset amino acid identifiers from the 1st position to the Lth position, that is, the predicted amino acid residue probability distribution at each position from the 1st position to the Lth position.

[0237] In practice, the 21 rows and L columns of logits matrix output by the Transformer in the pre-trained protein language model can be normalized by softmax to obtain the predicted amino acid residue probability distribution at each position in the first predicted amino acid residue probability distribution sequence and the second predicted amino acid residue probability distribution sequence.

[0238] It can be understood that, here, in addition to inputting the preferred multi-modal sequence and the non-preferred multi-modal sequence in the preference optimization multi-modal sequence pair into the pre-trained protein language model respectively,

[0239] The preferred multi-modal position encoding sequence or the non-preferred multi-modal position encoding sequence corresponding to the preferred multi-modal sequence or the non-preferred multi-modal sequence can also be inputted synchronously.

[0240] Here, since the preferred multi-modal sequence or the non-preferred multi-modal sequence includes five independent modalities, the corresponding preferred multi-modal position encoding sequence or non-preferred multi-modal position encoding sequence can be a preferred multi-modal position encoding sequence or non-preferred multi-modal position encoding sequence obtained by concatenating the rotation position encoding sequence of one modality for five times in succession, where the rotation position encoding sequence number of each modality is one-to-one corresponding, and the rotation position encoding of each modality is respectively a sequence of numbers: 0, 1, …, (L-1) corresponding rotation position encoding sequence.

[0241] Then, based on the first predicted amino acid residue probability distribution sequence and the second predicted amino acid residue probability distribution sequence, the direct preference loss is calculated.

[0242] Finally, the pre-trained protein language model is directly preference optimized by maximizing the direct preference loss.

[0243] In the above optional implementation, the direct preference optimization of the pre-trained protein language model is realized by the preference optimization multi-modal sequence pair including five independent modalities, which can make the pre-trained protein language model learn to generate mutant protein sequences in the direction of higher wet experiment scoring.

[0244] In some optional implementations, the following steps 211 and 212 can be performed after step 205 is performed or after step 208 is performed:

[0245] Step 211: for the protein wet experiment data in the protein wet experiment data set, the structure sequence, the multiple sequence alignment probability distribution sequence, the wild protein sequence, and the corresponding prior amino acid residue influence degree distribution sequence in the protein wet experiment data are concatenated in turn to obtain the corresponding proximal strategy optimization multi-modal sequence.

[0246] Here, the proximal policy optimization multi-modal sequence includes four modalities, wherein: the structure sequence in the protein wet experiment data and the wild protein sequence are both 1*L matrices, while the multiple sequence alignment probability distribution sequence and the prior amino acid residue influence degree distribution sequence can both be J*L matrices, and J may be 21, for example. Since the lengths of the four modalities are all L, and the dimensions of the multiple sequence alignment probability distribution sequence and the prior amino acid residue influence degree distribution sequence are the same, the pre-trained protein language model can easily understand the above four modalities.

[0247] In step 212, the pre-trained protein language model is proximal policy optimized based on the proximal policy optimization multi-modal sequence.

[0248] Here, the pre-trained protein language model can be a protein language model pre-trained through steps 201 to 203, or a protein language model supervised fine-tuned through steps 204 to 206.

[0249] Specifically, step 212 can be performed as follows:

[0250] First, the proximal policy optimization multi-modal sequence is input into the pre-trained protein language model to obtain a third predicted amino acid residue probability distribution sequence.

[0251] Here, assuming that the length of the wild protein sequence is L, the third predicted amino acid residue probability distribution sequence is arranged in order by L predicted amino acid residue probability distributions from the 1st position to the Lth position. The predicted amino acid residue probability distribution at each position includes the predicted probability of each preset amino acid residue identifier in the preset amino acid residue identifier set. Assuming that the preset amino acid residue identifier set has 20 natural amino acid identifiers and 1 preset alignment vacancy identifier, the third predicted amino acid residue probability distribution sequence includes the predicted probability value of each amino acid identifier in the 21 preset amino acid identifiers from the 1st position to the Lth position, that is, the predicted amino acid residue probability distribution at each position from the 1st position to the Lth position.

[0252] In practice, the 21 rows and L columns of logits matrix output by the Transformer in the pre-trained protein language model after softmax normalization can be used as the predicted amino acid residue probability distribution at each position of the third predicted amino acid residue probability distribution sequence.

[0253] It can be understood that, in addition to inputting the proximal policy optimization multi-modal sequence into the pre-trained protein language model, a proximal policy optimization multi-modal position encoding sequence corresponding to the proximal policy optimization multi-modal sequence can also be input simultaneously.

[0254] Here, since the proximal policy optimization multi-modal sequence includes four independent modalities, the corresponding proximal policy optimization multi-modal position encoding sequence can be a proximal policy optimization multi-modal position encoding sequence obtained by concatenating the rotation position encoding sequence of one modality four times in succession, where the rotation position encoding sequence number of each modality is one-to-one corresponding, and the rotation position encoding of each modality is respectively a sequence of 0, 1, …, (L-1).

[0255] Then, based on the third predicted amino acid residue probability distribution sequence, a proximal policy optimization loss is determined.

[0256] Here, in the experimental design, multiple scoring functions such as structural similarity and thermal stability score are introduced as core reward functions. The loss function used in the training process is the standard proximal policy optimization (PPO) clipping policy loss.

[0257] Finally, based on the proximal policy optimization loss, the pre-trained protein language model is optimized.

[0258] In the above optional implementation, the proximal policy optimization of the pre-trained protein language model is realized by the proximal policy optimization multi-modal sequence including four independent modalities, which can make the pre-trained protein language model learn to generate mutant protein sequences in the direction of higher wet experiment scoring.

[0259] The protein language model pre-training method provided by the above embodiments of the present disclosure includes: obtaining a sample protein data set, wherein the sample protein data includes a sample protein sequence, and corresponding multiple sequence alignment and structure sequence; for each sample protein data, performing the following multi-modal sequence generation operation: generating a multiple sequence alignment probability distribution sequence corresponding to the sample protein data according to the amino acid residue probability distribution of the multiple sequence alignment at each site in the sample protein data; concatenating the sample protein sequence in the sample protein data and the corresponding multiple sequence alignment probability distribution sequence and structure sequence in turn in a forward or reverse direction to generate a multi-modal sequence corresponding to the sample protein data; and finally, based on the multi-modal sequence corresponding to each sample protein data, performing autoregressive pre-training on the protein language model to obtain a pre-trained protein language model. The method can achieve the following technical effects, including but not limited to:

[0260] First, by calculating the multiple sequence alignment probability distribution sequence from the multiple sequence alignment, the problem that the existing multi-modal protein language model cannot use MSA as an independent modality is solved.

[0261] Second, using the multi-sequence alignment probability distribution sequence as an intermediate modality connecting protein sequence and protein structure can give the multimodal protein language model two thought chains, thereby improving the model's predictive performance.

[0262] Among them, the forward thinking chain corresponds to the structure prediction task, which is to generate a multimodal sequence corresponding to the protein data by sequentially linking the sample protein sequence, the corresponding multi-sequence alignment probability distribution sequence, and the structural sequence in the sample protein data. The protein language model is pre-trained by the forward multimodal sequence, which makes the structure prediction result more accurate in the process of predicting structure through protein sequence by the intermediate thinking process of MSA.

[0263] The reverse thinking chain corresponds to the reverse folding task, which involves sequentially concatenating the sample protein sequence, the corresponding multi-sequence alignment probability distribution sequence, and the structural sequence in the sample protein data to generate a reverse multimodal sequence corresponding to the sample protein data. In other words, the reverse multimodal sequence includes the structural sequence, the multi-sequence alignment probability distribution sequence, and the sample protein sequence from the sample protein data in a sequentially concatenated manner. By pre-training the protein language model with the reverse multimodal sequence, the multimodal protein language model can perform inference through the intermediate thinking process of MSA during the protein design process based on protein structure reverse folding, resulting in better protein design performance.

[0264] The following is for reference. Figure 7A The diagram illustrates a flow 700 of an embodiment of a protein mutation method according to the present disclosure. This protein mutation method includes the following steps:

[0265] Step 701: Obtain the protein sequence to be optimized, as well as the corresponding multiple sequence alignment and structural sequence.

[0266] Here, the protein sequence to be optimized, as well as the corresponding multiple sequence alignment and structural sequence, can also be obtained by the same or similar methods as described in step 201.

[0267] Step 702: Generate the corresponding multiple sequence alignment probability distribution sequence based on the amino acid residue probability distribution at each site of the multiple sequence alignment of the protein sequence to be optimized.

[0268] Here, we can use Figure 2A In the illustrated embodiment, the same or similar method described in step 2021 is used to execute step 702 to obtain the multiple sequence alignment probability distribution sequence corresponding to the protein sequence to be optimized.

[0269] In step 703, the structure sequence corresponding to the protein sequence to be optimized, the multiple sequence alignment probability distribution sequence, and the protein sequence to be optimized are sequentially concatenated to generate a protein multi-modal sequence to be optimized corresponding to the protein sequence to be optimized.

[0270] Here, the generated protein multi-modal sequence to be optimized includes three independent modalities: the structure sequence corresponding to the protein sequence to be optimized, the multiple sequence alignment probability distribution sequence, and the protein sequence to be optimized.

[0271] In step 704, the protein multi-modal sequence to be optimized is input into the pre-trained protein language model to obtain a first-round predicted mutant protein amino acid residue probability distribution sequence.

[0272] Here, the pre-trained protein language model can be obtained by pre-training the protein language model according to the protein language model pre-training method as shown in the embodiments and optional embodiments thereof. Figure 2A

[0273] Here, by concatenating the structure sequence corresponding to the protein sequence to be optimized, the multiple sequence alignment probability distribution sequence, and the protein sequence to be optimized in the reverse folding task thinking chain, the protein multi-modal sequence to be optimized is input into the pre-trained protein language model to obtain the first-round predicted mutant protein amino acid residue probability distribution sequence.

[0274] Here, assuming that the length of the protein sequence to be optimized is L, and the preset amino acid residue identifier set has J amino acid residue identifiers, then according to the above, the length of the structure sequence and the multiple sequence alignment probability distribution sequence is L, and each multi-modal alignment probability distribution in the multiple sequence alignment probability distribution sequence includes the probability of each preset amino acid residue identifier in the J preset amino acid residue identifiers. That is, the multiple sequence alignment probability distribution sequence can be represented as a J*L matrix. Here, the first-round predicted mutant protein amino acid residue probability distribution sequence includes the predicted probability of each preset amino acid residue identifier (for example, the jth preset amino acid residue identifier, j is a positive integer between 1 and J) in the J preset amino acid residue identifiers at each position (for example, the ith position, i is a positive integer between 1 and L) from the 1st position to the Lth position.

[0275] With the above first-round predicted mutant protein amino acid residue probability distribution sequence, a professional technical personnel can design mutations for the protein sequence to be optimized based on the above first-round predicted mutant protein amino acid residue probability distribution sequence using various methods, and obtain a plurality of mutant protein sequences.

[0276] In some optional embodiments, the protein mutation method can further include steps 705 to 709 of the flow 700.

[0277] ​Step 705: Obtain the first set of mutant protein sequences.

[0278] Here, the first-round set of mutant protein sequences is obtained by designing the first-round mutation of the protein sequence to be optimized based on the first-round predicted amino acid residue probability distribution sequence of the mutant protein.

[0279] For example, the first round of mutation design can be performed manually. Alternatively, various automated mutation methods can be used for the first round of mutation design, but this is not the focus of this invention and will not be elaborated here.

[0280] Then, wet experiments can be performed on each first-round mutated protein sequence, and a corresponding wet experiment score can be obtained. Here, the wet experiment score can be a score based on a preset wet experiment index, such as stability.

[0281] Step 706: Obtain the wet experiment score for each first-round mutant protein sequence.

[0282] Step 707: For each position in the first-round prior amino acid residue influence distribution sequence corresponding to the protein sequence to be optimized, based on the sum of the wet experimental scores of each first-round prior mutant protein sequence that is identified by the corresponding amino acid residue at that position in each first-round prior mutant protein sequence, determine the prior amino acid residue influence distribution of the first-round prior amino acid residue influence distribution sequence corresponding to the protein sequence to be optimized at that position.

[0283] Here, the first-round prior amino acid residue influence distribution sequence corresponding to the protein sequence to be optimized has the same sequence length as the protein sequence to be optimized.

[0284] Here, we can use Figure 2G In the illustrated embodiment, the same or similar method as step 2052 is used to execute step 702 to obtain the multiple sequence alignment probability distribution sequence corresponding to the protein sequence to be optimized. Specifically, each first-round prior mutant protein sequence in step 702 corresponds to each prior mutant protein sequence in step 2052, and the protein sequence to be optimized in step 702 corresponds to the wild-type protein sequence in the wet experimental data of that protein in step 2052.

[0285] Step 708: The structural sequence of the protein sequence to be optimized, the multi-sequence alignment probability distribution sequence, the protein sequence to be optimized, and the corresponding first-round amino acid residue prior probability distribution sequence are sequentially linked to obtain the first-round multimodal sequence to be optimized.

[0286] At step 709, taking the first round as the current round, based on the current round to-be-optimized multi-modal sequence and the pre-trained protein language model after supervised fine-tuning and direct preference optimization or after proximal strategy optimization, the next round mutation design is performed on the to-be-optimized protein sequence to obtain a next round mutant protein sequence set of the current round.

[0287] Here, various implementation manners are adopted to perform next round mutation design on the to-be-optimized protein sequence based on the current round to-be-optimized multi-modal sequence and the pre-trained protein language model after supervised fine-tuning and direct preference optimization or after proximal strategy optimization, to obtain a next round mutant protein sequence set of the current round. As an example, two implementation manners are given below:

[0288] Optionally, step 709 can be executed as follows: inputting the current round to-be-optimized multi-modal marker sequence into the pre-trained protein language model after supervised fine-tuning and direct preference optimization or after proximal strategy optimization to obtain a next round mutant protein marker sequence set of the current round. That is, here a direct generation manner can be adopted to directly obtain a next round mutant protein marker sequence set of the current round.

[0289] Optionally, step 709 can also be executed as follows:

[0290] First, obtain a target mutant protein prompt sequence for backbone generation of the to-be-optimized protein marker sequence based on the amino acid residue prior probability distribution sequence in the current round to-be-optimized multi-modal marker sequence.

[0291] Here, the target mutant protein prompt sequence is used to represent that at least one backbone position in the next round mutant protein sequence is not mutated.

[0292] As an example, the professional technician expects that the first to 100th positions in the next round mutant protein sequence are unchanged, the 101st to 105th positions are mutated, and the 106th to 200th positions are not mutated, according to the amino acid residue prior probability distribution sequence in the current round to-be-optimized multi-modal marker sequence, actual production needs, and conventional experience of mutation design. Then, the target protein prompt marker sequence can include the amino acid residue identifiers between the first to 100th positions which are not mutated, and the amino acid residue identifiers between the 106th to 200th positions.

[0293] Then, the target mutant protein prompt marker sequence is concatenated at the tail of the current round to-be-optimized multi-modal marker sequence.

[0294] After concatenation, the current round to-be-optimized multi-modal marker sequence includes, in sequence, the structure sequence of the to-be-optimized protein sequence, the multiple sequence alignment probability distribution sequence, the to-be-optimized protein sequence and the corresponding amino acid residue prior probability distribution sequence, and the target mutant protein prompt marker sequence.

[0295] Finally, input the current round of multi-modal label sequence to be optimized into the pre-trained protein language model after supervised fine-tuning and direct preference optimization or after proximal strategy optimization, to obtain the next round of mutant protein label sequence set of the current round.

[0296] The mutant protein sequence can be controlled in preference according to actual conditions by using the above skeleton generation method.

[0297] Through steps 705 to 709, the protein sequence to be optimized can be mutated based on the first round of wet experiment scoring to obtain the next round of mutant protein label sequence set of the first round, i.e., the second round of mutant protein label sequence set. Further, the above second round of mutant protein label sequence can be subjected to wet experiment, and the corresponding wet experiment score can be obtained.

[0298] In some optional embodiments, the above protein graph mutation method process 700 can further include steps 710 and 711:

[0299] Step 710: obtaining the wet experiment score of the wet experiment of each next round of mutant protein sequence of the current round, and updating the current round to the next round of the current round.

[0300] Step 711: performing next round of protein mutation design operation.

[0301] Here, the next round of protein mutation design operation can include the following steps 7101 to 710X as shown in Figure 7B

[0302] Step 7111: determining whether next round of protein mutation design is needed according to the wet experiment score of each current round of mutant protein sequence.

[0303] Here, the determination of whether next round of protein mutation design is needed according to the wet experiment score of each current round of mutant protein sequence can be made manually or by using various automatic determination methods. If it is determined that next round of protein mutation design is needed, step 7112 can be continued to be executed, otherwise step 711 can be ended.

[0304] Step 7112: for each position in the current round of prior amino acid residue influence degree distribution sequence corresponding to the protein sequence to be optimized, based on the sum of the wet experiment scores of each current round of prior mutant protein sequence in which the position is the corresponding amino acid residue identifier in each current round of prior mutant protein sequence, the prior amino acid residue influence degree distribution of the current round of prior amino acid residue influence degree distribution sequence corresponding to the protein sequence to be optimized at the position is determined.

[0305] ​Here, the specific operation of step 7112 and the technical effects produced thereby are basically the same as those of the operation and effects of step 706 described above, and thus will not be described here again.

[0306] Here, the sequence of the current round prior amino acid residue influence degree distribution corresponding to the protein sequence to be optimized is the same as the sequence length of the protein sequence to be optimized.

[0307] Step 7113: concatenating the sequence of the current round prior amino acid residue influence degree distribution to the tail of the last round of the multi-modal sequence to be optimized in the current round to obtain a multi-modal sequence to be optimized in the current round.

[0308] Step 7114: based on the multi-modal sequence to be optimized in the current round and the pre-trained protein language model optimized by supervised fine-tuning and direct preference optimization or by proximal strategy optimization, performing next round mutation design on the protein sequence to be optimized to obtain a set of next round mutant protein sequences in the current round.

[0309] Here, the specific operation of step 7114 and the technical effects produced thereby are basically the same as those of the operation and effects of step 708 described above, and thus will not be described here again.

[0310] That is, the multi-modal sequence to be optimized in the current round can include, in sequence: the structure sequence of the protein sequence to be optimized, the multi-sequence alignment probability distribution sequence, the protein sequence to be optimized, and the amino acid residue prior probability distribution sequence of each round from the first round to the current round.

[0311] For ease of understanding, please refer to Figure 7C , Figure 7C shows a schematic diagram of one embodiment of the multi-modal sequence to be optimized in the current round input to the pre-trained protein language model, as Figure 7C shown, the structure sequence of the protein sequence to be optimized (corresponding to "Structure Sequence" in Figure 8 ), the multi-sequence alignment probability distribution sequence (corresponding to "MSA Profile" in Figure 8 ), the protein sequence to be optimized (corresponding to "WT Sequence" in Figure 8 ), and the amino acid residue prior probability distribution sequence of each round from the first round (corresponding to "Profile D1" in Figure 8 ) to the current round (corresponding to "Profile D2" in Figure 8 ) are concatenated in sequence to obtain the multi-modal sequence to be optimized in the current round (i.e., the second round). Inputting the above multi-modal sequence to be optimized in the current round (i.e., the second round) into the pre-trained protein language model can obtain at least one next round mutant protein sequence (corresponding to Mut Sequence in Figure 8 ).

[0312] After step 7114 is executed, wet experiments can be performed on the next round of mutant protein marker sequences of each current round, and corresponding wet experiment scores are obtained.

[0313] Step 7115 obtains wet experiment scores of wet experiments performed on each next round of mutant protein sequences of the current round, and updates the current round to the next round of the current round, and continues to perform the next round of protein mutation design operation.

[0314] That is, after obtaining the wet experiment scores of the wet experiments performed on each next round of mutant protein sequences of the current round, and updating the current round to the next round of the current round, it goes to step 7111 to continue.

[0315] The protein mutation method provided by the above embodiments of the present disclosure can achieve the following technical effects, including but not limited to:

[0316] First, by introducing the multiple sequence alignment probability distribution sequence as an intermediate modality to form the forward thinking chain and the reverse thinking chain in the pre-training process of the protein language model, the thinking chain is extended in the process of mutation design of the protein, and the amino acid residue prior probability distribution sequence of each round in the mutation design optimization process is added as a new modality based on the reverse multi-modal sequence, which is equivalent to introducing a directional evolution mechanism, which can continuously recommend new and better mutant protein sequences by the pre-trained protein language model, and optimize the characteristics of the protein, such as stability, affinity and other characteristics.

[0317] Second, through the round-by-round optimization, the mutant protein gradually improves the performance in each iteration towards the target attribute. Here, the target attribute refers to the target attribute for which the wet experiment score is scored, such as stability, affinity and other characteristics.

[0318] Third, by using the multiple sequence alignment probability distribution sequence as the guide of the directional evolution direction, the generation direction can be controlled, and the cost of wet experiments can be reduced.

[0319] Fourth, it has generalization ability: this method is not limited to a single property such as thermal stability, and is suitable for multi-attribute optimization.

[0320] Fifth, in existing wet-and-dry fusion protein language models, before further optimizing a new round of mutant protein sequences based on the previous round of mutant protein sequences using the protein language model, it is necessary to fine-tune the protein language model using wet experimental scoring of the previous round of mutant protein sequences. In other words, a training cycle is required before the fine-tuned protein language model can recommend new rounds of mutant protein sequences. However, the protein mutation method provided in the above embodiments of this disclosure adopts a method similar to in-context learning (ICL) in natural language processing. It does not require any fine-tuning of the protein language model; it directly uses the pre-trained protein language model to perform round-by-round maturation optimization of the protein sequences to be optimized, saving costs and increasing the speed of the entire cycle. Sixth, through the fusion of experimental feedback, it truly achieves iterative optimization through collaborative efforts between the model and wet experiments.

[0321] Example 1. Mutation Effect Prediction Task (ProteinGym Benchmark):

[0322] The applicant evaluated the model on the standard ProteinGym Benchmark dataset, training the protein language model on data including different modalities. While maintaining a consistent model parameter size (150MB), the following comparative results were obtained: using only the protein sequence modality, the model scored 0.375; while using a protein sequence + structure dual modality, the model scored 0.406. Figure 2A The protein language model based on the bidirectional thinking chain structure, which consists of three independent modalities cascaded together, as shown in the example, achieved a score of 0.489. This demonstrates that introducing multiple sequence alignment probability distribution sequences as intermediate inference modalities significantly enhances the protein language model's ability to understand mutation effects.

[0323] Example 2. Protein 3D Structure Prediction Task (PDB Data Split Dataset):

[0324] In the structural prediction of 449 PDB test samples, the representative multimodal model ESM-3 (1.4B model parameters) had a TM-score of 0.794; while using Figure 2A The protein language model (650M parameters) based on the bidirectional thinking chain structure, which is a concatenated three independent modalities, shows that the TM-score is improved to 0.850 while significantly reducing the model complexity, demonstrating the superiority and efficiency of this method in structural modeling.

[0325] Example 3. Inverse Folding on PDB Dataset:

[0326] In the reverse folding test on 393 protein structure samples, the amino acid recovery rate of the ESM-3 (model parameter amount 1.4B) model reached 41.05%, and the TM-score was 0.862.

[0327] While the protein language model (model parameter amount 650M) based on the three independent modalities in series of the bidirectional thinking chain structure shown by the embodiment of Figure 2A achieved better performance while maintaining a smaller parameter size, with an amino acid recovery rate of 59.22% and a TM-score of 0.918.

[0328] In summary, the intermediate modality design of the multiple sequence alignment probability distribution sequence and the three-modality series autoregressive pre-training strategy significantly improve the generalization ability, prediction accuracy and controllability of the model in key protein design tasks. The related evaluation results not only surpass the existing level in multiple authoritative benchmarks, but also indicate the application potential and engineering transformation value of the technology in actual biological design.

[0329] Further referring to Figure 8 , as an implementation of the method shown in the above figures, the disclosure provides an embodiment of a protein language model pre-training device, which corresponds to the method embodiment shown in Figure 2A , and the device can be applied in various electronic devices.

[0330] As shown in Figure 8 , the protein language model pre-training device 800 of the embodiment includes a sample data acquisition module 801, a multi-modal sequence generation module 802, and a pre-training module 803. The sample data acquisition module 801 is configured to acquire a set of sample protein data, wherein the sample protein data includes a sample protein sequence, a corresponding multiple sequence alignment, and a structure sequence. The multi-modal sequence generation module 802 is configured to, for each sample protein data, perform the following multi-modal sequence generation operation: generating a multiple sequence alignment probability distribution sequence corresponding to the sample protein data according to the amino acid residue probability distribution of the multiple sequence alignment at each site in the sample protein data; and sequentially concatenating the sample protein sequence in the sample protein data, the corresponding multiple sequence alignment probability distribution sequence, and the structure sequence in series in the forward or reverse direction to generate a multi-modal sequence corresponding to the sample protein data. The pre-training module 803 is configured to perform autoregressive pre-training on a protein language model based on the multi-modal sequence corresponding to each of the sample protein data, to obtain a pre-trained protein language model.

[0331] In the embodiment, the specific processing of the sample data obtaining module 801, the multi-modal sequence generating module 802 and the pre-training module 803 of the protein language model pre-training apparatus 800 and the technical effects brought by the same can be respectively referred to the related descriptions of steps 201, 202 and 203 in the corresponding embodiments, which will not be repeated here. Figure 2A The related descriptions of steps 201, 202 and 203 in the corresponding embodiments will not be repeated here.

[0332] In some optional embodiments, the generating the multi-sequence alignment probability distribution sequence corresponding to the sample protein data according to the amino acid residue probability distribution of each position in the multi-sequence alignment in the sample protein data can include:

[0333] For each position of the multi-sequence alignment in the sample protein data, the following position amino acid residue probability distribution determination operation is performed: for each preset amino acid residue identifier in the preset amino acid residue identifier set, the number of sequences in which the amino acid residue identifier at the position in the multi-sequence alignment in the sample protein data is the sequence number of the preset amino acid residue identifier divided by the total number of sequences in the multi-sequence alignment in the sample protein data, and the probability of the multi-sequence alignment in the sample protein data at the position being the preset amino acid residue identifier is determined; and the amino acid residue probability distribution of the multi-sequence alignment in the sample protein data at the position is generated according to the probabilities of the multi-sequence alignment in the sample protein data at the position being each of the preset amino acid residue identifiers.

[0334] According to the order of each position in the multi-sequence alignment in the sample protein data, the multi-sequence alignment probability distribution sequence corresponding to the sample protein data is generated based on the amino acid residue probability distribution of the multi-sequence alignment in the sample protein data at the corresponding position.

[0335] In some optional embodiments, the apparatus 800 can further include:

[0336] The wet experiment data obtaining module 804 is configured to obtain a set of protein wet experiment data, wherein the protein wet experiment data includes a wild protein sequence and a corresponding multi-sequence alignment probability distribution sequence, a structure sequence and a set of mutant protein wet experiment data, and the mutant protein wet experiment data includes a mutant protein sequence and a corresponding wet experiment score.

[0337] The prior influence degree generation module 805 is configured to perform the following prior amino acid residue influence degree distribution sequence generation operation on the protein wet experiment data in the protein wet experiment data set: based on the wet experiment scores of each mutant protein sequence in the protein wet experiment data, selecting a preset number of mutant protein sequences from each mutant protein sequence in the protein wet experiment data to generate a prior mutant protein sequence set; for each position in the prior amino acid residue influence degree distribution sequence corresponding to the protein wet experiment data, determining the prior amino acid residue influence degree of the position based on the sum of the wet experiment scores of each mutant protein sequence with the corresponding amino acid residue identifier at the position in each of the prior mutant protein sequences, wherein the prior amino acid residue influence degree distribution sequence corresponding to the protein wet experiment data is the same as the sequence length of the wild protein sequence.

[0338] In some optional embodiments, the device 800 can further include:

[0339] The preference pair generation module 806 is configured to perform the following preference optimization multi-modal sequence pair generation operation on the protein wet experiment data in the protein wet experiment data set: selecting a preferred mutant protein sequence and a less preferred mutant protein sequence from the other mutant protein sequences in each of the mutant protein sequences of the protein wet experiment data except for each of the prior mutant protein sequences, wherein the wet experiment score of the preferred mutant protein sequence is greater than the wet experiment score of the less preferred mutant protein sequence; sequentially concatenating the structure sequence, the multiple sequence alignment probability distribution sequence, the wild protein sequence, and the corresponding prior amino acid residue influence degree distribution sequence of the protein wet experiment data and the preferred mutant protein sequence to obtain a corresponding preferred multi-modal sequence; sequentially concatenating the structure sequence, the multiple sequence alignment probability distribution sequence, the wild protein sequence, and the corresponding prior amino acid residue influence degree distribution sequence of the protein wet experiment data and the less preferred mutant protein sequence to obtain a corresponding less preferred multi-modal sequence; and taking the preferred multi-modal sequence and the less preferred multi-modal sequence of the protein wet experiment data as the preference optimization multi-modal sequence pair of the protein wet experiment data.

[0340] The direct preference optimization module 807 is configured to perform direct preference optimization on the pre-trained protein language model based on each of the preference optimization multi-modal sequence pairs.

[0341] In some optional embodiments, the device 800 can further include:

[0342] The proximal strategy optimization multi-modal sequence serial module 808 is configured to sequentially serialize the structure sequence, the multi-sequence alignment probability distribution sequence, the wild protein sequence, and the corresponding prior amino acid residue influence degree distribution sequence in the protein wet experiment data in the protein wet experiment data set to obtain the corresponding proximal strategy optimization multi-modal sequence.

[0343] The proximal strategy optimization module 809 is configured to perform proximal strategy optimization on the pre-trained protein language model based on the proximal strategy optimization multi-modal sequences.

[0344] It should be noted that the implementation details and technical effects of each module in the protein language model pre-training apparatus provided by the embodiments of the present disclosure can refer to the descriptions of other embodiments in the present disclosure, which will not be described here.

[0345] The following refers to Figure 9 As an implementation of the method shown in the above figures, the present disclosure provides an embodiment of a protein mutation apparatus, which corresponds to the method embodiment shown in Figure 7A The apparatus can be applied in various electronic devices.

[0346] As shown in Figure 9 The protein mutation apparatus 900 of the present embodiment includes a data to be optimized acquisition module 901, a multi-sequence alignment probability distribution sequence generation module 902, a multi-modal serial module 903, and a first round prediction module 904. The data to be optimized acquisition module 901 is configured to acquire a protein sequence to be optimized and corresponding multi-sequence alignment and structure sequence.

[0347] The multi-sequence alignment probability distribution sequence generation module 902 is configured to generate a corresponding multi-sequence alignment probability distribution sequence according to the amino acid residue probability distribution of the multi-sequence alignment of the protein sequence to be optimized at each site.

[0348] The multi-modal serial module 903 is configured to sequentially serialize the structure sequence corresponding to the protein sequence to be optimized, the multi-sequence alignment probability distribution sequence, and the protein sequence to be optimized to generate a protein multi-modal sequence to be optimized corresponding to the protein sequence to be optimized.

[0349] The first round prediction module 904 is configured to input the protein multi-modal sequence to be optimized into the pre-trained protein language model to obtain a first round predicted mutant protein amino acid residue probability distribution sequence. The pre-trained protein language model is obtained by pre-training the method described in the embodiments and optional implementation manners thereof. Figure 2A

[0350] ​In the embodiment, the specific processing of the data acquisition module 901, the multiple sequence alignment probability distribution sequence generation module 902, the multi-modal concatenation module 903, and the first round prediction module 904 of the protein mutation device 900 to be optimized and the technical effects brought by the same can be referred to Figure 7A The related descriptions of steps 701, 702, 703, and 704 in the corresponding embodiment will not be repeated here.

[0351] In some optional embodiments, the device 900 can further include:

[0352] The first round mutant sequence acquisition module 905 is configured to acquire a first round mutant protein sequence set, wherein the first round mutant protein sequence set is obtained based on the first round predicted mutant protein amino acid residue probability distribution sequence to the protein sequence to be optimized for first round mutation design;

[0353] The wet experiment score acquisition module 906 is configured to acquire wet experiment scores of wet experiments on each of the first round mutant protein sequences;

[0354] The first round prior influence degree generation module 907 is configured to, for each position in the first round prior amino acid residue influence degree distribution sequence corresponding to the protein sequence to be optimized, determine the prior amino acid residue influence degree distribution of the first round prior amino acid residue influence degree distribution sequence corresponding to the protein sequence to be optimized at the position based on the sum of the wet experiment scores of each first round prior mutant protein sequence in which the position is the corresponding amino acid residue identifier in each of the first round prior mutant protein sequences. The first round prior amino acid residue influence degree distribution sequence corresponding to the protein sequence to be optimized has the same sequence length as the protein sequence to be optimized;

[0355] The first round to-be-optimized multi-modal sequence concatenation module 908 is configured to sequentially concatenate the structure sequence of the protein sequence to be optimized, the multiple sequence alignment probability distribution sequence, the protein sequence to be optimized, and the corresponding first round amino acid residue prior probability distribution sequence to obtain a first round to-be-optimized multi-modal sequence.

[0356] The first round mutation module 909 is configured to take the first round as the current round, based on the current round to-be-optimized multi-modal sequence and the pre-trained protein language model optimized by supervised fine-tuning and direct preference optimization or by proximal strategy optimization, to perform next round mutation design on the protein sequence to be optimized to obtain a next round mutant protein sequence set of the current round.

[0357] In some optional embodiments, the device 900 can further include:

[0358] The first-round mutation wet experiment score acquisition module 910 is configured to acquire wet experiment scores of wet experiments performed on each next-round mutant protein sequence of the current round, and update the current round to a next round of the current round;

[0359] The next-round protein mutation design module 911 is configured to perform the following next-round protein mutation design operation: in response to a determination that next-round protein mutation design needs to be performed according to the wet experiment scores of the wet experiments performed on each current-round mutant protein sequence, for each position in the current-round prior amino acid residue influence degree distribution sequence corresponding to the protein sequence to be optimized, determining a prior amino acid residue influence degree distribution of the current-round prior amino acid residue influence degree distribution sequence corresponding to the protein sequence to be optimized at the position based on a sum of the wet experiment scores of each current-round prior mutant protein sequence in which the position is marked with a corresponding amino acid residue in each current-round prior mutant protein sequence, wherein the current-round prior amino acid residue influence degree distribution sequence corresponding to the protein sequence to be optimized has the same length as the protein sequence to be optimized; concatenating the current-round prior amino acid residue influence degree distribution sequence at the tail of the previous-round protein sequence to be optimized of the current round to obtain a current-round protein sequence to be optimized; performing next-round mutation design on the protein sequence to be optimized based on the current-round protein sequence to be optimized and the pre-trained protein language model that has been supervised and fine-tuned and directly preferred or optimized by a proximal strategy, to obtain a set of next-round mutant protein sequences of the current round; acquiring wet experiment scores of wet experiments performed on each next-round mutant protein sequence of the current round, and updating the current round to a next round of the current round, and continuing to perform the next-round protein mutation design operation.

[0360] It should be noted that the implementation details and technical effects of each module of the protein mutation device provided by the embodiments of the present disclosure can refer to the descriptions of other embodiments of the present disclosure, which will not be repeated here.

[0361] The following refers to Figure 10 which shows a structural schematic diagram of a computer system 1000 suitable for implementing an electronic device of the present disclosure. Figure 10 The computer system 1000 shown is merely an example and should not impose any limitations on the functions and use ranges of the embodiments of the present disclosure.

[0362] As Figure 10 As shown, the computer system 1000 can include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1001 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 1002 or loaded into a random access memory (RAM) 1003 from a storage device 1008. Various programs and data required for the operation of the computer system 1000 are also stored in the RAM 1003. The processing device 1001, the ROM 1002, and the RAM 1003 are connected to each other through a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0363] Generally, the following devices can be connected to the I / O interface 1005: input devices 1006 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, etc.; output devices 1007 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1008 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication devices 1009 can allow the computer system 1000 to communicate with other devices wirelessly or through wires to exchange data. Although Figure 10 The computer system 1000 is shown with various devices, but it should be understood that not all of the shown devices are required to implement or have the computer system 1000. More or fewer devices can alternatively be implemented or have.

[0364] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 1009, or installed from the storage devices 1008, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of embodiments of the present disclosure are performed.

[0365] It should be noted that the computer readable medium in the above disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of computer readable storage media can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present disclosure, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take many forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained in the computer readable medium can be transmitted by any suitable medium, including but not limited to a wire, a cable, an RF (radio frequency) or the like, or any suitable combination of the above.

[0366] The computer readable medium described above can be contained in the electronic device described above; or can exist separately and not be assembled into the electronic device.

[0367] The computer readable medium described above carries one or more programs, which, when executed by the electronic device, cause the electronic device to implement the method as shown in Figure 2A The protein language model pre-training method and / or the protein mutation method as shown in the embodiments and optional implementation modes thereof. Figure 7A The protein language model pre-training method and / or the protein mutation method as shown in the embodiments and optional implementation modes thereof.

[0368] Computer program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Python, Java, Smalltalk, C++, or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0369] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0370] The modules involved in the embodiments of the present disclosure can be implemented by software or by hardware. In some cases, the name of a module does not limit the module itself. For example, a sample data acquisition module can also be described as a "module for acquiring a sample protein data set".

[0371] The above description is merely that of preferred embodiments of the present disclosure and of the principles thereof. Those skilled in the art should understand that the scope of the disclosure is not limited to the specific combinations of technical features described above, and should also encompass other technical solutions formed by any combination of the above technical features or equivalent features, without departing from the above disclosed concept. For example, the above technical features can be replaced with technical features disclosed in the present disclosure (but not limited to) having similar functions to form technical solutions.< / sos> < / eor> < / sor> < / eom> < / som> < / eos> < / sos> < / eor> < / sor> < / eom> < / som> < / eos> < / sos> < / eos> < / sos> < / eom> < / som> < / eor> < / sor> < / eor> < / sor> < / eom> < / som> < / eos> < / sos> < / sor> < / eos> < / sos> < / eom> < / som> < / eor> < / sor> < / eos> < / sos> < / eom> < / som> < / eor> < / sor> ​​​< / eos> < / sos> < / eom> < / som> < / eor> < / sor> < / eos> < / sos> < / eom> < / som> < / eor> < / sor>

Claims

1. A protein language model pre-training method, comprising: Obtain a set of sample protein data, which includes sample protein sequences and corresponding multiple sequence alignments and structural sequences; For each sample protein data, the following multimodal sequence generation operation is performed: Based on the probability distribution of amino acid residues at each site in the multi-sequence alignment of the sample protein data, a multi-sequence alignment probability distribution sequence corresponding to the sample protein data is generated; The sample protein sequence, the corresponding multi-sequence alignment probability distribution sequence, and the structural sequence in the sample protein data are concatenated in either a forward or reverse manner to generate a multimodal sequence corresponding to the sample protein data. The protein language model is pre-trained by autoregression based on the multimodal sequences corresponding to the protein data of each sample, thus obtaining the pre-trained protein language model.

2. The method according to claim 1, wherein, The step of generating the multiple sequence alignment probability distribution sequence corresponding to the protein sample data based on the amino acid residue probability distribution at each site in the multiple sequence alignment of the protein sample data includes: For each site of multiple sequence alignment in the protein sample data, the following site amino acid residue probability distribution determination operation is performed: For each preset amino acid residue identifier in the preset amino acid residue identifier set, the probability that the multiple sequence alignment in the protein sample data identifies the amino acid residue at that site as the preset amino acid residue identifier is determined by dividing the number of sequences in the multiple sequence alignment in the protein sample data that identify the amino acid residue at that site by the preset amino acid residue identifier by the ratio of the number of sequences in the multiple sequence alignment in the protein sample data to the total number of sequences in the multiple sequence alignment in the protein sample data; Based on the probability that the multiple sequence alignment in the protein sample data identifies each preset amino acid residue identifier at that site, the amino acid residue probability distribution of the multiple sequence alignment in the protein sample data at that site is generated. Based on the order of the multiple sequence alignments in the protein sample data, and the probability distribution of amino acid residues at the corresponding sites in the multiple sequence alignments of the protein sample data, a multiple sequence alignment probability distribution sequence corresponding to the protein sample data is generated.

3. The method according to claim 1, wherein, The method further includes: A set of wet protein experimental data is obtained, including wild-type protein sequences and their corresponding multiple sequence alignment probability distribution sequences, structural sequences, and a set of wet protein experimental data for mutant proteins. The wet protein experimental data for mutant proteins includes mutant protein sequences and their corresponding wet experimental scores. For the wet protein experimental data in the set of wet protein experimental data, the following prior amino acid residue influence distribution sequence generation operation is performed: based on the wet experimental scores of each mutant protein sequence in the wet protein experimental data, a preset number of mutant protein sequences are selected from each mutant protein sequence in the wet protein experimental data to generate a set of prior mutant protein sequences; for each position in the prior amino acid residue influence distribution sequence corresponding to the wet protein experimental data, the prior amino acid residue influence distribution at that position is determined based on the sum of the wet experimental scores of each mutant protein sequence in each of the prior mutant protein sequences that is identified by the corresponding amino acid residue at that position, wherein the prior amino acid residue influence distribution sequence corresponding to the wet protein experimental data has the same sequence length as the wild-type protein sequence.

4. The method according to claim 3, wherein, The method further includes: For the protein wet experimental data in the aforementioned protein wet experimental dataset, the following preference-optimized multimodal sequence pair generation operation is performed: Among the mutant protein sequences in the protein wet experimental data, excluding the prior mutant protein sequences, preferred mutant protein sequences and undesirable protein sequences are selected, wherein the wet experimental score of the preferred mutant protein sequence is greater than the wet experimental score of the undesirable protein sequence; the structural sequence, multiple sequence alignment probability distribution sequence, wild-type protein sequence, and the corresponding prior amino acid residue influence distribution sequence from the protein wet experimental data are sequentially concatenated with the preferred mutant protein sequence to obtain the corresponding preferred multimodal sequence; the structural sequence, multiple sequence alignment probability distribution sequence, wild-type protein sequence, and the corresponding prior amino acid residue influence distribution sequence from the protein wet experimental data are sequentially concatenated with the undesirable mutant protein sequence to obtain the corresponding undesirable multimodal sequence; the preferred multimodal sequence and undesirable multimodal sequence from the protein wet experimental data are used as the preference-optimized multimodal sequence pair for the protein wet experimental data; Based on the aforementioned preferences, the pre-trained protein language model is directly optimized using multimodal sequence pairs.

5. The method according to claim 3, wherein, The method further includes: for the protein wet experimental data in the protein wet experimental dataset, sequentially concatenating the structural sequence, multiple sequence alignment probability distribution sequence, wild protein sequence, and corresponding prior amino acid residue influence distribution sequence in the protein wet experimental data to obtain the corresponding proximal strategy optimization multimodal sequence. Based on the aforementioned proximal strategies, multimodal sequences are optimized, and proximal strategy optimization is performed on the pre-trained protein language model.

6. A method for protein mutation, comprising: Obtain the protein sequence to be optimized, along with the corresponding multiple sequence alignment and structural sequence; Based on the probability distribution of amino acid residues at each site in the multiple sequence alignment of the protein sequence to be optimized, a corresponding multiple sequence alignment probability distribution sequence is generated. The structural sequence corresponding to the protein sequence to be optimized, the multiple sequence alignment probability distribution sequence, and the protein sequence to be optimized are sequentially concatenated to generate a multimodal sequence of the protein to be optimized corresponding to the protein sequence to be optimized. The multimodal sequence of the protein to be optimized is input into the pre-trained protein language model to obtain the first-round predicted amino acid residue probability distribution sequence of the mutant protein, wherein the pre-trained protein language model is pre-trained by the method described in any one of claims 1-5.

7. The method according to claim 6, wherein, The method further includes: A set of first-round mutant protein sequences is obtained, wherein the set of first-round mutant protein sequences is obtained by designing a first-round mutation of the protein sequence to be optimized based on the first-round predicted mutant protein amino acid residue probability distribution sequence; Obtain the wet experiment score for each of the first-round mutant protein sequences; For each position in the first-round prior amino acid residue influence distribution sequence corresponding to the protein sequence to be optimized, the prior amino acid residue influence distribution at that position is determined based on the sum of the wet experimental scores of each first-round prior mutant protein sequence that is identified by the corresponding amino acid residue at that position in each first-round prior mutant protein sequence. The first-round prior amino acid residue influence distribution sequence corresponding to the protein sequence to be optimized has the same sequence length as the protein sequence to be optimized. The structural sequence of the protein sequence to be optimized, the multiple sequence alignment probability distribution sequence, the protein sequence to be optimized, and the corresponding first-round amino acid residue prior probability distribution sequence are sequentially linked together to obtain the first-round multimodal sequence to be optimized. Taking the first round as the current round, based on the multimodal sequence to be optimized in the current round and the pre-trained protein language model after supervised fine-tuning and direct preference optimization or proximal strategy optimization, the next round of mutation design is carried out on the protein sequence to be optimized to obtain the set of next round mutant protein sequences for the current round.

8. The method according to claim 7, wherein, The method further includes: Obtain the wet experiment score for each subsequent round of the mutated protein sequence in the current round, and update the current round to the next round of the current round; The following next-round protein mutation design operation is performed: In response to determining that a next-round protein mutation design is required based on the wet experimental scores of the current-round mutated protein sequences, for each position in the current-round prior amino acid residue influence distribution sequence corresponding to the protein sequence to be optimized, the prior amino acid residue influence distribution at that position is determined based on the sum of the wet experimental scores of the current-round prior mutated protein sequences that are identified by the corresponding amino acid residues in the current-round prior mutated protein sequences. The prior amino acid residue influence distribution sequence corresponding to the protein sequence to be optimized is then used to determine the prior amino acid residue influence distribution at that position. The sequence length is the same as that of the protein sequence to be optimized; the prior amino acid residue influence distribution sequence of the current round is concatenated to the tail of the multimodal sequence to be optimized in the previous round of the current round to obtain the multimodal sequence to be optimized in the current round; based on the multimodal sequence to be optimized in the current round and the pre-trained protein language model optimized by supervised fine-tuning and direct preference optimization or by proximal strategy optimization, the next round of mutation design is performed on the protein sequence to be optimized to obtain the set of next round mutant protein sequences for the current round; the wet experiment score of each next round mutant protein sequence in the current round is obtained, and the current round is updated to the next round of the current round, and the next round protein mutation design operation is continued.

9. A protein language model pre-training device, comprising: The sample data acquisition module is configured to acquire a set of sample protein data, which includes sample protein sequences and corresponding multiple sequence alignments and structural sequences. The multimodal sequence generation module is configured to perform the following multimodal sequence generation operations for each sample protein data: generate a multi-sequence alignment probability distribution sequence corresponding to the sample protein data based on the amino acid residue probability distribution at each site of the multi-sequence alignment in the sample protein data; and concatenate the sample protein sequence, the corresponding multi-sequence alignment probability distribution sequence, and the structural sequence in the sample protein data in either a forward or reverse manner to generate a multimodal sequence corresponding to the sample protein data. The pre-training module is configured to perform autoregressive pre-training on the protein language model based on the multimodal sequences corresponding to the protein data of each sample, thereby obtaining a pre-trained protein language model.

10. A protein mutation device, comprising: The data acquisition module is configured to acquire the protein sequence to be optimized, as well as the corresponding multiple sequence alignment and structural sequence. The multiple sequence alignment probability distribution sequence generation module is configured to generate a corresponding multiple sequence alignment probability distribution sequence based on the amino acid residue probability distribution at each site of the multiple sequence alignment of the protein sequence to be optimized. The multimodal concatenation module is configured to sequentially concatenate the structural sequence corresponding to the protein sequence to be optimized, the multiple sequence alignment probability distribution sequence, and the protein sequence to be optimized to generate a multimodal sequence of the protein to be optimized corresponding to the protein sequence to be optimized. The first-round prediction module is configured to input the multimodal sequence of the protein to be optimized into the pre-trained protein language model to obtain the first-round predicted amino acid residue probability distribution sequence of the mutant protein, wherein the pre-trained protein language model is pre-trained by the method described in any one of claims 1-5.

11. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1-5 and / or the method as described in any one of claims 6-8.

12. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by one or more processors, it implements the method as described in any one of claims 1-5 and / or the method as described in any one of claims 6-8.

13. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the method as described in any one of claims 1-5 and / or the method as described in any one of claims 6-8.

Images