Protein language model training method, electronic device, computer-readable medium and program product

Abstract

Provided are a protein language model training method, a method for extracting an amino acid sequence representation by using a protein language model, a method for obtaining a new amino acid sequence by using a protein language model, a method for obtaining a related amino acid sequence by using a protein language model, a method for predicting an amino acid sequence perplexity by using a protein language model, an electronic device, a computer-readable medium, and a program product. The protein language model training method comprises: during a training process, executing at least one training sub-task amongst two training sub-tasks included in a first type of training task and a second type of training task; and, on the basis of a first loss value corresponding to the first type of training task and a loss value corresponding to the second type of training task, adjusting parameters of a protein language model to obtain a trained protein language model.

Description

Protein language model training method, electronic device, computer readable medium and program product

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese application CN202310832203.6 filed on July 6, 2023. The prior application is deemed to be part of the disclosure of this application and is incorporated into this application in its entirety. Technical Field

[0003] The present disclosure relates to the technical field of protein language model training, and specifically to a protein language model training method, a method for extracting amino acid sequence representation using a protein language model, a method for obtaining mutant amino acids or new amino acid sequences using a protein language model, a method for obtaining related amino acid sequences using a protein language model, a method for predicting amino acid sequence perplexity using a protein language model, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0004] Proteins are essential for life. Understanding their structure can help us understand their functions from a mechanistic perspective, thus facilitating subsequent target research and drug development. In the field of AI-assisted protein design and processing, key issues remain: how to represent proteins in mathematical form for use in AI algorithms and how to generate them.

[0005] The approaches described in this section are not necessarily approaches that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any approach described in this section is prior art simply by virtue of its inclusion in this section. Similarly, unless otherwise indicated, the issues raised in this section should not be considered as having been recognized in any prior art.

[0006] Summary of the Invention

[0007] In view of this, the present disclosure provides a method for training a protein language model, a method for extracting amino acid sequence representations using a protein language model, a method for obtaining mutant amino acids or new amino acid sequences using a protein language model, a method for obtaining related amino acid sequences using a protein language model, a method for predicting amino acid sequence perplexity using a protein language model, an electronic device, a computer-readable storage medium, and a computer program product.

[0008] According to one aspect of the present disclosure, a method for training a protein language model is provided, comprising: during a training process, performing at least one of two training subtasks included in a first type of training task and a second type of training task;

[0009] The first type of training task includes: obtaining a first amino acid sequence; performing a first preprocessing operation on the first amino acid sequence to obtain a first preprocessed amino acid sequence; the first preprocessing operation includes selecting one or more first sites in the first amino acid sequence and masking the word at the first site; inputting the first preprocessed amino acid sequence into the protein language model to obtain a first prediction result; calculating a first loss value based on the word at the first site in the first amino acid sequence and the element at the first calculation site in the first prediction result; the first calculation site is a site corresponding to the first site; the first training subtask in the second type of training task includes: obtaining a second amino acid sequence; performing a second preprocessing operation on the second amino acid sequence to obtain a second preprocessed amino acid sequence; the second preprocessing operation includes selecting one or more second sites in the second amino acid sequence and masking the word at the second site; inputting the second preprocessed amino acid sequence into the protein language model to obtain a second prediction result; calculating a first loss value based on the word at the second site in the second amino acid sequence and the second The second loss value is calculated for the element at the second calculation site in the prediction result; the second calculation site is selected from the site after the site aligned with the second site; the second training subtask in the second type of training task includes: obtaining a third amino acid sequence; performing a third preprocessing operation on the third amino acid sequence to obtain a third preprocessed amino acid sequence; the third preprocessing operation includes selecting one or more third sites located at the end of the sequence and adjacent to each other in the third amino acid sequence, and deleting the word units at the third sites; inputting the third preprocessed amino acid sequence into the protein language model to obtain a third prediction result; calculating the third loss value based on the word units at the third site in the third amino acid sequence and the elements at the third calculation site in the third prediction result; the third calculation site is selected from the site after the site aligned with the third site; adjusting the parameters of the protein language model based on the first loss value and the loss value corresponding to the second type of training task to obtain a trained protein language model, and the loss value corresponding to the second type of training task includes at least one of the second loss value and the third loss value.

[0010] According to another aspect of the present disclosure, a method for extracting amino acid sequence representations using a protein language model is provided. The protein language model is a trained protein language model obtained by training using the method provided by the present disclosure. The trained protein language model includes a representation extraction layer, a transformer block, and a classification layer. The method includes: obtaining a first target amino acid sequence; inputting the first target amino acid sequence into the representation extraction layer and the transformer block of the trained protein language model, and using the output of the protein language model transformer block as the representation of the first target amino acid sequence.

[0011] According to another aspect of the present disclosure, a method for obtaining a mutant amino acid or a new amino acid sequence using a protein language model is provided, wherein the protein language model is a trained protein language model obtained by training using the method provided by the present disclosure, and the method comprises: obtaining a second target amino acid sequence; replacing amino acids at some sites in the second target amino acid sequence to generate a new amino acid sequence; performing a second preprocessing operation on the second target amino acid sequence to obtain a second target preprocessed amino acid sequence; wherein the site to be replaced is used as the second site; the length of the second target preprocessed amino acid sequence is L3; and inputting the second target preprocessed amino acid sequence into the trained protein language model. language model; input the word unit output at the k-th position of the trained protein language model as the word unit input at the k+1-th position into the trained protein language model; wherein k is greater than L3; concatenate the second target preprocessed amino acid with the output word units from the L3+1-th position to the L3+n-th position of the trained protein language model to obtain a new amino acid sequence; n is the total number of new amino acids corresponding to the amino acid at the second position; or concatenate the second target preprocessed amino acid with the output word units from the L3+1-th position to the position -1 where the word unit output by the trained protein language model is a terminator to obtain a new amino acid sequence.

[0012] According to another aspect of the present disclosure, a method for obtaining a related amino acid sequence using a protein language model is provided, wherein the protein language model is a trained protein language model obtained by training using the method provided by the present disclosure, and the method comprises: inputting a third target preprocessed amino acid sequence into the trained protein language model; the length of the third target preprocessed amino acid sequence is L4, wherein L4 is greater than or equal to 0; inputting the word element output at the qth position of the trained protein language model as the word element input at the q+1th position into the trained protein language model; wherein q is greater than L4; using the word element output from the L4+1th position of the trained protein language model to the position -1 where the word element output by the trained protein language model is a terminator as the related amino acid sequence; or, using the word element output from the L4+1th position of the trained protein language model to a preset position as the related amino acid sequence; wherein the related amino acid sequence is an amino acid sequence continued based on the third target amino acid sequence; or, the related amino acid sequence is an amino acid sequence paired with the third target amino acid sequence.

[0013] According to another aspect of the present disclosure, a method for predicting the perplexity of an amino acid sequence using a protein language model is provided. The protein language model is a trained protein language model obtained by training using the method provided by the present disclosure. The trained protein language model includes a representation extraction layer, a transformer block, and a classification layer. The method includes: obtaining a fourth target amino acid sequence; sequentially taking each site in the fourth target amino acid sequence as a first site and performing a first preprocessing operation to obtain multiple fourth target preprocessed amino acid sequences; wherein each fourth target preprocessed amino acid sequence corresponds to a masked site; inputting a first current preprocessed amino acid sequence among the multiple fourth target preprocessed amino acid sequences into the protein language model to obtain a probability predicted by the classification layer; taking the probability corresponding to the first word at the first current masked site as the perplexity factor of the first current masked site; the first word is the word of the fourth target amino acid sequence at the first current masked site; the first current masked site is the masked site corresponding to the first current preprocessed amino acid sequence; and obtaining the perplexity of the fourth target amino acid sequence based on the perplexity factor of each site.

[0014] According to another aspect of the present disclosure, a method for predicting the perplexity of an amino acid sequence using a protein language model is provided. The protein language model is a trained protein language model obtained by training using the method provided by the present disclosure. The trained protein language model includes a representation extraction layer, a transformer block, and a classification layer. The method includes: obtaining a fifth target amino acid sequence; the length of the fifth target amino acid sequence is L5; taking each site in the fifth target amino acid sequence as a second site in turn, performing a second preprocessing operation to obtain multiple fifth target preprocessed amino acid sequences; wherein each fifth target preprocessed amino acid sequence corresponds to a masked site; inputting a second current preprocessed amino acid sequence from the multiple fifth target preprocessed amino acid sequences into the protein language model to obtain a probability predicted by the classification layer; taking the probability corresponding to the second word at the L5+1th site as the perplexity factor of the second current masked site; the second word is the word of the fifth target amino acid sequence at the second current masked site; the second current masked site is the masked site corresponding to the second current preprocessed amino acid sequence; and obtaining the perplexity of the fifth target amino acid sequence based on the perplexity factor of each site.

[0015] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above method.

[0016] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above method.

[0017] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the above method when executed by a processor.

[0018] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings illustrate exemplary embodiments and constitute a part of the specification. Together with the description of the specification, they serve to explain exemplary implementation of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals designate similar, but not necessarily identical, elements.

[0020] FIG1 shows a schematic diagram of an exemplary system in which various methods described herein may be implemented according to an embodiment of the present disclosure;

[0021] FIG2 shows a flow chart of a method for training a protein language model according to an exemplary embodiment of the present disclosure;

[0022] 3A-3C are schematic diagrams showing an exemplary training process of a protein language model according to an exemplary embodiment of the present disclosure;

[0023] FIG4 shows a flowchart of a method for extracting amino acid sequence representations using a protein language model according to an exemplary embodiment of the present disclosure;

[0024] FIG5 shows a flow chart of a method for obtaining a new amino acid sequence using a protein language model according to an exemplary embodiment of the present disclosure;

[0025] FIG6 shows a flowchart of a method for obtaining related amino acid sequences using a protein language model according to an exemplary embodiment of the present disclosure;

[0026] FIG7 shows a flowchart of a method for predicting amino acid sequence perplexity using a protein language model according to an exemplary embodiment of the present disclosure;

[0027] FIG8 shows a flowchart of a method for predicting amino acid sequence perplexity using a protein language model according to an exemplary embodiment of the present disclosure; and

[0028] FIG9 shows a block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure. DETAILED DESCRIPTION

[0029] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0030] In this disclosure, unless otherwise specified, the use of terms such as "first" and "second" to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements. Such terms are only used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of the element, while in some cases, based on the context of the description, they may also refer to different instances.

[0031] The terms used in the descriptions of the various examples described in this disclosure are for the purpose of describing specific examples only and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element may be one or more. In addition, the term "and / or" used in this disclosure encompasses any one and all possible combinations of the listed items.

[0032] Proteins are essential for life. Understanding their sequence and structure can help us understand their functions from a mechanistic perspective, thereby facilitating subsequent target research and drug development. In the field of AI-assisted protein design and processing, a key issue is how to represent proteins in mathematical form for use in AI algorithms and how to generate proteins.

[0033] In the related art, vector representation of protein amino acid sequences is a widely used approach. This approach is often analogous to the language model (LM) in natural language processing (NLP). The amino acid sequence is input into the LM, which then outputs the corresponding vector representation. Protein language models are typically trained using two approaches: masked language models (MLMs) and auto-regressive language models (ARLMs). A masked language model randomly replaces an amino acid sequence (e.g., replacing certain amino acids with a mask). The model is trained to predict the original masked amino acids. An auto-regressive language model predicts the next amino acid in an amino acid sequence from scratch (e.g., predicting the second half of a given sequence from the first half). A masked language model can provide a contextually meaningful vector representation for each amino acid in the sequence. However, an auto-regressive language model, because it predicts the next amino acid in the sequence, cannot fully represent the amino acid at the current position. One advantage of autoregressive language models is that they can be used to generate protein sequences, which is difficult for masked language models to do.

[0034] To address the above-mentioned issues, the present disclosure provides a method for training a protein language model. This method performs a first type of training task associated with training a masked language model and at least one second type of training task associated with training an autoregressive language model on an acquired first amino acid sequence, and adjusts the parameters of the protein language model based on the loss values ​​corresponding to the first type of training task and the loss values ​​corresponding to the second type of training task, thereby obtaining a trained protein language model. Thus, the present disclosure employs a hybrid training model that combines a masked language model with an autoregressive language model, enabling the trained protein language model to simultaneously represent each amino acid in an amino acid sequence with a vector and generate new amino acid sequences.

[0035] To better understand the technical solutions provided in the embodiments of this application, the following briefly introduces the application scenarios to which the technical solutions provided in the embodiments of this application are applicable. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of this application and are not limiting. In specific implementation, the technical solutions provided in the embodiments of this application can be flexibly applied according to actual needs.

[0036] The method provided in the embodiment of the present application can be applied to the application scenario shown in Figure 1. Referring to Figure 1, the exemplary system 100 in the application scenario includes multiple terminal devices 101, 102, 103, 104, 105 and 106, a server 120 and one or more communication networks 110 that couple one or more client devices to the server 120. The terminal devices 101, 102, 103, 104, 105 and 106 can be connected to the server 120 via a wired connection or a wireless connection and transmit data. For example, the terminal devices 101, 102, 103, 104, 105 and 106 can be connected to the server 120 via a data cable or via a wired network; the terminal devices 101, 102, 103, 104, 105 and 106 can also be connected to the server 120 via a radio frequency module, a WiFi module or a wireless network.

[0037] Terminal devices 101, 102, 103, 104, 105, and 106 may be computers, notebooks, PDAs, tablet computers, and the like. Server 120 may be a single server, a server cluster consisting of multiple servers, a cloud computing center, or a virtualization platform, or a personal computer, a large or medium-sized computer, or a computer cluster. Network 110 may be any type of network known to those skilled in the art, and may support data communications using any of a variety of available protocols (including, but not limited to, TCP / IP, SNA, IPX, and the like). By way of example only, one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0038] The system 100 may also include one or more databases 130. In some embodiments, these databases can be used to store data and other information. For example, one or more of the databases 130 can be used to store information such as audio files and video files. The data repository 130 can reside in a variety of locations. For example, the data repository used by the server 120 can be local to the server 120, or can be remote from the server 120 and can communicate with the server 120 via a network-based or dedicated connection. The data repository 130 can be of different types. In some embodiments, the data repository used by the server 120 can be a database, such as a relational database. One or more of these databases can store, update, and retrieve data to and from the database in response to commands.

[0039] In some implementations, one or more of the databases 130 may also be used by applications to store application data. The databases used by the applications may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0040] According to implementation needs, the application scenario in the embodiment of the present application may have any number of terminal devices, servers, networks and databases. This application does not make any special restrictions on this. The training method of the protein language model provided in the embodiment of the present application, the method of extracting amino acid sequence representation using the protein language model, the method of obtaining a new amino acid sequence using the protein language model, the method of obtaining related amino acid sequences using the protein language model, and the method of predicting amino acid sequence perplexity using the protein language model can be performed by the server 120, or can be performed by the terminal devices 101, 102, 103, 104, 105 and 106 in collaboration with the server 120.

[0041] To further illustrate the technical solutions provided by the embodiments of the present application, this is described in detail below with reference to the accompanying drawings and specific implementation methods. Although the embodiments of the present application provide method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on routine or no creative labor. In steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided by the embodiments of the present application. The method may be executed in the order of the methods shown in the embodiments or drawings or in parallel during the actual processing process or when the device is executed.

[0042] According to one aspect of the present disclosure, a method 200 for training a protein language model is provided. As shown in FIG2 , the method 200 includes: step S202, during a training process, executing at least one of two training subtasks included in a first type of training task and at least one second type of training task; step S203, adjusting parameters of the protein language model based on a first loss value and a loss value corresponding to the second type of training task, to obtain a trained protein language model, wherein the loss value corresponding to the second type of training task includes at least one of a second loss value and a third loss value.

[0043] As used in this disclosure, the term "training task" refers to inputting input samples into a model, obtaining actual output samples from the model, comparing the actual output samples with expected output samples, and adjusting or optimizing the model based on the comparison results. In the context of the protein language model of this disclosure, a training task refers to inputting a corresponding preprocessed amino acid sequence into a protein prediction model according to the corresponding task type, obtaining a corresponding predicted amino acid model output by the protein prediction model, calculating a corresponding loss value based on the tokens at useful sites in the corresponding preprocessed amino acid sequence and the corresponding predicted amino acid model, and thereby adjusting the model parameters to obtain a trained protein language model.

[0044] In step S202, the first type of training task includes: obtaining a first amino acid sequence; performing a first preprocessing operation on the first amino acid sequence to obtain a first preprocessed amino acid sequence; the first preprocessing operation includes selecting one or more first sites in the first amino acid sequence and masking the word at the first site; inputting the first preprocessed amino acid sequence into the protein language model to obtain a first prediction result; calculating a first loss value based on the word at the first site in the first amino acid sequence and the element at the first calculation site in the first prediction result; the first calculation site is a site aligned with the first site; the first training subtask in the second type of training task includes: obtaining a second amino acid sequence; performing a second preprocessing operation on the second amino acid sequence to obtain a second preprocessed amino acid sequence; the second preprocessing operation includes selecting one or more second sites in the second amino acid sequence and masking the word at the second site; the second preprocessing The processed amino acid sequence is input into the protein language model to obtain a second prediction result; a second loss value is calculated based on the word at the second site in the second amino acid sequence and the element at the second calculation site in the second prediction result; the second calculation site is selected from the site after the site aligned with the second site; the second training subtask in the second type of training task includes: obtaining a third amino acid sequence; performing a third preprocessing operation on the third amino acid sequence to obtain a third preprocessed amino acid sequence; the third preprocessing operation includes selecting one or more third sites in the third amino acid sequence that are located at the end of the sequence and adjacent to each other, and deleting the word at the third site; inputting the third preprocessed amino acid sequence into the protein language model to obtain a third prediction result; calculating a third loss value based on the word at the third site in the third amino acid sequence and the element at the third calculation site in the third prediction result; the third calculation site is selected from the site after the site aligned with the third site.

[0045] According to an embodiment of the present disclosure, a trained protein language model is obtained by performing a first type of training task associated with training a masked language model and at least one second type of training task associated with training an autoregressive language model on an acquired first amino acid sequence, and adjusting the parameters of the protein language model based on the loss values ​​corresponding to the first type of training task and the loss values ​​corresponding to the second type of training task. Thus, by utilizing a hybrid training model that combines a masked language model with an autoregressive language model, the trained protein language model simultaneously has the ability to represent each amino acid in an amino acid sequence with a vector and to generate new amino acid sequences, thereby being adaptable to a variety of downstream tasks and improving the processing performance of the protein language model in various downstream tasks.

[0046] In step S202, the first type of training task may include: step S2021a, obtaining a first amino acid sequence; step S2021b, performing a first preprocessing operation on the first amino acid sequence to obtain a first preprocessed amino acid sequence; step S2021c, inputting the first preprocessed amino acid sequence into a protein language model to obtain a first prediction result; step S2021d, calculating a first loss value based on the first amino acid sequence and the first prediction result.

[0047] In step S2021a, the first amino acid sequence can be obtained from a laboratory or various accessible databases, and the present disclosure does not impose any limitation on this.

[0048] The first amino acid sequence may include a sequence terminator. In subsequent processing, the sequence terminator is treated as a common type of amino acid without special processing.

[0049] The original amino acid sequences in the training data set are of different lengths, and each original amino acid sequence may include a sequence terminator at the end. The original amino acid sequences can be processed by concatenating each original amino acid sequence in the training data set end to end, and then truncating the concatenated amino acid sequences into amino acid sequences of a fixed length (e.g., 512, 1024). The first amino acid sequence is selected from the truncated amino acid sequence. In this way, calculations can be performed on amino acid sequences of fixed length, making full use of computing power. For example, the first amino acid sequence can be X1 X2 X3 X4 X5 EOS X6 X7, where EOS is the sequence terminator.

[0050] In step S2021b of the embodiment, the first preprocessing operation may include selecting one or more first positions in the first amino acid sequence and masking the word at the first position. Exemplarily, the masking includes replacing the amino acid at the first position with a first mask placeholder, that is, the first preprocessing operation does not change the length of the first amino acid sequence.

[0051] As used in this disclosure, the term "token" refers to the basic unit of text. In the context of protein language models, it particularly refers to the smallest constituent element of an amino acid sequence, which can be various amino acid identifiers used to represent amino acid types (i.e., amino acid IDs, such as L for leucine and A for alanine), or various tags (e.g., mask placeholders, sequence terminators, predicted start and end tags), etc. Therefore, it can be understood that in the context of this disclosure, when referring to various amino acid sequences, it can refer to amino acid sequences that are homogeneous with amino acid sequences that exist or have not yet been found in nature (i.e., the smallest constituent elements constituting the amino acid sequence only include amino acid IDs), or it can refer to amino acid sequences that are heterogeneous with amino acid sequences that exist or have not yet been found in nature (i.e., the smallest constituent elements constituting the amino acid sequence include or only include various tags). In the context of protein language models, tokens can only be selected from various amino acid identifiers and various tags. Therefore, each amino acid identifier and tag is an alternative token, and each amino acid identifier and various tags constitute each alternative token.

[0052] As used in this disclosure, the term "site" refers to the position of a basic unit of text in a text, and in the context of a protein language model, particularly refers to the position of the smallest constituent element of an amino acid sequence in an amino acid sequence.

[0053] As used in this disclosure, in the context of a protein language model, the term "mask" may refer to, for example, setting the amino acid ID at a given position (e.g., the first position) to a mask placeholder, thereby making it impossible to identify the amino acid type based on the word at that position.

[0054] The number and position of the masked words can be random or limited. For example, the number of masked words is about 15% of the length of the first amino acid sequence, and the masked position is, for example, the CDR region of the antibody.

[0055] Exemplarily, in step S2021c, the length of the first prediction result obtained by inputting the first preprocessed amino acid sequence into the protein language model (i.e., the number of elements or sites included in the prediction result) is the same as the length of the first preprocessed amino acid sequence, and the input word at each site has an output element corresponding to it.

[0056] It is understandable that the protein language model can predict the probability of which candidate word the site being predicted is. In the first type of task, the site being predicted is the covered site.

[0057] In the context of protein language models, the terms "prediction result" and "elements at a site in the prediction result" can be broadly understood. An element at a site in the prediction result can refer to one of the candidate tokens (e.g., the candidate token with the highest probability), the probability that the predicted site is a specific candidate token (e.g., the candidate token is the token at the input site corresponding to that site), the probability that the predicted site is each candidate token, or any combination of the three.

[0058] In an embodiment, step S2021d includes: calculating a first loss value based on the word at the first position in the first amino acid sequence and the element at the first calculation position in the first prediction result.

[0059] In an example, the first calculated site may be a site aligned with the first site, that is, the first calculated site may be a site in the first prediction result aligned with the first site in the first amino acid sequence.

[0060] In this example, the element at the first calculation site represents the probability that the output at the first calculation site is the corresponding input word at the first site. For example, there are three first sites in a first amino acid sequence, where the input word at the first site a is amino acid ID A, the input word at the first site b is amino acid ID L, and the input word at the first site c is amino acid ID U. The output element at the first calculation site a represents the probability of 0.7 that the amino acid ID is A at the first calculation site a, the output element at the first calculation site b represents the probability of 0.6 that the amino acid ID is L at the first calculation site b, and the output element at the first calculation site c represents the probability of 0.4 that the amino acid ID is U at the first calculation site c. The first loss value is determined based on these three probabilities.

[0061] In this example, the element at the first computation site represents the output word at the first computation site. For example, there are three first sites on the first amino acid sequence, where the input word at the first site a is amino acid ID A, the input word at the first site b is amino acid ID L, and the input word at the first site c is amino acid ID U. The output element at the first computation site a is A, the output element at the first computation site b is U, and the output element at the first computation site c is U. The first loss value is determined based on the tensor corresponding to each input word and the output element at the corresponding position (e.g., a 0th-order, 1st-order, or higher-order tensor, which can be determined by looking up a word-to-tensor mapping table).

[0062] Now turn to Figure 3A, which shows a schematic diagram of an exemplary training process for the first type of training task according to an embodiment of the present disclosure. It should be noted that the exemplary training process shown in Figure 3A is only an exemplary illustration of the first type of training task and is not intended to limit the first type of training task.

[0063] As shown in Figure 3A, box 301 represents a protein language model, such as the untrained protein language model described in method 200 with reference to Figure 2. Protein language model 301 receives a first preprocessed amino acid sequence 303a and outputs a first prediction result 304a. As an example, first amino acid sequence 302a includes five tokens, X0 to X4, and a first preprocessing operation masks tokens X1 and X3. That is, tokens X1 and X3 are located at first positions 305a in first amino acid sequence 302a, and there are two first positions 305a. In first prediction result 304a output by protein language model 301, elements X1' and X3' are located at positions corresponding to the positions of tokens X1 and X3 in first amino acid sequence 302a (i.e., first positions 305a). That is, elements X1' and X3' are located at first calculation sites 306a in first prediction result 304a, and there are also two first calculation sites 306a. Thus, a first loss value can be calculated based on the word elements (e.g., word elements X1 and X3) at the first position 305a in the first amino acid sequence 302a and the elements (e.g., word elements X1' and X3') at the first calculation position 306a in the first prediction result 304a.

[0064] It will be appreciated that the length of the first amino acid sequence 302a and the number of first sites 305a (and therefore first calculated sites 306a) in the example described above with reference to FIG3A are merely illustrative and are not intended to be limiting in the present disclosure. It will also be appreciated that the selected first sites may be located adjacent to one another, and the first site may be located anywhere in the first amino acid sequence, not just in the middle of the sequence.

[0065] In this example, a training sample (X, Y) can be generated for a first amino acid sequence according to the first type of task described above, where X is the first preprocessed amino acid sequence and Y is the content to be predicted (i.e., the true value, i.e., the masked amino acid ID at the first position). Assuming a given first amino acid sequence PSSLALSVGQKVTMSCKSSQSI, for the protein language model of the present disclosure (as an example and not limitation, the protein language model depicted in FIG. 3A ), the training sample can be constructed as, for example, (PSSL[MASK]LSVGQKV[MASK]MSCK[MASK]SQSI, ----A-------T----S----). Where "[MASK]" represents a mask, ----A-------T----S---- represents a word at the first position 305a in the first amino acid sequence 302a, i.e., the true value, and "-" is used to represent a position other than the first position, thereby determining the position of the first position.

[0066] In step S202 , the second type of training task may include at least one of two training subtasks.

[0067] In an embodiment, the first training subtask may include: step S2022a, obtaining a second amino acid sequence; step S2022b, performing a second preprocessing operation on the second amino acid sequence to obtain a second preprocessed amino acid sequence; step S2022c, inputting the second preprocessed amino acid sequence into a protein language model to obtain a second prediction result; step S2022d, calculating a second loss value based on the first amino acid sequence and the second prediction result.

[0068] The second amino acid sequence can also be selected from the truncated amino acid sequences described above.

[0069] In an embodiment, the second preprocessing operation may include selecting one or more second sites in the second amino acid sequence and masking the tokens at the second sites. Exemplarily, masking includes replacing the amino acid at the second site with a second mask placeholder. The second mask placeholder may be the same as or different from the first mask placeholder.

[0070] In an embodiment, step S2022d includes: calculating a second loss value based on the word at the second position in the first amino acid sequence and the element at the second calculation position in the second prediction result.

[0071] The steps in the first training subtask included in the above-mentioned second type of training task that are similar to those in the first type of training task can be found in the above description and will not be repeated here.

[0072] Exemplarily, the second calculated site is selected from a site following the site aligned with the second site.

[0073] In an embodiment, the second calculation site may include L1+1th to L1+Nth P2 +1 position; where L1 is the length of the second preprocessed amino acid sequence, N P2 is the second number of sites. It can be seen that the length of the second prediction result is greater than the length of the second preprocessed amino acid sequence.

[0074] The expected second prediction results are L1+1 to L1+N P2 The output word at the position is the word at the second position that is covered. P2 The output word at position L1+1 is the predicted terminator. P2 The output word of position N and the second position of position 1 to N P2The second positions may not correspond sequentially, but may correspond according to a preset correspondence. For example, if there are three second positions, the expected output word-grams at positions L1+1 to L1+3 are word-grams at second position a, second position c, and second position b, respectively (not necessarily second position a, second position b, and second position c), and the output word-gram at position L1+4 is the predicted terminator.

[0075] In this example, the element at the second calculation site represents the probability that the output word at the second calculation site is the corresponding input word at the second site. For example, there are two second sites on the first amino acid sequence, where the input word at the second site a is amino acid ID A, and the input word at the second site b is amino acid ID L. The output element at the L1+1th second calculation site (corresponding to the second site b) refers to the probability of 0.7 that the amino acid ID L is at the L1+1th second calculation site, the output element at the L1+2th second calculation site (corresponding to the second site a) refers to the probability of 0.6 that the amino acid ID A is at the L1+2th second calculation site, and the output element at the L1+3th calculation site refers to the probability of 0.4 that the predicted terminator is at the L1+3th calculation site. The second loss value is determined based on these three probabilities.

[0076] In the example, the element at the second calculation site represents the output word at the second calculation site. For example, there are two second sites on the second amino acid sequence, where the input word at the first site a is the amino acid ID A, the input word at the second site b is the amino acid ID L, the output word at the L1+1 second calculation site (corresponding to the second site b) is L, the output word at the L1+2 second calculation site (corresponding to the second site a) is U, and the output word at the L1+3 second calculation site is the predicted terminator. The second loss value is determined based on the tensors (e.g., 0-order, 1-order, or higher-order tensors, which can be determined by looking up a word-tensor-tensor mapping table) corresponding to each input word (L, A, predicted terminator, respectively) and the output word at the corresponding position (L, U, predicted terminator, respectively).

[0077] Now turning to Figure 3B, which shows a schematic diagram of an exemplary training process for the first training subtask in the second type of training task according to an exemplary embodiment of the present disclosure. It should be noted that the exemplary training process shown in Figure 3B is merely an exemplary illustration of the first training subtask in the second type of training task, and is not intended to limit the first training subtask in the second type of training task.

[0078] As shown in Figure 3B, box 301 represents a protein language model, such as the untrained protein language model described in method 200 with reference to Figure 2. Protein language model 301 receives a second preprocessed amino acid sequence 303b and outputs a second prediction result 304b. As an example, second amino acid sequence 302b includes five tokens, X0 to X4, and the second preprocessing operation masks tokens X1 and X2. That is, tokens X1 and X2 are located at second sites 305b in second amino acid sequence 302b, and there are two second sites 305b. By way of example and not limitation, a mask M can be used to mask two or more adjacent positions within second sites 305b. In the second prediction result 304b output by protein language model 301, corresponding tokens may be output at positions corresponding to the positions of token X0, mask M, tokens X3, and X4 in second amino acid sequence 302b, although the tokens output at these positions are irrelevant for calculating the second loss value. That is, the elements of the second prediction result 304b at positions corresponding to the positions of the word X0, mask M, word X3, and X4 in the second amino acid sequence 302b (i.e., positions 1 to L1) do not participate in the calculation of the second loss value. In the second prediction result 304b, the positions of elements X1' and X2' and the position after X2' are second calculation sites 306b, and the number of second calculation sites 306b is also two. Thus, the second loss value can be calculated at least in part based on the word (e.g., word X1 and X2) at the second site 305b in the first amino acid sequence 302b and the elements (e.g., word X1' and X2') at the second calculation site 306b in the second prediction result 304b. As described above, the second loss value is also calculated based on the output word and the predicted terminator at the last site of the second calculation site 306b.

[0079] It is understood that the length of the second amino acid sequence 302b and the number of second sites 305b (and therefore the number of second calculated sites 306b) in the example described above with reference to FIG3B are merely illustrative and are not intended to be limiting in the present disclosure. It is also understood that the selected second sites may be non-adjacent to each other and may be located anywhere in the second amino acid sequence, not just in the middle of the sequence.

[0080] In an embodiment, the second training subtask may include: step S2023a, obtaining a third amino acid sequence; step S2023b, performing a third preprocessing operation on the third amino acid sequence to obtain a third preprocessed amino acid sequence; step S2023c, inputting the third preprocessed amino acid sequence into the protein language model to obtain a third prediction result; step S2023d, calculating a third loss value based on the third amino acid sequence and the third prediction result.

[0081] In an embodiment, the third preprocessing operation may include selecting one or more third sites in the third amino acid sequence that are located at the end of the sequence and adjacent to each other, and deleting the word units at the third sites.

[0082] In an embodiment, step S2023d: calculating a third loss value based on the word at the third position in the first amino acid sequence and the element at the third calculation position in the third prediction result.

[0083] The steps in the second training subtask included in the above-mentioned second type of training task that are similar to those in the first type of training task can be found in the above description and will not be repeated here.

[0084] Exemplarily, the third calculated site is selected from a site following the site aligned with the third site.

[0085] In an embodiment, the third calculation site may include L2+1th to L2+Nth P3 +1 position; where L2 is the length of the third preprocessed amino acid sequence, N P3 is the number of the third site. It can be seen that the length of the third prediction result is greater than the length of the third preprocessed amino acid sequence.

[0086] The expected third prediction result is L2+1 to L2+N P3 The output word at the position is the word at the third position that is deleted, and the L2+N P3 The output word at position L2+1 is the predicted terminator. P3 The output word of position N and the third position of position 1 to N P3 The third positions can correspond to each other not in sequence, but according to a preset correspondence. For example, if there are three third positions, the expected output word-units at positions L2+1 to L2+3 are the word-units at positions L2+1, L2+3, and L2+4, respectively (not necessarily the word-units at positions L2+4). The output word-unit at position L2+4 is the predicted terminator.

[0087] The calculation of the third loss value can refer to the description of the second loss value and will not be repeated here.

[0088] Now turning to Figure 3C, which shows a schematic diagram of an exemplary training process for the second training subtask in the second type of training task according to an exemplary embodiment of the present disclosure. It should be noted that the exemplary training process shown in Figure 3C is merely an exemplary illustration of the first training subtask in the second type of training task, and is not intended to limit the first training subtask in the second type of training task.

[0089] As an example, the third amino acid sequence 302c includes five tokens, X0 to X4. The third preprocessing operation deletes tokens X3 and X4, resulting in a third preprocessed amino acid sequence 303c (including X0, X1, and X2). Figure 3C differs from Figure 3B in that the third preprocessed amino acid sequence 303c does not include any mask M. Similar to the first training subtask, in the third prediction result 304c output by the protein language model 301, corresponding tokens can be output at positions corresponding to the positions of tokens X0 to X2 in the third amino acid sequence 302c (i.e., positions 1 to L2), although the tokens output at these positions are irrelevant to the calculation of the third loss value. That is, the elements at positions in the third prediction result 304c corresponding to the positions of tokens X0 to X2 in the third amino acid sequence 302c do not participate in the calculation of the third loss value. In the third prediction result 304c, the positions of elements X3' and X4' are included in the third calculation site 306c. Thus, a third loss value can be calculated based at least in part on the word-grams at the third position 305c in the third amino acid sequence 302c (e.g., word-grams X3 and X4) and the elements at the third calculation position 306c in the third prediction result 304c (e.g., word-grams X3' and X4'). As described above, the third loss value is also calculated based on the output word-gram and the predicted terminator at the last position in the third calculation position 306c.

[0090] It is understood that the length of the first amino acid sequence 302c and the number of third sites 305c (and therefore third calculation sites 306c) in the example described above with reference to FIG3C are merely illustrative, and the present disclosure does not impose any limitation thereto.

[0091] In step S203, the parameters of the protein language model may be adjusted based on the first loss value and the loss value corresponding to the second type of training task to obtain a trained protein language model.

[0092] The second type of training task may include at least one of a first training subtask and a second training subtask, and the loss value corresponding to the second type of training task may include at least one of a second loss value and a third loss value.

[0093] In the example, the parameters of the protein language model may include parameters related to the training process of the protein language model (such as parameters related to the convergence of the objective function of the model), etc., and the present disclosure does not impose any limitations on this.

[0094] It is understood that the first amino acid sequence, the second amino acid sequence, and the third amino acid sequence may be the same or different. A first type of training task and at least one second type of training task may be performed on the same amino acid sequence, or only one training task (e.g., one of the first type of training task, the first training subtask, and the second training subtask) may be performed on the same amino acid sequence, thereby obtaining multiple sets of loss values ​​corresponding to the training tasks experienced by the amino acid sequence. The parameters of the protein language model are then adjusted based on the multiple sets of loss values ​​to obtain a trained protein language model.

[0095] In step S202, during the training process, at least one of the two training subtasks included in the first type of training task and the second type of training task can be executed in parallel; or, during the training process, at least one of the two training subtasks included in the first type of training task and the second type of training task can be executed sequentially. The present disclosure does not limit the execution order of training tasks and training subtasks.

[0096] Exemplarily, step S220 may include:

[0097] In the first training phase, the first type of training task is executed; in the second training phase, the second type of training task is executed. For example, in the first training phase (epochs 1-N), the first type of training task is executed; in the second training phase (epochs N+1-M), the first subtask and / or the second subtask are executed. It is understood that the learning rate can be different in different training phases.

[0098] Exemplarily, step S220 may include:

[0099] In the first training stage, the first type of training task is performed; in the second training stage, the first type of training task and the second type of training task are performed; in the third training stage, the second type of training task is performed. For example, in the first training stage (1-N epochs), the first type of training task is performed; in the second training stage (N+1-M epochs), the first type of training task and at least one of the first subtask and the second subtask are performed; in the third training stage (M+1 epochs and above), at least one of the first subtask and the second subtask is performed. It is understandable that the learning rate of different training stages can be different. Exemplarily, the learning rate in the second training stage is lower than the learning rate in the first training stage and the third training stage, so that the training task can be smoothly switched from the first type to the second type.

[0100] Exemplarily, step S220 includes: first performing the first type of training task and at least one of the two training subtasks included in the second type of training task on the first dataset; then performing at least one of the first type of training task, the first training subtask, and the second training subtask on the second dataset. For example, s may be first trained on a protein dataset and then fine-tuned on an antibody dataset.

[0101] According to an embodiment of the present disclosure, in the above-mentioned method 200, step 2022c may include: inputting the second preprocessed amino acid sequence into the protein language model, and inputting the masked amino acids as word units from the L1+2th to the L1+second number of sites+1 position into the protein language model; and / or, step 2023c may include: inputting the third preprocessed amino acid sequence into the protein language model, and inputting the deleted amino acids as word units from the L2+2th to the L2+third number of sites+1 position into the protein language model.

[0102] Exemplarily, inputting the second preprocessed amino acid sequence into the protein language model may be inputting the second preprocessed amino acid sequence as a word unit from the 1st position to the L1th position into the model; inputting the third preprocessed amino acid sequence into the protein language model may be inputting the third preprocessed amino acid sequence as a word unit from the 1st position to the L2th position into the model.

[0103] Exemplarily, step 2022c may further include: inputting the second placeholder predicted start symbol into the model as the word element at position L1+1; and / or inputting the second placeholder into the model as the word element at position L2+1. The second placeholder may be a predicted start symbol, prompting the model to predict the start and output the desired word element at the position where the second placeholder is positioned. The second placeholder may include a start prediction symbol S as shown in Figures 3B and 3C.

[0104] It is understandable that in the second type of training task, the protein language model is trained as an autoregressive model. The characteristic of the autoregressive model is that when a word is input at the k-th position, the model will output the word at the k-th position as a prediction of the word at the k+1-th position, and will use the word output at the k-th position as the input word at the k+1-th position. The model will output the word at the k+1-th position as a prediction of the word at the k+2-th position. During the training phase, in order to enable the model to more accurately predict the words at the later positions, and not to cause excessive deviations in the predictions of the words at the later positions due to prediction errors in the first few positions, we use the true value as the input word at the k+1-th position, rather than using the word output at the k-th position as the input word at the k+1-th position. Specifically, in the first training subtask, the masked amino acids (i.e., L1+1 to L1+N P2The true value corresponding to the word at each position is output) as the L1+2th to L1+Nth P2 +1 position word unit is input into the protein language model; in the second training subtask, the deleted amino acids (i.e., L2+1 to L2+N P3 The true value corresponding to the word at the output position) is used as the L2+2th to L2+Nth P3 +1 position of the word is input to the protein language model.

[0105] It is understood that in an autoregressive model, if a word is not input at the kth position, the model will not output the word at the kth position. In some specific embodiments, during the training phase, in order to stop the output of the model in a timely manner without continuing to output backward, one of the following two methods can be adopted:

[0106] Method 1: In the first training subtask, the masked amino acids are taken as L1+2th to L1+Nth P2 +1 position word unit input protein language model, in the L1+N P2 +2 positions are no longer input into the word unit. And / or, in the second training subtask, the deleted amino acids are used as L2+2 to L2+N P3 +1 position word unit input protein language model, in the L1+N P2 +2 positions no longer input word element.

[0107] Method 2: In the first and second training subtasks, if the output token at a certain position is not a predicted terminator, the output token at that position is used as the input token at the next position; otherwise, there is no input token at the next position.

[0108] It should be noted that although FIG3B shows that the word element at the second calculation site 306 b is X1′ and X2′ representing the (new) amino acid ID output by the protein language model for the masked amino acid ID at the second site 305 b in the first amino acid sequence 302 b, it is understandable that based on the mechanism of the autoregressive model described above, the following special cases may exist: (a) the protein language model may output, for the masked amino acid ID at the second site in the first amino acid sequence, elements representing the (new) amino acid ID exceeding the number of the second site (for example, in addition to X1′ and X2′, also output X3′, X4′, etc.); (b) the protein language model may output, for the masked amino acid ID at the second site in the first amino acid sequence, elements representing the (new) amino acid ID less than the number of the second site (for example, only output X1′); or (c) the protein language model may output zero elements representing the (new) amino acid ID for the masked amino acid ID at the second site in the first amino acid sequence (for example, directly outputting a predicted terminator).

[0109] For the special case (a), the L1+1th to L1+Nth prediction results from the second prediction output by the protein language model can be selected. P2 +1 position is used as the second calculation site, and the loss value starting from L1+N is not used P2 +2 elements to output.

[0110] For special cases (b) and (c), assuming that the number of second sites in the second amino acid sequence is N, and in the second prediction result, the number of elements representing the (new) amino acid ID output by the protein language model for the masked amino acid ID at the second site in the second amino acid sequence is M, and M is less than N, then each site in the second calculation site except for the M word elements representing the (new) amino acid ID (i.e., the NM sites at which no word element representing the (new) amino acid ID is output) can be set to a given value to facilitate the calculation of the second loss value.

[0111] Similarly, it should be noted that although FIG3C shows that the word element at the third calculation site 306 c is X3′ and X4′ representing the (related) amino acid ID output by the protein language model for the deleted amino acid ID at the third site 305 c in the first amino acid sequence 302 c, it is understandable that based on the mechanism of the autoregressive model described above, the following special cases may exist: (a) the protein language model may output, for the deleted amino acid ID at the third site in the third amino acid sequence, elements representing (related) amino acid IDs exceeding the number of the third sites (for example, in addition to X3′ and X4′, also output X5′, X6′, etc.); (b) the protein language model may output, for the deleted amino acid ID at the third site in the first amino acid sequence, elements representing (related) amino acid IDs less than the number of the third sites (for example, only output X3′); or (c) the protein language model may output zero elements representing (related) amino acid IDs for the masked amino acid ID at the third site in the third amino acid sequence (for example, directly outputting a predicted terminator).

[0112] For the special case (a), the L2+1th to L2+Nth prediction results of the third prediction output by the protein language model can be selected. P3 +1 position is used as the third calculation site, and the loss value starting from L2+N is not used P3 +2 elements to output.

[0113] For special cases (b) and (c), assuming that the number of third sites in the third amino acid sequence is N, and in the third prediction result, the number of elements representing (related) amino acid IDs output by the protein language model for the deleted amino acid ID at the third site in the third amino acid sequence is M, and M is less than N, then each site in the third calculation site except for these M word elements representing (related) amino acid IDs (i.e., the NM sites at which no word element representing (related) amino acid ID is output) can be set to a given value to facilitate the calculation of the third loss value.

[0114] According to an embodiment of the present disclosure, in the above-mentioned method 200, masking the amino acid at the first site may include: for each first site, using a first mask placeholder to correspondingly replace the amino acid at the first site; and / or, masking the amino acid at the second site may include: taking a second site that is not adjacent to other second sites or multiple second sites that are adjacent to each other as a group of second sites, and using a second mask placeholder to correspondingly replace the word elements at a group of second sites.

[0115] The first mask placeholder and the second mask placeholder may be the same or different.

[0116] For example, when the site where X0 X1 in X0 X1 X2 X3 X4 is located is used as the first site, the masked amino acid sequence is MM X2 X3 X4; when the site where X0 X1 in X0 X1 X2 X3 X4 is located is used as the second site, the masked amino acid sequence is M X2 X3 X4.

[0117] It is hoped that the first type of training task will enable the model to extract amino acid sequence representations, and it is necessary to maintain a correct understanding of the position of each amino acid. Therefore, in the first type of training task, the masking operation does not change the length of the amino acid sequence. It is hoped that the first subtask in the second type of training task will enable the model to generate different amino acid sequences based on an amino acid sequence, and may even generate two amino acids from one amino acid. Therefore, a certain degree of flexibility needs to be maintained. In the second type of training task, the masking operation can change the length of the amino acid sequence.

[0118] According to an embodiment of the present disclosure, in the above method 200, the third amino acid sequence can be obtained by splicing the light chain and heavy chain of the antibody, or can be obtained by splicing the light chain variable region and heavy chain variable region of the antibody; selecting one or more third sites in the third amino acid sequence that are located at the tail of the sequence and adjacent to each other includes: selecting the site in the third amino acid sequence corresponding to the chain located at the tail of the sequence as the third site; the chain located at the tail of the sequence is one of the light chain, the light chain variable region, the heavy chain, and the heavy chain variable region.

[0119] It is understandable that by training the model using the second subtask in the second type of training task, the model is expected to be able to perform two types of inference tasks. Task 1: predict the second half of an amino acid sequence based on the first half. Task 2: predict another amino acid sequence that can pair with an amino acid sequence based on an amino acid sequence. A specific example of Task 2 is predicting the pairing of one antibody chain based on its other chain, such as predicting VH based on VL or VL based on VH. When predicting VH based on VL, VL-VH can be spliced ​​into a single amino acid sequence, with the site where VH is located selected as the third site. When predicting VL based on VH, VH-VL can be spliced ​​into a single amino acid sequence, with the site where VL is located selected as the third site. For example, if VH is X1 X2 X3 X4 EOS and VL is X5 X6 X7 EOS, when predicting VL based on VH, the spliced ​​amino acid sequence is X1 X2 X3 X4 EOS X5 X6 X7 EOS, and the deleted site is X5 X6 X7 EOS.

[0120] According to an embodiment of the present disclosure, the protein language model in any of the above embodiments may include a representation extraction layer, a transformer block, and a classification layer.

[0121] The representation extraction layer can be used to determine the representation corresponding to the word-gram based on the word-gram, and the representation corresponding to the word-gram can be obtained by the word-gram representation (representation determined only based on the word-gram content) and the position representation (representation determined based on the position of the word-gram) (for example, the word-gram representation and the position representation are added together). The position representation can include a relative position representation and / or an absolute position representation. Both the word-gram representation and the position representation can be obtained by looking up the mapping table. For details, please refer to the description in GLM, which will not be repeated here. The only difference from GLM is that in the mapping table of the word-gram representation, the embodiment of the present disclosure records the mapping relationship between various amino acid IDs and various tags and the word-gram representation.

[0122] The transformer block can be a transformer block, such as one with a GLM-like architecture, which unifies the masked language model and the autoregressive model through different attention mask matrices. The disclosed embodiments unify the comprehension and generation tasks through a special attention mask matrix and hybrid training of the two types of tasks.

[0123] Figure 4 shows a method 400 for extracting amino acid sequence representations using a protein language model according to an embodiment of the present disclosure. The protein language model can be a trained protein language model obtained by training using any of the method embodiments described above. According to an embodiment of the present disclosure, the trained protein language model may include a representation extraction layer, a transformer block, and a classification layer. As shown in Figure 4, the method 400 includes: step S401, obtaining a first target amino acid sequence; step S402, inputting the first target amino acid sequence into the representation extraction layer and transformer block of the trained protein language model, and using the output of the protein language model transformer block as the representation of the first target amino acid sequence.

[0124] It should be noted that, on the one hand, the transformer block can encode the representation corresponding to the corresponding word element extracted by the extraction layer of the model (such as word element representation, position representation or fragment representation, etc.) together with the representation of other words (such as adjacent words), thereby encoding useful context information into the corresponding representation of the corresponding word element. On the other hand, the output of the transformer block (for example, a sequence of representation vectors composed of the representation vectors of each word element in the first target amino acid sequence) can be matrix multiplied with the classification matrix of the classification layer (that is, the representation vector of each word element output by the transformer block is vector-inner-producted with each row vector or column vector (depending on the specific arrangement) of the classification matrix), thereby obtaining the classification of each candidate word element in the first target amino acid sequence output by the classification layer (for example, what is the amino acid ID of the word element or what kind of tag the word element represents). Thus, the output of the protein language model transformer block can be obtained as the representation of the first target amino acid sequence. Such a sequence representation integrates the contextual relationship between each word element in the sequence and is beneficial to the application of downstream modules. Furthermore, the representation of the first target amino acid sequence may be a representation vector sequence, and the representation vectors in the representation vector sequence are in one-to-one correspondence with the word units in the first target amino acid sequence.

[0125] For example, after obtaining a representation of the first target amino acid sequence, it can be used for a variety of downstream tasks, such as protein structure prediction tasks and protein physicochemical index prediction tasks. The representation of the first target amino acid sequence can be used for downstream tasks by inputting the representation of the first target amino acid sequence into an MLP (multi-layer perceptron) corresponding to the downstream task to obtain the downstream task results. The MLP can be pre-trained. The trained protein language model can also be further fine-tuned for specific downstream tasks.

[0126] Figure 5 illustrates a method 500 for generating new amino acid sequences using a protein language model, according to an embodiment of the present disclosure. The protein language model can be a trained protein language model obtained by training using any of the method embodiments described above. Method 500 can be used, for example, to generate a mutated sequence at a specified position in a known amino acid sequence, or to generate an amino acid sequence in a CDR region that differs from a known sequence.

[0127] As shown in FIG5 , the method 500 includes: step S501, obtaining a second target amino acid sequence (the goal of the method 500 is to replace amino acids at some sites in the second target amino acid sequence to generate a new amino acid sequence); step S502, performing a second preprocessing operation on the second target amino acid sequence to obtain a second target preprocessed amino acid sequence; wherein the site to be replaced is used as the second site; and the length of the second target preprocessed amino acid sequence is L3; step S503, inputting the second target preprocessed amino acid sequence into a trained protein language model; and inputting the word element output at the kth position of the trained protein language model as the word element input at the k+1th position into the trained protein language model; wherein k is greater than L3. It is understandable that in step S503, the predicted start symbol can also be used as the input word element at the L3+1th position. Step S504: concatenate the second target preprocessed amino acid with the output from the L3+1th position to the L3+nth position of the trained protein language model to obtain a new amino acid sequence; n is the total number of new amino acids corresponding to the amino acid at the second position; or, concatenate the second target preprocessed amino acid with the output word from the L3+1th position to the position -1 where the word output by the trained protein language model is a terminator, to obtain a new amino acid sequence.

[0128] For example, a mutant amino acid or a new amino acid sequence can be generated based on a preset mutation position and number of mutations from an amino acid sequence of length 6. For example, it is desired to mutate the amino acids at positions 2 and 5 of the amino acid sequence, with the amino acid at position 2 mutated to one new amino acid and the amino acid at position 5 mutated to two new amino acids. In this case, n = 3. The amino acid sequence of length 6 is X1 X2 X3 X4 X5 EOS, and its corresponding preprocessed amino acid sequence is X1 M1 X3X4 M2 EOS, with L3 = 6. In step 503, the six tokens in the preprocessed amino acid sequence are used as input for positions 1-6, respectively, the predicted start symbol S is used as input for position 7, the output token of the model at position 7 is used as input for position 8, and the output token of the model at position 8 is used as input for position 9. In step 504, the output word of the model at the 7th position is used to replace M1 in the preprocessed amino acid sequence, and the output word of the model at the 8th and 9th positions is used to replace M2 in the preprocessed amino acid sequence to obtain a mutation result.

[0129] In the inference phase of the model, the characteristic of the autoregressive model is that we use the word element output at the kth position as the input word element at the k+1th position. Exemplarily, to avoid unnecessary output, k can also be made smaller than Lmax. Lmax is, for example, L3+n, or the smaller of L3+n and the position where the word element output by the trained protein language model is the prediction terminator. It is understandable that when the last output position of the model is L3+n, the number of words predicted by the model has reached the required number of words, and at this time, there is no need to input a word element to the model at the next position for the model to continue outputting. When the model outputs the terminator, it means that the model has stopped predicting, and there is no need to input a word element to the model at the next position for the model to continue outputting. In the above two cases, the model can stop outputting words, thereby saving computing resources.

[0130] According to an embodiment of the present disclosure, the trained protein language model may include a representation extraction layer, a transformer block, and a classification layer. As described above, the model can obtain the probability that each candidate word is at the site required for prediction. Further, for at least one position in the position used for splicing (from the L3+1 position to the L3+n position, or from the L3+1 position to the position -1 where the predicted terminator is located), an alternative word that meets the preset conditions can be selected as the output word for that position. The alternative output word includes, for example, 20 amino acids and various tags. The preset condition may be that the corresponding probability meets at least one of the preset probability ranking condition, the probability threshold condition, and the alternative word content condition. The selection may be randomly selected from the alternative words that meet the preset conditions. For example, select from the first few alternative words whose sum of probabilities reaches 90%, randomly select from the words that are not predicted terminators, select the alternative word ranked second in probability, etc. The probability is predicted by the classification layer. In this way, the desired new amino acid types can be flexibly selected based on the word units predicted by the model, which not only makes it easy to obtain an amino acid sequence different from the second target amino acid sequence, but also enables multiple amino acid sequences different from the second target amino acid sequence to be obtained when the setting strategies of the preset conditions are different.

[0131] It is understandable that the classification layer includes a classification matrix (linear layer) and a softmax function. After the output of the transformer block is matrix multiplied with the classification matrix of the classification layer, the matrix multiplication result is passed through the softmax function to obtain the probability corresponding to each candidate word at each position. Under normal circumstances, the candidate word with the highest probability at that position can be used as the output of that position. However, in the scenario of generating a new amino acid sequence, the amino acid type with the highest probability has a greater probability of being consistent with the original amino acid type. Therefore, a word can be selected as the output word from the candidate words that meet the preset conditions to ensure that there is a greater probability of generating an amino acid different from the original amino acid type, and when the preset conditions and selection strategies are different, multiple new amino acids can be generated.

[0132] Figure 6 shows a method 600 for obtaining relevant amino acid sequences using a protein language model according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the protein language model can be a trained protein language model obtained by training using any of the method embodiments described above. As shown in Figure 6, the method 600 includes: step S601, inputting the third target preprocessed amino acid sequence into the trained protein language model; the length of the third target preprocessed amino acid sequence is L4, where L4 is greater than or equal to 0; inputting the word element output at the qth position of the trained protein language model as the word element input at the q+1th position into the trained protein language model; where q is greater than L4. It is understandable that in step S601, the predicted start symbol can also be used as the input word element at the L4+1th position. Step S602: The word element output from the L4+1th position of the trained protein language model to the position where the word element output by the trained protein language model is the terminator - 1 position is used as the relevant amino acid sequence; or, the word element output from the L4+1th position of the trained protein language model to the preset position is used as the said relevant amino acid sequence.

[0133] In an embodiment, the related amino acid sequence is an amino acid sequence continued based on the third target amino acid sequence. In this case, the third target amino acid sequence can be an amino acid sequence containing amino acid types, or it can be empty (in this case, the amino acid sequence is directly predicted from the beginning). Alternatively, the related amino acid sequence is an amino acid sequence paired with the third target amino acid sequence. The amino acid sequence from the L4+1 position to the predicted terminator can be used as the related amino acid sequence, or the amino acid sequence of the required length (L4+1 position to the preset position) can be intercepted as the related amino acid sequence.

[0134] For example, when the task of obtaining the relevant amino acid sequence is to give the first half of the amino acid sequence and the model is required to predict the second half of the amino acid sequence (i.e., continue the amino acid sequence), the first half of the amino acid sequence can be used as the third target amino acid sequence; when the task is to give the heavy chain or heavy chain variable region of an antibody and the model is required to predict the paired light chain or light chain variable region, the heavy chain or heavy chain variable region of the antibody can be used as the third target amino acid sequence.

[0135] For example, the third target amino acid sequence is X1 X2 X3 X4 X5. In step 601, X1 X2 X3 X4 X5 is used as the model's input tokens at positions 1 to 5, EOS is used as the model's input token at position 6, the model's output token at position 6 is used as the model's input token at position 7, the model's output token at position 7 is used as the model's input token at position 8, and so on, until the model's output token at position 10 is a predicted terminator. The output tokens from positions 6 to 9 are used as the relevant amino acid sequence.

[0136] Exemplarily, a trained protein language model includes a representation extraction layer, a transformer block, and a classification layer. For at least one position between the L4+1th position and the position -1th position where the token output by the trained protein language model is a terminator, a candidate token that meets a preset condition is selected as the output token for that position. The preset condition may be that the corresponding probability satisfies at least one of a preset probability ranking condition, a probability threshold condition, and a token content condition. The candidate tokens are selected from different types of amino acids and tags. The probabilities are predicted by the classification layer. See the previous description and will not be repeated here.

[0137] In the field of protein design, the perplexity of designed proteins is often of interest. Perplexity reflects the manufacturability of a protein. The amino acid sequences of proteins that exist in nature tend to have lower perplexity and higher manufacturability, while the amino acid sequences of proteins that are unlikely to exist in nature tend to have higher perplexity and lower manufacturability. Determining perplexity through wet experiments is expensive, so it's desirable to predict the perplexity of amino acid sequences through models.

[0138] According to an embodiment of the present disclosure, the trained protein language model may include a representation extraction layer, a transformer block, and a classification layer. Furthermore, the method 600 may further include: step S603: for each position from the L4+1th position to the position -1th position where the trained protein language model outputs the terminator, the probability corresponding to the output word at that position is used as the perplexity factor for that position; and step S604: based on the perplexity factor for each position, obtaining the perplexity of the relevant amino acid sequence.

[0139] For example, if the probability of outputting a word at position L4+1 is 80%, the probability of outputting a word at position L4+2 is 70%, and the probability of outputting a word at the last position is 50%, then the perplexity of the amino acid sequence can be obtained based on the probabilities at each position. Obtaining the perplexity of the amino acid sequence based on the perplexity factor at each position is conventional, for example, by multiplying the perplexities at each position.

[0140] As can be seen, for an amino acid sequence generated by method 600, its perplexity can be obtained while the amino acid sequence is generated. For an amino acid sequence not generated by method 600, the perplexity of the amino acid sequence can also be predicted using a protein language model. For an amino acid sequence generated by method 600, its perplexity can also be predicted using a protein language model in an additional step, rather than obtaining the perplexity while generating the amino acid sequence.

[0141] In particular, when the amino acid sequence is obtained by method 500 or method 600, when the perplexity obtained by method 500 or method 600 is higher than the perplexity threshold, method 500 or method 600 can be repeatedly performed until a new amino acid sequence that meets the perplexity requirement is obtained.

[0142] FIG7 shows a method 700 for predicting amino acid sequence perplexity using a protein language model according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the protein language model may be a trained protein language model obtained by training using any of the method embodiments described above. According to an embodiment of the present disclosure, the trained protein language model may include a representation extraction layer, a transformer block, and a classification layer. As shown in FIG7 , the method 700 includes:

[0143] Step S701: Obtain a fourth target amino acid sequence; the fourth target amino acid sequence is an amino acid sequence whose perplexity is to be determined.

[0144] Step S702: sequentially performing the first preprocessing operation on each site in the fourth target amino acid sequence as the first site to obtain a plurality of fourth target preprocessed amino acid sequences; wherein each fourth target preprocessed amino acid sequence corresponds to a masked site;

[0145] For example, the fourth target amino acid sequence is X1 X2 X3 X4 X5 EOS, and the plurality of fourth target pre-processed amino acid sequences are:

[0146] M X2 X3 X4 X5 EOS;

[0147] X1 M X3 X4 X5 EOS;

[0148] X1 X2 M X4 X5 EOS;

[0149] X1 X2 X3 M X5 EOS;

[0150] X1 X2 X3 X4 M EOS.

[0151] The fourth target amino acid sequence may further include X1 X2 X3 X4 X5 M.

[0152] Step S703: Input a first current preprocessed amino acid sequence from the plurality of fourth target preprocessed amino acid sequences into a protein language model to obtain a probability predicted by the classification layer; use the probability corresponding to the first word at the first current masked site as a perplexity factor for the first current masked site; the first word is the word of the fourth target amino acid sequence at the first current masked site; the first current masked site is the masked site corresponding to the first current preprocessed amino acid sequence;

[0153] For example, M X2 X3 X4 X5 EOS is input into the protein language model. At the first output position, the model classification layer predicts the probability of each candidate word (including 20 amino acids, tags, etc.) at this position, and takes the probability corresponding to X1 as the confusion factor of the first position.

[0154] Input X1 M X3 X4 X5 EOS into the protein language model. At the second output position, the model classification layer predicts the probability of each candidate word at this position, and uses the probability corresponding to X2 as the confusion factor of the second position.

[0155] In this way, the perplexity factor at each site can be obtained.

[0156] And step S704, obtaining the perplexity of the fourth target amino acid sequence according to the perplexity factor of each site.

[0157] It can be seen that method 700 determines the perplexity in a manner similar to the first type of training task. Specifically, on the premise that the target amino acid sequence to be evaluated for perplexity is obtained, for the target amino acid sequence, the word element at each site can be sequentially covered from the starting site to the last site of the sequence, thereby obtaining multiple target pre-processed amino acid sequences. It should be noted that since the first pre-processing operation is performed on the target amino acid sequence, the model will give the probability that the covered site is each candidate word element at the site corresponding to the covered site in the target amino acid sequence, and use the probability corresponding to the real word element of the covered site as the perplexity factor corresponding to the covered site. Finally, the perplexity of the amino acid sequence is obtained based on the perplexity factor of each site. This effectively simplifies the calculation of perplexity and expands the use of the disclosed model.

[0158] FIG8 shows a method 800 for predicting amino acid sequence perplexity using a protein language model according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the protein language model may be a trained protein language model obtained by training using any of the method embodiments described above. According to an embodiment of the present disclosure, the trained protein language model may include a representation extraction layer, a transformer block, and a classification layer. As shown in FIG8 , the method 800 includes:

[0159] Step S801, obtaining a fifth target amino acid sequence; the length of the fifth target amino acid sequence is L5;

[0160] For example, the fifth target amino acid sequence is X1 X2 X3 X4 X5 EOS, and L5=6.

[0161] Step S802: sequentially performing the second preprocessing operation on each site in the fifth target amino acid sequence as the second site to obtain a plurality of fifth target preprocessed amino acid sequences; wherein each fifth target preprocessed amino acid sequence corresponds to a masked site;

[0162] For example, a plurality of fifth target pre-processed amino acid sequences are:

[0163] M X2 X3 X4 X5 EOS;

[0164] X1 M X3 X4 X5 EOS;

[0165] X1 X2 M X4 X5 EOS;

[0166] X1 X2 X3 M X5 EOS;

[0167] X1 X2 X3 X4 M EOS.

[0168] The fifth target amino acid sequence may further include X1 X2 X3 X4 X5 M.

[0169] Step S803: Input the second current preprocessed amino acid sequence from the plurality of fifth target preprocessed amino acid sequences into the protein language model to obtain the probability predicted by the classification layer; use the probability corresponding to the second word at the L5+1th position as the perplexity factor for the second current masked position; the second word is the word of the fifth target amino acid sequence at the second current masked position; and the second current masked position is the masked position corresponding to the second current preprocessed amino acid sequence. It is understood that when the fifth target preprocessed amino acid sequence is input as the input word from position 1 to position L5 into the model, the predicted start symbol can also be input as the input word at position L5+1 into the model.

[0170] For example, M X2 X3 X4 X5 EOS is input into the protein language model. At the L5+1th output position, the model classification layer predicts the probability of each candidate word at this position, and uses the probability corresponding to X1 as the confusion factor of the current masked site (the position where X1 is located).

[0171] Input X1 M X3 X4 X5 EOS into the protein language model. At the L5+1th output position, the model classification layer predicts the probability of each candidate word at this position, and uses the probability corresponding to X2 as the confusion factor of the current masked site (the position where X2 is located).

[0172] In this way, the perplexity factor at each site can be obtained.

[0173] And step S804, obtaining the perplexity of the fifth target amino acid sequence according to the perplexity factor of each site.

[0174] It can be seen that method 800 determines the perplexity in a manner similar to the first training subtask in the second type of training task. Specifically, on the premise of obtaining the target amino acid sequence for which the perplexity is to be evaluated, for the target amino acid sequence, the word element at each site can be sequentially covered from the starting site to the last site of the sequence, thereby obtaining multiple target pre-processed amino acid sequences. Then, for each of these multiple target pre-processed amino acid sequences, it is sent to the trained protein language model to obtain the result output by the classification layer of the model (such as a matrix representation). It should be noted here that since the second pre-processing operation is performed on the target amino acid sequence, the model will give the probability that the covered site is each candidate word element after the original length of the target amino acid sequence, and use the probability corresponding to the real word element of the covered site as the perplexity factor corresponding to the covered site. Finally, the perplexity of the amino acid sequence is obtained based on the perplexity factor of each site. This effectively simplifies the calculation of perplexity and expands the use of the disclosed model.

[0175] According to an embodiment of the present disclosure, an electronic device is also provided. The electronic device may include at least one processor and a memory communicatively connected to the at least one processor. The memory may store instructions executable by the at least one processor, and the instructions may be executed by the at least one processor to enable the at least one processor to perform the method disclosed in any of the method embodiments described above.

[0176] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided. The computer instructions can be used to cause a computer to execute the method disclosed in any of the method embodiments described above.

[0177] According to an embodiment of the present disclosure, a computer program product is also provided. The computer program product may include a computer program. When executed by a processor, the computer program may implement the method disclosed in any of the method embodiments described above. According to an embodiment of the present disclosure, an electronic device, a readable storage medium, and a computer program product are also provided. Referring to Figure 9, a block diagram of an electronic device 900 that can serve as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0178] As shown in Figure 9, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. Various programs and data required for the operation of the device 900 can also be stored in the RAM 903. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0179] Multiple components in the device 900 are connected to the I / O interface 905, including: an input unit 906, an output unit 907, a storage unit 908, and a communication unit 909. The input unit 906 can be any type of device that can input information to the device 900. The input unit 906 can receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a trackpad, a trackball, a joystick, a microphone and / or a remote control. The output unit 907 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator and / or a printer. The storage unit 908 can include but is not limited to a magnetic disk, an optical disk. The communication unit 909 allows the device 900 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as Bluetooth TMdevices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0180] The computing unit 901 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning network algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as the neural network training method and / or the method for predicting protein structure. For example, in some embodiments, the neural network training method and / or the method for predicting protein structure can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the neural network training method and / or the method for predicting protein structure described above can be performed. Alternatively, in other embodiments, the computing unit 901 may be configured in any other appropriate manner (eg, by means of firmware) to execute the neural network training method and / or the protein structure prediction method.

[0181] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0182] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0183] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0184] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0185] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0186] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship is established by computer programs running on the respective computers, establishing a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosts and VPS services ("Virtual Private Servers" or simply "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0187] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0188] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but is only limited by the claims after authorization and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. In addition, the steps may be performed in an order different from that described in this disclosure. Further, the various elements in the embodiments or examples may be combined in various ways. It is important that as technology evolves, many of the elements described herein may be replaced by equivalent elements that appear after this disclosure.

Claims

1. A method for training a protein language model, comprising: During the training process, performing at least one of the two training subtasks included in the first type of training task and the second type of training task; The first type of training task includes: obtaining a first amino acid sequence; performing a first preprocessing operation on the first amino acid sequence to obtain a first preprocessed amino acid sequence; the first preprocessing operation includes selecting one or more first positions in the first amino acid sequence and masking the word units at the first positions; inputting the first preprocessed amino acid sequence into a protein language model to obtain a first prediction result; calculating a first loss value according to the word units at the first position in the first amino acid sequence and the elements at the first calculation position in the first prediction result; the first calculation position is a position aligned with the first position; The first training subtask in the second type of training task includes: obtaining a second amino acid sequence; performing a second preprocessing operation on the second amino acid sequence to obtain a second preprocessed amino acid sequence; the second preprocessing operation includes selecting one or more second sites in the second amino acid sequence and masking the word units at the second sites; inputting the second preprocessed amino acid sequence into a protein language model to obtain a second prediction result; calculating a second loss value according to the word units at the second site in the second amino acid sequence and the elements at the second calculation site in the second prediction result; the second calculation site is selected from the site after the site aligned with the second site; The second training subtask in the second type of training task includes: obtaining a third amino acid sequence; performing a third preprocessing operation on the third amino acid sequence to obtain a third preprocessed amino acid sequence; the third preprocessing operation includes selecting one or more third sites located at the end of the sequence and adjacent to each other in the third amino acid sequence, and deleting the word element at the third site; inputting the third preprocessed amino acid sequence into the protein language model to obtain a third prediction result; calculating a third loss value according to the word element at the third site in the third amino acid sequence and the element at the third calculation site in the third prediction result; the third calculation site is selected from the site after the site aligned with the third site; The parameters of the protein language model are adjusted based on the first loss value and the loss value corresponding to the second type of training task to obtain a trained protein language model, wherein the loss value corresponding to the second type of training task includes at least one of the second loss value and the third loss value.

2. The method according to claim 1, wherein: During the training process, performing at least one of the two training subtasks included in the first type of training task and the second type of training task includes: In the first training phase, a first type of training task is performed; In the second training phase, a second type of training task is performed.

3. The method according to claim 1, wherein: During the training process, performing at least one of the two training subtasks included in the first type of training task and the second type of training task includes: In the first training phase, a first type of training task is performed; In the second training phase, the first type of training tasks and the second type of training tasks are performed; In the third training phase, a second type of training task is performed.

4. The method according to claim 2 or 3, wherein: At least one of the two training subtasks included in the execution of the first type of training task and the second type of training task is performed on the first data set; After performing at least one of two training subtasks included in the first type of training task and the second type of training task on the first data set, the method further includes performing the first type of training task and / or the second type of training task on the second data set.

5. The method according to claim 1, wherein: The step of inputting the second preprocessed amino acid sequence into a protein language model to obtain a second prediction result includes: inputting the second preprocessed amino acid sequence into a protein language model, and inputting the masked amino acids into the protein language model as word units from positions L1+2 to L1+second number of sites+1; and / or, The step of inputting the third preprocessed amino acid sequence into a protein language model to obtain a third prediction result comprises: inputting the third preprocessed amino acid sequence into a protein language model, and inputting the deleted amino acids into the protein language model as word units from the L2+2th to the L2+third number of sites+1th position.

6. The method according to any one of claims 1 to 5, wherein: Masking the amino acid at the first position comprises: for each first position, using a first mask placeholder to correspondingly replace the amino acid at the first position; and / or, Masking the amino acid at the second site includes: taking a second site that is not adjacent to other second sites or a plurality of second sites that are adjacent to each other as a group of second sites, and using a second mask placeholder to correspondingly replace the word units at the group of second sites.

7. The method according to any one of claims 1 to 6, wherein: Inputting the second preprocessed amino acid sequence into a protein language model to obtain a second prediction result further includes: inputting the second placeholder as a word at the L1+1th position into the protein language model; and / or, Inputting the third preprocessed amino acid sequence into a protein language model to obtain a third prediction result also includes: inputting the second placeholder as a word element at the L2+1th position into the protein language model.

8. The method according to claim 1, wherein: The third amino acid sequence is obtained by splicing the light chain and heavy chain or the light chain variable region and heavy chain variable region of the antibody; Selecting one or more third sites in the third amino acid sequence that are located at the end of the sequence and adjacent to each other comprises: selecting a site in the third amino acid sequence that corresponds to the chain located at the end of the sequence as the third site; The chain located at the tail of the sequence is the light chain, light chain variable region, heavy chain or heavy chain variable region.

9. The method according to claim 1, wherein: The protein language model includes a representation extraction layer, a transformer block and a classification layer.

10. A method for extracting amino acid sequence representation using a protein language model, wherein the protein language model is a trained protein language model obtained by training using the method according to any one of claims 1 to 9, and the trained protein language model comprises a representation extraction layer, a transformer block and a classification layer; The method comprises: obtaining a first target amino acid sequence; The first target amino acid sequence is input into the representation extraction layer and the transformer block of the trained protein language model, and the output of the protein language model transformer block is used as the representation of the first target amino acid sequence.

11. A method for generating a new amino acid or mutant amino acid sequence using a protein language model, wherein the protein language model is a trained protein language model obtained by training using the method according to any one of claims 1 to 9, the method comprising: obtaining a second target amino acid sequence; The amino acids at some positions in the second target amino acid sequence are replaced to generate new amino acids or mutant amino acid sequences; Performing the second preprocessing operation on the second target amino acid sequence to obtain the second target preprocessed amino acid sequence; wherein the site to be replaced is used as the second site; and the length of the second target preprocessed amino acid sequence is L3; Inputting the second target preprocessed amino acid sequence into the trained protein language model; Input the word unit output at the kth position of the trained protein language model as the word unit input at the k+1th position into the trained protein language model; wherein k is greater than L3; The second target preprocessed amino acid is concatenated with the output word unit from the L3+1th position to the L3+nth position of the trained protein language model to obtain the new amino acid or mutant amino acid sequence; n is the total number of new amino acids or mutant amino acids corresponding to the second site amino acid; or, the second target preprocessed amino acid is concatenated with the output word unit from the L3+1th position of the trained protein language model to the position -1 where the word unit output by the trained protein language model is a terminator to obtain the new amino acid or mutant amino acid sequence.

12. The method according to claim 11, wherein: The trained protein language model includes a representation extraction layer, a transformer block and a classification layer; for at least one of the positions used for splicing, an alternative word that meets a preset condition is selected as the output word of the position; the preset condition can be that the corresponding probability meets at least one of a preset probability ranking condition, a probability threshold condition, and an alternative word content condition; the alternative word is selected from different types of amino acids and tags; and the probability is predicted by the classification layer.

13. A method for obtaining a related amino acid sequence using a protein language model, wherein the protein language model is a trained protein language model obtained by training using the method according to any one of claims 1 to 9, the method comprising: Inputting the third target preprocessed amino acid sequence into the trained protein language model; The length of the third target preprocessed amino acid sequence is L4, where L4 is greater than or equal to 0; Input the word unit output at the qth position of the trained protein language model as the word unit input at the q+1th position into the trained protein language model; wherein q is greater than L4; The word unit output from the position L4+1 to the position -1 where the word unit output by the trained protein language model is the terminator is used as the word unit Related amino acid sequence; or, the word unit output from the L4+1th position to the preset position of the trained protein language model is used as the related amino acid sequence; Wherein, the related amino acid sequence is an amino acid sequence continued based on the third target amino acid sequence; or, the related amino acid sequence is an amino acid sequence paired with the third target amino acid sequence.

14. The method according to claim 13, wherein: The trained protein language model includes a representation extraction layer, a transformer block and a classification layer. For at least one position from the L4+1th position to the position -1th position where the word unit output by the trained protein language model is a terminator, an alternative word unit that meets a preset condition is selected as the output word unit of the position; the preset condition can be that the corresponding probability satisfies at least one of a preset probability ranking condition, a probability threshold condition, and an alternative word unit content condition; the alternative word unit is selected from different types of amino acids and tags; and the probability is predicted by the classification layer.

15. The method according to claim 14, further comprising: For each position from the L4+1th position to the position -1th position where the trained protein language model outputs the terminator, the probability corresponding to the output word at the position is used as the perplexity factor of the position; According to the perplexity factor of each position, the perplexity of the related amino acid sequence is obtained.

16. A method for predicting amino acid sequence perplexity using a protein language model, wherein the protein language model is a trained protein language model obtained by training using the method of any one of claims 1 to 9, wherein the trained protein language model comprises a representation extraction layer, a transformer block and a classification layer, and the method comprises: obtaining a fourth target amino acid sequence; Sequentially taking each site in the fourth target amino acid sequence as the first site, performing the first preprocessing operation to obtain a plurality of fourth target preprocessed amino acid sequences; wherein each fourth target preprocessed amino acid sequence corresponds to a masked site; Inputting the first current preprocessed amino acid sequence among the plurality of fourth target preprocessed amino acid sequences into the protein language model to obtain the probability predicted by the classification layer; using the probability corresponding to the first word at the first current masked site as the perplexity factor of the first current masked site; the first word is the word of the fourth target amino acid sequence at the first current masked site; the first current masked site is the masked site corresponding to the first current preprocessed amino acid sequence; According to the perplexity factors of the respective sites, the perplexity of the fourth target amino acid sequence is obtained.

17. A method for predicting amino acid sequence perplexity using a protein language model, wherein the protein language model is a trained protein language model obtained by training using the method of any one of claims 1 to 9, wherein the trained protein language model comprises a representation extraction layer, a transformer block and a classification layer, and the method comprises: obtaining a fifth target amino acid sequence; The length of the fifth target amino acid sequence is L5; Sequentially taking each site in the fifth target amino acid sequence as the second site, performing the second preprocessing operation to obtain a plurality of fifth target preprocessed amino acid sequences; wherein each fifth target preprocessed amino acid sequence corresponds to a masked site; Inputting the second current preprocessed amino acid sequence among the plurality of fifth target preprocessed amino acid sequences into the protein language model to obtain the probability predicted by the classification layer; using the probability corresponding to the second word at the L5+1th site as the perplexity factor of the second current masked site; the second word is the word of the fifth target amino acid sequence at the second current masked site; the second current masked site is the masked site corresponding to the second current preprocessed amino acid sequence; According to the perplexity factors of the respective sites, the perplexity of the fifth target amino acid sequence is obtained.

18. An electronic device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 17.

19. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-17.

20. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 17 is implemented.

Images