Antibody language model training method, antibody sequence prediction method and device

Abstract

The invention discloses an antibody language model training method and device and an antibody sequence prediction method and device, and the method comprises the steps: obtaining a training sample set, and enabling training samples in the training sample set to be generated through a pre-constructed antibody multivariate coding vocabulary; performing structured corpus construction processing on training samples in the training sample set to obtain a training corpus data set; and according to the training corpus data set, performing dual-task cooperative training processing on a language model, and generating a pre-training antibody language model for antibody design. The data processing capability of the antibody language model can be improved, and the method can be widely applied to the technical field of artificial intelligence.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an antibody language model training method, an antibody sequence prediction method, and an apparatus thereof. Background Technology

[0002] With the development of high-throughput sequencing technology, massive amounts of antibody sequence data have been generated, providing a valuable data foundation for understanding antibody function and characteristics through artificial intelligence methods. Large-scale language models based on the Transformer architecture (such as BERT and GPT) capture deep semantic and grammatical patterns in the data by mapping discrete symbols (such as words) into continuous vector representations. Among related technologies, some treat protein or antibody sequences as a "biological language" composed of amino acid letters, developing various antibody language models. However, in practical applications, it has been found that these antibody language models suffer from insufficient information integration during information learning and lack mutation sensing capabilities, resulting in poor antibody optimization and design capabilities and affecting the prediction efficiency of antibody sequences.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0004] The main objective of this application is to propose an antibody language model training method, an antibody sequence prediction method, and an apparatus thereof, which can improve the prediction efficiency of antibody sequences.

[0005] To achieve the above objectives, one aspect of this application proposes an antibody language model training method, the method comprising: A training sample set is obtained, wherein the training samples in the training sample set are generated by a pre-constructed antibody multivariate coding vocabulary; The training samples in the training sample set are processed to construct a structured corpus, resulting in a training corpus dataset. The language model is trained using a dual-task collaborative training process based on the training corpus dataset to generate a pre-trained antibody language model for antibody design.

[0006] In some embodiments, obtaining the training sample set includes the following steps: Obtain antibody sequence samples; The amino acid letters of the antibody sequence sample are encoded according to the antibody multivariate coding vocabulary to obtain antibody sequence coding data; The antibody sequence sample is initially encoded, and the positional encoding data is obtained by encoding the positional data using the multivariate encoding vocabulary based on the positional annotations in the initial encoding data. The gene recombination information of the antibody sequence sample is encoded according to the antibody multivariate coding vocabulary to obtain gene recombination coding data; The antibody sequence encoding data, the position encoding data, and the gene recombination encoding data are combined with identifiers to generate training samples.

[0007] In some embodiments, the process of constructing a structured corpus from the training samples in the training sample set to obtain a training corpus dataset includes the following steps: The training samples were subjected to sequence analysis to obtain antibody sequences, germline sequences, recombinant gene information, and mutation information. The training corpus dataset is generated by performing question-and-answer combination structure processing on the training samples based on the antibody sequence, the germline sequence, the recombinant gene information, and the mutation information.

[0008] In some embodiments, the step of performing dual-task collaborative training on the language model based on the training corpus to generate a pre-trained antibody language model for antibody design includes the following steps: The training corpus is input into the language model for training; the training samples in the training corpus include contextual content and predicted content. The context content is subjected to masked language prediction processing to obtain a first prediction result; The predicted content is then subjected to next-word prediction processing to obtain a second prediction result; The loss value of the language model is calculated based on the first prediction result and the second prediction result, and the parameters of the language model are updated based on the calculated loss value to obtain the pre-trained antibody language model.

[0009] In some embodiments, performing masked language prediction processing on the context content to obtain a first prediction result includes the following steps: The context content is randomly masked to obtain mask data; The mask data is subjected to self-attention feature update processing to obtain updated features; Based on the updated features, the content at the mask position in the mask data is predicted to obtain the first prediction result.

[0010] In some embodiments, performing next-word prediction processing on the predicted content to obtain a second prediction result includes the following steps: Determine the task prompt based on the predicted content; Based on the context content and the task prompt, lexical prediction processing is performed to obtain the second prediction result.

[0011] To achieve the above objectives, another aspect of this application proposes an antibody sequence prediction method, comprising the following steps: Obtain the antibody sequence to be predicted; The antibody sequence to be predicted is input into a pre-trained antibody language model for sequence completion or sequence error correction to obtain the target antibody sequence; the antibody language model is trained using the antibody language model training method described above.

[0012] To achieve the above objectives, another aspect of this application proposes an antibody language model training system, the system comprising: The data acquisition module is used to acquire a training sample set, wherein the training samples in the training sample set are generated by a pre-constructed antibody multivariate coding vocabulary; The corpus construction module is used to perform structured corpus construction processing on the training samples in the training sample set to obtain the training corpus dataset; The task training module is used to perform dual-task collaborative training on the language model based on the training corpus dataset to generate a pre-trained antibody language model for antibody design.

[0013] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0014] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0015] To achieve the above objectives, another aspect of this application provides a computer program product that stores a computer program, which, when executed by a processor, implements the aforementioned method.

[0016] The embodiments of this application include at least the following beneficial effects: This application provides an antibody language model training method, an antibody sequence prediction method, and an apparatus thereof. This scheme obtains a training sample set, where the training samples are generated from a pre-constructed antibody multi-dimensional coding vocabulary. This allows for the integration of multi-dimensional information through the antibody multi-dimensional coding vocabulary to construct training samples, providing the model with rich biological context. Furthermore, this scheme obtains a training corpus dataset by performing structured corpus construction processing on the training samples in the training sample set. This allows for deep fusion of multi-dimensional antibody information, further enriching the training corpus and enhancing the model's generalization ability and versatility. Moreover, this scheme performs dual-task collaborative training on the language model based on the training corpus dataset, generating a pre-trained antibody language model for antibody design. This enables the model to not only enhance its understanding of contextual information but also to generate targeted question answers, improving the data processing capability of the antibody language model and enabling accurate prediction of antibody sequences. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of an implementation environment provided in the embodiments of this application; Figure 2 This is a flowchart of an antibody language model training method provided in an embodiment of this application; Figure 3 This is an example diagram of a corpus construction process provided in an embodiment of this application; Figure 4 This is an example diagram illustrating the inferability of antibody multivariate information provided in an embodiment of this application; Figure 5 This is a schematic diagram of a dual-task collaborative training provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an antibody language model training system provided in an embodiment of this application; Figure 7 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of systems and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0019] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0020] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0022] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0023] 1) Artificial intelligence (AI) is the theory, methods, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence.

[0024] 2) Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close relationship with linguistic research. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0025] With the development of high-throughput sequencing technology, massive amounts of antibody sequence data have been generated, providing a valuable data foundation for understanding antibody function and characteristics through artificial intelligence methods. In recent years, large-scale language models based on the Transformer architecture (such as BERT and GPT) have achieved great success in the field of natural language processing. Their core idea is to map discrete symbols (such as words) into continuous vector representations, thereby capturing deep semantic and grammatical patterns in the data. Inspired by this, researchers have treated protein or antibody sequences as a "biological language" composed of amino acid letters, developing various Protein Language Models (PLMs).

[0026] For example, general protein language models such as ESM and ProtTrans are pre-trained on massive amounts of general protein sequences, learning universal protein sequence rules. Although these models can be applied to antibodies to some extent, the proportion of antibody sequences in their training data is limited, and they do not consider the specific biological background of antibodies. Therefore, their performance is not optimal for antibody-specific tasks.

[0027] Specialized antibody sequence language models, such as AntiBERTy and IgLM, are pre-trained on millions of antibody sequences, better capturing the statistical distribution characteristics of antibody sequences. However, most related antibody language models typically only use the antibody sequence itself as input, and their learning process heavily relies on sequence information. For example, each antibody evolves through V(D)J rearrangements (resulting in a specific germline sequence) and somatic high-frequency mutations (SHMs). Ignoring its germline origin and mutation history, the model cannot distinguish between conserved framework regions inherited from the germline and key complementarity-determining regions (CDRs) formed by acquired mutations that determine specificity, resulting in imprecise and biologically meaningful representations. The model struggles to deeply understand the impact mechanisms of somatic high-frequency mutations, limiting its application potential in antibody optimization and design. Furthermore, related technologies cannot fully exploit the multi-dimensional and multi-level information contained in sequence data, resulting in room for improvement in the generalization ability and interpretability of the learned feature representations.

[0028] In view of this, this application provides an antibody language model training method, an antibody sequence prediction method, and an apparatus thereof. This scheme obtains a training sample set, performs structured corpus construction processing on the training samples in the training sample set to obtain a training corpus dataset. This dataset can construct a pre-training framework that deeply integrates multidimensional antibody information. The antibody sequence, its corresponding germline sequence, VDJ gene tag, and detailed mutation information are used as inputs to the model, providing rich biological context. Furthermore, the language model is subjected to dual-task collaborative training based on the training corpus dataset to generate a pre-trained antibody language model for antibody design. This model combines the dual self-supervised learning objectives of Masked Language Modeling (MLM) and Next Token Prediction (NTP), enabling the model to learn not only the contextual semantics of amino acids (through MLM) but also the generation rules and evolutionary trajectories of antibody sequences (through NTP), thereby achieving a more comprehensive and deeper understanding of antibody sequences.

[0029] This application provides an antibody language model training method, relating to the field of artificial intelligence technology. This antibody language model training method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing an antibody language model training method, but is not limited to the above forms.

[0030] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0031] Figure 1 This is a schematic diagram illustrating the implementation environment of a method provided in an embodiment of this application. (Refer to...) Figure 1 The main hardware and software components of this implementation environment include a terminal 101 and a server 102, which are communicatively connected. The method can be executed based on the interaction between the terminal 101 and the server 102. Furthermore, the terminal 101 and the server 102 can be nodes in a blockchain; this embodiment does not specifically limit this.

[0032] Figure 2 This is an optional flowchart of an antibody language model training method provided in an embodiment of this application. Figure 2 The method may include, but is not limited to, steps S201 to S203.

[0033] Step S201: Obtain a training sample set, wherein the training samples in the training sample set are generated by a pre-constructed antibody multivariate coding vocabulary; Step S202: Perform structured corpus construction processing on the training samples in the training sample set to obtain the training corpus dataset; Step S203: Perform dual-task collaborative training on the language model based on the training corpus dataset to generate a pre-trained antibody language model for antibody design.

[0034] Steps S201 to S203, as illustrated in this embodiment, generate training samples using a pre-constructed antibody multivariate coding vocabulary, thereby constructing a training sample set. This antibody multivariate coding vocabulary integrates multivariate information, enabling the generated training samples to quickly retrieve corresponding phylogenetic sequences, mutation information, and associated VDJ gene information. Then, structured corpus construction is performed on the training samples in the training sample set. Reasonable and inferable question-and-answer corpora are constructed using the amino acid sequence information, phylogenetic sequences, mutation information, and associated VDJ gene information from the training samples, further enriching the training corpus. Finally, the training corpus dataset is input into a language model for dual-task collaborative training, generating a pre-trained antibody language model for antibody design. The core architecture of this language model can be constructed using a transformer model.

[0035] In some embodiments, obtaining the training sample set includes the following steps: Obtain antibody sequence samples; The amino acid letters of the antibody sequence sample are encoded according to the antibody multivariate coding vocabulary to obtain antibody sequence coding data; The antibody sequence sample is initially encoded, and the positional encoding data is obtained by encoding the positional data using the multivariate encoding vocabulary based on the positional annotations in the initial encoding data. The gene recombination information of the antibody sequence sample is encoded according to the antibody multivariate coding vocabulary to obtain gene recombination coding data; The antibody sequence encoding data, the position encoding data, and the gene recombination encoding data are combined with identifiers to generate training samples.

[0036] In this embodiment, the original antibody sequence sample is obtained, and the antibody sequence sample is encoded using a pre-constructed antibody multivariate coding vocabulary to generate training samples, thereby constructing a training sample set. Specifically, this embodiment encodes the amino acid letters of the antibody sequence sample using the antibody multivariate coding vocabulary. Specifically, the sequence in the antibody sequence sample is encoded using 20 standard amino acid single-letter representations combined with the unknown amino acid representation letter X, resulting in antibody sequence coding data. Furthermore, by associating and encoding the amino acid positions of the antibody sequence sample, precise association of amino acid positions is established using IMGT standard position numbers, and special structural positions outside the IMGT standard numbers are marked using 26 lowercase letters. This embodiment also directly encodes VDJ gene recombination information, using 466 known VDJ genotypes + unknown V genotypes + unknown D genotypes + unknown J genotypes for gene recombination information encoding. This embodiment also establishes statement identifiers by dividing the module boundaries of multi-source information; for example, it can construct... <ab> , <gl> , <mut> , <vdj>The identifiers represent antibody sequence symbols, phylogenetic sequence symbols, mutation information symbols, and VDJ tag symbols, respectively. Furthermore, a multivariate antibody coding vocabulary can be used to handle common technical scenarios and construct general-purpose special tokens, including... <unk> , <mask> , <pad> , <end>These represent unknown characters, mask characters, padding characters, and end characters, respectively. Embodiments of this application can also define the overall structure of the input using content delimiters, including... <content> , <answer>, representing the context content identifier and the answer identifier, respectively.

[0037] In antibody language models, conventional vocabularies consist only of "amino acids (uppercase letters)" and "general special tokens," resulting in a single information composition and limited information content. However, the embodiments of this application construct antibody sequences using a multivariate antibody coding vocabulary, which integrates diverse information. After obtaining the antibody sequence, the corresponding germline sequence, mutation information, and the associated VDJ gene can be quickly retrieved, providing a data foundation for subsequent model training.

[0038] In some embodiments, the process of constructing a structured corpus from the training samples in the training sample set to obtain a training corpus dataset includes the following steps: The training samples were subjected to sequence analysis to obtain antibody sequences, germline sequences, recombinant gene information, and mutation information. The training corpus dataset is generated by performing question-and-answer combination structure processing on the training samples based on the antibody sequence, the germline sequence, the recombinant gene information, and the mutation information.

[0039] In the embodiments of this application, such as Figure 3 As shown, by analyzing the training samples using tools, we can obtain information such as their phylogenetic sequence, the VDJ gene to which they belong, and where mutations have occurred compared to the phylogenetic sequence. We can also obtain antibody sequences, phylogenetic sequences, recombinant gene information, and mutation information. This allows us to construct some reasonable question-and-answer corpora that satisfy inference. Here, context represents the context content input to the model, and answer represents the content that the model needs to predict. By combining the context content and the predicted content into a question-and-answer structure, we can generate the training corpus dataset.

[0040] Taking Content=antibody + germline + mutation, Answer=vdj as an example, a corpus of combined antibody information question-and-answer structures is as follows: ' <content> ', ' <ab>', 'Q', 'V', 'Q', 'L', 'V', 'Q', ......, ' <gl>', 'Q', 'V', 'Q', 'L', 'V', 'Q', ......, ' <mut>', '62', 'N', '107', 'E', '108', 'K', '111', 'c', 'R', '112', 'c', 'N', '112', 'b', 'H', ' <answer> ',' <vdj>', 'IGHV1-2*04', 'IGHD5-18*01', 'IGHJ6*02', ' <end>'; This application embodiment uses statement identifiers and content separators to separate and differentiate between questions and answers and multi-dimensional information. This allows for a more intuitive understanding of the corpus format, and the structured input is beneficial for model understanding.

[0041] For example, this corpus can be viewed as a "question-answer pair", composed of... <content>and <answer>It consists of two main parts, separated by specific labels to facilitate model parsing. Among them, <content>This is the input part of the model, containing all the known information needed to answer the question. In this example, it contains three pieces of information: the antibody sequence... <ab>phylogenetic sequence <gl>and mutation information <mut> 。 <answer>It is the learning objective or output part of the model, which the model needs to learn based on... <content>The predicted content in this corpus is the predicted VDJ gene.

[0042] in <mut>In the segment, '62', 'N' indicates that, compared to the Germline sequence, the 62nd site in the IMGT encoding is mutated to N. '111', 'c', 'R' indicates that, compared to the Germline sequence, the 111c site in the IMGT encoding is mutated to R.

[0043] The corpus allows for information redundancy, specifically meaning that the information dimensions contained in the input content may exceed the key information actually needed by the model to infer the target answer. However, this redundancy will not interfere with the model's answer prediction because, during training, the model can automatically identify and focus on the "core information necessary for inferring the answer." Even if the content contains additional related information, it will not negatively affect its judgment. For example: Content = antibody + germline + mutation, Answer = vdj. The VDJ gene can be inferred from the antibody sequence or germline sequence alone.

[0044] Content = antibody + vdj + mutation, Answer = germline, which allows us to infer the germline sequence from the antibody sequence alone.

[0045] However, the corpus does not allow for random combinations; it requires biological plausibility and inference. Specifically, the following types of erroneous combinations exist: Content=vdj, Answer=antibody, is not allowed because the mutation information is missing, making it impossible to infer the complete antibody sequence from the VDJ gene alone.

[0046] Content=germline, Answer=mutation, is not allowed because the antibody sequence is missing, making it impossible to infer mutation information from the germline sequence alone.

[0047] like Figure 4 As shown, in this embodiment, any one of the following information—VDJ / Germline / Mutation—can be derived from Antibody; however, Germline + Mutation are required to derive Antibody. When constructing the corpus, additional information can be added to ensure the inferability of the result. The following syntax is correct: Antibody->VDJ, satisfying the inferability; Antibody + Germline -> VDJ satisfies inferability, where Germline is additional information that does not affect inferability; Antibody+Germline+Mutation->VDJ satisfies inferability, where Germline and Mutation are additional information that does not affect inferability; While ensuring inferability, this application example can randomly add additional information to further enrich the corpus combinations and expand the training data volume. This application embodiment, by processing the training samples using a question-and-answer combination structure, can further enrich the training corpus and enhance the model's generalization ability and versatility.

[0048] In some embodiments, the step of performing dual-task collaborative training on the language model based on the training corpus to generate a pre-trained antibody language model for antibody design includes the following steps: The training corpus is input into the language model for training; the training samples in the training corpus include contextual content and predicted content. The context content is subjected to masked language prediction processing to obtain a first prediction result; The predicted content is then subjected to next-word prediction processing to obtain a second prediction result; The loss value of the language model is calculated based on the first prediction result and the second prediction result, and the parameters of the language model are updated based on the calculated loss value to obtain the pre-trained antibody language model.

[0049] In this embodiment of the application, the language model is subjected to dual-task collaborative training based on the training corpus, such as... Figure 5 As shown, by employing dual self-supervised learning objectives, different parts of the structured input are collaboratively optimized to generate a pre-trained antibody language model for antibody design.

[0050] In some embodiments, performing masked language prediction processing on the context content to obtain a first prediction result includes the following steps: The context content is randomly masked to obtain mask data; The mask data is subjected to self-attention feature update processing to obtain updated features; Based on the updated features, the content at the mask position in the mask data is predicted to obtain the first prediction result.

[0051] In this embodiment of the application, the context content is randomly masked. A portion of the input context content is randomly selected and replaced with a mask marker [mask] as input to the language model. The language model then predicts each masked content so that it can learn the distribution characteristics of the context content.

[0052] In one feasible embodiment, by... <content>Within a segment, a certain percentage (e.g., 15%) of tokens are randomly masked. These tokens can be amino acids, position numbers, or even the statement identifier itself. The model predicts the masked token based on the entire context of the masked position. The antibody language model is based on the Transformer architecture and uses a self-attention feature update module. Based on the input corpus, it dynamically updates the features of each token, ultimately predicting the correct token at the masked position. The self-attention feature update mechanism can be represented as: ; ; In the formula, X represents the input corpus, and W represents the learnable parameters in the model. The process of training an MLM can be represented as: ; in, It is the set of indices of the masked elements. It is a masked element. It is an unmasked element. These are model parameters. It is the probability distribution predicted by the model. The loss function is cross-entropy. After calculating the loss value, the model parameters are optimized through backpropagation, with the goal of minimizing the cross-entropy loss function. .

[0053] In some embodiments, performing next-word prediction processing on the predicted content to obtain a second prediction result includes the following steps: Determine the task prompt based on the predicted content; Based on the context content and the task prompt, lexical prediction processing is performed to obtain the second prediction result.

[0054] In the embodiments of this application, according to the provided <content>The segment content and the task prompt are used to perform next-word prediction processing on the predicted content. The task prompt is... <answer>The first token after that is generated autoregressively. <answer>The content of a segment. For example, a specific input corpus can be: ' <content> ', ' <ab>', 'Q', 'V', 'Q', 'L', 'V', 'Q', ......, ' <gl>', 'Q', 'V', 'Q', 'L', 'V', 'Q', ......, ' <mut>', '62', 'N', '107', 'E', '108', 'K', '111', 'c', 'R', '112', 'c', 'N', '112', 'b', 'H', ' <answer> ',' <vdj>', 'IGHV1-2*04', 'IGHD5-18*01', 'IGHJ6*02', ' <end>'; The input corpus includes the context content and the task prompt (including all content + answer identifier + VDJ task prompt), denoted as T1. The model predicts the next token 'IGHV1-2*04' based on T1, denoted as T2. Based on T1 and T2, the model predicts the next token 'IGHD5-18*01', denoted as T3. Based on T1, T2, and T3, the model predicts the next token 'IGHJ6*02', denoted as T4. Finally, the model predicts the next token based on T1, T2, T3, and T4. <end>', when the model predicts the next token as ' <end>At this point, autoregressive generation will stop, and a second prediction result will be obtained. During training, similar to MLM, the loss function minimizing cross-entropy is used as the loss value for word prediction, which can be expressed as: ; The total loss function for dual-task collaborative training is In this embodiment, the model parameters of the language model are updated based on the calculated loss value, thereby obtaining the trained antibody language model.

[0055] This application embodiment pre-trains by combining masked language learning and next-word prediction as dual tasks. Masked language learning is used to enhance the antibody language model's understanding of contextual information, and the antibody language model generates targeted answers to questions during next-word prediction, thereby improving the antibody language model's processing efficiency for antibody sequences. The trained antibody language model can then predict missing or erroneous antibody sequences.

[0056] To achieve the above objectives, another aspect of this application proposes an antibody sequence prediction method, comprising the following steps: Obtain the antibody sequence to be predicted; The antibody sequence to be predicted is input into a pre-trained antibody language model for sequence completion or sequence error correction to obtain the target antibody sequence; the antibody language model is trained using the antibody language model training method described above.

[0057] It is understood that the antibody language model in this application embodiment is trained by the antibody language model training method described above. The content of the antibody speech model training method embodiment described above is applicable to this antibody sequence prediction embodiment. The specific functions implemented by this prediction method embodiment are the same as those of the above model training method embodiment, and the beneficial effects achieved are also the same as those achieved by the above model training method embodiment.

[0058] The solutions of this application embodiment will be described in detail and explained below with reference to specific application examples: The antibody language model training method and antibody sequence prediction method disclosed in this application can be based on artificial intelligence. For AI-based antibody language model training and antibody sequence prediction methods, they can learn the biological context information of antibody sequences in a manner similar to humans analyzing the amino acid residue sequence of antibody sequences, thereby designing or predicting antibody sequences based on the learned knowledge. Artificial intelligence, through research on the design principles and implementation methods of various intelligent machines, enables the disclosed antibody language model training and antibody sequence prediction methods to optimize the results of antibody humanization, improve multiple indicators including antibody thermostability, immunogenicity, and affinity for antigen binding, and increase the success rate of antibody humanization. The antibody language model trained in this application can be used as a base model, fine-tuned using various downstream data to obtain a model specialized for specific downstream tasks, including but not limited to species classification, affinity prediction, stability, and structure prediction. Specifically, the antibody language model trained in this application can be used for sequence completion and error correction based on the trained antibody language model. By inputting antibody sequences containing missing or incorrect values ​​into the antibody language model, a complete and accurate antibody sequence is predicted and output.

[0059] This application's embodiments achieve comprehensive coverage of multi-source antibody information by combining a customized vocabulary with structured corpus construction: integrating mature antibody sequences, Germline sequences, VDJ gene information, and mutation information (including mutation site coordinates and post-mutation amino acids) into a unified input. Furthermore, by combining a masked language model (MLM) and next-token prediction (NTP) with dual self-supervised learning objectives, the model can not only learn the contextual semantics of amino acids (through MLM) but also learn the generation rules and evolutionary trajectories of antibody sequences (through NTP), thereby achieving a more comprehensive and deeper understanding of antibody sequences.

[0060] Please see Figure 6 This application also provides an antibody language model training system that can implement the above-described antibody language model training method. The system includes: The data acquisition module 601 is used to acquire a training sample set, wherein the training samples in the training sample set are generated by a pre-constructed antibody multivariate coding vocabulary. The corpus construction module 602 is used to perform structured corpus construction processing on the training samples in the training sample set to obtain a training corpus dataset; The task training module 603 is used to perform dual-task collaborative training on the language model based on the training corpus dataset to generate a pre-trained antibody language model for antibody design.

[0061] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0062] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0063] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0064] Please see Figure 7 , Figure 7 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 701 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 702 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 702 and is called and executed by the processor 701. The input / output interface 703 is used to implement information input and output; The communication interface 704 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 705 transmits information between various components of the device (e.g., processor 701, memory 702, input / output interface 703, and communication interface 704); The processor 701, memory 702, input / output interface 703, and communication interface 704 are connected to each other within the device via bus 705.

[0065] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0066] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0067] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0068] This application provides an antibody language model training method, an antibody sequence prediction method, and an apparatus thereof. This scheme acquires a training sample set, which is generated from a pre-constructed antibody multivariate coding vocabulary. This allows for the integration of multivariate information through the antibody multivariate coding vocabulary to construct training samples, providing the model with rich biological context. Furthermore, this scheme obtains a training corpus dataset by performing structured corpus construction processing on the training samples in the training sample set. This allows for deep fusion of multidimensional antibody information, further enriching the training corpus and enhancing the model's generalization ability and versatility. Moreover, this scheme performs dual-task collaborative training on the language model based on the training corpus dataset, generating a pre-trained antibody language model for antibody design. This enables the model to not only enhance its understanding of contextual information but also to generate targeted question answers, improving the data processing capability of the antibody language model and enabling accurate antibody sequence prediction.

[0069] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0070] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0071] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0072] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0073] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0074] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0075] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0076] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0077] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0078] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0079] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.< / end> < / end> < / end> < / vdj> < / answer> < / mut> < / gl> < / ab> < / content> < / answer> < / answer> < / content> < / content> < / mut> < / content> < / answer> < / mut> < / gl> < / ab> < / content> < / answer> < / content> < / end> < / vdj> < / answer> < / mut> < / gl> < / ab> < / content> < / answer> < / content> < / end> < / pad> < / mask> < / unk> < / vdj> < / mut> < / gl> < / ab>

Claims

1. A method for training an antibody language model, characterized in that, The method includes the following steps: A training sample set is obtained, wherein the training samples in the training sample set are generated by a pre-constructed antibody multivariate coding vocabulary; The training samples in the training sample set are processed to construct a structured corpus, resulting in a training corpus dataset. The language model is trained using a dual-task collaborative training process based on the training corpus dataset to generate a pre-trained antibody language model for antibody design.

2. The method according to claim 1, characterized in that, Obtaining the training sample set includes the following steps: Obtain antibody sequence samples; The amino acid letters of the antibody sequence sample are encoded according to the antibody multivariate coding vocabulary to obtain antibody sequence coding data; The antibody sequence sample is initially encoded, and the positional encoding data is obtained by encoding the positional data using the multivariate encoding vocabulary based on the positional annotations in the initial encoding data. The gene recombination information of the antibody sequence sample is encoded according to the antibody multivariate coding vocabulary to obtain gene recombination coding data; The antibody sequence encoding data, the position encoding data, and the gene recombination encoding data are combined with identifiers to generate training samples.

3. The method according to claim 1, characterized in that, The process of constructing a structured corpus from the training samples in the training sample set to obtain a training corpus dataset includes the following steps: The training samples were subjected to sequence analysis to obtain antibody sequences, germline sequences, recombinant gene information, and mutation information. The training corpus dataset is generated by performing question-and-answer combination structure processing on the training samples based on the antibody sequence, the germline sequence, the recombinant gene information, and the mutation information.

4. The method according to claim 1, characterized in that, The step of performing dual-task collaborative training on the language model based on the training corpus to generate a pre-trained antibody language model for antibody design includes the following steps: The training corpus is input into the language model for training; the training samples in the training corpus include contextual content and predicted content. The context content is subjected to masked language prediction processing to obtain a first prediction result; The predicted content is then subjected to next-word prediction processing to obtain a second prediction result; The loss value of the language model is calculated based on the first prediction result and the second prediction result, and the parameters of the language model are updated based on the calculated loss value to obtain the pre-trained antibody language model.

5. The method according to claim 4, characterized in that, The process of performing masked language prediction on the context content to obtain a first prediction result includes the following steps: The context content is randomly masked to obtain mask data; The mask data is subjected to self-attention feature update processing to obtain updated features; Based on the updated features, the content at the mask position in the mask data is predicted to obtain the first prediction result.

6. The method according to claim 4, characterized in that, The step of performing next-word prediction processing on the predicted content to obtain a second prediction result includes the following steps: Determine the task prompt based on the predicted content; Based on the context content and the task prompt, lexical prediction processing is performed to obtain the second prediction result.

7. An antibody sequence prediction method, characterized in that, Includes the following steps: Obtain the antibody sequence to be predicted; The antibody sequence to be predicted is input into a pre-trained antibody language model for sequence completion or sequence error correction to obtain the target antibody sequence; the antibody language model is trained by the antibody language model training method as described in any one of claims 1 to 6.

8. A training system for an antibody language model, characterized in that, The system includes: The data acquisition module is used to acquire a training sample set, wherein the training samples in the training sample set are generated by a pre-constructed antibody multivariate coding vocabulary; The corpus construction module is used to perform structured corpus construction processing on the training samples in the training sample set to obtain the training corpus dataset; The task training module is used to perform dual-task collaborative training on the language model based on the training corpus dataset to generate a pre-trained antibody language model for antibody design.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images