A method and device for training biological language model

Abstract

The present invention provides a method and device for training a biological language model; the method includes: extracting features from multimodal data corresponding to each biological molecule to obtain joint multimodal features; converting the biological molecule sequence and the joint multimodal features to text based on an extended vocabulary to output text features; and training a supervised language model according to natural language requests based on text features and description texts corresponding to the biological molecules to obtain a biological language model. Thus, this embodiment extracts features from multimodal data corresponding to biological molecules, and performs model training after converting the obtained joint multimodal features and biological molecule sequences into text features, so that the language model can more comprehensively understand the structure and function of biological molecules, thereby resolving the semantic conflict between natural language and biological molecule representation in the prior art, and improving the language model's ability to understand and design biological molecules.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a method and device for training a biological language model. Background Art

[0002] Understanding and designing biomolecules is a key step in natural science research. However, how to understand biomolecules through natural language and design them according to human intentions remains an unsolved problem. Although large language models have made significant progress in following human instructions and generating human-like text, they have encountered challenges when processing biomolecules. These challenges mainly stem from the differences between biomolecules and natural language, as well as the complexity of large language models in processing data beyond text input.

[0003] In current research, models such as AlphaFold have greatly promoted the development of the field of biomolecular structure prediction and design. However, large language models still have shortcomings in how to understand and design biomolecules through natural language. There are two reasons: 1. General large language models have limitations in understanding the professional domain knowledge of biomolecules due to the lack of specialized training data; 2. The vocabulary and semantic differences between the language used to describe biomolecules and natural language also increase the difficulty of understanding for large language models. For example, in English, the letter "C" may mean the letter C, but in chemical molecules, it means a carbon atom, and in the context of proteins, it means cysteine.

[0004] In order to solve the above problems, the present invention proposes a biological language model for understanding and designing biological molecules. Summary of the invention

[0005] In response to the above-mentioned problems existing in the prior art, an embodiment of the present invention provides a method and device for training a biological language model; this method enables the biological language model to better understand the structure and function of biological molecules, thereby resolving the semantic conflict between natural language and biological molecules in the prior art, and significantly improving the biological language model's ability to understand and design biological molecules.

[0006] According to a first aspect of an embodiment of the present invention, a method for training a biological language model is provided, the method comprising: performing feature extraction processing on multimodal data corresponding to each biological molecule to obtain joint multimodal features; performing text conversion processing on the biological molecule sequence and the joint multimodal features based on an extended vocabulary, and outputting text features; based on the text features and the descriptive text corresponding to the biological molecule, performing supervised language model training according to natural language requests to obtain a biological language model.

[0007] Optionally, the biological molecules include small molecule organic matter and proteins; when the biological molecules are small molecule organic matter, the multimodal data corresponding to the biological molecules include a 2D image and a 3D structure corresponding to the small molecule organic matter; when the biological molecules are proteins, the multimodal data corresponding to the biological molecules include a 1D sequence and a 3D structure corresponding to the protein.

[0008] Optionally, the method comprises: taking the text features as training samples and the description text corresponding to the biological molecule as the true label, performing supervised language model training based on natural language requests to obtain a first biological language model; taking the description text corresponding to the biological molecule as training samples and the text features as the true label, performing supervised language model training based on natural language requests to obtain a second biological language model.

[0009] Optionally, the method further includes: using the first biological language model to perform text description prediction on the text features corresponding to the biological molecule to be tested, and outputting the text description corresponding to the biological molecule to be tested; using the second biological language model to perform biological molecule prediction on the text description corresponding to the biological molecule to be tested, and outputting the molecular structure expression corresponding to the biological molecule to be tested.

[0010] Optionally, the multimodal data corresponding to each biological molecule is subjected to feature extraction processing to obtain a joint multimodal feature; including: for any modal data in the multimodal data corresponding to the biological molecule: encoding the modal data to generate a modal representation corresponding to the modal data; determining a motif prompt corresponding to the biological molecule; and based on all modal representations and motif prompts corresponding to the biological molecule, performing a joint feature extraction process to generate a joint multimodal feature corresponding to the biological molecule.

[0011] Optionally, determining the motif prompt corresponding to the biological molecule includes: acquiring a motif representation corresponding to the biological molecule; performing motif prompt processing on the modal representation based on a motif prompt extractor, and outputting a motif prompt corresponding to the biological molecule.

[0012] Optionally, the method performs a joint feature extraction process based on all modal representations and motif prompts corresponding to the biological molecule to generate a joint multimodal feature corresponding to the biological molecule; including: concatenating the modal representations corresponding to all modal data in the biological molecule to generate a fused modal representation; performing a joint multimodal feature extraction process on the fused modal representation and the motif prompt using a Transformer encoder-decoder architecture to output a joint multimodal feature corresponding to the biological molecule.

[0013] Optionally, the text conversion processing of the joint multimodal features based on the extended vocabulary to output text features includes: obtaining a first vocabulary corresponding to the biological molecule sequence and a second vocabulary corresponding to the joint multimodal features from the extended vocabulary; marking the biological molecule sequence using the first vocabulary to generate a biological molecule marker sequence corresponding to the biological molecule; marking the joint multimodal features using the second vocabulary to generate a joint multimodal marker feature corresponding to the biological molecule; splicing the biological molecule marker sequence corresponding to the biological molecule and the joint multimodal marker feature to output text features.

[0014] According to the second aspect of an embodiment of the present invention, there is also provided an apparatus for training a biological language model, the apparatus comprising: a feature extraction module, for performing feature extraction processing on multimodal data corresponding to each biological molecule to obtain joint multimodal features; a text conversion module, for performing text conversion processing on biological molecule sequences and joint multimodal features based on an extended vocabulary, and outputting text features; a language model training module, for performing supervised language model training according to natural language requests based on the text features and the descriptive text corresponding to the biological molecules to obtain a biological language model.

[0015] According to a third aspect of an embodiment of the present invention, there is also provided an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; the processor is configured to read the executable instructions from the memory and execute the instructions to implement the method described in the first aspect.

[0016] According to a fourth aspect of an embodiment of the present invention, there is further provided a computer-readable medium on which a computer program is stored. When the program is executed by a processor, the method described in the first aspect is implemented.

[0017] The present invention provides a method and device for training a biological language model; the method includes at least the following steps: first, feature extraction processing is performed on the multimodal data corresponding to each biological molecule to obtain a joint multimodal feature; second, text conversion processing is performed on the biological molecule sequence and the joint multimodal feature based on an extended vocabulary to output text features; then, supervised language model training is performed according to a natural language request based on the text features and the description text corresponding to the biological molecule to obtain a biological language model. Therefore, this embodiment extracts features from the multimodal data corresponding to the biological molecule, and performs model training after converting the obtained joint multimodal features and the biological molecule sequence into text features, so that the language model can more comprehensively understand the structure and function of the biological molecule, thereby solving the semantic conflict between the natural language and biological molecule representation in the prior art, and improving the language model's ability to understand and design biological molecules. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Hereinafter, some specific embodiments of the present invention will be described in detail in an exemplary and non-limiting manner with reference to the accompanying drawings. The same reference numerals in the accompanying drawings indicate the same or similar components or parts. It should be understood by those skilled in the art that these drawings are not necessarily drawn to scale. In the accompanying drawings:

[0019] Figure 1 A flowchart of a method for training a biological language model provided by an embodiment of the present invention;

[0020] Figure 2 It is a schematic diagram of a process of performing feature extraction processing on multimodal data corresponding to each biomolecule in one embodiment of the present invention;

[0021] Figure 3 A schematic diagram of the structure of an apparatus for training a biological language model provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0023] like Figure 1 , which is a flow chart of a method for training a biological language model provided by one embodiment of the present invention.

[0024] A method for training a biological language model, the method comprising at least the following steps:

[0025] S101, performing feature extraction processing on the multimodal data corresponding to each biomolecule to obtain a joint multimodal feature;

[0026] S102, performing text conversion processing on the biomolecule sequence and the joint multimodal features based on the extended vocabulary, and outputting text features;

[0027] S103, based on the text features and the description text corresponding to the biological molecule, supervised language model training is performed according to the natural language request to obtain a biological language model.

[0028] In S101, there is no limitation on the method of feature extraction processing, and feature extraction processing can be performed based on a neural network algorithm or a pre-trained model to generate joint multimodal features. Pre-trained models include: GIN model, Uni-Mol model, ESM2-35M model, and SaProt-35M model.

[0029] The biomolecules include small molecule organic matter and / or protein. When the biomolecule is a small molecule organic matter, the multimodal data corresponding to the biomolecule includes a 2D graph and a 3D structure corresponding to the small molecule organic matter; when the biomolecule is a protein, the multimodal data corresponding to the biomolecule includes a 1D sequence and a 3D structure corresponding to the protein. Small molecule organic matter is used to indicate an organic molecule with a total number of atoms less than 50. Proteins are generally organic molecules with an atomic number of not less than 50.

[0030] In S102, the vocabulary corresponding to the biomolecule category is searched from the extended vocabulary table; the searched vocabulary, biomolecule sequence, and joint multimodal features are spliced ​​and processed to output text features.

[0031] For any category of biological molecules: obtain a first vocabulary corresponding to the biological molecule sequence and a second vocabulary corresponding to the joint multimodal feature from the extended vocabulary; use the first vocabulary to mark the biological molecule sequence to generate a biological molecule marker sequence corresponding to the biological molecule; use the second vocabulary to mark the joint multimodal feature to generate a joint multimodal marker feature corresponding to the biological molecule; splice the biological molecule marker sequence corresponding to the biological molecule and the joint multimodal marker feature to output the text feature.

[0032] In order to better distinguish different forms of modal data, different words are selected from the extended vocabulary to mark different modalities; for example: using " <selfies> "as well as"< / selfies> " is used to mark the biological molecule sequence corresponding to the small molecule organic matter; <fasta> "as well as"< / fasta> " is used to mark the biological molecule sequence corresponding to the protein; <mol> "as well as"< / mol>" is used to mark the joint multimodal features of small molecule organic matter; <prot> "as well as"< / prot> ” to label the joint multimodal features of proteins.

[0033] For example: text feature expressions corresponding to small molecule organic matter As described in the following formula (1):

[0034] Formula (1);

[0035] The text feature expression corresponding to the protein is as follows (2):

[0036] Formula (2);

[0037] in, [SELFIES sequence] represents the biological molecule sequence corresponding to the small molecule organic matter; [FASTA sequence] represents the biological molecule sequence corresponding to the protein; Indicates modal features. Represents the number of learnable queries in a Transformer encoder-decoder architecture.

[0038] In addition, the specific prefix "[C]" can be introduced to indicate a carbon atom; the specific prefix " ” to represent each amino acid.

[0039] At this time, the text feature expression corresponding to the small molecule organic matter is as follows:

[0040] Formula (3);

[0041] The text feature expression corresponding to the protein is as follows (4):

[0042] Formula (4).

[0043] Thus, small molecule organics or proteins are integrated into the text format based on an extended vocabulary combined with joint multimodal features and biomolecule sequences, thereby enhancing the ability of the language model to understand and interpret biomolecules.

[0044] In S103, based on the text features and description text corresponding to the biological molecule, supervised language model training is performed using a pre-trained model according to a natural language request to obtain a biological language model.

[0045] The pre-trained model is used to indicate a pre-trained language model that is compatible with any GPT style; for example: the OPT-350M language model.

[0046] There are two forms of language model training. Depending on the selection of training samples, the prediction functions of the obtained biological language model are also different. For example: using the text features as training samples and the description text corresponding to the biological molecule as the true label, supervised language model training is performed based on natural language requests to obtain a first biological language model; using the description text corresponding to the biological molecule as training samples and the text features as the true label, supervised language model training is performed based on natural language requests to obtain a second biological language model.

[0047] Here, the first biological language model is used to output description text; the second biological language model is used to output biological molecule sequence.

[0048] Natural language request example: What is the function of this protein?

[0049] Descriptive text Example: This molecule is a member of the group benzene, a member of the group sulfonyl, and a member of the group triazole.

[0050] This embodiment extracts features from multimodal data corresponding to biological molecules, and performs model training after converting the obtained joint multimodal features and biological molecule sequences into text features, so that the language model can have a more comprehensive understanding of the structure and function of biological molecules, thereby resolving the semantic conflict between natural language and biological molecule representation in the prior art, and improving the language model's ability to understand and design biological molecules.

[0051] In a preferred implementation of this embodiment, the method further includes: using the first biological language model to predict the text description of the text features corresponding to the biological molecule to be tested, and outputting the text description corresponding to the biological molecule to be tested; using the second biological language model to predict the biomolecule sequence of the text description corresponding to the biological molecule to be tested, and outputting the molecular structure expression corresponding to the biological molecule to be tested. Here, the molecular structure expression is used to indicate the biomolecule sequence.

[0052] As a result, effective alignment between natural language and biological molecules is achieved through the biological language model, which significantly improves the biological language model's ability to understand and design biological molecules.

[0053] like Figure 2 FIG. 1 is a flow chart showing a process of performing feature extraction processing on multimodal data corresponding to each biological molecule in one embodiment of the present invention.

[0054] The multimodal data corresponding to each biomolecule is subjected to feature extraction processing, which includes at least the following steps:

[0055] S201, for any modal data in the multimodal data corresponding to the biomolecule: encode the modal data to generate a modal representation corresponding to the modal data;

[0056] S202, determining a motif prompt corresponding to the biomolecule;

[0057] S203, based on all modal representations and motif hints corresponding to the biological molecules, a joint feature extraction process is performed to generate a joint multimodal feature corresponding to the biological molecules.

[0058] In S201, different encoding processing methods are used for different modal data of the same biological molecule.

[0059] Exemplary: when the biomolecule is a small molecule organic substance, the 2D graph corresponding to the small molecule organic substance is encoded based on GIN to generate a 2D molecular modal representation; the 3D structure corresponding to the small molecule organic substance is encoded based on Uni-Mol to generate a 3D molecular modal representation;

[0060] When the biological molecule is a protein, the 1D sequence corresponding to the protein is encoded based on the pre-trained ESM2-35M to generate a 1D protein modal representation; the 3D structure corresponding to the protein is encoded based on SaProt-35M to generate a 3D protein modal representation.

[0061] Specifically, for small molecule organic matter, the 2D graph is defined as ;in, Represents an atomic node, represents the chemical bonds between these nodes. The 3D structure is defined as ;in, represents a group of atoms, Represent the spatial coordinates of these atoms.

[0062] Using pre-trained 5-layer GIN as 2D graph encoder , and Uni-Mol as a 3D structure encoder , to obtain the molecular modal representation; specifically as shown in the following formula (5):

[0063] Formula (5);

[0064] in, is a 2D molecular modal representation; It is a 3D molecular modal representation.

[0065] For proteins, the 1D sequence is defined as ,in represents an amino acid. The 3D structure is defined as , where S is an amino acid sequence, Represents the coordinates of the four backbone atoms in each amino acid (N, C, CA, O).

[0066] Use pre-trained ESM2-35M as 1D sequence encoder , and SaProt-35M as a 3D structure encoder , to obtain the protein modal representation; specifically as shown in the following formula (6):

[0067] Formula (6);

[0068] in, is a 1D protein modal representation; is a 3D protein modality representation.

[0069] In S202, in small molecule organic compounds, a motif usually represents a functional group or substructure, which plays an important role in enhancing the understanding of the molecule. Similarly, in proteins, a motif is a continuous amino acid sequence that carries a specific biological function and constitutes the basic element of protein function. In order to integrate basic prior knowledge within the motif, this application introduces motif hints, which are intended to highlight key areas in biological molecules that are critical to understanding their functions and interactions.

[0070] Exemplarily, a motif representation corresponding to the biomolecule is obtained; a motif hint extractor is used to process the motif representation, and a motif hint corresponding to the biomolecule is output. More specifically, when the biomolecule is a small molecule organic compound, a first motif representation corresponding to the small molecule organic compound is determined based on a functional fingerprint FCFP, and a motif hint extractor is used to process the first motif representation, and a motif hint corresponding to the small molecule organic compound is generated; when the biomolecule is a protein, a second motif representation corresponding to the protein is queried based on a UniProt database, and a motif hint extractor is used to process the second motif representation, and a motif hint corresponding to the protein is generated.

[0071] For example: The first motif is represented by , the second motif is represented by ;in, ∈{0,1} represents the The presence of motifs in small organic molecules ( =1) or does not exist ( =0); ∈{0,1} represents the The presence of motifs in proteins ( =1) or does not exist ( =0), and denote the total number of predefined motifs in microorganisms and proteins, respectively.

[0072] Motif tips corresponding to small organic matter The expression of is shown in formula (7):

[0073] Formula (7);

[0074] Motif hints corresponding to proteins The expression of is shown in formula (8):

[0075] Formula (8);

[0076] in, are the model parameters of the small molecule motif cue extractor; are the model parameters of the protein motif hint extractor.

[0077] Thus, motif cues based on biomolecule correspondences can guide the multimodal feature extraction process towards features associated with the identified motifs, thereby increasing the biological relevance of the extracted multimodal features.

[0078] In S203, here, joint feature extraction processing is performed based on an existing neural network or a pre-trained model. For example, the modal representations corresponding to all modal data in the biomolecule are concatenated to output a fused modal representation; then the fused modal representation and the motif prompt are input into a Transformer encoder-decoder architecture for joint multimodal feature extraction processing, and a joint multimodal feature corresponding to the biomolecule is output.

[0079] Specifically, in order to obtain a more informative and robust representation than a single modality representation, based on the complementarity and redundancy of each modality representation, a multi-layer perceptron (MLP) is used to concatenate all modality representations in the biomolecule and output a fused modality representation.

[0080] Fusion modal representation corresponding to small molecule organic matter It is obtained by the following formula (9):

[0081] Formula (9);

[0082] Fusion mode representation corresponding to the protein It is obtained by the following formula (10):

[0083] Formula (10);

[0084] in, Represents a join operation.

[0085] When the biomolecule is a small organic molecule, As the initial input to the Transformer encoder, As the input of the Transformer decoder. The Transformer encoder-decoder architecture is used to perform joint multimodal feature extraction on small molecule organic matter, and the joint multimodal features corresponding to the small molecule organic matter are output. The joint multimodal feature expression corresponding to the small molecule organic matter As shown in the following formula (11):

[0086] Formula (11);

[0087] When the biomolecule is a protein, As the initial input to the Transformer encoder, As the input of the Transformer decoder. The Transformer encoder-decoder architecture is used to extract joint multimodal features of proteins and output joint multimodal features corresponding to proteins. The joint multimodal feature expression corresponding to proteins is As shown in the following formula (12):

[0088] Formula (12);

[0089] Here, Q represents a set of learnable queries in the Transformer encoder-decoder architecture.

[0090] The Transformer encoder-decoder architecture is used to effectively extract and fuse these heterogeneous source features. The self-attention mechanism and multi-layer perceptron MLP based on the Transformer encoder-decoder architecture can dynamically weight the modal representation corresponding to each modal data, so that it can focus on the most relevant features across modalities and within modalities. The Transformer encoder captures salient features, and the Transformer decoder combines these features to construct joint multimodal features. Therefore, this embodiment can extract features from fused modal representations and motif cues based on the Transformer encoder-decoder architecture, thereby improving the accuracy of joint multimodal feature extraction.

[0091] The present embodiment is described in detail below in conjunction with specific application scenarios.

[0092] When the biological molecule is a small molecule organic substance, the 2D image corresponding to the small molecule organic substance is encoded based on GIN to generate a 2D molecular modal representation; the 3D structure corresponding to the small molecule organic substance is encoded based on Uni-Mol to generate a 3D molecular modal representation. The first motif representation corresponding to the small molecule organic substance is determined based on the functional fingerprint FCFP, and the first motif representation is processed with a motif prompt based on the small molecule motif prompt extractor to generate a motif prompt corresponding to the small molecule organic substance. The 2D molecular modal representation and the 3D molecular modal representation are spliced ​​and processed to output a fused modal representation; then the fused modal representation and the motif prompt are input into the Transformer encoder-decoder architecture to perform joint multimodal feature extraction and output a joint multimodal feature corresponding to the small molecule organic substance. A first vocabulary corresponding to a biological molecule sequence corresponding to the small molecule organic substance and a second vocabulary corresponding to a joint multimodal feature are obtained from an extended vocabulary; the biological molecule sequence is marked using the first vocabulary to generate a biological molecule marker sequence corresponding to the small molecule organic substance; the joint multimodal feature is marked using the second vocabulary to generate a joint multimodal marker feature corresponding to the small molecule organic substance; the biological molecule marker sequence corresponding to the small molecule organic substance and the joint multimodal marker feature are spliced, and a text feature corresponding to the small molecule organic substance is output.

[0093] When the biological molecule is a protein, the 1D sequence corresponding to the protein is encoded based on the pre-trained ESM2-35M to generate a 1D protein modal representation; the 3D structure corresponding to the protein is encoded based on SaProt-35M to generate a 3D protein modal representation. Based on the UniProt database, the second motif representation corresponding to the protein is queried, and the second motif representation is processed with a motif prompt based on the protein motif prompt extractor to generate a motif prompt corresponding to the protein. The 1D protein modal representation and the 3D protein modal representation are concatenated to output a fused modal representation; then the fused modal representation and the motif prompt are input into the Transformer encoder-decoder architecture to perform joint multimodal feature extraction and output joint multimodal features corresponding to the protein. A first vocabulary corresponding to the biological molecule sequence corresponding to the protein and a second vocabulary corresponding to the joint multimodal feature are obtained from the extended vocabulary; the biological molecule sequence is marked with the first vocabulary to generate a biological molecule marker sequence corresponding to the protein; the joint multimodal feature is marked with the second vocabulary to generate a joint multimodal marker feature corresponding to the protein; the biological molecule marker sequence corresponding to the protein and the joint multimodal marker feature are spliced, and the text feature corresponding to the protein is output.

[0094] First, the text features corresponding to small molecule organics are used as the first training samples, and the text features corresponding to proteins are used as the second training samples. Secondly, the description text corresponding to the small molecule organics is used as the true label, and the OPT-350M language model is used to perform supervised language model training on several first training samples; at the same time, the description text corresponding to the protein is used as the true label, and the OPT-350M language model is used to perform supervised language model training on several second training samples; and a first biological language model is generated. Finally, the first biological language model is used to predict the text description of the text features corresponding to the small molecule organics or proteins to be tested, and a text description corresponding to the small molecule organics or proteins to be tested is generated.

[0095] Alternatively, first, the description text corresponding to the small molecule organic substance is used as the first training sample, and the description text corresponding to the protein is used as the second training sample. Secondly, the text features corresponding to the small molecule organic substance are used as the true label, and the OPT-350M language model is used to perform supervised language model training on several first training samples; at the same time, the text features corresponding to the protein are used as the true label, and the OPT-350M language model is used to perform supervised language model training on several second training samples; and a second biological language model is generated. Finally, the second biological language model is used to perform sequence prediction on the description text corresponding to the small molecule organic substance to be tested or the protein to be tested, and a biological molecule sequence corresponding to the small molecule organic substance to be tested or the protein to be tested is generated.

[0096] In this embodiment, a plurality of first training samples may be used to train a language model, and a corresponding biological language model for predicting small molecule organic substances to be tested may be generated; or a second training sample may be used to train a language model, and a corresponding biological language model for predicting proteins to be tested may be generated.

[0097] This embodiment adopts a knowledge-guided multimodal feature extraction method; first, a pre-trained lightweight encoder is used to process each modal data separately; second, multimodal features are fused through a motif prompt extractor; finally, specific vocabulary is introduced based on an extended vocabulary to ensure the integrity of each modal feature; thereby, the semantic conflict between natural language and biomolecule representation is resolved; thus, the biolanguage model can achieve effective alignment between natural language and biomolecules, significantly improving the understanding and design capabilities of biomolecules.

[0098] like Figure 3 , which is a schematic diagram of the structure of an apparatus for training a biological language model provided by one embodiment of the present invention.

[0099] A device for training a biological language model, the device 300 includes: a feature extraction module 301, used to perform feature extraction processing on multimodal data corresponding to each biological molecule to obtain joint multimodal features; a text conversion module 302, used to perform text conversion processing on biological molecule sequences and joint multimodal features based on an extended vocabulary, and output text features; a language model training module 303, used to perform supervised language model training according to natural language requests based on the text features and the description text corresponding to the biological molecules to obtain a biological language model.

[0100] In a preferred implementation of this embodiment, the biological molecules include small molecule organic matter and proteins; when the biological molecules are small molecule organic matter, the multimodal data corresponding to the biological molecules include a 2D image and a 3D structure corresponding to the small molecule organic matter; when the biological molecules are proteins, the multimodal data corresponding to the biological molecules include a 1D sequence and a 3D structure corresponding to the protein.

[0101] In a preferred implementation of this embodiment, the language model training module includes: a first biological language model training module, which is used to use the text features as training samples and the descriptive text corresponding to the biological molecule as the true label, and perform supervised language model training based on natural language requests to obtain a first biological language model; a second biological language model training module, which is used to use the descriptive text corresponding to the biological molecule as training samples and the text features as the true label, and perform supervised language model training based on natural language requests to obtain a second biological language model.

[0102] In a preferred implementation of this embodiment, the device also includes: a first prediction module, which is used to use the first biological language model to predict the text description of the text features corresponding to the biological molecule to be tested, and output the text description corresponding to the biological molecule to be tested; a second prediction module, which is used to use the second biological language model to perform biological molecule prediction on the text description corresponding to the biological molecule to be tested, and output the molecular structure expression corresponding to the biological molecule to be tested.

[0103] In a preferred implementation of this embodiment, the feature extraction module includes: an encoding processing unit, which is used to encode any modal data in the multimodal data corresponding to the biological molecule: to generate a modal representation corresponding to the modal data; a determination unit, which is used to determine the motif prompt corresponding to the biological molecule; and a joint feature extraction processing unit, which is used to perform joint feature extraction processing based on all modal representations and motif prompts corresponding to the biological molecule, to generate a joint multimodal feature corresponding to the biological molecule.

[0104] In a preferred implementation of this embodiment, the determination unit includes: an acquisition subunit, used to acquire the motif representation corresponding to the biological molecule; and a motif prompt processing subunit, used to perform motif prompt processing on the modal representation based on a motif prompt extractor, and output a motif prompt corresponding to the biological molecule.

[0105] In a preferred implementation of this embodiment, the joint feature extraction processing unit includes: a splicing processing subunit, which is used to splice the modal representations corresponding to all modal data in the biological molecule to generate a fused modal representation; a joint feature extraction processing subunit, which is used to perform joint multimodal feature extraction processing on the fused modal representation and the motif prompt using the Transformer encoder-decoder architecture, and output a joint multimodal feature corresponding to the biological molecule.

[0106] In a preferred implementation of this embodiment, the text conversion module includes: a selection unit, which is used to obtain a first vocabulary corresponding to a biological molecule sequence and a second vocabulary corresponding to a joint multimodal feature from an extended vocabulary; a first generation unit, which is used to mark the biological molecule sequence using the first vocabulary to generate a biological molecule marker sequence corresponding to the biological molecule; a second generation unit, which is used to mark the joint multimodal feature using the second vocabulary to generate a joint multimodal marker feature corresponding to the biological molecule; and a splicing unit, which is used to splice the biological molecule marker sequence corresponding to the biological molecule and the joint multimodal marker feature to output a text feature.

[0107] The above device can execute a method for training a biological language model provided by an embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing a method for training a biological language model. For technical details not described in detail in this embodiment, please refer to a method for training a biological language model provided by an embodiment of the present invention.

[0108] The present invention also provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement a method for training a biological language model described in the present invention.

[0109] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method according to various embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0110] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user computing device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0111] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the method according to the following embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0112] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0113] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0114] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0115] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0116] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for training a biological language model, characterized in that: include: Perform feature extraction on the multimodal data corresponding to each biomolecule to obtain joint multimodal features; Acquire a first vocabulary corresponding to a biomolecule sequence and a second vocabulary corresponding to a joint multimodal feature from an extended vocabulary; label the biomolecule sequence using the first vocabulary to generate a biomolecule labeling sequence corresponding to the biomolecule; label the joint multimodal feature using the second vocabulary to generate a joint multimodal labeling feature corresponding to the biomolecule; concatenate the biomolecule labeling sequence corresponding to the biomolecule and the joint multimodal labeling feature to output a text feature; Based on the text features and the description text corresponding to the biological molecule, supervised language model training is performed according to natural language requests to obtain a biological language model.

2. The method according to claim 1, characterized in that The biomolecules include small organic molecules and proteins; When the biomolecule is a small molecule organic substance, the multimodal data corresponding to the biomolecule includes a 2D image and a 3D structure corresponding to the small molecule organic substance; When the biomolecule is a protein, the multimodal data corresponding to the biomolecule includes a 1D sequence and a 3D structure corresponding to the protein.

3. The method according to claim 1, characterized in that The method of performing supervised language model training based on the text features and the description text corresponding to the biomolecule according to the natural language request to obtain the biological language model comprises: Using the text features as training samples and the description text corresponding to the biomolecule as a true label, performing supervised language model training based on natural language requests to obtain a first bio-language model; The description text corresponding to the biological molecule is used as a training sample and the text feature is used as a true label, and supervised language model training is performed based on a natural language request to obtain a second biological language model.

4. The method according to claim 3, characterized in that Also includes: Using the first biological language model to predict text descriptions of text features corresponding to the biological molecule to be tested, and outputting text descriptions corresponding to the biological molecule to be tested; The second biological language model is used to perform biological molecule prediction on the text description corresponding to the biological molecule to be detected, and a molecular structure expression corresponding to the biological molecule to be detected is output.

5. The method according to claim 1, characterized in that The method of extracting features from the multimodal data corresponding to each biomolecule to obtain a joint multimodal feature comprises: For any modal data in the multimodal data corresponding to the biomolecule: encoding the modal data to generate a modal representation corresponding to the modal data; determining a motif hint corresponding to the biomolecule; Based on all modal representations and motif prompts corresponding to the biomolecule, a joint feature extraction process is performed to generate a joint multimodal feature corresponding to the biomolecule.

6. The method according to claim 5, characterized in that Determining the motif prompt corresponding to the biomolecule comprises: Obtaining a motif representation corresponding to the biomolecule; The modality representation is subjected to motif hint processing based on a motif hint extractor, and a motif hint corresponding to the biomolecule is output.

7. The method according to claim 5, characterized in that The method of performing joint feature extraction processing based on all modal representations and motif prompts corresponding to the biomolecule to generate a joint multimodal feature corresponding to the biomolecule comprises: splicing the modal representations corresponding to all modal data in the biomolecule to generate a fused modal representation; The fused modality representation and the motif prompt are processed by joint multimodal feature extraction using a Transformer encoder-decoder architecture to output joint multimodal features corresponding to the biomolecule.

8. A device for training a biological language model, characterized in that: include: A feature extraction module is used to perform feature extraction processing on the multimodal data corresponding to each biomolecule to obtain a joint multimodal feature; A text conversion module is used to obtain a first vocabulary corresponding to a biomolecule sequence and a second vocabulary corresponding to a joint multimodal feature from an extended vocabulary; use the first vocabulary to mark the biomolecule sequence to generate a biomolecule marker sequence corresponding to the biomolecule; use the second vocabulary to mark the joint multimodal feature to generate a joint multimodal marker feature corresponding to the biomolecule; splice the biomolecule marker sequence corresponding to the biomolecule and the joint multimodal marker feature to output a text feature; The language model training module is used to perform supervised language model training according to natural language requests based on the text features and the description text corresponding to the biological molecule to obtain a biological language model.

9. A computer readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images