Multi-modal large language model training method, system and device and storage medium

Abstract

The invention provides a multi-modal large language model training method, system and device and a storage medium, and belongs to the technical field of biological medicine. The invention provides a method for training a multi-modal large language model in the field of biological medicine. The method comprises an entity recognition module, a hypertext embedding module, a lexical element division module and a model training module. According to the training method provided by the invention, names or anaphora of genes, proteins and molecules are automatically identified from natural text descriptions of literatures and patents by adopting a general large language model, then embedding is carried out, identification is carried out on the basis of real natural texts in the literatures instead of a mode of artificially constructing sentences from data, and thus the identification efficiency is greatly improved. And more natural modal fusion can be realized. According to the method, the matched multi-modal sequence data is embedded into the text according to the marking rule, so that the original text semantics are not damaged while the text information is enriched, and a better fitting effect on the data is realized.

Description

[0001] The present invention claims the priority of the Chinese patent application filed with the Chinese Patent Office on July 29, 2024, with application number 2024110200205 and invention name “Training method, system, device and storage medium for multimodal large language model”, the entire contents of which are incorporated by reference in the application. Technical Field

[0002] The present invention belongs to the field of biomedicine technology, and specifically relates to a training method, system, device and storage medium for a multimodal large language model. Background Art

[0003] ChatGPT has aroused public attention to large language models (LLMs), which are gradually affecting all aspects of life, including assisting copywriting, polishing, knowledge explanation, automatic code writing, etc. However, at this stage, large language models cannot effectively handle multiple modal information such as biological sequences and chemical structures in the biomedical field. In particular, the modal information in the biomedical field corresponds to the modality of drugs, which refers to sequence data in the form of drugs such as small molecules, antibodies, and nucleic acids, usually including DNA / RNA nucleic acid sequences, protein amino acid sequences, and SMILES sequences of small molecules.

[0004] In order to use large language models to model multimodal data in the biomedical field, there are two main existing approaches: 1) Sequences of each modality, such as DNA / RNA nucleic acid sequences, protein amino acid sequences, and small molecule SMILES sequences, are modeled separately to construct representation or generation models for each sequence. The disadvantage of this approach is that it is not flexible enough and the model can only predict sequences of a specific modality; 2) Sequences of different modalities, such as DNA / RNA nucleic acid sequences and protein sequences, are treated as ordinary texts and directly fed into the large language model. The disadvantage is that this cannot be effectively distinguished from descriptive texts in natural language, which will destroy the semantic information of natural language, making model training difficult and making it more likely to produce "hallucinations" (outputting content that does not conform to grammatical rules).

[0005] Therefore, in order to address the problem that the current large language models in the biomedical field can either only process unimodal sequence data or have poor effects in processing multimodal sequence data, it is necessary to provide a new large language model training method for multimodal data in the biomedical field. Summary of the invention

[0006] In view of the problem that the current large language models in the biomedical field can only process unimodal sequence data or have poor effects in processing multimodal sequence data, the present invention proposes a large language model training method for multimodal data in the biomedical field. The training method of the multimodal large language model in the biomedical field provided by the present invention includes: an entity recognition module, a hypertext embedding module, a word unit division module and a model training module. The training method provided by the present invention uses a general large language model to automatically identify the names or references of genes, proteins and molecules from the natural text descriptions of documents and patents, and then embeds them. This method of identifying based on the real natural text in the document rather than artificially constructing sentences from the data can achieve a more natural modal fusion. The matched multimodal sequence data is embedded into the text according to the marking rules, thereby enriching the text information without destroying the original text semantics, thereby achieving a better fitting effect on the data.

[0007] 1. Entity Recognition Module

[0008] There are a lot of names or references to genes, proteins or molecules in biomedical literature and patent corpora. The present invention identifies the names or references of related genes, proteins or molecules to prepare for embedding more detailed biological sequence and molecular structure information.

[0009] 2. Hypertext Embedding Module

[0010] The structural information of the corresponding molecules or the sequence information of nucleic acids and proteins are matched and embedded into the name or reference according to the rules of "markup language" to construct a new corpus, which is hypertext.

[0011] 3. Token Segmentation Module

[0012] The basis of the language model is to predict tokens, which are simply the constituent units of sentences. GPT appears to be somewhat intelligent by continuously predicting the next token. Data of different modalities are composed of different tokens. For example, the "base adenine A" in the nucleic acid sequence (DNA / RNA) and the "alanine A" in the protein sequence have very different meanings from the letter A in the natural text "Trichostatin A can resist infection"; the "cytosine C" in the nucleic acid sequence (DNA / RNA) and the "cysteine ​​C in the protein sequence" have very different meanings from the letter C in the "carbon atom C" in the molecule. However, because they are all in text form, large models in general fields are often mixed together, resulting in difficulties in semantic distinction and poor model effects. Here, the present invention specifically designs different tokens for sequences of each different modality to better distinguish data of different modalities.

[0013] 4. Model training module

[0014] Hypertext embedded with data from different modalities is used as the corpus for training large language models to achieve fusion between modalities without causing semantic confusion.

[0015] The technical solution of the present invention includes:

[0016] On the one hand, the present invention provides a method for training a multimodal large language model in the field of biomedicine, characterized in that the training method comprises an entity recognition module, a hypertext embedding module, a word unit segmentation module and a model training module;

[0017] The entity recognition module comprises the following steps:

[0018] S1. Identify gene entities, protein entities, molecular entities or language text entities in natural language text;

[0019] S2, modal information matching gene entities, protein entities, molecular entities or language text entities;

[0020] The hypertext embedding module comprises the following steps:

[0021] S3, embedding the matched modal information into a gene entity, a protein entity, a molecule entity or a language text entity to construct a hypertext;

[0022] The word unit segmentation module comprises the following steps:

[0023] S4, designing word-units according to the modal information, embedding the word-units into the hypertext, and constructing the embedded word-unit hypertext;

[0024] The model training module includes the following steps:

[0025] S5. Use the embedded word-meta hypertext as the corpus for training the large language model.

[0026] Specifically, the modal information described in steps S2-S4 includes gene sequence information, protein sequence information, molecular structure information or natural text information.

[0027] More specifically, the word unit design formula of the gene sequence information is: *(|base|**|base|**|base|) n *; where n≥1.

[0028] Preferably, the bases include A, G, C, T or U;

[0029] The A represents adenine, the G represents guanine, the C represents cytosine, the T represents thymine, and the U represents uracil.

[0030] More specifically, the word unit design formula of protein sequence information is: <|amino acid|>n; where n≥1.

[0031] Preferably, the amino acids include A, R, N, D, C, Q, E, G, H, I, L, K, M, F, P, S, T, W, Y or V;

[0032] The A represents alanine, the R represents arginine, the N represents asparagine, the D represents aspartic acid, the C represents cysteine, the Q represents glutamine, the E represents glutamic acid, the G represents glycine, the H represents histidine, the I represents isoleucine, the L represents leucine, the K represents lysine, the M represents methionine, the F represents phenylalanine, the P represents proline, the S represents serine, the T represents threonine, the W represents tryptophan, the Y represents tyrosine, and the V represents valine.

[0033] More specifically, the word unit design formula of the molecular sequence information is: {|x|}n; where n≥1;

[0034] Preferably, the x comprises a molecular linear sequence; the molecular linear sequence comprises letter units such as SMILES and SELFIES.

[0035] Further preferably, the molecular linear sequence includes C, N, O, S, F, Cl, Br, I, H, natural numbers, -, =, #, [], %, $, @, / , \;

[0036] The C represents carbon, the N represents nitrogen, the O represents oxygen, the S represents sulfur, the F represents fluorine, the Cl represents chlorine, the Br represents bromine, the I represents iodine, the H represents hydrogen, the natural number represents the index of the ring, the [] is used to represent the charge, the % represents the separator for multi-fragment molecules, the $ represents a multimer or repeating unit of the molecule, the @ represents the stereochemistry of the atom, the / represents the stereochemistry of the bond is left-handed, and the \ represents the stereochemistry of the bond is right-handed.

[0037] The natural numbers represent the index of the ring, for example, the key between the numbers 1 and 2 represents the beginning and the end of a ring.

[0038] The [] is used to represent electric charge, for example, [O-] represents an oxygen atom with a negative charge.

[0039] Specifically, the natural language text described in step S1 includes paper materials, patent materials, experimental data materials or knowledge graph materials.

[0040] Preferably, the natural language text described in step S1 includes paper materials, patent materials, experimental data materials or knowledge graph materials in the field of biomedicine.

[0041] Specifically, the recognition or matching described in step S1 or step S2 is performed using a large language model.

[0042] Preferably, the large language model includes the Mistral-7B large language model, the MolT5-Large large language model, the Meditron-7B large language model or the llama2 large language model.

[0043] Further preferably, the large language model is the Mistral-7B large language model.

[0044] Preferably, the large language model described in step S5 includes the Mistral-7B large language model, the MolT5-Large large language model, the Meditron-7B large language model or the llama2 large language model.

[0045] Further preferably, the large language model described in step S5 is a llama2 large language model.

[0046] Specifically, the training described in step S5 includes: pre-training the embedded word-meta hypertext.

[0047] Preferably, the pre-training includes:

[0048] S1. Initialize the model parameters, such as weights and biases. These parameters will be adjusted during the training process.

[0049] S2. Design pre-training tasks suitable for the model. Common pre-training tasks include masked language model (MLM), next sentence prediction (NSP), entity recognition, etc. These tasks can help the model learn the inherent laws of language.

[0050] S3. Use a large amount of computing resources (such as GPU or TPU) to train the model. During the training process, the model will continue to learn and adjust its parameters to better complete the pre-training task.

[0051] On the other hand, the present invention provides a training system for a multimodal large language model, wherein the training system includes using the above-mentioned training method to train the corpus.

[0052] On the other hand, the present invention provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned training method when executing the computer program.

[0053] On the other hand, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above-mentioned training method when executed by a processor.

[0054] The technical effects achieved by the present invention are:

[0055] (1) Aiming at the characteristics of multimodal sequence data in the biomedical field, the present invention designs a method for embedding sequence data of multiple modalities into text to construct a hypertext corpus, thereby training a large language model.

[0056] (2) The present invention uses a general large language model to automatically identify the names or references of genes, proteins and molecules from the natural text descriptions of documents and patents, and then embeds them. This method of identifying based on the real natural text in the documents rather than manually constructing sentences from data can achieve a more natural modal fusion.

[0057] (3) The present invention embeds the matched multimodal sequence data into the text according to the marking rules, thereby enriching the text information without destroying the original text semantics, thereby achieving a better fitting effect on the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 Flowchart of the multimodal hypertext large language model training method for the biomedical field. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0060] Example 1 Training method of multimodal large language model in the field of biomedicine

[0061] 1. Entity Recognition Module

[0062] 1.1 Get the dataset

[0063] The present invention firstly obtains 8 million documents from the public biomedical literature dataset PubMed Central (PMC).

[0064] 1.2 Matching molecular structural information or nucleic acid and protein sequence information

[0065] The open source large language model mistral-7b is used to identify the names or references of genes, proteins or molecules in the text, and match the corresponding molecular structure information or nucleic acid and protein sequence information.

[0066] Entity recognition module example:

[0067] Trichostatin A is a histone deacetylase (HDAC) inhibitor that upregulates the expression of p21.

[0068] Molecular Entity: Trichostatin A;

[0069] Protein entity: Histone deacetylase;

[0070] Gene entity: p21.

[0071] 2. Hypertext Embedding Module

[0072] The structural information of the corresponding molecules or the sequence information of nucleic acids and proteins are matched and embedded into the name or reference according to the rules of "markup language" to construct a new corpus, which is hypertext.

[0073] Hypertext embedding module example:

[0074]

[0075]

[0076] 3. Token Segmentation Module

[0077] The molecular structure information or sequence information of nucleic acids and proteins of different modalities in the hypertext are divided according to different words to construct a hypertext embedded with data of different modalities.

[0078] An example of a word segmentation module is shown in Table 1:

[0079] Table 1 Example of word segmentation module

[0080]

[0081] 4. Model training module

[0082] Hypertext embedded with data from different modalities is used as the corpus for training large language models to achieve fusion between modalities without causing semantic confusion.

[0083] (1) Hypertext is used as the corpus of the large language model, and the word-merging module is used to process the corpus into the input format of the large language model llama2, that is, the fusion of sequence and natural text.

[0084] <molecule>

[0085] <mark>曲古抑菌素A< / mark>

[0086] <smiles>|O|}{|=|}{|C|}{|(|}{|N|}{|O|}{|)|}{|\|}{|C|}{|=|}{|C|}{|\|}{|C|}{|(|}{|=|}{|C|}{|\[|}{|C|}{|@|}{|H|}{|]|}{|(|}{|C|}{|(|}{|=|}{|O|}{|)|}{|c|}{|1|}{|c|}{|c|}{|(|}{|N|}{|(|}{|C|}{|)|}{|C|}{|)|}{|c|}{|c|}{|1|}{|)|}{|C|}{|)|}{|C|}< / smiles>

[0087] < / molecule>

[0088] is a

[0089] <protein>

[0090] <mark>组蛋白脱乙酰基酶(HDAC)< / mark>

[0091] <amino><|G|><|A|><|M|><|T|><|K|><|P|><|F|><|T|><|T|><|G|><|D|><|M|><|L|><|K|><|H|><|Q|>…< / amino>

[0092] < / protein>

[0093] inhibitor leads to the upregulation of

[0094] <gene>

[0095] <mark>p21< / mark>

[0096] <nucleic>*|A|**|T|**|C|**|G|**|G|**|C|**|A|**|A|**|T|**|T|**|T|**|A|**|G|**|A|**|C|*… <nucleic>

[0097] < / nucleic> < / nucleic> < / gene>

[0098] (2) Pre-train llama2 on hypertext corpus; first, initialize the model parameters with the open source llama2 parameters. Pre-training uses the next word prediction method. Use a large amount of computing resources to train the model. During the training process, the model will continue to learn and adjust its parameters to more accurately predict the next word.

[0099] (3) Fine-tune the algorithm on the tasks of generating or optimizing molecules based on language descriptions (Table 2) and generating text descriptions of structural features and potential properties of given molecules (Table 3), and test it on the public basic datasets ChEBI and L+M-24.

[0100] Table 2 Generate or optimize molecules based on language description

[0101]

[0102]

[0103] Note: In the table, ↑ means the higher the index, the better the model effect; ↓ means the higher the index, the better the model effect; * means the results are taken from the original paper; # means the results are taken from the competition ranking list; Ours means the model of this invention.

[0104] Table 3 Structural features and potential properties of generated molecules Text description

[0105]

[0106] Note: In the table, ↑ means the higher the index, the better the model effect; ↓ means the higher the index, the better the model effect; * means the results are taken from the original paper; # means the results are taken from the competition ranking list; Ours means the model of this invention.

[0107] The effect of the model of the present invention is better than the current best model in most indicators, demonstrating the effectiveness of the solution.

[0108] The above detailed description is a specific description of one of the feasible embodiments of the present invention, and the embodiment is not intended to limit the scope of the present invention. It should be pointed out that any equivalent implementation or change that does not deviate from the present invention should be included in the scope of the technical solution of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached requirements.

Claims

1. A method for training a multimodal large language model, characterized in that: The training method includes an entity recognition module, a hypertext embedding module, a word unit segmentation module and a model training module; The entity recognition module comprises the following steps: S1. Identify gene entities, protein entities, molecular entities or language text entities in natural language text; S2, modal information matching gene entities, protein entities, molecular entities or language text entities; The hypertext embedding module comprises the following steps: S3, embedding the matched modal information into a gene entity, a protein entity, a molecule entity or a language text entity to construct a hypertext; The word unit segmentation module comprises the following steps: S4, designing word-units according to the modal information, embedding the word-units into the hypertext, and constructing the embedded word-unit hypertext; The model training module includes the following steps: S5. Use the embedded word-meta hypertext as the corpus for training the large language model.

2. The training method according to claim 1, characterized in that: The modal information described in steps S2-S4 includes gene sequence information, protein sequence information, molecular structure information or natural text information.

3. The training method according to claim 2, characterized in that: The word element design formula of the gene sequence information is: *(|base|**|base|**|base|) n *; wherein n≥1; the base comprises A, G, C, T or U; The A represents adenine, the G represents guanine, the C represents cytosine, the T represents thymine, and the U represents uracil.

4. The training method according to claim 2, characterized in that: The word element design formula of the protein sequence information is: <|amino acid|>n; wherein n≥1; the amino acid includes A, R, N, D, C, Q, E, G, H, I, L, K, M, F, P, S, T, W, Y or V; The A represents alanine, the R represents arginine, the N represents asparagine, the D represents aspartic acid, the C represents cysteine, the Q represents glutamine, the E represents glutamic acid, the G represents glycine, the H represents histidine, the I represents isoleucine, the L represents leucine, the K represents lysine, the M represents methionine, the F represents phenylalanine, the P represents proline, the S represents serine, the T represents threonine, the W represents tryptophan, the Y represents tyrosine, and the V represents valine.

5. The training method according to claim 2, characterized in that: The word element design formula of the molecular structure information is: {|x|}n; wherein n≥1; the x includes a molecular linear sequence representation; the molecular linear sequence representation includes C, N, O, S, F, Cl, Br, I, H, natural numbers, -, =, #, [], %, $, @, / , \; The C represents carbon, the N represents nitrogen, the O represents oxygen, the S represents sulfur, the F represents fluorine, the Cl represents chlorine, the Br represents bromine, the I represents iodine, the H represents hydrogen, the natural number represents the index of the ring, the [] is used to represent the charge, the % represents the separator for multi-fragment molecules, the $ represents a multimer or repeating unit of the molecule, the @ represents the stereochemistry of the atom, the / represents the stereochemistry of the bond is left-handed, and the \ represents the stereochemistry of the bond is right-handed.

6. The training method according to claim 1, characterized in that: The natural language text in step S1 includes paper materials, patent materials, experimental data materials or knowledge graph materials; The recognition or matching described in step S1 or step S2 is performed using a large language model, and the large language model includes a Mistral-7B large language model, a MolT5-Large large language model, a Meditron-7B large language model or an llama2 large language model.

7. The training method according to claim 1, characterized in that: The large language model described in step S5 includes the Mistral-7B large language model, the MolT5-Large large language model, the Meditron-7B large language model or the llama2 large language model; the training described in step S5 includes: pre-training the embedded word-unit hypertext.

8. A multimodal large language model training system, characterized in that: The training system includes using the training method described in any one of claims 1-7 to train the corpus.

9. A terminal device, characterized in that: The terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the training method described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the training method according to any one of claims 1 to 7 is implemented.

Images