Training data generation method and device based on language model
By identifying and expanding semantic networks to generate training datasets, the problem of high cost of acquiring training data for large language models is solved, achieving efficient and low-cost data acquisition and quality assurance, which is applicable to specific fields.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BGI RES BEIJING
- Filing Date
- 2024-10-22
- Publication Date
- 2026-04-24
AI Technical Summary
In existing technologies, acquiring training data for large language models is costly and time-consuming, especially in specific domains where data demand is high, unstructured data is problematic, and noise issues are severe. Manual annotation is also costly and difficult to scale.
By inputting a text set into the target language model, entities, concepts, and relationships are identified, an initial semantic network is generated, and the language model is updated based on the expanded semantic network to generate a training dataset for the target domain. An automatic annotation function is used to reduce reliance on manual annotation.
It significantly reduces the cost and time of training data acquisition, improves data acquisition efficiency, generates training data of higher quality and greater diversity, adapts to specific domain needs, and reduces reliance on personnel with specialized knowledge.
Smart Images
Figure CN121920447A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of natural language processing technology or other related fields, and more specifically, to a method and apparatus for generating training data based on a language model. Background Technology
[0002] Large Language Models (LLMs), natural language processing models based on deep learning techniques, have made significant progress in the field of Natural Language Processing (NLP). These models learn from massive amounts of text data, acquiring profound language understanding capabilities and enabling them to perform various tasks, including text generation, summarization, translation, and question answering. With advancements in computing power and algorithm optimization, LLMs are increasingly adept at understanding complex semantics and uncovering deep linguistic relationships.
[0003] With the continuous development of LLM, the current direction of model development is how to automatically extract and construct semantic networks from large-scale text data to more effectively represent and reason about knowledge. For example, semantic networks automatically generated by LLM can be used to improve the relevance judgment of search engines, enhance the dialogue understanding ability of chatbots, or improve the accuracy of personalized recommendations in recommendation systems.
[0004] While large language models have demonstrated impressive capabilities in related technologies, fine-tuning them for specific tasks or domains still faces the challenge of high data requirements. In particular, training data for various models presents several problems. For example, with publicly available training datasets, issues include: 1) Unstructured and noisy data: Public datasets are often unstructured, containing a large amount of irrelevant information or noise, which affects the correct construction of semantic networks. 2) Insufficient domain-specific data: Although public datasets are large in volume, data in specific domains is often limited, restricting the ability to build refined semantic networks in these domains. For manually annotated training data: 1) High cost and time consumption: While manual annotation provides high-quality data, it is costly and time-consuming, especially for large-scale datasets. 2) Difficulty in scaling: The speed and flexibility of manual annotation often cannot keep up with the needs of continuously updated and expanded datasets. 3) Difficulty in annotating specialized domains: Finding suitable annotators can be difficult for domains requiring in-depth expertise.
[0005] There is currently no effective solution to the above problems. Summary of the Invention
[0006] This invention provides a method and apparatus for generating training data based on a language model, which at least solves the technical problem of high cost and time consumption when acquiring training data for large language models in related technologies.
[0007] According to one aspect of the present invention, a method for generating training data based on a language model is provided, comprising: inputting a text set into a target language model to identify the entity set, concept set, and relation set of all texts; mapping the identified domain entities, text concepts, and the relationships between domain entities to a data relation description framework to generate an initial semantic network, wherein the initial semantic network includes a node set and an edge set, the node set including nodes represented by each domain entity and text concept, and the edge set including edges represented by each relationship; expanding the initial semantic network, and updating the target language model based on the expanded target semantic network, thereby generating a training dataset for the target domain from the updated target language model.
[0008] Optionally, the step of inputting the text set into the target language model includes: establishing network connections with N trusted data sources related to the target domain, and reading trusted data text from each of the trusted data sources to obtain the text set, wherein the trusted data sources include at least: public databases, academic paper websites, and N is a positive integer; preprocessing the text set, wherein the preprocessing includes at least one of the following: text cleaning, word segmentation, and standardization, wherein the text cleaning includes at least: removing duplicate text data and removing noise data, and the standardization includes unifying the naming conventions of entities in all domains in all texts; and inputting the preprocessed text set into the target language model.
[0009] Optionally, the steps of identifying the entity set, concept set, and relation set of all texts include: analyzing all texts in the text set using the target language model, identifying entity words related to the target domain in each text and the category to which the entity words belong, obtaining the domain entities and the categories to which the entities belong, and obtaining the entity set; labeling the domain entities and the categories to which the entities belong to the corresponding source texts; obtaining the text concepts to which the labeled domain entities belong in the texts, and obtaining the concept set; obtaining the interactions or associations between the labeled domain entities in the texts, and obtaining the relation set containing the associations between all domain entities.
[0010] Optionally, after identifying the entity set, concept set, and relation set of all texts, the method further includes: standardizing the mapping of the domain entities, text concepts, and relationships based on domain knowledge data of the target domain to generate a standardized data relationship description framework.
[0011] Optionally, the step of extending the initial semantic network includes: generating text paragraphs related to the target domain based on the initial semantic network using the target language model; analyzing the generated text paragraphs to extract new domain entities, new text concepts, and new relationships, and mapping the new domain entities, new text concepts, and new relationships to the data relationship description framework; and integrating the mapped new domain entities, new text concepts, and new relationships into the initial semantic network to obtain the target semantic network.
[0012] Optionally, the step of updating the target language model based on the extended target semantic network and generating a training dataset from the updated target language model includes: generating a text dataset associated with the target domain based on the extended target semantic network and the target language model; updating the target language model using the generated text dataset; and annotating the text dataset using the target language model to generate the training dataset.
[0013] Optionally, the step of annotating the text dataset by the target language model to generate a training dataset includes: calling an automatic annotation function by the target language model, using the automatic annotation function to annotate the text dataset, and generating the labeled training dataset.
[0014] Optionally, the target language model is a BERT model or a GPT model, the data relationship description framework is a schema framework, and the initial semantic network is a semantic network modeled using graph neural network technology.
[0015] According to another aspect of the present invention, a training data generation apparatus based on a language model is also provided, comprising: a text recognition unit, configured to input a text set into a target language model and recognize the entity set, concept set, and relation set of all texts; a semantic network generation unit, configured to map the identified domain entities, text concepts, and the relationships between domain entities into a data relation description framework to generate an initial semantic network, wherein the initial semantic network includes a node set and an edge set, the node set including nodes represented by each domain entity and text concept, and the edge set including edges represented by each relationship; and a training data generation unit, configured to extend the initial semantic network and update the target language model based on the extended target semantic network, thereby generating a training dataset for the target domain from the updated target language model.
[0016] Optionally, the text recognition unit includes: a network connection establishment module, used to establish network connections with N trusted data sources in the associated target domain, and read trusted data text from each of the trusted data sources to obtain the text set, wherein the trusted data sources include at least: public databases, academic paper websites, and N is a positive integer; a text preprocessing module, used to preprocess the text set, wherein the preprocessing includes at least one of the following: text cleaning, word segmentation, and standardization, wherein the text cleaning includes at least: removing duplicate text data and removing noise data, and the standardization includes unifying the naming conventions of entities in all domains in all texts; and a text input module, used to input the preprocessed text set into the target language model.
[0017] Optionally, the text recognition unit further includes: a text analysis module, used to analyze all texts in the text set using the target language model, identify entity words related to the target domain in each text and the category to which the entity words belong, obtain the domain entities and the categories to which the entities belong, and obtain an entity set; a text annotation module, used to annotate the domain entities and the categories to which the entities belong to the corresponding source texts respectively; a text concept acquisition module, used to acquire the text concepts to which the annotated domain entities belong in the text, and obtain a concept set; and a relationship acquisition module, used to acquire the interaction or association relationships between the annotated domain entities in the text, and obtain a relationship set containing the association relationships between all domain entities.
[0018] Optionally, the language model-based training data generation device further includes: a mapping unit, used to perform standardized mapping on the domain entities, the text concepts and the relationships based on domain knowledge data of the target domain after recognizing the entity set, concept set and relationship set of all texts, to generate a standardized data relationship description framework.
[0019] Optionally, the training data generation unit includes: a text paragraph generation module, used to generate text paragraphs related to the target domain based on the initial semantic network using the target language model; a text paragraph analysis module, used to analyze the generated text paragraphs, extract new domain entities, new text concepts, and new relationships, and map the new domain entities, the new text concepts, and the new relationships to the data relationship description framework; and an integration module, used to integrate the mapped new domain entities, the new text concepts, and the new relationships into the initial semantic network to obtain the target semantic network.
[0020] Optionally, the training data generation unit includes: a dataset generation module, used to generate a text dataset associated with the target domain based on the expanded target semantic network and the target language model; and a training data generation module, used to update the target language model with the generated text dataset, and to annotate the text dataset by the target language model to generate the training dataset.
[0021] Optionally, the training data generation module includes a labeling submodule, which is used by the target language model to call an automatic labeling function, and use the automatic labeling function to label the text dataset to generate the labeled training dataset.
[0022] Optionally, the target language model is a BERT model or a GPT model, the data relationship description framework is a schema framework, and the initial semantic network is a semantic network modeled using graph neural network technology.
[0023] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute any of the above-described language model-based training data generation methods.
[0024] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the language model-based training data generation method described above.
[0025] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of the language model-based training data generation method described in any one of the preceding embodiments.
[0026] In this disclosure, a text set is input into a target language model to identify the entity set, concept set, and relation set of all texts. The identified domain entities, text concepts, and the relationships between domain entities are mapped to a data relation description framework to generate an initial semantic network. The initial semantic network is then expanded, and the target language model is updated based on the expanded target semantic network. The updated target language model generates a training dataset for the target domain. The initial semantic network contains a node set and an edge set. The node set includes nodes represented by domain entities and text concepts, and the edge set includes edges represented by relationships.
[0027] In this disclosure, semantic networks can be automatically constructed and training datasets can be generated, significantly reducing reliance on manual resources and thus greatly improving the efficiency of data acquisition. This solves the technical problem of high cost and time consumption when acquiring training data for large language models in related technologies. Attached Figure Description
[0028] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:
[0029] Figure 1 This is a flowchart of an optional language model-based training data generation method according to an embodiment of the present invention;
[0030] Figure 2 This is a flowchart of an optional method for generating a training dataset based on a large language model (LLM) according to an embodiment of the present invention;
[0031] Figure 3 This is a schematic diagram of an optional language model-based training data generation device according to an embodiment of the present invention;
[0032] Figure 4 This is a hardware structure block diagram of an electronic device (or mobile device) that performs a language model-based training data generation method according to an embodiment of the present invention. Detailed Implementation
[0033] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0034] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0035] To facilitate understanding of the present invention by those skilled in the art, some terms or nouns involved in the various embodiments of the present invention are explained below:
[0036] A schema framework is a tool used to describe data structures and relationships, typically used in database design and data model building. It defines the structure and attributes of elements such as tables, fields, relationships, and constraints, as well as the relationships between them.
[0037] It should be noted that the language model-based training data generation method and apparatus disclosed herein can be used in the field of natural language processing technology when generating corpora and extracting training datasets based on natural language models, and can also be used in any field other than natural language processing technology (e.g., intelligent question answering, personalized content recommendation, medical text retrieval, knowledge graph construction, etc.). In the case of generating corpora and extracting training datasets based on natural language models, this disclosure does not limit the application field of the language model-based training data generation method and apparatus.
[0038] The following embodiments of the present invention can be applied to various systems / applications / devices for generating training data based on language models. The present invention generates rich semantic network training data through Large Language Models (LLM), enabling large-scale data generalization from a limited initial dataset. This provides sufficient training corpus for fine-tuning the LLM model while significantly reducing the need and cost of manual annotation. In data-scarce fields (e.g., biomedicine), it effectively promotes the breadth and depth of LLM applications, reducing reliance on large amounts of manually annotated data while ensuring model performance.
[0039] This invention provides robust support for the customized application and fine-tuning of large language models, particularly suitable for applications with scarce data or in specialized fields. By automatically generating schema frameworks, constructing and expanding semantic networks, and generating high-quality training datasets based on these frameworks, it effectively addresses the problems of high data preparation costs, inconsistent data quality, and limited domain coverage in existing technologies. In specialized fields such as biomedicine, the schema framework of this invention not only enhances the accuracy of semantic network construction but also supports the ability to update and expand the network in real time, ensuring that the network can keep up with the rapid changes in knowledge within the domain.
[0040] This invention proposes an automated method for constructing and expanding semantic networks based on large-scale language models, specifically designed to generate high-quality training data for specific domains (such as biomedicine). This approach leverages the deep language understanding capabilities of LLM (Large Language Modeling), combined with a customized schema framework and mathematical model, to automatically generate a semantic network from initial text data. Through iterative expansion, the network content is further optimized and enriched, resulting in a high-quality training dataset that meets the specific needs of the domain. This not only solves the problems of data noise and unstructured data but also overcomes the limitations of high cost, time consumption, and scalability associated with manual annotation, providing a more efficient and economical solution for constructing high-quality semantic networks.
[0041] The present invention will now be described in detail with reference to various embodiments.
[0042] Example 1
[0043] According to an embodiment of the present invention, an embodiment of a training data generation method based on a language model is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.
[0044] This embodiment provides a process for efficient semantic network extraction using large language models (LLMs), and for automatically generating high-quality biomedical domain labeled data based on this information. This method places particular emphasis on improving the quality and diversity of automatically generated data, aiming to reduce reliance on experts and lower the cost of manual annotation.
[0045] Figure 1 This is a flowchart of an optional language model-based training data generation method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:
[0046] Step S101: Input the text set into the target language model to identify the entity set, concept set, and relation set of all texts.
[0047] In this embodiment, the first step is to obtain the text dataset associated with the target domain, and then preprocess the obtained text dataset.
[0048] Optionally, the step of inputting the text set into the target language model includes: establishing network connections with N trusted data sources related to the target domain, and reading trusted data text from each trusted data source to obtain a text set, wherein the trusted data sources include at least: public databases and academic paper websites, and N is a positive integer; preprocessing the text set, wherein the preprocessing includes at least one of the following: text cleaning, word segmentation, and standardization, wherein text cleaning includes at least: removing duplicate text data and removing noise data, and standardization includes unifying the naming conventions of entities in each domain in all texts; and inputting the preprocessed text set into the target language model.
[0049] Various types of data texts are retrieved from trusted data sources within the target domain. These trusted data sources, in addition to the aforementioned public databases and research paper websites, can also include electronic libraries, electronic archives, and scientific paper repositories within the target domain. From these trusted data sources, N core trusted data texts are selected, represented as a text set (T = {t_1, t_2, ..., t_N}).
[0050] Next, these data texts undergo preprocessing, including data cleaning (removing irrelevant information and noise to ensure low noise levels, such as noise removal and duplicate content removal), word segmentation (segmenting the text into words to facilitate LLM word-by-word analysis), and standardization (unifying entity naming conventions, such as standardizing "STAT1" into a uniform annotation format). The preprocessed text can be represented as the set resulting from the preprocessing function `Preprocess(·)`.
[0051] T'={Preprocess(t_1), Preprocess(t_2),..., Preprocess(t_N)}.
[0052] Optionally, the steps of identifying the entity set, concept set, and relation set of all texts include: analyzing all texts in the text set using a target language model, identifying entity words related to the target domain in each text and the category to which the entity words belong, obtaining domain entities and their categories, and obtaining an entity set; labeling the domain entities and their categories to their corresponding source texts; obtaining the text concepts to which the labeled domain entities belong in the text, and obtaining a concept set; obtaining the interactions or relationships between the labeled domain entities in the text, and obtaining a relation set containing the relationships between all domain entities.
[0053] For example, by analyzing a preprocessed text set T' using LLM, potential key entity set E, concept set C, and relation set R can be identified. This process can be represented by the action of the LLM information extraction function LLMextract(·): [E,C,R]=LLMextract(T').
[0054] Optionally, after identifying the entity set, concept set, and relation set of all texts, the method further includes: standardizing the mapping of domain entities, text concepts, and relationships based on domain knowledge data of the target domain to generate a standardized data relationship description framework.
[0055] For example, based on the identified entities E, concepts C, and relations R, and combined with domain knowledge, a standardized schema framework is automatically generated. This framework can be represented as a set of mappings to E, C, and R:
[0056] Sschema={map(E),map(C),map(R)}.
[0057] Here, map(·) represents a standardized mapping of domain entities, text concepts, and relationships based on domain knowledge.
[0058] This embodiment achieves a high degree of domain adaptability and interpretability in the generation and application of the schema framework. The generation and mapping of entities, concepts, and relationships through a formulaic approach ensures the standardization and consistency of the entire process.
[0059] It should also be noted that in this embodiment, the specific type of the target language model is not limited. For example, the target language model can be a BERT model or a GPT model, the data relationship description framework is a schema framework, and the initial semantic network is a semantic network modeled using graph neural network technology.
[0060] Step S102: Map the identified domain entities, text concepts, and relationships between domain entities to a data relationship description framework to generate an initial semantic network. The initial semantic network includes a node set and an edge set. The node set includes nodes represented by domain entities and text concepts, and the edge set includes edges represented by relationships.
[0061] For example, the initially identified domain entities, textual concepts, and relationships between these entities are mapped into the generated schema framework. This mapping process can be represented by the function MapToSchema(·): [Emapped,Cmapped,Rmapped]=MapToSchema(E,C,R,Sschema).
[0062] When constructing the semantic network, the process includes: building an initial semantic network Sinit based on the mapped entities Emapped, concepts Cmapped, and relations Rmapped. This initial semantic network can be represented as a semantic network graph G(V,E) containing nodes and edges.
[0063] Sinit=G(V,E)=G(Emapped∪Cmapped,Rmapped).
[0064] Where V = Emapped∪Cmapped is the set of nodes in the semantic network graph, and E = Rmapped is the set of edges pointing to the semantic network.
[0065] It should be noted that, in addition to generating semantic networks using text data, this embodiment can also integrate multimodal data (such as images, videos, and audio), utilize large-scale multimodal language models for information extraction and training data generation, enrich the dimensions of training data, and improve the model's performance when processing multimodal data.
[0066] Furthermore, this embodiment can also utilize graph neural network (GNN) technology to model and optimize the semantic network, replacing or assisting traditional LLM methods. GNNs can process graph-structured data more effectively and may provide better performance in some cases.
[0067] Step S103: Expand the initial semantic network and update the target language model based on the expanded target semantic network. The updated target language model generates a training dataset for the target domain.
[0068] It should be noted that the target domain mentioned in this embodiment can refer to a domain with high requirements for model training data, generally requiring professional manual annotation of the training data, such as the biomedical field. Furthermore, this embodiment can also pre-train the model in a specific domain before constructing the semantic network and generating data, which can optimize the model's understanding and processing capabilities of specific domain knowledge, thereby improving the quality of the final training data.
[0069] In this step, the large language model is updated through an extended semantic network, which facilitates the automatic expansion and updating of the training dataset in the target domain by the updated large language model.
[0070] Optionally, the step of extending the initial semantic network includes: generating text paragraphs related to the target domain based on the initial semantic network using the target language model; analyzing the generated text paragraphs to extract new domain entities, new text concepts, and new relationships, and mapping the new domain entities, new text concepts, and new relationships to a data relationship description framework; and integrating the mapped new domain entities, new text concepts, and new relationships into the initial semantic network to obtain the target semantic network.
[0071] For example, LLM is used to generate text segments related to the network content based on the current semantic network Sinit (defined as Dgen in this embodiment). This process can be represented as: Dgen = LLMgenerate(Sinit). Then, the generated text segment Dgen is analyzed to extract new entities Enew, concepts Cnew, and relationships Rnew, and these are mapped into the schema framework to ensure they conform to the semantic standards within the domain. For example, by analyzing the newly generated text segment, a previously unreported association between a gene and a new disease may be discovered, and this association will be added to the existing semantic network. [Enew, Cnew, Rnew] = LLMextract(Dgen).
[0072] Furthermore, this embodiment also provides a dynamic semantic network update mechanism: the semantic network is dynamically updated based on newly collected data or the model's performance in practical applications to reflect the latest knowledge and data. This ensures that training data continuously adapts to the changing information environment.
[0073] Optionally, the step of updating the target language model based on the extended target semantic network and generating a training dataset from the updated target language model includes: generating a text dataset associated with the target domain based on the extended target semantic network and the target language model; updating the target language model using the generated text dataset; and annotating the text dataset using the target language model to generate a training dataset.
[0074] The newly extracted entities, concepts, and relationships are integrated into the initial semantic network Sinit, and the updated target semantic network Supdated is represented as follows:
[0075] Supdated=G(V',E')=G(V∪(Enew∪Cnew),E∪Rnew);
[0076] Where V'=V∪(Enew∪Cnew) is the updated node set, and E'=E∪Rnew is the updated edge set.
[0077] Data generation based on semantic networks: Guided by the expanded and updated target semantic network Supdated, LLM is used to generate more domain-related high-quality text datasets Dtrain, where Dtrain = LLMgenerate(Supdated).
[0078] This embodiment can generate a large amount of high-quality training data. By using the formula Dtrain = LLMgenerate(Supdated), it ensures that the generated data is highly adaptable to the domain and provides reliable data support for model fine-tuning.
[0079] Optionally, the step of annotating the text dataset by the target language model to generate the training dataset includes: calling an automatic annotation function by the target language model, using the automatic annotation function to annotate the text dataset, and generating a labeled training dataset.
[0080] For example, the model can be further fine-tuned using the generated text dataset Dtrain. LLM automatically labels this data to generate the final training dataset Dfinal. Dfinal = LLMlabel(Dtrain), where LLMlabel(·) is the automatic labeling function used to generate labeled training data.
[0081] From text data extraction to semantic network expansion and training data generation, the entire process is highly transparent and repeatable. Automated annotation, through the formula Dfinal = LLMlabel(Dtrain), ensures consistency and efficiency in annotation.
[0082] Through the above steps, a text set can be input into the target language model to identify the entity set, concept set, and relation set of all texts. The identified domain entities, text concepts, and the relationships between domain entities are mapped to a data relationship description framework to generate an initial semantic network. The initial semantic network is then expanded, and the target language model is updated based on the expanded target semantic network. The updated target language model generates a training dataset for the target domain. The initial semantic network contains a node set and an edge set. The node set includes nodes representing domain entities and text concepts, and the edge set includes edges representing relationships. In this embodiment, the semantic network can be automatically constructed and the training dataset generated, significantly reducing reliance on manual resources and greatly improving the efficiency of data acquisition. This solves the technical problem of high cost and time consumption when acquiring training data for large language models in related technologies.
[0083] The invention will now be described in conjunction with another optional implementation.
[0084] This embodiment illustrates how to efficiently extract semantic networks using a large language model (LLM), and the process of automatically generating high-quality biomedical domain labeled data based on this information. Emphasis is placed on improving the quality and diversity of automatically generated data, aiming to reduce reliance on experts and lower the cost of manual annotation.
[0085] Figure 2 This is a flowchart of an optional method for generating a training dataset based on a large language model (LLM) according to an embodiment of the present invention, such as... Figure 2 As shown, it includes:
[0086] The first step is to obtain the tag data, including data entities and data relationships.
[0087] For example, select the text example: "Another STAT family member, STAT1, possesses cancer-inhibitory properties and can promote apoptosis in tumor cells upon activation.", and then perform text preprocessing, entity annotation, and relation extraction (determine the interactions or relationships between the annotated entities in the text).
[0088] The preprocessing step involves inputting text into an LLM file. The goal is to automatically identify and label key entities and their categories, and further determine the relationships between all entities in the sentence. This preprocessing step includes:
[0089] Text cleaning: Remove irrelevant information and noise to ensure data purity.
[0090] Word segmentation: The text is segmented into words to facilitate word-by-word analysis by the LLM.
[0091] Standardization: Unify the naming conventions for entities, for example, standardize "STAT1" into a unified label format.
[0092] The entity extraction steps include:
[0093] Entity recognition: Using LLM to analyze text and identify biological entities within it.
[0094] In this example, the identified entities and their categories include:
[0095] Protein: STAT, STAT1;
[0096] Phenotype:cancer-inhibitory properties,apoptosis;
[0097] Cell: tumor cells.
[0098] Entity annotation: Based on the recognition results, entities and their categories are annotated in the original text. For example, the instance text will become:
[0099] Another <protein> STAT< / protein> family member, <protein> STAT1< / protein> ,possesses <phenotype> cancer-inhibitory properties< / phenotype> and can promote <phenotype> apoptosis< / phenotype> in <cell> tumor cells< / cell> "upon activation."
[0100] The relationship extraction steps include:
[0101] In this example, LLM is used to analyze the semantic relationships between entities and extract the following relationships:
[0102] STAT1 ("Protein") and cancer-inhibitory properties ("Phenotype"): have inhibitory effects;
[0103] STAT1 ("Protein") and apoptosis ("Phenotype"): Facilitating relationship;
[0104] STAT1 ("Protein") and tumor cells ("Cell"): act on and are related to the two phenotypes mentioned above.
[0105] The second step is to integrate entities and relationships to generate a semantic network.
[0106] To ensure that the identified entities and relationships conform to domain-specific standardization requirements, the extracted entities and relationships are mapped to a domain-specific schema framework to generate a semantic network. For example, Protein entities are mapped to the "Molecular" category in the schema framework, Phenotype is mapped to the "Characterize" category, and Cell is mapped to the "Cell Component" category. This mapping ensures the uniformity and consistency of the semantic network structure and provides a reliable foundation for generating high-quality training data.
[0107] The third step is to combine the prompt word and the source semantic network.
[0108] LLM is used to generate domain-specific text paragraphs based on a source semantic network. New entities and relations are then extracted from these paragraphs and mapped into a schema framework to ensure compliance with domain-specific semantic standards. The newly extracted entities, concepts, and relations are then concatenated with prompt words and integrated into the source semantic network.
[0109] The fourth step is to generate a training dataset with labeled information using a large language model.
[0110] Based on the identified entities and their relationships, LLM then automatically generates a series of new training datasets. These samples, while retaining the original entities and their relationships, increase the diversity of the data through different representations. Below are some of the generated samples:
[0111] " <protein> STAT1< / protein> is responsible for producing <phenotype> apoptosis< / phenotype> and also interacts with <cell> tumor cells< / cell> while possessing <phenotype> cancer-inhibitory properties< / phenotype> .".
[0112] The interaction between <protein> STAT1< / protein> and <cell> tumor cells< / cell> It enhances apoptosis and exerts <phenotype> cancer-inhibitory properties< / phenotype> .".
[0113] The generated training data not only encompasses various expressions of entities and relationships but also enriches the data through diverse text formats. This automatically generated training data provides a large, high-quality corpus for biomedical research, which can be used for further training and fine-tuning of machine learning models.
[0114] This invention automatically generates high-quality semantic network training data through Large Language Models (LLM), bringing several beneficial effects. The main beneficial effects include:
[0115] Significantly Improved Efficiency: By automating the construction of semantic networks and the generation of training data, this invention significantly reduces the need for human intervention, thereby greatly improving the efficiency and productivity of data acquisition. This makes it possible to obtain large amounts of high-quality data in a short time, accelerating the model development and fine-tuning process.
[0116] Improved quality and accuracy: Leveraging the deep language understanding capabilities of LLM, the generated training data is guaranteed in terms of both quality and diversity. This data reflects the subtle differences and complex relationships within the domain, providing rich corpus for model training and thus improving the feasibility of training high-quality vertical domain models.
[0117] Significant cost reduction: Compared with traditional manual annotation, this invention reduces reliance on human resources, thereby significantly reducing data acquisition costs.
[0118] Expanding the scope of application: By automatically generating domain-specific labeled data, this invention expands the application potential of LLM in various fields, especially in fields where labeled data is scarce.
[0119] The following is a detailed description with reference to another embodiment.
[0120] Example 2
[0121] The language model-based training data generation device provided in this embodiment includes multiple implementation units, each of which corresponds to a specific implementation step in Embodiment 1 above.
[0122] Figure 3 This is a schematic diagram of an optional language model-based training data generation device according to an embodiment of the present invention, such as... Figure 3 As shown, the language model-based training data generation device may include: a text recognition unit 31, a semantic network generation unit 32, and a training data generation unit 33.
[0123] The text recognition unit 31 is used to input the text set into the target language model and recognize the entity set, concept set and relation set of all texts.
[0124] The semantic network generation unit 32 is used to map the identified domain entities, text concepts and the relationships between domain entities to the data relationship description framework to generate an initial semantic network. The initial semantic network includes a set of nodes and a set of edges. The set of nodes includes nodes represented by each domain entity and text concept, and the set of edges includes edges represented by each relationship.
[0125] The training data generation unit 33 is used to extend the initial semantic network and update the target language model based on the extended target semantic network, and generate the target domain training dataset from the updated target language model.
[0126] The aforementioned language model-based training data generation device can input a text set into the target language model through the text recognition unit 31, recognizing the entity set, concept set, and relation set of all texts. The semantic network generation unit 32 maps the recognized domain entities, text concepts, and relationships between domain entities into a data relationship description framework, generating an initial semantic network. The training data generation unit 33 expands the initial semantic network and updates the target language model based on the expanded target semantic network. The updated target language model generates a training dataset for the target domain. The initial semantic network includes a node set and an edge set. The node set includes nodes representing domain entities and text concepts, and the edge set includes edges representing relationships. In this embodiment, the semantic network can be automatically constructed and the training dataset generated, significantly reducing reliance on manual resources and greatly improving data acquisition efficiency. This solves the technical problem of high cost and time consumption when acquiring training data for large language models in related technologies.
[0127] Optionally, the text recognition unit includes: a network connection establishment module, used to establish network connections with N trusted data sources in the associated target domain, and read trusted data text from each trusted data source to obtain a text set, wherein the trusted data sources include at least: public databases and academic paper websites, and N is a positive integer; a text preprocessing module, used to preprocess the text set, wherein the preprocessing includes at least one of the following: text cleaning, word segmentation, and standardization, wherein text cleaning includes at least: removing duplicate text data and removing noise data, and standardization includes unifying the naming conventions of entities in all domains in all texts; and a text input module, used to input the preprocessed text set into the target language model.
[0128] Optionally, the text recognition unit further includes: a text analysis module, used to analyze all texts in the text set using a target language model, identify entity words related to the target domain in each text and the category to which the entity words belong, obtain domain entities and the categories to which the entities belong, and obtain an entity set; a text annotation module, used to annotate the domain entities and the categories to which the entities belong to the corresponding source texts; a text concept acquisition module, used to acquire the text concepts to which the annotated domain entities belong in the text, and obtain a concept set; and a relationship acquisition module, used to acquire the interaction or association relationships between the annotated domain entities in the text, and obtain a relationship set containing the association relationships between all domain entities.
[0129] Optionally, the language model-based training data generation device further includes a mapping unit, which, after recognizing the entity set, concept set, and relation set of all texts, performs standardized mapping of domain entities, text concepts, and relationships based on domain knowledge data of the target domain to generate a standardized data relationship description framework.
[0130] Optionally, the training data generation unit includes: a text paragraph generation module, used to generate text paragraphs related to the target domain based on the initial semantic network using the target language model; a text paragraph analysis module, used to analyze the generated text paragraphs, extract new domain entities, new text concepts, and new relationships, and map the new domain entities, new text concepts, and new relationships to the data relationship description framework; and an integration module, used to integrate the mapped new domain entities, new text concepts, and new relationships into the initial semantic network to obtain the target semantic network.
[0131] Optionally, the training data generation unit includes: a dataset generation module, used to generate a text dataset related to the target domain based on the expanded target semantic network and through the target language model; and a training data generation module, used to update the target language model with the generated text dataset, and to annotate the text dataset by the target language model to generate a training dataset.
[0132] Optionally, the training data generation module includes a labeling submodule, which is used by the target language model to call an automatic labeling function to label the text dataset and generate a labeled training dataset.
[0133] Optionally, the target language model is a BERT model or a GPT model, the data relationship description framework is a schema framework, and the initial semantic network is a semantic network modeled using graph neural network technology.
[0134] Optionally, the target field is the biomedical field.
[0135] The aforementioned language model-based training data generation device may further include a processor and a memory. The aforementioned text recognition unit 31, semantic network generation unit 32, training data generation unit 33, etc., are all stored in the memory as program units, and the processor executes the aforementioned program units stored in the memory to realize the corresponding functions.
[0136] The aforementioned processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and by adjusting kernel parameters, training datasets can be automatically generated based on a large language model.
[0137] The aforementioned memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.
[0138] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute the language model-based training data generation method of any one of the above embodiments.
[0139] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the language model-based training data generation method of any one of the above embodiments.
[0140] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the language model-based training data generation method described in various embodiments of this application.
[0141] This application also provides a computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the language model-based training data generation method described in various embodiments of this application.
[0142] Figure 4 This is a hardware structure block diagram of an electronic device (or mobile device) that performs a language model-based training data generation method according to an embodiment of the present invention. Figure 4 As shown, an electronic device may include one or more ( Figure 4 The processor (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and memory 404 for storing data are illustrated using 402a, 402b, ..., 402n. In addition, it may include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a keyboard, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 4 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device described above. For example, the electronic device may also include components that are more... Figure 4 The more or fewer components shown, or having the same Figure 4 The different configurations shown.
[0143] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0144] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0145] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.
[0146] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0147] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0148] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.
[0149] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A method for generating training data based on a language model, characterized in that, include: Input the text set into the target language model to identify the entity set, concept set, and relation set of all texts; The identified domain entities, text concepts, and relationships between domain entities are mapped to a data relationship description framework to generate an initial semantic network. The initial semantic network includes a set of nodes and a set of edges. The set of nodes includes nodes represented by each domain entity and text concept, and the set of edges includes edges represented by each relationship. The initial semantic network is expanded, and the target language model is updated based on the expanded target semantic network. The updated target language model is then used to generate a training dataset for the target domain.
2. The method for generating training data based on a language model according to claim 1, characterized in that, The steps of inputting a text set into a target language model include: Establish network connections with N trusted data sources in the relevant target domain, and read trusted data text from each of the trusted data sources to obtain the text set, wherein the trusted data sources include at least: public databases and academic paper websites, and N is a positive integer; The text set is preprocessed, wherein the preprocessing includes at least one of the following: text cleaning, word segmentation, and standardization. The text cleaning includes at least: removing duplicate text data and removing noise data. The standardization includes unifying the naming conventions of entities in all domains in all texts. The preprocessed text set is input into the target language model.
3. The method for generating training data based on a language model according to claim 1, characterized in that, The steps for identifying the entity set, concept set, and relation set of all text include: The target language model is used to analyze all texts in the text set, identify the entity words in each text that involve the target domain and the category to which the entity words belong, obtain the domain entities and the categories to which the entities belong, and obtain the entity set; The domain entities and their respective categories are labeled in the corresponding source text. Obtain the text concepts in the text to which the domain entities belong, and obtain a concept set; Obtain the interactions or relationships between the domain entities that have been annotated in the text, and obtain a relationship set containing all the relationships between domain entities.
4. The method for generating training data based on a language model according to claim 1, characterized in that, After identifying the entity set, concept set, and relation set of all texts, it also includes: Based on the domain knowledge data of the target domain, the domain entities, the text concepts, and the relationships are standardized and mapped to generate a standardized data relationship description framework.
5. The method for generating training data based on a language model according to claim 1, characterized in that, The step of extending the initial semantic network includes: The target language model is used to generate text paragraphs related to the target domain based on the initial semantic network; The generated text paragraphs are analyzed to extract new domain entities, new text concepts, and new relationships, and these new domain entities, text concepts, and relationships are mapped to the data relationship description framework. The mapped new domain entities, the new text concepts, and the new relationships are integrated into the initial semantic network to obtain the target semantic network.
6. The method for generating training data based on a language model according to claim 1, characterized in that, The steps of updating the target language model based on the expanded target semantic network and generating a training dataset from the updated target language model include: Based on the expanded target semantic network, a text dataset associated with the target domain is generated through the target language model; The target language model is updated using the generated text dataset, and the target language model is used to annotate the text dataset to generate the training dataset.
7. The method for generating training data based on a language model according to claim 6, characterized in that, The steps of annotating the text dataset using the target language model to generate a training dataset include: The target language model calls the automatic annotation function to annotate the text dataset, generating the labeled training dataset.
8. The method for generating training data based on a language model according to any one of claims 1 to 7, characterized in that, The target language model is either the BERT model or the GPT model, the data relationship description framework is a schema framework, and the initial semantic network is a semantic network modeled using graph neural network technology.
9. A training data generation device based on a language model, characterized in that, include: The text recognition unit is used to input a text set into the target language model and recognize the entity set, concept set, and relation set of all texts; A semantic network generation unit is used to map the identified domain entities, text concepts, and the relationships between domain entities to a data relationship description framework to generate an initial semantic network. The initial semantic network includes a set of nodes and a set of edges. The set of nodes includes nodes represented by each domain entity and text concept, and the set of edges includes edges represented by each relationship. The training data generation unit is used to extend the initial semantic network and update the target language model based on the extended target semantic network, thereby generating a training dataset for the target domain from the updated target language model.
10. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the language model-based training data generation method according to any one of claims 1 to 8.