A document-level relation extraction data synthesis method based on large model reverse generation

Abstract

The application discloses a document-level relation extraction data synthesis method based on a large model reverse generation, which has the characteristics of adopting a knowledge graph sampling method based on random walk to sample a topic coherent subgraph from an external knowledge graph, and dynamically adjusting the sampling probability of a triple, and specifically comprises the following steps: sampling a triple set from a knowledge graph, dynamically adjusting the sampling probability, data enhancement, generating corresponding document data based on the triple set, and a reflection correction mechanism, etc.Compared with the prior art, the application has the advantages that no model needs to be trained, only the reasoning ability of a large language model itself is relied on to generate synthetic data, the high accuracy and high recall rate of the document-level relation extraction synthetic data label are ensured, the data credibility is improved, a solid foundation is provided for subsequent model training, the human cost is reduced, the problem that it is difficult to obtain enough high-quality labeled data in specific fields such as education and medical treatment is solved, and the application has a good application prospect.

Description

Technical Field

[0001] This invention relates to the field of document-level relation extraction technology, specifically a method for synthesizing document-level relation extraction data based on reverse generation of a large model. Background Technology

[0002] Relation extraction is a key task in natural language processing, aiming to identify and extract relationships between entities from unstructured text, typically represented as structured triples. This task has wide applications in information retrieval, knowledge graph construction, and intelligent question-answering systems. Existing research mainly focuses on sentence-level relation extraction, i.e., identifying entities and their relationships from only a single sentence. However, in real-world scenarios, information usually appears in document form, and the entities and their relationships often span multiple sentences. Furthermore, the same entity may be referred to in different forms within a document. Therefore, document-level relation extraction is more complex than sentence-level extraction, requiring models with stronger reasoning capabilities and relying more heavily on high-quality, sufficient training data. However, annotating document-level data is difficult and costly, especially in professional fields such as education and healthcare, where obtaining sufficient high-quality labeled data for training is extremely challenging.

[0003] To address the issue of insufficient data, existing methods typically utilize large language models to generate synthetic data and their labels for training. However, these methods have limited performance in document-level relation extraction tasks, and the synthetic labeled data often contains significant noise, leading to mislabeling and negatively impacting model training. Furthermore, these methods struggle to guarantee the diversity of synthetic data, frequently resulting in numerous similar samples and data redundancy, thus failing to effectively improve model performance. Therefore, designing a synthetic method capable of generating high-quality, highly diverse document-level relation extraction data has become a pressing challenge. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by providing a document-level relation extraction and data synthesis method based on large-scale model inverse generation. This method employs a knowledge graph sampling approach based on random walks to sample topic-coherent subgraphs from an external knowledge graph and dynamically adjusts the sampling probability of triples to ensure balanced coverage of all relation types. Furthermore, subsampling and knowledge graph completion methods are introduced to enhance the sampled triple set, generating a high-quality, diverse synthetic dataset. In the generated document data, this method uses prompt templates to guide the large language model to distribute start and end entities across multiple sentences and makes extensive use of pronouns, making the synthesized data more closely resemble real-world document scenarios. Through reflective correction strategies, it ensures consistency between labels and documents, effectively solving the problems of poor data quality, insufficient diversity, and inaccurate labels in existing technologies. This significantly reduces data annotation costs and has promising application prospects.

[0005] The specific technical solution to achieve the purpose of this invention is: a method for synthesizing document-level relation extraction data based on large model reverse generation, characterized in that the method specifically includes the following steps:

[0006] Step 1: Sample a set of triples from the knowledge graph

[0007] Select a starting entity e0 from the knowledge graph (e.g., WikiData) as the sampling starting point, and use a backtracking-based random walk strategy to traverse adjacent entities and their relationships starting from this entity, gradually sampling a specified number of N triples, and saving them to the triple set T represented by the following equation (a):

[0008]

[0009] in, and Representing the head entity and tail entity respectively, r (i) This indicates a corresponding relationship.

[0010] After sampling the triplet at each step At that time, one of the following schemes will be randomly selected for the next sampling step:

[0011] ① Randomly backtrack to a head entity in the sampled triplet set T. Then, it is used as the new head entity for triple sampling to obtain a new triplet.

[0012] ② Directly using the tail entity of the current triple Triples are sampled as new head entities to obtain triples.

[0013]

[0014] This backtracking-based random walk strategy can effectively ensure the coverage and thematic coherence of the triple set.

[0015] Step 2: Dynamically adjust the sampling probability

[0016] After each sampling of triples, the relation type distribution of the sampled triples is statistically analyzed, and the sampling probability of each relation in the next round of sampling is dynamically adjusted. The sampling probability P of relation r is... r After every k steps of sampling, the triplet sampling probability P is dynamically updated based on the relation frequency f(r). r Calculated by the following formula (d):

[0017]

[0018] Where ε is the smoothing factor.

[0019] During the sampling process, for the current triplet Will be based on probability To be retained. The probability of discarding certain relationships is considered. This dynamic sampling strategy ensures that low-frequency relationships are sampled to guarantee a balanced distribution of relationships in the synthesized data and prevents long-tail problems.

[0020] At the start of each round of sampling of relation triplet sets, the current sampling probability P r It will also be used to determine the initial relation triplet to be used in this round of random walk. Specifically, entities associated with low-frequency relations with lower sampling probabilities will be selected as the sampling starting point for the current round of random walk, further ensuring that long-tail relations can be covered.

[0021] Step 3: Data Augmentation

[0022] Further data augmentation is performed on the sampled triplet set T to enhance the diversity of the synthesized data. The data augmentation mainly adopts the following two schemes:

[0023] ① Subsampling: From the already sampled triplet set T, a small portion of the triplets are randomly discarded to obtain the enhanced triplet set T represented by the following equation (b). aug :

[0024]

[0025] ② Knowledge graph completion: Use the knowledge graph completion model to complete the set of triples T that may be missing relation triples, resulting in the following equation (c) representing the enhanced set of triples T. aug :

[0026]

[0027] Step 4: Generate corresponding document data based on the set of triples.

[0028] The enhanced set of triples T aug Input a large language model, and synthesize document relation extraction data text based on the constructed prompt template. The constructed prompt template is as follows:

[0029] You will receive a set of relation triples {triplet_set}. Please write a fluent, Wikipedia-style text paragraph based on these triples, with the following requirements:

[0030] (1) Distribute the head and tail entities of the triples in multiple different sentences as much as possible.

[0031] (2) Use pronouns to replace some frequently occurring entities to ensure sentence fluency.

[0032] (3) Each sentence describes the relationship in one or more triples to maintain the document’s coherence and logic.

[0033] (4) Ensure that the paragraph description is semantically complete, containing all information of the relation triples, and that nothing is omitted.

[0034] (5) The number of sentences shall not be less than 6.

[0035] Example: {example_list}

[0036] Where {triplet_set} is the set of input triplets T aug The placeholder for {example_list} is an optional placeholder for context examples that can be further provided to large language models to guide the constraints on their output.

[0037] Step 5: Reflection and Correction Mechanism

[0038] To ensure the consistency and correctness of the generated document and the set of triples, a reflective correction mechanism is introduced. After the large language model generates the initial document, it performs a self-check by comparing it with the original set of triples to ensure that all relations are accurately reflected in the document. The specific reflective prompt template is as follows:

[0039] Please check that the following text correctly describes all the given triplet relations. If there are any omissions or errors, please correct them: Text: {generated_text} Triples: {triplet_set}

[0040] Here, {generated_text} is a placeholder for the input text, and {triplet_set} is a placeholder for the set of input triplets.

[0041] Compared with the prior art, the present invention has the following significant technical advancements and beneficial technical effects:

[0042] (1) This invention can effectively ensure the high accuracy and high recall of document-level relation extraction and synthetic data labels by sampling a set of topic-coherent relation triples from an external knowledge graph and generating corresponding text based on these triples using a large language model, thereby improving the credibility of the data and providing a solid foundation for subsequent model training.

[0043] (2) This invention does not require training a model, but relies solely on the reasoning ability of the large language model itself to generate synthetic data. This strategy significantly reduces the cost of model training while maintaining the high quality standard of the data.

[0044] (3) This invention can independently synthesize high-quality labeled data, reducing the reliance on professional human annotators. This not only further reduces labor costs, but also solves the problem of difficulty in obtaining sufficient high-quality labeled data in specific fields (such as education and medical care). Attached Figure Description

[0045] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0046] The present invention will be further described in detail below with reference to the specific embodiments and accompanying drawings. Except for the contents specifically mentioned below, the processes, conditions, and experimental methods for implementing the present invention are all common knowledge and general knowledge in the art, and the present invention does not have any particular limitations.

[0047] Example 1

[0048] See Figure 1 The present invention specifically includes the following steps:

[0049] Step 1: Sample a set of triples from the knowledge graph

[0050] The WikiData knowledge graph, containing approximately 2,700,000 different entities and nearly 800 different relations, was preloaded and preprocessed to serve as the raw graph for performing triple set sampling. Python was then used to construct it into a graph structure, resulting in graph G = (E, R, T), where E is the set of entities, E = {e1, e2, ..., e...}. n}, each e i R represents an entity; R is a set of relations, R = {r1, r2, ..., r} m}, each r j This represents a certain relationship between entities; T is a set of triples. Each triplet (e sub ,r,eobj ) represents entity e sub With entity e obj There is a relationship r between them.

[0051] In this embodiment, the starting entity e0 for sampling is selected as Albert Einstein, the size N of the sampled triplet set is selected as 4, and the triplet set T is used to store all sampled triplets. First, using the starting entity e0 as the head entity, a relation triplet is randomly sampled to obtain the triplet. Then, a backtracking-based random walk sampling algorithm is used to sample triples at each step, randomly employing one of the following two schemes to ensure the coverage and thematic coherence of the triple set:

[0052] ① Randomly backtrack to a head entity in the sampled triplet set T. Then, it is used as the new head entity for triple sampling to obtain a new triplet.

[0053] ② Directly using the tail entity of the current triple Triples are sampled as new head entities to obtain triples.

[0054]

[0055] In this embodiment, these two sampling schemes were randomly used in each subsequent sampling step, resulting in a set of triples T = {(Albert Einstein, educated at, University of Zurich),}

[0056] (Albert Einstein,received,Nobel Prize in Physics),(University of Zurich,located in,Zurich),

[0057] (Nobel Prize in Physics,awarded by,Royal Swedish Academy ofSciences)}.

[0058] Step 2: Dynamically adjust the sampling probability

[0059] After sampling triplets every k steps, the relation type distribution of the sampled triplets is statistically analyzed, and the sampling probability of each relation in the next round of sampling is dynamically adjusted. The sampling probability P of relation r is... r After every k steps of sampling, the triplet sampling probability P is dynamically updated based on the relation frequency f(r). rCalculated by the following formula (d):

[0060]

[0061] Where ε is the smoothing factor, P r The initial values ​​are set to 1, ε is set to 0.005, and k is set to 4.

[0062] In this embodiment, based on the sampling result set T from the previous round, the frequencies of the sampled relations are updated as follows: f(r = "educatedat") = 1, f(r = "received") = 1, f(r = "located in") = 1, f(r = "awarded by") = 1. The probabilities of the corresponding relations are also updated, resulting in...

[0063]

[0064] Dynamically adjusting the sampling probability after every k steps of sampling ensures that low-frequency relationships are sampled as much as possible during subsequent random walk sampling. This effectively ensures that the relationship categories contained in the final document-level relationship extraction and synthesis data are balanced, avoiding long-tail problems that could affect the model training effect.

[0065] In this embodiment, the triplet set T for this round has been sampled. After completing all subsequent steps to obtain the synthesized data, it will re-enter the next round of sampling to generate a new data text. Therefore, the sampling probability dynamically adjusted in step 2 will also affect the selection of the initial sampling starting point in the next round of sampling. Specifically, entities associated with low-frequency relationships with lower sampling probabilities will be preferentially selected as the sampling starting point for the next round of random walks, further ensuring that long-tailed relationships can be covered.

[0066] Step 3: Data Augmentation

[0067] Data augmentation is performed on the set of triples T obtained from step 1 sampling to obtain the augmented set of triples T. aug This enhances the diversity of the subsequently generated document-level relationship extraction data. The main approaches are as follows:

[0068] ① Subsampling: From the already sampled set of triplets T, a small subset of triplets is randomly discarded to obtain...

[0069] In this embodiment, if a subsampling strategy is adopted, and a triplet is randomly discarded, the following enhanced triplet T can be obtained."# :

[0070] T "# ={(Albert Einstein,educated at,University of Zurich),

[0071] (Albert Einstein,received,Nobel Prize in Physics),

[0072] (University of Zurich, located in, Zurich)}.

[0073] ② Knowledge Graph Completion: Using the knowledge graph completion model Llama 3.1-8B, missing relation triples in the triple set T are completed, resulting in... The corresponding knowledge graph completion prompt template is as follows:

[0074] Given a set of triplets: {triplet_set}, predict potentially missing triplets to complete a knowledge graph. Each triplet consists of a "head entity", a "relation", and a "tail entity". Ensure that the completed triplets are logically sound and consistent with existing triplets. Return the complete set of triplets. Requirements: Complete potentially missing triplets and ensure that all triplets have reasonable logical relationships. The returned complete set of triplets is T. "# It should include the triples in the initial set of triples and the completed set of triples.

[0075] For example, if there is an unlabeled relationship between two entities, express this relationship by completing the annotation.

[0076] Output: Please return the complete set of triples T after completion. "# Here, {triplet_set} represents the placeholder for the input triplet to be completed.

[0077] In this embodiment, if a knowledge graph completion strategy is adopted, the following enhanced triplet set T can be obtained. "# :

[0078] T "# ={(Albert Einstein,educated at,University of Zurich),

[0079] (Albert Einstein,received,Nobel Prize in Physics),

[0080] (University of Zurich,located in,Zurich),

[0081] (Nobel Prize in Physics,awarded by,Royal Swedish Academy ofSciences),

[0082] (Albert Einstein,wrote,Theory of Relativity),

[0083] (Royal Swedish Academy of Sciences,founded in,1739),

[0084] (Theory of Relativity,related to,Physics)}.

[0085] Step 4: Generate corresponding document data based on the set of triples.

[0086] The enhanced set of triples T aug Input a large language model, and synthesize document relation extraction data text based on the constructed prompt template. The constructed prompt template is as follows:

[0087] You will receive a set of relation triplets {triplet_set}. Please write a fluent, Wikipedia-style text paragraph based on these triplets, with the following requirements:

[0088] (1) Distribute the head and tail entities of the triples in multiple different sentences as much as possible.

[0089] (2) Use pronouns to replace some frequently occurring entities to ensure sentence fluency.

[0090] (3) Each sentence describes the relationship in one or more triples to maintain the document’s coherence and logic.

[0091] (4) Ensure that the paragraph description is semantically complete, containing all information of the relation triples, and that nothing is omitted.

[0092] (5) The number of sentences shall not be less than 6.

[0093] Example: {example_list}

[0094] Where {triplet_set} is the set of input triplets T augThe placeholder for {example_list} is an optional placeholder for context examples that can be further provided to large language models to guide constraints on their output.

[0095] In this embodiment, the set of triples T obtained in step 3 is... aug After inputting into the large language model and receiving prompts, the final output text document is as follows:

[0096] Albert Einstein, the renowned physicist, pursued his education at the University of Zurich, which is located in Zurich. He later received the NobelPrize in Physics, awarded by the Royal Swedish Academy of Sciences, for his groundbreaking work on the Theory of Relativity.

[0097] Step 5: Reflection and Correction Mechanism

[0098] To ensure that the text document generated in step 4 and the set of triples T obtained in step 3 are consistent, aug To ensure consistency and correctness, a reflective correction mechanism is introduced. After the large language model generates the initial document, it performs a self-check by comparing it with the original set of triples to ensure that all relations are accurately reflected in the document. The specific reflective prompt template is as follows:

[0099] Please check that the following text correctly describes all the given triplet relations. If there are any omissions or errors, please correct them: Text: {generated_text} Triples: {triplet_set}. Where {generated_text} is a placeholder for the input text, and {triplet_set} is a placeholder for the set of input triples.

[0100] In this embodiment, Document and T aug The input is fed into a large language model, and prompts are used to allow the model to reflect and correct itself, resulting in the corrected document.fixed Finally, these were added as input text and labels to the document-level relation extraction dataset D.

[0101] The above is merely one embodiment of the present invention and is not intended to limit the present invention. Any modifications and improvements made without departing from the concept of the present invention shall fall within the scope of protection of the claims of the present invention.

Claims

1. A method for synthesizing document-level relation extraction data based on large-scale model reverse generation, characterized in that, The method specifically includes: Step 1: Sample a set of triples from the knowledge graph 1-1: Select a starting entity e0 from the knowledge graph as the sampling starting point, and use a backtracking-based random walk strategy to traverse adjacent entities and their relationships starting from this entity, and gradually sample a specified number of triples N, to obtain the triple set T represented by the following equation (a): in, These are the head entity and the tail entity, respectively; r (i) This is a corresponding relationship; 1-2: For each triplet completed: During sampling, the following method is used randomly for the next sampling step to obtain new triples: 1) Randomly backtrack to a head entity in the sampled triplet set T. Then, it is used as the new head entity for triple sampling, resulting in a new triple: 2) Directly using the tail entity of the current triple Sample the new head entity into a triplet to obtain another new triplet: Step 2: Dynamically adjust the sampling probability 2-1: After each sampling of triples, the relation type distribution of the sampled triples is statistically analyzed, and the sampling probability P of each relation r in the next round of sampling is dynamically adjusted. r After k steps of sampling, the data is dynamically updated based on the relational frequency f(r); 2-2: During the sampling process, for the current triplet: With probability P r(i) To be retained, 1-P r(i) The probability of [the probability] is discarded; 2-3: At the start of each round of relation triplet set sampling, the triplet sampling probability P r When selecting the sampling starting point for the next round of random walks, priority should be given to entities associated with low-frequency relationships. Step 3: Data Augmentation Data augmentation of the triple set T is performed using subsampling or knowledge graph completion methods to obtain the augmented triple set T. aug The subsampling method involves randomly discarding 20% ​​of the triplets in the set T, resulting in the enhanced set T represented by the following equation (b). aug : The knowledge graph completion uses a knowledge graph completion model to supplement potentially missing relation triples in the triple set T, resulting in the enhanced triple set T represented by the following equation (c). aug : Step 4: Generate corresponding document data based on the set of triples. The enhanced set of triples T aug Inputting a large language model, the system synthesizes document relation extraction data text based on the constructed prompt template to generate a corresponding preliminary document. The constructed prompt template is based on the received triple set T. aug Given the placeholder {triplet_set}, write a text paragraph with the following requirements: 1) Distribute the head and tail entities of the triples in multiple different sentences as much as possible; 2) Use pronouns to replace some frequently occurring entities; 3) Each sentence describes a relation in one or more triples; 4) The text paragraph must contain information from all relation triples, and the number of sentences must be no less than 6; Step 5: Reflection and Correction Mechanism The initial document generated by the large language model is compared with the set of triples T to check whether the text correctly describes all the given triple relations. If there are omissions or errors, the text is corrected to obtain the corrected document. fixed The text and labels are added as input to the document-level relation extraction dataset D.

2. The document-level relation extraction and data synthesis method based on large model reverse generation according to claim 1, characterized in that, The random walk sampling in step 1 specifically includes the following steps: 1) Randomly select the head entity of the sampled triples as the new head entity to generate new triples; 2) Use the tail entity of the current triple as the new head entity to continue generating new triples.

3. The document-level relation extraction and data synthesis method based on large model reverse generation according to claim 1, characterized in that, The triplet sampling probability P in step 2 r Calculated by the following formula (d): Where ε is the smoothing factor and f(r) is the relation frequency.

4. The document-level relation extraction and data synthesis method based on large model reverse generation according to claim 1, characterized in that, The knowledge graph completion model in step 3 uses the large language model Llama 3.1-8B.

5. The document-level relation extraction and data synthesis method based on large model reverse generation according to claim 1, characterized in that, The prompt template in step 4 specifically includes the following features of the document-level relationship extraction data: 1) The text should contain more than 50% of all relational triples across sentences; 2) The text contains an average of 3 to 5 core-referenced entities; 3) The text must contain at least 6 sentences; 4) Each sentence in the text may contain information from more than one relation triple.

Images