Document-level relation extraction data synthesis method based on large model reverse generation

Abstract

The invention discloses a document-level relation extraction data synthesis method based on large model reverse generation, which is characterized in that a knowledge graph sampling method based on random walk is adopted, sub-graphs with coherent themes are sampled from an external knowledge graph, and the sampling probability of triples is dynamically adjusted, so that the knowledge graph sampling efficiency is improved. The method specifically comprises the steps of sampling a triple set from a knowledge graph, dynamically adjusting the sampling probability, performing data enhancement, generating corresponding document data based on the triple set, performing a reflection correction mechanism and the like. Compared with the prior art, the method has the advantages that model training is not needed, the synthetic data is generated only depending on the reasoning ability of a large language model, high accuracy and high recall rate of document-level relation extraction synthetic data labels are ensured, the credibility of the data is improved, a solid foundation is provided for subsequent model training, and the method is worthy of popularization and application. The method not only reduces the labor cost, but also solves the problem that it is difficult to obtain enough high-quality annotation data in the specific fields of education, medical treatment and the like, and has a good application prospect.

Description

Technical Field

[0001] The present invention relates to the technical field of document-level relationship extraction, and in particular to a document-level relationship extraction data synthesis method based on large model reverse generation. Background Art

[0002] Relation extraction is a key task in the field of natural language processing. It aims to identify and extract the relationship between entities from unstructured text, usually in the form of structured triples. This task has wide application value in information retrieval, knowledge graph construction, intelligent question-answering systems and other fields. Existing research mainly focuses on sentence-level relationship extraction, that is, identifying entities and their relationships from a single sentence. However, in actual scenarios, information usually appears in the form of documents, and the entities and their relationships involved often span multiple sentences. At the same time, the same entity may be referred to in different forms in a document. Therefore, document-level relationship extraction is more complex than sentence-level, requiring the model to have stronger reasoning ability and a stronger dependence on high-quality and sufficient training data. However, document-level data is difficult to annotate and has high annotation costs, especially in professional fields such as education and medical care. It is extremely difficult to obtain enough high-quality annotated data for training.

[0003] In order to solve the problem of insufficient data, existing methods usually use large language models to generate synthetic data and its labels for training. However, due to the limited performance of these methods in document-level relationship extraction tasks, the synthetic labeled data often contains a lot of noise, resulting in the generation of wrong labels, which in turn affects the training effect of the model. In addition, such methods also find it difficult to ensure the diversity of synthetic data, and often a large number of similar samples appear, resulting in data redundancy and unable to effectively improve model performance. Therefore, how to design a synthesis method that can generate high-quality and high-diversity document-level relationship extraction data has become a problem that needs to be solved urgently. Summary of the invention

[0004] The purpose of the present invention is to provide a data synthesis method for document-level relationship extraction method based on reverse generation of a large model in response to the deficiencies of the prior art. The method adopts a knowledge graph sampling method based on random walks to sample subgraphs with coherent themes from an external knowledge graph, and dynamically adjusts the sampling probability of triples to ensure that all relationship types are evenly covered, and further introduces subsampling and knowledge graph completion methods to enhance the sampled triple sets and generate high-quality and diverse synthetic data sets. In the generated document data, this method uses prompt templates to guide the large language model to distribute the head and tail entities in multiple sentences, and uses a large number of pronouns to make the synthetic data closer to the real document scene. Through the reflection and correction strategy, the consistency of the label and the document is ensured, which effectively solves the problems of poor quality, insufficient diversity and inaccurate labels of the synthetic data in the prior art, greatly reduces the cost of data annotation, and has good application prospects.

[0005] The specific technical solution to achieve the purpose of the present invention is: a document-level relationship extraction data synthesis method based on reverse generation of a large model, which is characterized in that the implementation of the method specifically includes the following steps:

[0006] Step 1: Sample triple sets from the knowledge graph

[0007] Select a starting entity e0 from the knowledge graph (such as WikiData) as the sampling starting point, use a backtracking-based random walk strategy to traverse adjacent entities and their relationships from this entity, and gradually sample a specified number of N triples, and save them to the triple set T represented by the following formula (a):

[0008]

[0009] in, and Represents the head entity and the tail entity respectively, r (i) Indicates the corresponding relationship.

[0010] After sampling the triplet at each step When , randomly use one of the following schemes for the next step of sampling:

[0011] ① Randomly trace back to a head entity in the sampled triple set T And use it as the new head entity for triple sampling to get a new triple

[0012] ② Directly use the tail entity of the current triple As the new head entity, triple sampling is performed to obtain the triple

[0013]

[0014] This backtracking-based random walk strategy can effectively ensure the coverage and topic coherence of the triple set.

[0015] Step 2: Dynamically adjust sampling probability

[0016] After sampling each triple, the relationship type distribution of the sampled triple is counted, and the sampling probability of each relationship in the next round of sampling is dynamically adjusted. The sampling probability P of relationship r r After every k steps of sampling, the triple sampling probability P is dynamically updated according to the relationship frequency f(r). r Calculated by the following formula (d):

[0017]

[0018] Here, ε is the smoothing factor.

[0019] During the sampling process, for the current triple With probability To reserve, This dynamic sampling strategy ensures that low-frequency relations can be sampled to ensure that the relationship distribution of synthetic data is balanced and there will be no long-tail problem.

[0020] At the beginning of each round of relation triple set sampling, the current sampling probability P r It will also be used to determine the initial relationship triples used in this round of random walk. Specifically, entities associated with low-frequency relationships with lower sampling probabilities will be prioritized as the sampling starting points for the current round of random walk, further ensuring that long-tail relationships can be covered.

[0021] Step 3: Data Augmentation

[0022] Further data enhancement is performed on the sampled triple set T to improve the diversity of synthetic data. Data enhancement mainly adopts the following two schemes:

[0023] ① Subsampling: Randomly discard a small number of triplets from the sampled triple set T to obtain the enhanced triple set T represented by the following formula (b): aug :

[0024]

[0025] ② Knowledge graph completion: Use the knowledge graph completion model to supplement the relationship triples that may be missing in the triple set T, and obtain the following formula (c) to represent the enhanced triple set T aug :

[0026]

[0027] Step 4: Generate corresponding document data based on the triple set

[0028] The enhanced triple set T aug Input the large language model and synthesize the document relationship extraction data text according to the constructed prompt template. The constructed prompt template is as follows:

[0029] You will receive a set of relation triplets {triplet_set}. Please write a Wikipedia-style, fluent text paragraph based on these triplets, requiring:

[0030] (1) Distribute the head entity and tail entity of the triple into multiple different sentences as much as possible.

[0031] (2) Use pronouns to replace some frequently occurring entities to ensure smooth sentence flow.

[0032] (3) Each sentence describes the relationship between one or more triples, maintaining document coherence and logic.

[0033] (4) Ensure that the paragraph description is semantically complete and contains information on all relationship triples without any omissions.

[0034] (5) The number of sentences should not be less than 6.

[0035] Example: {example_list}

[0036] Where {triplet_set} is the input triple set T aug ; {example_list} is a placeholder for optional context examples, which can be further provided to the large language model to guide and constrain its resulting output.

[0037] Step 5: Reflect on the correction mechanism

[0038] In order to ensure the consistency and correctness of the generated documents and triple sets, a reflection correction mechanism is introduced. After the large language model generates the preliminary document, it performs a self-check by comparing the original triple set to ensure that all relationships are accurately reflected in the document. The specific reflection prompt template is as follows:

[0039] Please check that the following text correctly describes all given triplet relations. If there are omissions or errors, please correct them: Text: {generated_text} Triplet: {triplet_set}

[0040] Among them, {generated_text} is the placeholder for the input text, and {triplet_set} is the placeholder for the input triplet set.

[0041] Compared with the prior art, the present invention has the following significant technical advances and beneficial technical effects:

[0042] (1) The present invention can effectively ensure the high accuracy and high recall rate of document-level relation extraction synthetic data labels by sampling a set of topic-coherent relation triples from an external knowledge graph and using a large language model to generate corresponding text based on these triple sets, thereby improving the credibility of the data and providing a solid foundation for subsequent model training.

[0043] (2) The present invention does not require model training and only relies on the reasoning ability of the large language model itself to generate synthetic data. This strategy significantly reduces the cost of model training while also maintaining high data quality standards.

[0044] (3) The present invention can independently synthesize high-quality annotated data, reducing the reliance on professional human annotators. This not only further reduces labor costs, but also solves the problem of difficulty in obtaining sufficiently high-quality annotated data in specific fields (such as education and medical care). BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0046] The present invention is further described in detail with reference to the following specific examples and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are all common knowledge and common common sense in the art, and the present invention has no particular limitations.

[0047] Example 1

[0048] See also Figure 1 , the present invention specifically comprises the following steps:

[0049] Step 1: Sample triple sets from the knowledge graph

[0050] The WikiData knowledge graph is preloaded and preprocessed, which contains about 2.7 million different entities and nearly 800 different relationships. It is used as the original graph for performing triple set sampling. It is constructed into a graph structure using Python, and the graph G = (E, R, T) is obtained. Where E is the set of entities, E = {e1, e2, ..., e n}, each e i represents an entity; R is a set of relations, R = {r1, r2, ..., r m}, each r j Represents a certain relationship between entities; T is a set of triples, Each triple (e sub ,r,eobj ) represents entity e sub With entity e obj There is a relationship r between them.

[0051] In this embodiment, the starting entity e0 of the sampling is selected as Albert Einstein, the size of the sampled triple set N is selected as 4, and the triple set T is used to store all the sampled triples. First, take the starting entity e0 as the head entity, randomly sample a relation triple, and obtain the triple After that, a backtracking-based random walk sampling algorithm is used to sample triples at each step, randomly adopting one of the following two schemes to ensure the coverage and topic coherence of the triple set:

[0052] ① Randomly trace back to a head entity in the sampled triple set T And use it as the new head entity for triple sampling to get a new triple

[0053] ② Directly use the tail entity of the current triple As the new head entity, triple sampling is performed to obtain the triple

[0054]

[0055] In this embodiment, the two sampling schemes are randomly used in each subsequent sampling step to obtain a triple set T = {(Albert Einstein, educated at, University of Zurich),

[0056] (Albert Einstein,received,Nobel Prize in Physics),(University of Zurich,located in,Zurich),

[0057] (Nobel Prize in Physics,awarded by,Royal Swedish Academy ofSciences)}.

[0058] Step 2: Dynamically adjust sampling probability

[0059] After sampling triples every k steps, the relationship type distribution of the sampled triples is counted, and the sampling probability of each relationship in the next round of sampling is dynamically adjusted. The sampling probability P of relationship r r After every k steps of sampling, the triple sampling probability P is dynamically updated according to the relationship frequency f(r). rCalculated by the following formula (d):

[0060]

[0061] Where ε is the smoothing factor, P r The initial value is set to 1, the initial value of ε is set to 0.005, and k is set to 4.

[0062] In this embodiment, according to the sampling result set T of the previous round, the frequencies of the sampled relations are updated to f(r="educatedat")=1, f(r="received")=1, f(r="located in")=1, f(r="awarded by")=1. And the corresponding relation probabilities are also updated, and we get

[0063]

[0064] Dynamically adjusting the sampling probability after every k steps of sampling can ensure that low-frequency relations can be sampled as much as possible in the subsequent random walk sampling process, effectively ensuring that the relationship categories contained in the final document-level relationship extraction synthetic data are balanced, avoiding the long-tail problem and affecting the effect of model training.

[0065] In this embodiment, the triple set T of this round has been sampled, and after all subsequent steps to obtain synthetic data, the next round of sampling process will be re-entered to generate a new data text. For this reason, the sampling probability dynamically adjusted in step 2 will also affect the selection of the initial sampling starting point in the next round of sampling. Specifically, entities associated with low-frequency relationships with low sampling probabilities will be preferentially selected as the sampling starting points for the next round of random walks, further ensuring that long-tail relationships can be covered.

[0066] Step 3: Data Augmentation

[0067] Perform data enhancement on the triplet set T sampled in step 1 to obtain the enhanced triplet set T aug , in order to enhance the diversity of document-level relation extraction data generated subsequently. The following two solutions are mainly adopted:

[0068] ① Subsampling: Randomly discard a small number of triplets from the sampled triple set T to obtain

[0069] In this embodiment, if a subsampling strategy is adopted and a triplet (Nobel Prize in Physics, awarded by, Royal Swedish Academy of Sciences) is randomly discarded, the following enhanced triplet T can be obtained."# :

[0070] T "# ={(Albert Einstein,educated at,University of Zurich),

[0071] (Albert Einstein,received,Nobel Prize in Physics),

[0072] (University of Zurich, located in, Zurich)}.

[0073] ② Knowledge graph completion: Use the knowledge graph completion model Llama 3.1-8B to supplement the relationship triples that may be missing in the triple set T, and get The corresponding knowledge graph completion prompt template is as follows:

[0074] Given a triplet set: {triplet_set}, predict the possible missing triples to complete the knowledge graph. Each triplet consists of "head entity", "relationship", and "tail entity". Please ensure that the completed triples are logically reasonable and consistent with the existing triples. Return the completed triplet set. Requirements: Complete the possible missing triples and ensure that all triples have reasonable logical relationships. The returned completed triplet set T "# It should contain the triples in the initial set of triples and the completed triples.

[0075] For example, if there is an unlabeled relationship between two entities, express this relationship through completion.

[0076] Output: Please return the completed set of triples T "# . Among them, {triplet_set} represents the placeholder of the input triple to be completed.

[0077] In this embodiment, if the knowledge graph completion strategy is adopted, the following enhanced triple set T can be obtained: "# :

[0078] T "# ={(Albert Einstein,educated at,University of Zurich),

[0079] (Albert Einstein,received,Nobel Prize in Physics),

[0080] (University of Zurich,located in,Zurich),

[0081] (Nobel Prize in Physics,awarded by,Royal Swedish Academy ofSciences),

[0082] (Albert Einstein,wrote,Theory of Relativity),

[0083] (Royal Swedish Academy of Sciences,founded in,1739),

[0084] (Theory of Relativity,related to,Physics)}.

[0085] Step 4: Generate corresponding document data based on the triple set

[0086] The enhanced triple set T aug Input the large language model and synthesize the document relationship extraction data text according to the constructed prompt template. The constructed prompt template is as follows:

[0087] You will receive a set of relation triplets {triplet_set}. Please write a fluent text paragraph in Wikipedia style based on these triplets. The requirements are as follows:

[0088] (1) Distribute the head entity and tail entity of the triple into multiple different sentences as much as possible.

[0089] (2) Use pronouns to replace some frequently occurring entities to ensure smooth sentence flow.

[0090] (3) Each sentence describes the relationship between one or more triples, maintaining document coherence and logic.

[0091] (4) Ensure that the paragraph description is semantically complete and contains information on all relationship triples without any omissions.

[0092] (5) The number of sentences should not be less than 6.

[0093] Example: {example_list}

[0094] where {triplet_set} is the input triplet set T aug{example_list} is a placeholder for optional context examples that can be further provided to the large language model to guide and constrain its resulting output.

[0095] In this embodiment, the triple set T obtained in step 3 is aug After inputting into the large language model, the final output text document can be obtained as shown below:

[0096] Albert Einstein, the renowned physicist, pursued his education at the University of Zurich, which is located in Zurich. He later received the NobelPrize in Physics, awarded by the Royal Swedish Academy of Sciences, for his groundbreaking work on the Theory of Relativity.

[0097] Step 5: Reflect on the correction mechanism

[0098] To ensure that the text Document generated in step 4 and the triple set T obtained in step 3 aug To ensure consistency and correctness, a reflection correction mechanism is introduced. After the large language model generates a preliminary document, it performs a self-check by comparing the original triple set to ensure that all relationships are accurately reflected in the document. The specific reflection prompt template is as follows:

[0099] Please check whether the following text correctly describes all the given triple relations. If there are omissions or errors, please correct them: Text: {generated_text} Triplet: {triplet_set}. Among them, {generated_text} is a placeholder for the input text, and {triplet_set} is a placeholder for the input triple set.

[0100] In this embodiment, Document and T aug Input it into the big language model, use prompts to let the big model reflect and correct it, and get the corrected result Documentfixed Finally, they are added as input text and labels to the document-level relation extraction dataset D.

[0101] The above is only an implementation mode of the present invention and is not intended to limit the present invention. Any modifications and improvements made without departing from the concept of the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A document-level relationship extraction data synthesis method based on reverse generation of a large model, characterized in that: The method specifically includes: Step 1: Sample triple sets from the knowledge graph 1-1: Select a starting entity e0 from the knowledge graph as the sampling starting point, use a backtracking-based random walk strategy to traverse adjacent entities and their relationships from this entity, and gradually sample a specified number of N triplets to obtain a triple set T represented by the following formula (a): in, are the head entity and the tail entity respectively; r (i) For the corresponding relationship; 1-2: For each completed triple: When sampling, the following method is randomly used to perform the next step of sampling to obtain a new triplet: 1) Randomly trace back to a head entity in the sampled triple set T And use it as the new head entity for triple sampling to get a new triple: 2) Directly use the tail entity of the current triple Sampling the triplet as the new head entity, we get another new triplet: Step 2: Dynamically adjust sampling probability 2-1: After sampling each triple, count the relationship type distribution of the sampled triples, and dynamically adjust the sampling probability P of each relationship r in the next round of sampling r , after k steps of sampling, it is dynamically updated according to the relation frequency f(r); 2-2: During the sampling process, for the current triplet: With probability P r(i) Keep, 1-P r(i) The probability of is discarded; 2-3: At the beginning of each round of relation triple set sampling, the triple sampling probability P r When selecting the sampling starting point for the next round of random walk, entities associated with low-frequency relationships are given priority as the sampling starting point; Step 3: Data Augmentation Use subsampling method or knowledge graph completion method to perform data enhancement on triple set T to obtain enhanced triple set T aug The subsampling method randomly discards 20% of the triplets in the triple set T, and obtains the enhanced triple set T represented by the following formula (b): aug : The knowledge graph completion uses the knowledge graph completion model to supplement the relationship triples that may be missing in the triple set T, and the following formula (c) is obtained to represent the enhanced triple set T aug : Step 4: Generate corresponding document data based on the triple set The enhanced triple set T aug Input the large language model, synthesize the document relationship extraction data text according to the constructed prompt template, and generate the corresponding preliminary document. The constructed prompt template is based on the received triple set T aug Placeholder {triplet_set}, write the following required text paragraph: 1) Distribute the head entity and tail entity of the triple into multiple different sentences as much as possible; 2) Use pronouns to replace some frequently occurring entities; 3) Each sentence describes a relationship among one or more triples; 4) The text paragraph must contain information about all relation triples, and the number of sentences must be no less than 6; Step 5: Reflect on the correction mechanism Compare the preliminary document generated by the large language model with the triple set T to check whether the text correctly describes all given triple relationships. If there are omissions or errors, correct the text and obtain the corrected result Document fixed , added as input text and labels to the document-level relation extraction dataset D.

2. The document-level relationship extraction data synthesis method based on large model reverse generation according to claim 1 is characterized in that: The random walk sampling in step 1 specifically includes the following steps: 1) Randomly select the head entity of the sampled triple as the new head entity to generate a new triple; 2) Take the tail entity of the current triple as the new head entity and continue to generate new triples.

3. The document-level relationship extraction data synthesis method based on large model reverse generation according to claim 1 is characterized in that: The triplet sampling probability P of step 2 r Calculated by the following formula (d): Among them, ε is the smoothing factor; f(r) is the relationship frequency.

4. The document-level relationship extraction data synthesis method based on large model reverse generation according to claim 1 is characterized in that: The knowledge graph completion model in step 3 adopts the large language model Llama 3.1-8B.

5. The document-level relationship extraction data synthesis method based on large model reverse generation according to claim 1 is characterized in that: The prompt template of step 4 specifically includes the following features of document-level relationship extraction data: 1) The cross-sentence triples contained in the text should account for more than 50% of all relation triples; 2) The text contains an average of 3 to 5 coreference entities; 3) The text length must be at least 6 sentences; 4) Each sentence in the text may contain information about more than one relation triple.

Images