Large model-based relation triple extraction method
By combining a small pre-trained language model with a large model pipeline structure, the CNRel framework solves the problems of insufficient recall and precision of large models in relation triple extraction. Through entity pair extraction and noise filtering modules, it significantly improves the extraction effect and stability.
Patent Information
- Application Number
- CN202510596941.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-09
- Publication Date
- 2025-09-23
AI Technical Summary
The existing technology of relation triple extraction based on large models has the problems of low recall rate and insufficient precision. Especially when processing complex and long texts, it is easy to lose information and is easily affected by hallucination phenomena, resulting in erroneous extraction and repeated extraction.
CNRel, a relation triple extraction framework with a pipeline structure, combines a small pre-trained language model and a large model. Through entity pair extraction, relation classification and noise filtering modules, it uses fine-tuning datasets and the EMA attention mechanism to generate rich candidate entity pairs and eliminate incorrect triples.
It significantly improves the recall and precision of large models, alleviates the hallucination problem, improves the accuracy and stability of triple extraction, and has good portability and versatility.
Smart Images
Figure CN120688607A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of natural language processing knowledge graphs, and specifically relates to a relation triple extraction method based on a large model, which aims to realize the extraction and filtering of relation triples in natural language texts. Background Art
[0002] As a key sub-technology for information extraction in the knowledge graph field, relation triple extraction (RTE) aims to extract triples of the form (Subject, Relation, Object) from text. Current research on relation triple extraction primarily relies on small deep learning neural network models such as BERT. Trained on a designed and constructed training dataset, these models can effectively extract a sufficient number of entity pairs and triples from text. Based on the structure of the model, two approaches can be categorized: pipeline extraction and joint extraction. Pipeline extraction divides the relation triple extraction task into two independent subtasks: entity extraction and relation classification. Entity extraction aims to extract named entities from text and construct candidate entity pairs, while relation classification aims to predict the relationships between entity pairs. Due to inherent limitations, pipeline-based relation triple extraction methods struggle to effectively handle single entity overlap (SEO) and entity pair overlap (EPO) during training and inference. Joint extraction approaches treat relation triple extraction as an end-to-end task, simultaneously extracting entities and relations by training a single deep learning model. However, this method faces great challenges in the joint modeling of entity extraction and relationship extraction, and it is difficult to accurately detect and divide the boundaries of named entity words in text sequences, which leads to errors in entity word extraction and the omission of a lot of valuable information. The model has good recall performance but poor precision.
[0003] Recently, large models have demonstrated powerful text understanding and generation capabilities, and have performed well in various natural language processing tasks. Previous work has shown that large models can effectively extract triples from text with the help of zero-shot learning, few-shot learning, or fine-tuning methods. However, the current large-model-based relation triple extraction still faces two challenges. (1) Like most natural language processing tasks, it is unnatural and inconsistent with language logic to regard the relation triple extraction task as a text-to-text task, because there is a data offset between the pre-training data and the inference data of the large model. Relying solely on simple instruction, context learning (ICL) methods or fine-tuning is usually not enough to help large models effectively extract triples, especially in long texts with complex structures. This method may lead to the loss of valuable information, thereby limiting the recall rate of the large-model-based relation triple extraction model. (2) Due to its limitations that are inconsistent with human knowledge and its robustness issues in different question-answering scenarios, large models are susceptible to hallucinations, including factual hallucinations and faithfulness hallucinations. As a result, large models are prone to errors or duplicate triplets in triple extraction tasks, and lack the ability to distinguish and filter out erroneous extraction results. This greatly limits the accuracy of large models for relation triple extraction.
[0004] Therefore, there is an urgent need for an efficient and robust large-scale model-based relation triple extraction method. This paper addresses this need by proposing a pipelined relation triple extraction framework, CNRel, which aims to improve the recall and precision of large-scale model-based relation triple extraction models. Summary of the Invention
[0005] To solve the above problems, the present invention aims to propose a relation triple extraction method based on a large model, the steps of which include:
[0006] Get the original text information and segment it into sentences;
[0007] Extract entity pairs from the segmented text information using an entity pair extraction model based on a pre-trained language model;
[0008] Predefine the relationship type for entity pair classification and generate a relationship triple extraction task description. Generate a question template for the large model based on the relationship type, task description, and original text information.
[0009] Construct a fine-tuning dataset based on the question template, and use the fine-tuned large model to extract relation triples;
[0010] The large model is used to determine the correctness of the extracted relationship triples to eliminate erroneous relationship triples.
[0011] Furthermore, the entity pairs are screened by a pre-trained entity pair filtering model based on the EMA attention mechanism.
[0012] Furthermore, the rotation position embedding is added as auxiliary embedding information and the attention score in EMA is calculated based on it, and entity pairs that do not meet the set score are eliminated.
[0013] Furthermore, the classification task of the relation triples is defined as a binary classification task, and a prompt template for the large model is generated. The prompt template consists of a list of the relation triples and a task instruction requiring the large model to make a correctness judgment on the input relation triples. A fine-tuning dataset is constructed based on the prompt template, and the fine-tuned large model is used to judge the correctness of the extracted relation triples.
[0014] The present invention also proposes a relation triple extraction system based on a large model, comprising:
[0015] The text data acquisition module obtains the original text information and segments it into sentences;
[0016] The entity pair extraction module extracts entity pairs from the segmented text information through an entity pair extraction model based on a pre-trained language model;
[0017] The relation and triple extraction module generates a question template for the large model based on the relation type, the relation triple extraction task description, and the original text information. It then constructs a fine-tuning dataset based on the question template and uses the fine-tuned large model to extract relation triples.
[0018] The noise filtering module judges the correctness of the extracted relation triples through the large model shown and eliminates the erroneous relation triples.
[0019] Furthermore, it also includes an entity pair filtering module, which filters the entity pairs through a pre-trained entity pair filtering model based on the EMA attention mechanism.
[0020] Furthermore, the entity pair filtering module adds the rotation position embedding as auxiliary embedding information and calculates the attention score in EMA accordingly, and eliminates entity pairs that do not meet the set score.
[0021] Furthermore, the noise filtering module defines the classification task of the relationship triples as a binary classification task, and generates a prompt template for the large model. The prompt template consists of two parts: a list of the relationship triples and a task instruction requiring the large model to make a correctness judgment on the input relationship triples. A fine-tuning dataset is constructed based on the prompt template, and the fine-tuned large model is used to judge the correctness of the extracted relationship triples.
[0022] The present invention further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing the above method.
[0023] The present invention also provides a storage medium storing a computer program, wherein the computer program implements the above method when executed by a computer.
[0024] The beneficial effects of the present invention are as follows:
[0025] This paper proposes a large-model-based relational triple extraction framework, CNRel (Candidate Prompt Enhancement and Noise Filtering Relational Triple Extraction Framework Based on Large Language Models). This framework combines small pre-trained language models with large models for relational triple extraction and filtering. First, a candidate entity pair extraction and filtering module based on the small pre-trained language model is used to extract and refine all possible entity pairs in the text, ensuring that as much valuable information as possible is captured. Then, a fine-tuned open-source large model (such as LLaMA, Qwen, etc.) is used to predict the relationships between candidate entity pairs and extract as many triples as possible from the range outside the candidate entity pair set. Finally, another fine-tuned open-source large model is used to determine the correctness of the extracted triples and remove erroneous triples. This paper provides a unified framework combining large and small models for triple extraction, significantly improving the triple extraction performance of large models and providing a solution for automatically denoising relational triple extraction results.
[0026] The present invention combines a small pre-trained language model with a large model, and extracts and filters as many high-quality candidate entity pairs as possible from the text through a BERT-based entity pair extraction and filtering block. These candidate entity pairs are used as part of the large model instructions and are input into the relationship and triple extraction module based on the large model. Relationships are assigned to the candidate entity pairs and as many triples as possible are extracted, which significantly improves the hallucination problem of the large model while improving the recall rate of the relationship triple extraction model. By constructing appropriate instructions and fine-tuning the large model, the error between the two stages of the traditional pipeline structure relationship triple extraction model is significantly reduced. Through the triple noise filtering module based on the large model, the correctness of the triple extraction results is judged, and erroneous triples are eliminated, which significantly improves the accuracy of the relationship triple extraction model.
[0027] 1. Accuracy: By fine-tuning the dataset and combining it with Lora's fine-tuning of the open-source large model, we combined the high-quality candidate entity pair sets provided by the candidate entity pair extraction and filtering module with the intelligently designed instruction template. This multi-faceted enrichment and optimization of the large model's instructions ensures the accuracy of the large model for relation triple extraction.
[0028] 2. Robustness: This invention uses high-quality candidate entity pairs extracted and filtered by the small model to enrich the instructions of the large model, effectively alleviating the hallucination problem of the large model and improving the stability of the large model.
[0029] 3. Portability: This invention improves the portability of the code of each component module in the framework, enabling it to effectively connect different large-scale language model bases and be applicable to various natural language processing tasks, with strong general capabilities.
[0030] In summary, the present invention provides an accurate, stable, and versatile relation triple extraction scheme by combining the training, fine-tuning, and prompting techniques of a small pre-trained language model and a large language model, effectively improving the ability of the large model for triple extraction while effectively alleviating the hallucination problem of the large model. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 It is a process framework diagram of a specific implementation method of the present invention.
[0032] Figure 2 This is a framework diagram of the candidate entity pair extraction and filtering module according to a specific embodiment of the present invention.
[0033] Figure 3 It is a framework diagram of the relationship and triple extraction module of the specific implementation method of the present invention.
[0034] Figure 4 This is a detailed template for asking questions to a large model in the relation and triple extraction module of a specific embodiment of the present invention.
[0035] Figure 5 This is a framework diagram of a noise filtering module according to a specific embodiment of the present invention.
[0036] Figure 6 The figures are comparative experimental data between the present invention and the prior art. DETAILED DESCRIPTION
[0037] The present invention will be described in further detail below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.
[0038] The overall structure of the present invention is shown in FIG. Figure 1 shown.
[0039] This framework mainly consists of three modules: candidate entity pair extraction and filtering module, relation and triple extraction module, and noise filtering module. The design of each module and the overall process of the framework are as follows:
[0040] 1. The candidate entity pair extraction and filtering module is designed to extract and filter high-quality candidate entity pairs from the text to enrich the instructions of the large model. The input text data is segmented into sentences after data preprocessing and input into the candidate entity pair extraction and filtering module. The preprocessing includes cleaning the content of the original corpus, removing unnecessary symbols and garbled characters, and retaining neatly formatted sentences. Through the entity pair extraction model based on a small pre-trained language model and the entity pair filtering model based on the EMA attention mechanism, high-quality candidate entity pairs are screened and integrated into the candidate entity pair list.
[0041] 2. The Relation and Triple Extraction module aims to classify the relationship type for each candidate entity pair and extract as many triples as possible from outside the candidate entity pair list. This module combines the candidate entity pair list, raw text information, and relationship list output from the previous module with the triple extraction task instructions. This is then embedded into a pre-configured large-scale model Q&A template. The module then uses a fine-tuned open-source large-scale model to extract triples. The extracted triples are then output and integrated into the triple list.
[0042] 3. The noise filtering module is designed to judge the correctness of the extracted triples and eliminate erroneous triples, thereby significantly improving the accuracy of the triple extraction model and reducing the cost of subsequent manual review.
[0043] 1. Entity pair extraction and filtering module design
[0044] The overall design of the entity pair extraction and filtering module is as follows Figure 2 As shown:
[0045] For the input sentence S=[x1,x2,…x N ], first use the small pre-trained language model BERT\RoBERTa as the English and Chinese PLM (Product Lifecycle Management) to vectorize the sentence words:
[0046] [h1,h2,…,h N ]=PLM[x1,x2,…,x N ]
[0047] The entity pair extraction model is then used to extract entity pairs that may have relationships from the sentence. However, there are many meaningless entity pairs, and the quality is poor. To address this, an entity pair filtering model based on the EMA (Efficient Multi-Scale Attention) attention mechanism is designed to score the entity pairs in the entity pair list and retain the high-quality entity pairs to form a candidate entity pair list:
[0048] [g1,g2,…,g N ]=EMA[h1,h2,…,h N ]
[0049] In addition, the rotated position embedding (RoPE) is added as auxiliary embedding information and the attention score in EMA is calculated based on it:
[0050]
[0051] Where, d2=64, q i and k i are the query and value vectors for the ith token in the sentence, computed by two linear layers:
[0052] q i =W q h i +b q
[0053] k i =W k h i +b k
[0054] Among them, W q , W k , b q and b k is a trainable parameter, and the loss function of the entity pair filtering model is:
[0055]
[0056] Among them, σ is the sigmoid function, y ij Represents the entity pair (x i ,x j ), then the entity pair set output by the entity pair extraction model is input into the entity pair filtering model and each entity pair (h, t) in the set is scored:
[0057]
[0058] Among them, h m and t m Is the head entity collection and tail entity collection Finally, the entity pairs with scores F(h,t)>0 are retained and constructed as a set of candidate entity pairs.
[0059] Some important hyperparameter settings of the entity pair extraction model and the EMA-based entity pair filtering model during the training phase are shown in Table 1:
[0060] Table 1
[0061] epochs 3 batch_size 16 learning_rate 5e-5 optimizer AdamW
[0062] 2. Relation and triple extraction module design
[0063] The overall design of the relation and triple extraction module is as follows Figure 3 As shown:
[0064] Integrate the candidate entity pair list output by the previous module into the large model instruction on the left side of the above figure. First, define the relationship type list and generate the task description. Then, based on the relationship type list, task description and original text information, generate a template for asking questions to the large model. The detailed template for asking questions to the large model is as follows: Figure 4 As shown, where x rel Represents a set of predefined relationship types, x can represents the candidate entity pair set, x sen Represents the original sentence. A fine-tuning dataset is constructed based on this template, and LORA is used to fine-tune the open source large model (Qwen-7B). Some key hyperparameter settings in the fine-tuning stage are shown in Table 2:
[0065] Table 2
[0066] per_device_train_batch_size 2 per_device_eval_batch_size 1 gradient_accumulation_steps 8 save_strategy steps save_steps 1000 learning_rate 1e-5 weight_decay 0.1 logging_steps 1 2048
[0067] During the reasoning phase, the fine-tuned Qwen-7B can focus on referring to a set of candidate entity pairs and a predefined set of relationship types, and assign relationships to entity pairs in the candidate entity pair set based on the contextual semantics of the input sentence. It can also extract as many relationship triplets as possible from the input sentence and output them in the form of a triple list.
[0068] 3. Noise filtering module design
[0069] The overall design of the noise filtering module is as follows model_max_length As shown:
[0070] Although the prompt template was carefully designed in the previous module, there are still inevitably erroneous triplets in the extracted triple list, which will bring significant noise to the extraction results and increase the subsequent manual processing costs. In order to evaluate the correctness of the output results of the large model of the previous module and filter out erroneous triplets, a noise filtering module is designed. Specifically, the triple classification task is defined as a two-classification task, and a prompt template for the large model is designed as shown in the upper half of the above figure. It consists of two parts: the triple list output by the previous module and the task instructions. In the task instructions, the large model is clearly required to make a correctness judgment on the input triples and can only answer "Yes, it's true" or "No, it's wrong". A fine-tuning dataset is constructed using this template and LORA is used to fine-tune the large model (Qwen-7B). Some key hyperparameter settings in the fine-tuning stage are shown in Table 3:
[0071] Table 3
[0072] Figure 5 2 per_device_train_batch_size 1 per_device_eval_batch_size 8 gradient_accumulation_steps save_strategy steps 2000 save_steps 1e-3 learning_rate 0.1 weight_decay 1 logging_steps 1024
[0073] The fine-tuned large model can evaluate and filter the correctness of triple extraction results, significantly improving the accuracy of the model and significantly reducing the subsequent manual screening costs, improving the triple extraction effect while increasing audit efficiency.
[0074] This example further discloses that the precision, recall, and F1 value of the method (CNRel) of the present invention on the relation extraction datasets NYT10, NYT11-HRL, and HacRED surpass those of the advanced baseline models TPLinker, CasRel, RERE, and Ding et al. The specific data are as follows: model_max_length Figure 6 shown.
Claims
1. A method for extracting relation triples based on a large model, comprising the following steps: Get the original text information and segment it into sentences; Extract entity pairs from the segmented text information using an entity pair extraction model based on a pre-trained language model; Predefine the relationship type for entity pair classification and generate a relationship triple extraction task description. Generate a question template for the large model based on the relationship type, task description, and original text information. Construct a fine-tuning dataset based on the question template, and use the fine-tuned large model to extract relation triples; The large model is used to determine the correctness of the extracted relationship triples to eliminate erroneous relationship triples.
2. The method according to claim 1, wherein The entity pairs are filtered by a pre-trained entity pair filtering model based on the EMA attention mechanism.
3. The method according to claim 2, wherein The rotation position embedding is added as auxiliary embedding information and the attention score in EMA is calculated based on it, and entity pairs that do not meet the set score are eliminated.
4. The method according to any one of claims 1 to 3, characterized in that: The classification task of the relation triples is defined as a binary classification task, and a prompt template is generated for the large model. The prompt template consists of a list of the relation triples and a task instruction requiring the large model to make a correctness judgment on the input relation triples. A fine-tuning dataset is constructed based on the prompt template, and the fine-tuned large model is used to judge the correctness of the extracted relation triples.
5. A relation triple extraction system based on a large model, comprising: The text data acquisition module obtains the original text information and segments it into sentences; The entity pair extraction module extracts entity pairs from the segmented text information through an entity pair extraction model based on a pre-trained language model; The relation and triple extraction module generates a question template for the large model based on the relation type, the relation triple extraction task description, and the original text information. It then constructs a fine-tuning dataset based on the question template and uses the fine-tuned large model to extract relation triples. The noise filtering module judges the correctness of the extracted relation triples through the large model shown and eliminates the erroneous relation triples.
6. The system according to claim 5, wherein: It also includes an entity pair filtering module, which filters the entity pairs through a pre-trained entity pair filtering model based on the EMA attention mechanism.
7. The system according to claim 6, wherein: The entity pair filtering module adds the rotation position embedding as auxiliary embedding information and calculates the attention score in EMA based on it, and eliminates entity pairs that do not meet the set score.
8. The system according to any one of claims 5 to 7, wherein: The noise filtering module defines the classification task of the relation triples as a binary classification task, and generates a prompt template for the large model. The prompt template consists of a list of the relation triples and a task instruction requiring the large model to make a correctness judgment on the input relation triples. A fine-tuning dataset is constructed based on the prompt template, and the fine-tuned large model is used to judge the correctness of the extracted relation triples.
9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, the computer program being configured to be executed by the processor, the computer program comprising instructions for executing the method according to any one of claims 1 to 4.
10. A storage medium storing a computer program, wherein when the computer program is executed by a computer, the method according to any one of claims 1 to 4 is implemented.