Small-sample relation extraction method based on large language model
By constructing a sample library and using strategies of text similarity and random sampling, scenario learning examples are constructed, which solves the problem of large language models being expensive when dealing with relationships between multiple entities, and realizes efficient and precise relationship extraction, and reduces training overhead.
Patent Information
- Application Number
- CN202411805368.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-10
- Publication Date
- 2025-03-18
AI Technical Summary
The existing relationship extraction method based on large language models needs to generate a large number of thinking chain outputs when processing the relationship between multiple entities in long texts, resulting in large overhead and affecting the system's response speed and efficiency.
By constructing a pre-sample library and combining sample selection strategies that combine text similarity and random sampling, a scenario learning example that is closely related to the original text semantics and has a variety of samples is constructed, and the boot model follows the few-sample instructions to extract the relationship of the specified type.
Relief on a large amount of labeled data is reduced, and the efficient, accurate and generalized relationship triple extraction is achieved. There is no need to retrain the model when the relationship category changes. It only needs to update the sample library and recluster, which greatly reduces training overhead.
Smart Images
Figure CN119646237A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of natural language technology, and in particular to a small sample relationship extraction method based on a large language model. Background Art
[0002] In the field of natural language processing, traditional relation extraction methods often define this task as a series of classification problems, the core of which is to build a classification model that can identify specific relationships between entities. This method relies on providing a large amount of manually annotated data for each specific relationship type, which is largely limited by the quality and diversity of the data, and greatly limits the model's ability to migrate to new domains or tasks. In addition, this isolated learning method seriously hinders knowledge sharing between different information extraction subtasks, affecting the rapid adaptation and expansion of relation extraction systems in diverse application scenarios.
[0003] In recent years, due to its excellent performance and flexibility, Large Language Model (LLM) has quickly become the mainstream model in the field of natural language processing, and is widely used in various tasks such as machine translation, text summarization, question-answering systems, and text classification. In the pre-training stage, LLM captures the rich underlying patterns, structures, and semantic knowledge contained in the text corpus through self-supervised learning tasks. Furthermore, by fine-tuning these pre-trained models on a small amount of labeled data, they can be adapted to specific downstream tasks. This training-fine-tuning paradigm of large language models significantly reduces the reliance on large amounts of labeled data, reducing the cost and time of manual annotation.
[0004] Currently, many relation extraction research works based on large language models rely on Chain of Thought (CoT) to improve performance. However, these Chain of Thought-based methods need to generate a complete chain of thought output for each pair of relationship entities. This pair-by-pair generation of chains of thought will result in a huge overhead, especially when dealing with the relationship between multiple entities in long texts. For systems that require high real-time performance, this may become a performance bottleneck, affecting the response speed and efficiency of the system. Summary of the invention
[0005] The purpose of the present invention is to provide a small sample relationship extraction method based on a large language model to solve the problems raised in the above background technology.
[0006] In view of the above problems, the technical solution proposed by the present invention is:
[0007] The small sample relation extraction method based on the large language model includes the following steps:
[0008] Step 1: model fine-tuning, the model fine-tuning includes the following steps:
[0009] Dataset construction: Collect Chinese relation extraction datasets from CFETER, CRED, and CMRE, and combine them with manually annotated data from multiple fields to build a training dataset that covers a wide range of relation extraction scenarios; Input and label generation: For a given text x and relation type r, generate prompt prompt = f(x, r) as fine-tuning input; For all relation triplets corresponding to type r in the text x, generate a set of Sort the head entity and tail entity by their positions in the original text and convert them into fine-tuned target labels in the specified output format Loss function and optimization: using cross entropy loss function Use the LORA solution to speed up the fine-tuning process;
[0010] Specifically, this step enables the large language model to initially have the ability to extract a given type of relationship from unstructured text and represent it in a designed output format.
[0011] Step 2: Sample library construction and maintenance, the sample library construction and maintenance includes the following steps: Text feature extraction: using the BERT pre-trained model to extract the text feature vector f of the original text x;
[0012] Relation type annotation: manually annotate the original text x, the triple set of all relation types R involved in the downstream task in the original text After sorting, it is converted into a JSON list Data labeling and quality scoring: manual Rating Composing sample library data
[0013] Clustering and sample library construction: Based on the K-means unsupervised clustering algorithm, all texts in the sample library are divided into categories according to text features, and the category center feature vector c is obtained;
[0014] Specifically, this step enables the examples in scenario learning to provide richer information, thereby further improving the performance of relationship extraction.
[0015] Step 3: context learning sample selection, the context learning sample selection includes the following steps: obtaining document features and calculating similarity: for the text x of the relationship to be extracted and the given relationship type r i , obtain the document feature f of the text x, calculate the cosine similarity between f and the central feature vector c of each category in the sample library, and determine the category c that is closest to the input document semantics j = argmax m similarity(f,c m );
[0016] Random sampling and screening: Randomly sample N = 10 texts from category cj to form a candidate set, and select n = 3 texts with higher scores on relation type ri and their relation labels from the candidate set to form a scenario learning sample set;
[0017] Specifically, this step aims to ensure that the examples learned in context are closely related to the input text in terms of semantics while having a certain quality and diversity, so as to improve the accuracy and generalization of relation extraction.
[0018] Step 4: Relation extraction, the relationship extraction includes the following steps:
[0019] Design prompt template: The prompt template contains four parts: target (clearly extract triple type), constraint (define output format and rules), example (provide relevant sample pairs), and input (current text to be extracted);
[0020] Output format design: Use JSON dictionary format to encode the triple extraction results, encapsulate each pair of given type relationships in a JSON dictionary to form a JSON list. The dictionary annotates entities and their types, and outputs all given type relationship triplets at once;
[0021] Model prediction: The model extracts relations from the input text based on the prompt template and scenario learning examples, and generates output in JSON format.
[0022] Compared with the prior art, the beneficial effects of the present invention are as follows: the few-sample relationship extraction method based on a large language model, through a pre-constructed sample library, combines text similarity with a random sampling sample selection strategy to construct a scenario learning sample that is closely related to the original text semantics and has a variety of samples, and guides the model to follow the few-sample instructions to extract the specified type of relationship. It reduces the dependence on a large amount of annotated data and achieves efficient, accurate and generalized relationship triple extraction. In addition, the construction and update of the sample library makes it unnecessary for this solution to retrain the model when the relationship category changes, but only needs to update the sample library and re-cluster, which greatly reduces the training overhead and enhances the generalization ability and robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 It is a general block diagram of a method for extracting relations from a small number of samples based on a large language model disclosed in an embodiment of the present invention;
[0024] Figure 2 A method block diagram of sample selection for a small sample relationship extraction method based on a large language model disclosed in an embodiment of the present invention;
[0025] Figure 3 A method block diagram of constructing a sample library for a small sample relationship extraction method based on a large language model disclosed in an embodiment of the present invention. DETAILED DESCRIPTION
[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0027] See also Figure 1-Figure 3 The present invention provides a technical solution: a method for extracting relations from a small number of samples based on a large language model, comprising the following steps:
[0028] Step 1: model fine-tuning, the model fine-tuning includes the following steps:
[0029] Dataset construction: Collect Chinese relation extraction datasets from CFETER, CRED, and CMRE, and combine them with manually annotated data from multiple fields to build a training dataset that covers a wide range of relation extraction scenarios; Input and label generation: For a given text x and relation type r, generate prompt prompt = f(x, r) as fine-tuning input; For all relation triplets corresponding to type r in the text x, generate a set of Sort the head entity and tail entity by their positions in the original text and convert them into fine-tuned target labels in the specified output format Loss function and optimization: using cross entropy loss function Use the LORA solution to speed up the fine-tuning process;
[0030] Specifically, this step enables the large language model to initially have the ability to extract a given type of relationship from unstructured text and represent it in a designed output format.
[0031] Step 2: Sample library construction and maintenance, the sample library construction and maintenance includes the following steps: Text feature extraction: using the BERT pre-trained model to extract the text feature vector f of the original text x;
[0032] Relation type annotation: manually annotate the original text x, the triple set of all relation types R involved in the downstream task in the original text After sorting, it is converted into a JSON list Data labeling and quality scoring: manual Rating Composing sample library data
[0033] Clustering and sample library construction: Based on the K-means unsupervised clustering algorithm, all texts in the sample library are divided into categories according to text features, and the category center feature vector c is obtained;
[0034] Specifically, this step enables the examples in scenario learning to provide richer information, thereby further improving the performance of relationship extraction.
[0035] Step 3: context learning sample selection, the context learning sample selection includes the following steps: obtaining document features and calculating similarity: for the text x of the relationship to be extracted and the given relationship type r i , obtain the document feature f of the text x, calculate the cosine similarity between f and the central feature vector c of each category in the sample library, and determine the category c that is closest to the input document semantics j = argmax m similarity(f,c m );
[0036] Random sampling and screening: Randomly sample N = 10 texts from category cj to form a candidate set, and select relationship type r from the candidate set i The n=3 texts with higher scores and their relationship tags constitute the scenario learning sample set;
[0037] Specifically, this step aims to ensure that the examples learned in context are closely related to the input text in terms of semantics while having a certain quality and diversity, so as to improve the accuracy and generalization of relation extraction.
[0038] Step 4: Relation extraction, the relationship extraction includes the following steps:
[0039] Design prompt template: The prompt template contains four parts: target (clearly extract triple type), constraint (define output format and rules), example (provide relevant sample pairs), and input (current text to be extracted);
[0040] Output format design: Use JSON dictionary format to encode the triple extraction results, encapsulate each pair of given type relationships in a JSON dictionary to form a JSON list. The dictionary annotates entities and their types, and outputs all given type relationship triplets at once;
[0041] Model prediction: The model extracts relations from the input text based on the prompt template and scenario learning examples, and generates output in JSON format.
[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0043] See also Figure 1-Figure 3The present invention provides a technical solution: a small sample relationship extraction method based on a large language model, including.
[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0045] See also Figure 1-Figure 3 The present invention provides a technical solution: a small sample relation extraction method based on a large language model, using precision (Precision, P), recall (Recall, R) and F1 score (F1 Score) as evaluation indicators, which are specifically defined as:
[0046]
[0047] Among them, TP (True Positives) is the number of samples correctly predicted as positive, FP (False Positives) is the number of negative samples incorrectly predicted as positive, and FN (False Negatives) is the number of positive samples incorrectly predicted as negative. Since this task involves multiple relationship types, this paper uses Micro P, Micro R, and Micro F1 indicators to comprehensively evaluate the performance of the model on each relationship type.
[0048] The experiment uses the Qwen2-7B-Instruct model as the base and is tested on a manually labeled data center operation-related database. The experimental results show that this scheme has significantly improved performance compared to existing extraction methods.
[0049]
[0050]
Claims
1. A small sample relation extraction method based on a large language model, characterized in that: The following steps are involved: Step 1: model fine-tuning, the model fine-tuning includes the following steps: Dataset construction: Collect the CFETER, CRED, and CMRE Chinese relation extraction datasets, and combine them with manually annotated data from multiple fields to build a training dataset that covers a wide range of relation extraction scenarios; Input and label generation: For a given text x and relation type r, generate prompt prompt = f(x, r) as fine-tuning input; for all relation triplets corresponding to type r in the text x Sort the head entity and tail entity by their positions in the original text and convert them into fine-tuned target labels in the specified output format Loss function and optimization: using cross entropy loss function Use the LORA solution to speed up the fine-tuning process; Step 2: Sample library construction and maintenance, the sample library construction and maintenance includes the following steps: Text feature extraction: Use the BERT pre-trained model to extract the text feature vector f of the original text x; Relation type annotation: manually annotate the original text x, the triple set of all relation types R involved in the downstream task in the original text After sorting, it is converted into a JSON list Data labeling and quality scoring: manual Rating Composing sample library data Clustering and sample library construction: Based on the K-means unsupervised clustering algorithm, all texts in the sample library are divided into categories according to text features, and the category center feature vector c is obtained; Step 3: context learning sample selection, the context learning sample selection includes the following steps: obtaining document features and calculating similarity: for the text x of the relationship to be extracted and the given relationship type r i , obtain the document feature f of the text x, calculate the cosine similarity between f and the central feature vector c of each category in the sample library, and determine the category c that is closest to the input document semantics j = argmax m similarity(f,c m ); Random sampling and screening: From category c j Randomly sample N = 10 texts to form a candidate set, and select the relationship type r from the candidate set i The n=3 texts with higher scores and their relationship tags constitute the scenario learning sample set; Step 4: Relation extraction, the relationship extraction includes the following steps: Design prompt template: The prompt template contains four parts: target (clearly extract triple type), constraint (define output format and rules), example (provide relevant sample pairs), and input (current text to be extracted); Output format design: Use JSON dictionary format to encode triple extraction results, encapsulate each pair of given type relations in a JSON dictionary to form a JSON list, the dictionary annotates entities and their types, and outputs all given type relationship triplets at once; Model prediction: The model extracts relations from the input text based on the prompt template and scenario learning examples, and generates output in JSON format.