Relation extraction method based on prompt label collaboration
By constructing synonym labels and injecting relational knowledge, the problem of knowledge transfer barriers in relation extraction of pre-trained language models is solved, more efficient relational label mapping and semantic parsing are achieved, and the generalization ability of the model is improved.
Patent Information
- Application Number
- CN202510743452.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-05
- Publication Date
- 2025-09-16
AI Technical Summary
Existing pre-trained language models face knowledge transfer barriers in the pre-training and fine-tuning stages in relation extraction tasks. The complexity of label semantics increases the difficulty of mask prediction, and fixed label semantic representations are difficult to cover the relationship mapping between entity pairs in different contexts.
Two sets of semantically identical synonym labels are constructed, relational knowledge is injected through a pre-trained language model, the representation of the hint template part is restored using a masked language model, and regularization is performed by minimizing the KL divergence to achieve collaborative mapping of labels.
It effectively reduces the difficulty of mapping different context entities to relational labels, improves the generalization and semantic parsing capabilities of the model, and enhances the ability to recognize label semantics.
Smart Images
Figure CN120654699A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a relationship extraction method based on prompt tag collaboration. Background Art
[0002] With the explosive growth of the internet and the surge in digital information, humanity is facing an unprecedented flood of data, encompassing a rich array of content, including news reports, social media updates, and academic papers. This vast expanse of text holds valuable insights and knowledge, but the sheer volume of data makes traditional manual screening and analysis inadequate. To address this challenge, information extraction technology—a solution designed to automatically parse unstructured text and extract structured information—is emerging.
[0003] One of the key components of information extraction is relation extraction, whose goal is to accurately detect and extract associations between entities in text, such as job affiliation, place of residence, or family relationships. In recent years, the application of pre-trained language models (PLMs) in the field of relation extraction has shown extraordinary potential because they can extract a wide range of language structure, semantic understanding, grammatical knowledge, and background information from massive amounts of text. By further training (i.e., fine-tuning) the PLM on task-specific datasets, the model can be guided to direct its rich internal knowledge towards solving specific NLP tasks. This feature of PLMs enables them to effectively understand the context of sentences and gain insight into the implicit connections between entities.
[0004] However, despite the remarkable success of fine-tuning strategies in the field of natural language processing (NLP), research indicates that the fundamental difference in task format between pre-training and fine-tuning may pose a barrier to knowledge transfer. Pre-training often focuses on cloze-like tasks, such as predicting missing words in a text sequence; whereas fine-tuning addresses diverse tasks such as classification, text generation, or sequence labeling. This inconsistency may hinder the effective application of knowledge accumulated by pre-training learning models (PLMs) in downstream tasks. To address this, researchers have proposed an innovative approach called "prompt learning" to reconcile the differences between pre-training and fine-tuning. Prompt learning leverages the general language understanding capabilities of PLMs acquired on large-scale datasets, combined with carefully designed prompts (such as "ElonMusk[MASK]Tesla") and a set of candidate vocabulary (e.g., "founded," "located in," and "manufactured"), to reshape the relation extraction task into a cloze problem consistent with the pre-training task format.
[0005] In relation extraction, label text has rich semantic features, making it difficult to interpret relation labels using a single word from the PLM vocabulary. The semantic complexity of labels increases the difficulty of mask prediction. While information captured by the template, such as entity type and location, can provide more contextual clues and impose structural constraints on the mask, it cannot provide more semantic information about the label itself. Furthermore, a fixed label semantic representation struggles to encompass the relationship mapping between entity pairs in different contexts. Summary of the Invention
[0006] The purpose of the present invention is to provide a relationship extraction method based on prompt tag collaboration, construct two sets of synonym tags for relationship tags, and calibrate their representations by perceiving the context to obtain the best prompts, effectively reducing the difficulty of mapping different context entity pairs to relationship tags.
[0007] To achieve the above objectives, the present invention provides a relationship extraction method based on prompt tag collaboration, comprising the following steps:
[0008] S1. Obtain a text dataset, extract relation mention sentences containing a pair of entities with known semantic relationship categories between the entities from the text dataset, and construct an input template based on the relation mention sentences;
[0009] S2. Input the input template into the pre-trained language model to form a sentence vector;
[0010] S3. Construct two sets of semantically identical synonym labels for each type of relational label. Inject relational knowledge into the two sets of synonym labels through a pre-trained language model to obtain an abstract semantic representation of each word in the same word set.
[0011] S4. Extract the prompt template part from the sentence vector mapping, introduce the two sets of synonym labels after injecting relational knowledge, and use the masked language model of the pre-trained language model to restore the representation of the prompt template part;
[0012] S5. Map the vector output of the prompt template part to the relationship type corresponding to the relationship label through the classification head.
[0013] Preferably, constructing an input template according to the relation mention statement in step S1 includes:
[0014] Extract the two entities contained in the relation mention sentence, add a specific placeholder [MASK] to the two entities as a prompt to form a prompt template, and splice the prompt template into the original relation mention sentence to form the final input template. The template is represented as:
[0015] S=(s1,s2,…,entity1,…,entity2,…,sn.entity1,[MASK],entity2);
[0016] Among them, S represents the final template, entity1 and entity2 represent two entities in the sentence, and [MASK] represents the prompt placeholder.
[0017] Preferably, step S3 specifically includes:
[0018] For each type of relationship label, construct two sets of semantically identical synonym labels Y1 and Y2, where Y1 uses the default label of the dataset and Y2 uses the synonym label;
[0019] Use the pre-trained language model to encode the semantic knowledge of the two sets of synonym labels and create two sets of answer word sets V1 and V2 to fully represent the corresponding two sets of synonym labels Y1 and Y2.
[0020] Preferably, the method for creating the answer word set includes:
[0021] Decompose the relations in the default label Y1 of the dataset into semantic words, and initialize them according to the average value of the PLM word embedding of the semantic words obtained by decomposing the relations in the default label Y1 of the dataset to inject relational knowledge and obtain the answer word set V1. Use the same method to decompose all the relations in another set of synonym labels Y2 to obtain the answer word set V2.
[0022] Preferably, step S4 specifically includes: taking out the vector representation part X of the prompt template part [MASK] in the sentence vector matrix X m , introduce the answer word sets V1 and V2 after injecting knowledge relations, and use the mask language model of the pre-trained language model to recover the representation of the two sets of [MASK] from the answer words according to the inner product similarity.
[0023] Preferably, the masked language model of the pre-trained language model is provided with two classification heads and is trained using a cross entropy loss function.
[0024] Preferably, the probability distributions P1(y1|x) and P2(y2|x) of the position of [MASK] are obtained based on the two sets of [MASK];
[0025] Minimize the bidirectional KL divergence between the probability distributions of the same samples and regularize the model to map the vector output of the relation hint part to the relation type corresponding to the relation label through the classification head.
[0026] Therefore, the present invention adopts the above-mentioned relationship extraction method based on prompt tag collaboration, which has the following beneficial effects:
[0027] (1) Construct two sets of synonym labels for the relationship label and integrate the rich semantic features between the synonym labels into the mask model, so that the mask prediction can obtain richer label semantic information to improve the semantic parsing ability of the mask, effectively reducing the difficulty of mapping different context entity pairs to relationship labels;
[0028] (2) Expand the model’s understanding of different words in the corpus, explore the rich semantics and prior knowledge between synonymous labels, and ultimately achieve semantic sharing of different labels, thereby improving the model’s generalization ability;
[0029] (3) Compared with the traditional one-to-one label mapping in prompt learning, when the masked generated words are biased but have similar semantics, the correct category can still be predicted, which enables synergy between different labels with similar semantics and improves the model's ability to recognize label semantics.
[0030] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 is a flow chart of a method according to an embodiment of the present invention;
[0032] Figure 2 A roadmap for the extraction method of an embodiment of the present invention;
[0033] Figure 3 This is an extraction model diagram of an embodiment of the present invention. DETAILED DESCRIPTION
[0034] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings herein can be arranged and designed in various different configurations. In the description of the present invention, it should be noted that the orientation or position relationship indicated by the terms "upper", "lower", "inside", "outside", etc. is based on the orientation or position relationship shown in the drawings, or is the orientation or position relationship in which the product of the invention is usually placed when in use. It is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.
[0035] Example
[0036] Reference Figure 1-3 The present invention provides a relationship extraction method based on prompt tag collaboration, the steps comprising:
[0037] S1. Obtain the SemEvaltask 2010 dataset, extract relation mention sentences containing a pair of entities with known semantic relationship categories from the dataset, and construct input templates based on the relation mention sentences. Constructing input templates based on relation mention sentences includes:
[0038] Extract the two entities contained in the relation mention sentence, add a specific placeholder [MASK] to the two entities as a prompt to form a prompt template, and splice the prompt template into the original relation mention sentence to form the final input template. The template is represented as:
[0039] S=(s1,s2,…,entity1,…,entity2,…,s n .entity1,[MASK],entity2);
[0040] Among them, S represents the final template, entity1 and entity2 represent two entities in the sentence, and [MASK] represents the prompt placeholder.
[0041] S2. The input template is fed into a pre-trained language model (PLM) to form a sentence vector. The PLM uses the BERT architecture, which includes a word embedding layer, a multi-head attention layer, layer normalization, and a feedforward neural network. The word embedding layer includes word embedding and positional encoding. The prompt template first passes through the word embedding layer to convert the discrete input into a continuous representation. Multi-head attention is then used to filter the semantic information of the sentence in the input prompt template, helping the model understand the semantic relationships between entities. Finally, layer normalization and a feedforward neural network are used to further enhance the semantic expression capabilities of the PLM.
[0042] S3. For each type of relationship tag, construct two sets of synonym tags with the same semantics. Through PLM, inject relationship knowledge into the two sets of synonym tags to obtain the abstract semantic representation of each word in the same word set. Specifically:
[0043] For each type of relationship label, two sets of semantically identical synonym labels Y1 and Y2 are constructed, as shown in Table 1. Y1 uses the default label of the dataset, and Y2 uses the synonym label.
[0044] The PLM is used to encode the semantic knowledge of the two sets of synonym labels, creating two sets of answer words V1 and V2 to fully represent the corresponding two sets of synonym labels Y1 and Y2. The answer word sets are created by decomposing the relations in the dataset's default label Y1 into semantic words. The answer word sets are initialized with the average PLM word embeddings of the semantic words obtained by decomposing the relations in the dataset's default label Y1 to inject relational knowledge, resulting in the answer word set V1.
[0045] For example, for the relationship "message-topic" in Y1, decompose it into semantic words to get "message messageoftopic topic", and then initialize the answer word. The formula is: in is an answer word in the answer word set V1, Emb [·] BERT word embedding.
[0046] Use the same method to decompose all the relationships in another set of synonym labels Y2 to obtain the answer word set V2.
[0047] Table 1 Some examples of original relation labels and synonym labels
[0048]
[0049] S4. Extract the prompt template part from the sentence vector mapping, introduce the two sets of synonym labels after injecting relational knowledge, and use the masked language model of PLM to restore the representation of the prompt template part. Specifically, it includes:
[0050] Take out the vector representation part X of the prompt template part [MASK] in the sentence vector matrix X m , in order to m To predict relationships, we introduce a set of answer words V1 and V2, which have been injected with knowledge relationships, to represent the relationship to be tested. We then use the head layer (·) of the PLM's Masked Language Model (MLM) to recover the representations of the two sets [MASK] from the answer words based on inner product similarity. The PLM's masked language model has two classification heads and is trained using the cross-entropy loss function.
[0051] S5. Map the vector output of the prompt template part to the relationship type corresponding to the relationship label through the classification head.
[0052] Based on the two sets of [MASK], we obtain the probability distributions P1(y1|x) and P2(y2|x) for the [MASK] position. During this training step, the probability distributions of the two sets of [MASK] are not completely consistent due to the dropout operation randomly discarding units in the model and the fact that [MASK] predicts answer words with different labels with similar semantics. The model is then regularized by minimizing the bidirectional Kullback-Leibler (KL) divergence between the probability distributions of the same samples. This allows the classification head to map the vector output of the relation hint part to the relation type corresponding to the relation label.
[0053] The KL divergence calculation formula is as follows:
[0054]
[0055] Where D KL (P1(y1|x)||P2(y2|x)) represents the KL divergence between two distributions P1 and P2.
[0056] The classification head output formula is:
[0057]
[0058] Where R is the total number of answer word v categories, and the learnable words can adaptively obtain the best representation of prompt learning through answer word optimization; r is the current category number; j is the category sum index variable; y r is the true label of the sample; is the cross entropy loss; h M It is the characteristic variable of the [MASK] position.
[0059] To make the solution of the present invention clearer, the text data set "this book was written by Lu Xun" is taken as an example for explanation.
[0060] Select the text "This book was written by Lu Xun" from the dataset. The two entities in the sentence are entity 1 "book" and entity 2 "Lu Xun," and the semantic relationship between the entities is "Product-Producer." Next, construct a prompt template "book? [MASK], Lu Xun" and append it to the end of the sentence to form the input template. [MASK] is a specific placeholder.
[0061] The input template is fed into PLM to form a sentence vector.
[0062] Construct another set of synonym tags "Creation-Creator" for the relationship tag "Product-Producer", and then encode the original relationship tags and synonym tags through PLM to form two sets of answer words V1 and V2.
[0063] Take out the vector representation part X with [MASK] in the sentence vector matrix X m , in order to m To predict the relationship, the head layer head(·) of the MLM of PLM is used to recover the representation of the two sets of [MASK] from the answer words according to the inner product similarity.
[0064] Based on the two sets of [MASK], the probability distribution of the [MASK] position is obtained, and then the bidirectional KL divergence between the probability distributions of the same samples is minimized to regularize the model. The vector output of the relationship prompt part is mapped to the relationship type corresponding to the relationship label through the classification head.
[0065] To verify the effectiveness of the method of the present invention, the datasets SemEval-2010 task 8, TACRED and Re-TACRED were used for verification. The results are shown in Table 2.
[0066] Table 2 Experimental comparison results
[0067]
[0068] As shown in Table 2, the method of the present invention obtains F1 values of 91.2%, 91.3% and 72.7% respectively, which has better performance.
[0069] Therefore, the present invention adopts the above-mentioned relationship extraction method based on prompt label collaboration to construct two sets of synonym labels for relationship labels, and integrates the rich semantic features between synonym labels into the mask model, so that richer label semantic information can be obtained during mask prediction to improve the semantic parsing ability of the mask, effectively reducing the difficulty of mapping different context entity pairs to relationship labels.
[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solutions of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.
Claims
1. A relationship extraction method based on prompt tag collaboration, characterized in that the steps include: S1. Obtain a text dataset, extract relation mention sentences containing a pair of entities with known semantic relationship categories between the entities from the text dataset, and construct an input template based on the relation mention sentences; S2. Input the input template into the pre-trained language model to form a sentence vector; S3. Construct two sets of semantically identical synonym labels for each type of relational label. Inject relational knowledge into the two sets of synonym labels through a pre-trained language model to obtain an abstract semantic representation of each word in the same word set. S4. Extract the prompt template part from the sentence vector mapping, introduce the two sets of synonym labels after injecting relational knowledge, and use the masked language model of the pre-trained language model to restore the representation of the prompt template part; S5. Use the mask language model of the pre-trained language model to map the vector output of the prompt template part to the relationship type corresponding to the relationship label.
2. The relationship extraction method based on prompt tag collaboration according to claim 1 is characterized in that: In step S1, constructing an input template based on the relation mention statement includes: Extract the two entities contained in the relation mention sentence, add a specific placeholder [MASK] to the two entities as a prompt to form a prompt template, and splice the prompt template into the original relation mention sentence to form the final input template. The template is represented as: S=(s1,s2,…,entity1,…,entity2,…,s n .entity1,[MASK],entity2); Among them, S represents the final template, entity1 and entity2 represent two entities in the sentence, and [MASK] represents the prompt placeholder.
3. The relationship extraction method based on prompt tag collaboration according to claim 2 is characterized in that: Step S3 specifically includes: For each type of relationship label, construct two sets of semantically identical synonym labels Y1 and Y2, where Y1 uses the default label of the dataset and Y2 uses the synonym label; Use the pre-trained language model to encode the semantic knowledge of the two sets of synonym labels and create two sets of answer word sets V1 and V2 to fully represent the corresponding two sets of synonym labels Y1 and Y2.
4. The relationship extraction method based on prompt tag collaboration according to claim 3 is characterized in that: The method for creating the answer word set includes: Decompose the relations in the default label Y1 of the dataset into semantic words, and initialize them according to the average value of the PLM word embedding of the semantic words obtained by decomposing the relations in the default label Y1 of the dataset to inject relational knowledge and obtain the answer word set V1. Use the same method to decompose all the relations in another set of synonym labels Y2 to obtain the answer word set V2.
5. The relationship extraction method based on prompt tag collaboration according to claim 4 is characterized in that: Step S4 specifically includes: taking out the vector representation part X of the prompt template part [MASK] in the sentence vector matrix X m , introduce the answer word sets V1 and V2 after injecting knowledge relations, and use the mask language model of the pre-trained language model to recover the representation of the two sets of [MASK] from the answer words according to the inner product similarity.
6. The relationship extraction method based on prompt tag collaboration according to claim 5 is characterized by: The masked language model of the pre-trained language model is provided with two classification heads and is trained using a cross entropy loss function.
7. The relationship extraction method based on prompt tag collaboration according to claim 6 is characterized in that: Step S5 specifically includes: Based on the two sets of [MASK], the probability distribution of the [MASK] position P1(y1|x) and P2(y2|x) are obtained; Minimize the bidirectional KL divergence between the probability distributions of the same samples and regularize the model to map the vector output of the relation hint part to the relation type corresponding to the relation label through the classification head.