A relation extraction method based on hint learning in low-resource scenarios

Abstract

The present invention discloses a relationship extraction method based on prompt learning in a low-resource scenario, comprising the following steps: 1) decomposing a complex relationship into a joint representation of multiple perspectives, wherein the perspectives include: characters, locations, actions, and active and passive relationships; 2) using virtual words to represent each perspective relationship obtained by the decomposition; 2.1) representing the representation relationship of each perspective; 2.2) sampling the representation relationship of each perspective; 2.3) using a mask pre-training model, creating m different relationship virtual words for each relationship r in its vocabulary, each relationship virtual word focusing on a different perspective of a relationship; 3) obtaining the final complex relationship representation based on the representation of each perspective relationship and the generation probability of each perspective relationship representation. The present invention proposes a relationship extraction method using multi-perspective decoupling learning to decouple complex relationships into a joint distribution of multiple perspectives, which can effectively alleviate the problem of insufficient data in low-resource scenarios.

Description

Technical Field

[0001] The present invention relates to natural language processing technology, and in particular to a relationship extraction method based on prompt learning in a low-resource scenario. Background Art

[0002] Given that hint learning can efficiently utilize the knowledge of upstream models without training, existing relation extraction methods based on low-resource scenarios often use this technology. There are usually two ways to use this technology to complete relation extraction. (1) Use a combination of existing words to represent the relationship. This type of method assumes that each relationship can be represented by a word in the existing vocabulary, or a combination of several words. Then this method often requires manual construction of these combinations, which is difficult to construct for complex relationships and difficult to map perfectly to this relationship. (2) Construct new virtual words to represent the relationship. This type of method uses model learning to learn a new virtual word for each relationship to represent the relationship. This method requires providing these words with a good context and sufficient data samples. Otherwise, the virtual words obtained are prone to underfitting, resulting in incorrect mapping of the relationship. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a relationship extraction method based on prompt learning in a low-resource scenario in view of the defects in the prior art.

[0004] The technical solution adopted by the present invention to solve the technical problem is: a relationship extraction method based on prompt learning in a low-resource scenario, comprising the following steps:

[0005] 1) Decompose complex relationships into joint representations of multiple perspectives, including characters, locations, actions, and active and passive relationships;

[0006] Assume that xi represents the sentence to be queried, which contains the head entity ws and the tail entity wo to be queried, yi represents the relationship between the head entity and the tail entity, and i represents the sequence number; in the relationship representation for a single sample pair (xi, yi) It is decomposed into a joint representation of multiple perspective relations h, as follows:

[0007]

[0008] Here, it is assumed that the relational representation of a sample pair can be decomposed into a joint representation of many “sub-representations”, which are referred to as the various perspectives of the relation in the present invention, and p(h|xi) is the posterior probability of these perspectives.

[0009] 2) Use virtual words to represent the perspective relationships obtained by disassembly;

[0010] 2.1) Represent the representation relationship of each perspective;

[0011] When using prompt learning, the text x needs to be converted into a masked format T(x)=x:ws:[MASK]:wo, where the colon ":" represents string concatenation;

[0012] Use the cloze ability of the upstream pre-trained model to obtain the latent space representation of the mask

[0013] Decode to get the word v(y) related to this relationship;

[0014] 2.2) Sampling the representation relationship of each perspective;

[0015] 2.3) Use RoBERTa-large as the training model and create m different relational virtual words v for each relation r in its vocabulary 1...m (r), each relational subjunctive focuses on a different perspective of a relation;

[0016] Multi-view masks and their representation

[0017] Use multiple consecutive [MASK] as the decomposition method, and then follow the sentence order from left to right to the i-th [MASK] corresponding to the virtual word v j (r), the specific expression is as follows:

[0018] T(x)=x:w s :[MASK] 1...m :w o

[0019] Posterior probability Represents the generation probability of each view relationship representation, and uses a linear layer to learn to obtain the posterior probability The formula is as follows:

[0020]

[0021] 3) The final complex relationship representation is obtained based on the representation of each perspective relationship and the generation probability of each perspective relationship representation.

[0022] According to the above scheme, the loss function used in the training of the RoBERTa-large model in step 2.3) is as follows:

[0023]

[0024] in, The loss function corresponding to each data pair.

[0025]

[0026] According to the above scheme, the loss function used in the training of the RoBERTa-large model in step 2.3) is as follows:

[0027]

[0028] in, is the local loss function, is the global loss function;

[0029]

[0030] in, represents the set of all relations, The vector representation of the i-th relation virtual word of relation r, sim represents the similarity between two vectors, α and β are hyperparameters, which are set to 1.2 and 0.7 respectively in the present invention.

[0031] According to the above scheme, in the mask pre-training model RoBERTa-large, the relational virtual words are initialized as follows:

[0032] First, manually construct simple description information for each relationship, then initialize the state of each relationship virtual word, use the mask pre-training model to directly predict the word at each mask position, and then use this word to initialize the relationship virtual word at the corresponding position.

[0033] The beneficial effects produced by the present invention are:

[0034] 1. The present invention proposes a relationship extraction method using multi-perspective decoupling learning to decouple complex relationships into a joint distribution of multiple perspectives, and uses the sampled multi-perspective multiple vector representations to describe the relationships in low-resource scenarios, thereby making more full use of the knowledge of the mask pre-training model and alleviating the problem of insufficient data in low-resource scenarios.

[0035] 2. The present invention proposes a global-local loss function for use in combination to provide more robust training and encourage the training of relational virtual words. Relational virtual words corresponding to the same relationship have similar representations, and relational virtual words corresponding to different relationships have dissimilar representations.

[0036] 3. The present invention designs a matching dynamic initialization technology that can encourage each relational virtual word to learn different perspectives of the relationship. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0038] Figure 1 It is a flow chart of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0040] like Figure 1 As shown, a relation extraction method based on prompt learning in a low-resource scenario includes the following steps:

[0041] 1) Decompose complex relationships into joint representations of multiple perspectives, including characters, locations, actions, and active and passive relationships;

[0042] Assume x i Represents the sentence to be queried, which contains the head entity (subject) w to be queried s and tail entity (object) w o ,y i Indicates the relationship between the head entity and the tail entity, i represents the sequence number; for a single sample pair (x i ,y i ) relationship representation, from the original complex relationship representation It is decomposed into a joint representation of multiple perspective relations h, as follows:

[0043]

[0044] Here, it is assumed that the relational representation of a sample pair can be decomposed into a joint representation of many “sub-representations”, which are referred to as the various perspectives of the relation in this paper. i ) are the posterior probabilities of these perspectives.

[0045] 2) Use virtual words to represent the perspective relationships obtained by disassembly;

[0046] 2.1) Represent the representation relationship of each perspective;

[0047] When using prompt learning, the text x needs to be converted into a format with a mask T(x) = x:w s :[MASK]:w o , where the colon ":" indicates string concatenation;

[0048] Use the cloze ability of the upstream pre-trained model to obtain the latent space representation of the mask

[0049]

[0050] Decode to get the word v(y) related to this relationship;

[0051] 2.2) Sampling the representation relationship of each perspective;

[0052] The usual hint learning uses this relational latent space to represent the probability of the predicted category as follows:

[0053]

[0054]

[0055] p M represents the mask pre-training model. In order to modify h in a smoother and less modified way xi Sampling is performed. The present invention starts from the above formula and performs the following operations on "[MASK] = v(y i )” to obtain samplers with different perspectives by variable control, as shown in the following formula:

[0056]

[0057] Among them, [MASK] is disassembled into multiple different [MASK] j , they represent the same relational information but have different relational perspectives. Therefore, we also need to convert v(y i ) is disassembled into different v j (y i ) to the corresponding [MASK] j Complete the mapping between the vocabulary and the latent space.

[0058] 2.3) Use RoBERTa-large as the training model and create m different relational virtual words v for each relation r in its vocabulary 1...m (r), each relational subjunctive focuses on a different perspective of a relation;

[0059] Since a normal complex relationship may require a long paragraph to describe clearly, in order to obtain multiple masks and their mask representations, the present invention directly uses multiple consecutive [MASK] as a decomposition method, and then the i-th [MASK] corresponds to the virtual word v from left to right in the order of the sentence. j (r), the specific expression is as follows:

[0060] T(x)=x:w s :[MASK] 1...m :w o

[0061] Posterior probability Represents the generation probability of each view relationship representation, and uses a linear layer to learn to obtain the posterior probability The formula is as follows:

[0062]

[0063] The loss function used in RoBERTa-large model training is as follows:

[0064]

[0065] in, The loss function corresponding to each data pair.

[0066]

[0067] The loss function used in the training of the RoBERTa-large model in step 2.3) is as follows:

[0068]

[0069] in, represents the set of all relations, The vector representation of the i-th relational virtual word of relation r, sim represents the similarity between two vectors, α and β are hyperparameters, which are set to 1.2 and 0.7 respectively in the present invention;

[0070]

[0071] In the mask pre-training model RoBERTa-large, the relational virtual words are initialized as follows:

[0072] First, manually construct simple description information for each relationship, then initialize the state of each relationship virtual word, use the mask pre-training model to directly predict the word at each mask position, and then use this word to initialize the relationship virtual word at the corresponding position.

[0073] 3) The final complex relationship representation is obtained based on the representation of each perspective relationship and the generation probability of each perspective relationship representation.

[0074] The present invention provides a low-resource relationship extraction method based on prompt learning. The relationship representation is disassembled into a joint distribution of multiple perspectives through decoupling learning, and virtual words are used to represent these perspectives, so as to convert the description of the relationship from the traditional single vector representation to a joint representation of multiple vectors of multiple perspectives. At the same time, a global-local loss and dynamic initialization method adapted to this representation are designed to achieve efficient low-resource relationship extraction.

[0075] experiment

[0076] Experiments show that the relation extraction method of the present invention can achieve better results than existing mainstream methods. The experiments are evaluated on three datasets: SemEval (a traditional relation extraction dataset, covering 9 active and passive relations and one other relation), TACRED (a large-scale sentence annotation dataset, covering 41 relations and one other relation), and TACREV (a large-scale sentence annotation dataset, covering 41 relations and one other relation).

[0077] Table 1 Dataset information

[0078] Dataset Train Dev Test Relation SemEval 6507 1493 2717 19 TACRED 68124 22631 15509 42 TACREV 68124 22631 15509 42

[0079] The experimental part aims to evaluate the effectiveness of the low-resource relationship extraction method proposed in this invention. The relationship extraction index is evaluated by the harmonic mean score of the accuracy and recall of the relationship classification (hereinafter referred to as F1). Among them, K is responsible for simulating the number of each category in the low-resource scenario, and K equals 1, 5, and 16, respectively, indicating that the number of each relationship category is only 1, 5, and 16. The MVRE band is used here to represent the model proposed in this invention. GL and DI respectively represent the global-local loss and dynamic initialization method of the supporting modules proposed in this invention, and SI represents the static initialization method. MVRE (w / o GL&DL) represents the model that does not use these two supporting modules.

[0080] Table 2 Indicator results for low resource scenarios

[0081]

[0082] From the index results of low-resource scenarios of various data sets, it can be observed that the method proposed by the present invention outperforms all other models in low-resource scenarios, especially in extremely low-resource scenarios, which shows that the design of multi-perspective decoupling learning of relationships is effective. At the same time, when the supporting modules are removed, the present invention can also show good results and is still ahead of other benchmark models.

[0083] Table 3 Effect analysis of each technology of the present invention

[0084]

[0085] It can be seen from Table 3 that the design of each part of the present invention is effective and necessary. In particular, the dynamic initialization technology (DI) has a huge increase in the final effect of the method of the present invention. This shows that using the model's original guess information of multiple masks in a sentence is conducive to multi-view decoupling learning of relationships.

[0086] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A relation extraction method based on prompt learning in low-resource scenarios, characterized in that: The following steps are involved: 1) Decompose complex relationships into joint representations of multiple perspectives, including characters, locations, actions, and active and passive relationships; Assume x i Represents the sentence text to be queried, which contains the head entity w to be queried s and tail entity w o ,y i Indicates the relationship between the head entity and the tail entity, i represents the sequence number; for a single sample pair (x i ,y i ) It is decomposed into a joint representation of multiple perspective relations h, as follows: Among them, p represents probability; Here we assume that the relational representation of a sample pair can be decomposed into a joint representation of many sub-representations, where the sub-representations are the various perspectives of the relation, p(h|x i ) are the posterior probabilities of these perspectives; 2) Use virtual words to represent the perspective relationships obtained by disassembly; 2.1) Represent the relationship between each perspective; When using prompts to learn, you need to ask the sentence text x i , converted into the format T(x i )=x i :w s :[MASK]:w o , where the colon ":" indicates string concatenation; Use the cloze function of the upstream pre-trained model to obtain the latent space representation h of the mask. x =M(T(x i )) [MASK] ; Decoding and this relationship y i Related words v(y i ); 2.2) Sampling the representations of each view relationship; 2.3) Use the mask pre-training model to create m different relational virtual words v for each relation r in its vocabulary 1...m (r), each relational subjunctive focuses on a different perspective of a relation; Among them, the multi-view mask and its representation are as follows: Use multiple consecutive [MASK] as the decomposition method, and then follow the sentence order from left to right to the i-th [MASK] corresponding to the virtual word v i (r), the specific expression is as follows: T(x i )=x i :w s :[MASK] 1...m :w o ; Using the posterior probability Represents the generation probability of each view relationship representation, and uses a linear layer to learn to obtain the posterior probability The formula is as follows: in, Represents x i The j-th relation representation from the perspective of ; 3) The final complex relationship representation is obtained based on the representation of each perspective relationship and the generation probability of each perspective relationship representation.

2. According to claim 1, the relationship extraction method based on prompt learning in low-resource scenarios is characterized in that: In step 2.3), the mask pre-training model uses RoBERTa-large as the training model.

3. The method for relation extraction based on prompt learning in low-resource scenarios according to claim 2, characterized in that: The loss function used in the training of the RoBERTa-large model in step 2.3) is as follows: in, The loss function corresponding to each data pair; 4. The method for relation extraction based on prompt learning in low-resource scenarios according to claim 3 is characterized in that The loss function used in the training of the RoBERTa-large model in step 2.3) is as follows: in, is the local loss function, is the global loss function; in, represents the set of all relations, is the vector representation of the i-th relational virtual word of relation r, is the vector representation of the j-th relational virtual word of relation r, sim represents the similarity between two vectors, and α and β are hyperparameters.

5. The method for relation extraction based on prompt learning in low-resource scenarios according to claim 2, characterized in that: In the mask pre-training model RoBERTa-large, the relational virtual words are initialized as follows: First, manually construct simple description information for each relationship, then initialize the state of each relationship virtual word, use the mask pre-training model to directly predict the word at each mask position, and then use this word to initialize the relationship virtual word at the corresponding position.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images