Unified low sample relation extraction method and device based on multiple selection matching network

Abstract

The application discloses a unified low sample relation extraction method and device based on a multiple-choice matching network. The method comprises the following steps: a common coding and matching mechanism based on a pre-training language model and multiple-choice marking relation description and relation instance; a triple obtained through open information extraction of large-scale pure text and a paraphrase text generated through a generative pre-training language model, and a triple-paraphrase pre-training mode based on the same; and an online meta-learning training mode based on a small sample under a new task. The mechanism based on the multiple-choice matching network can model various scenes in the low sample relation extraction task, and provides an efficient and fast network architecture, so that the model is more in line with the multiple requirements of model performance and speed in actual application.

Description

Technical Field

[0001] This invention relates to a low-sample relation extraction method, and more particularly to a low-sample relation extraction method and apparatus based on a multi-select matching network and its pre-training and online training methods, belonging to the field of natural language processing technology. Background Technology

[0002] Relation extraction is an important task in information extraction and natural language understanding. This task aims to identify the type of relationship expressed by a pair of entities, given a context. For example, given the context "A is the founder of Company B," and the entity pair "A" and "Company B," a relation extraction system should be able to correctly identify the entity pair expressing the relationship "is the founder of...".

[0003] In recent years, with the development of deep learning methods, the accuracy of relation extraction systems has been greatly improved. However, these methods require a large amount of high-quality labeled data for model training, resulting in excessive labeling costs. Furthermore, the models obtained using these methods have poor generalization ability and cannot be transferred to new relation categories and domains.

[0004] Therefore, low-shot relation extraction has received widespread attention in recent years. This task aims to transfer knowledge from existing relation extraction models to target relation extraction tasks with only a few labeled samples, or even no labeled samples. Common low-shot relation extraction tasks include zero-shot relation extraction and few-shot relation extraction. In zero-shot relation extraction tasks, the model needs to generalize the existing relation extraction model to the target task with only additional knowledge (usually descriptions or names of relation categories) and no labeled instances. Few-shot relation extraction models, on the other hand, rely on only a small number of labeled instances (usually 1 or 5 per new relation category) to generalize to the target relation extraction task.

[0005] Traditional unified low-sample relation extraction can be divided into two main categories. The first category models relation extraction as a text entailment task, which involves concatenating and co-encoding text containing relation category information or labeled instances with instances to be classified, and then determining whether they match to extract the relation. The second category models relation extraction as a machine reading comprehension task, which uses templates to generate questions containing relation category information and queries the instances to be classified. If the generated or extracted answer is correct, it is considered to express the corresponding relation category.

[0006] However, the above methods suffer from high computational complexity at the encoding or decoding end, resulting in excessive computational cost for relation extraction and making it difficult for the program to run at the required speed. Summary of the Invention

[0007] To overcome the problem of high computational cost in existing unified low-sample relation extraction methods, this invention proposes a unified relation extraction method and apparatus based on a multi-select matching network. The relation extraction is modeled as a computationally convenient multi-select matching task, and the multi-select matching network is optimized and trained using a task-specific pre-training method, thereby avoiding the problem of high model computational cost.

[0008] To achieve the above objectives, the specific technical solution adopted by the present invention is as follows:

[0009] A unified low-sample relation extraction method based on multi-select matching networks includes the following steps:

[0010] Concatenate all the relationship descriptions of the current task into a single multiple-choice statement;

[0011] The instances to be classified are combined with the above multiple-choice statements and input together into the encoder of the multiple-choice matching network;

[0012] The representation of the instance to be classified after being encoded by the encoder is matched with the representation of each relation in the multiple-choice statement, and the relation with the highest similarity is used as the prediction result to obtain the relation extraction result.

[0013] Furthermore, the trained multi-select matching network is obtained through the following steps:

[0014] a) In large-scale open-domain text, the (subject, predicate, object) triples are extracted using the OpenIE tool, where each part of the triple is a text fragment from the original text.

[0015] b) The extracted triples are used to generate corresponding paraphrased texts containing the semantics of the triples through a pre-trained generative language model.

[0016] c) Extract the predicates from the above triples as relation descriptions for the corresponding generated paraphrased text. Then, using a negative sampling strategy, sample multiple negative relation examples for each instance. Perform relation prediction and extraction according to the multi-select matching network method described above, and calculate the relation extraction loss function. Here, "negative relation example" refers to the semantic relation expressed by the predicates in the negative triples.

[0017] d) For each new task containing a small number of labeled samples, use the labels of these small number of labeled samples to calculate the loss function of the relation extraction result output by the multi-select matching network, and update the model of the multi-select matching network through online learning.

[0018] Furthermore, the encoder includes models such as Bert and RoBERTa.

[0019] Furthermore, the large-scale open-domain text includes the text contained in Wikipedia articles.

[0020] Furthermore, the OpenIE tool includes: the open-source OpenIE 5.0 tool.

[0021] Furthermore, the generative language models include models such as T5 and BART.

[0022] A unified low-sample relation extraction device based on a multi-select matching network employing the above method includes:

[0023] The model training module is used to train the multi-select matching network;

[0024] The relation extraction module is used to concatenate all relation descriptions of the current task into a multiple-choice statement, merge the instance to be classified with the multiple-choice statement, and input them into the encoder of the trained multiple-choice matching network. The representation of the instance to be classified after being encoded by the encoder is matched with the representation of each relation in the multiple-choice statement, and the relation with the highest similarity is taken as the prediction result to obtain the relation extraction result.

[0025] Compared with the prior art, the positive effects of the present invention are:

[0026] 1) Based on the multi-select matching network, the computational cost and speed of relation extraction models can be greatly reduced.

[0027] 2) Based on the text-triple-text AND training method, it can make full use of large-scale text data, fully initialize the model, and enhance the generalization of the model.

[0028] 3) Based on online learning training, it can make full use of a small number of labeled samples in downstream tasks to learn the model online, further fit the current task, and improve model performance. Attached Figure Description

[0029] Figure 1 This is a model diagram of a multi-select matching network.

[0030] Figure 2 This is a framework diagram for pre-training and online training methods for multi-select matching networks. Detailed Implementation

[0031] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, and to make the objectives, features, and advantages of the present invention more apparent and understandable, the core technologies of the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0032] The unified low-sample relation extraction method based on multi-select matching network of the present invention includes the following key steps:

[0033] 1) A unified low-sample relation extraction model based on multi-select matching network is used to model relation extraction as a multi-select matching task, which greatly reduces the computational complexity of the model.

[0034] 2) Based on the above model, a text-triple-text pre-training method is proposed, which utilizes large-scale unsupervised text to pre-train the model. Simultaneously, a small number of labeled instances included in the new task are used to further train the model online.

[0035] 3) Based on the model obtained above, perform multi-select matching on instances under the new task to obtain the results of low-sample relationship extraction.

[0036] This invention relates to the following key elements:

[0037] 1. Multi-select matching network for low-sample relation extraction

[0038] In low-sample relation extraction tasks, most methods are based on textual entailment or machine reading comprehension paradigms. These paradigms typically require each relation and each instance to be encoded and extracted separately, resulting in high computational complexity and slow speed, making the models difficult to apply in practice.

[0039] To address this, this invention proposes a more efficient unified low-sample relation extraction model: a multi-select matching network. The multi-select matching network is a neural network model that matches the instance to be classified with multiple options and selects the most similar option as the output. The encoder is a module within the multi-select matching network. By modeling relation extraction as a multi-select matching task, this multi-select matching network significantly reduces the computational complexity of the model and improves its parallel efficiency. Figure 1 As shown, for any new relation extraction task, all target relations are first concatenated using the special identifier "[choice]" to form a multi-choice statement for each query instance. For each query instance to be classified, the head and tail entities are distinguished from other characters by inserting markers on both sides "[e1][ / e1]" and "[e2][ / e2]" respectively, and the processed instance is then appended to the multi-choice statement.

[0040] For example, for the task of extracting three types of relationships ("is an employee of...", "is the CEO of...", "other"), and the query instance "Cook is the CEO of Apple", it is processed as: "[cls][choice]is an employee of...[choice]is the CEO of...[choice]other[sep][e1]Cook[ / e1]is[e2]the CEO of Apple[ / e2]". Then, this processed instance is input into the encoder module of the multi-select matching network for encoding.

[0041]

[0042] Where h is the representation vector of each character after being encoded by the encoder, and d is the dimension of the vector.

[0043] After encoding, the average of the representations of the head and tail entities is used as the representation of the entire instance:

[0044] h ins =avg(h [e1] h [e2] )

[0045] Where h [e1] and h [e2] These are the encoded representation vectors of the head and tail entities, respectively.

[0046] The special identifier "[choice]" in the multi-choice statement is used as the representation of each candidate relation. The similarity between the candidate relation representation and the instance representation is calculated separately.

[0047]

[0048] Where D(.) is the distance function, Represent the candidate relations and calculate the probability of each relation category based on similarity:

[0049]

[0050] Where θ is the model parameter, N is the number of relation categories, z is a multi-select statement containing all relation information, and y i Let P(y) represent the i-th relation category. i |x,z;θ) indicates that the current paraphrase instance x belongs to the relation y. i The probability of the relationship is calculated. Finally, the relationship with the highest similarity is selected as the prediction result, and the relationship extraction result is output:

[0051]

[0052] 2. Pre-training method based on triplet-rehearsal

[0053] To leverage large-scale unlabeled corpora, improve the generalization of multi-choice matching networks, and provide the model with optimized initialization parameters, this invention proposes a corresponding triplet-paraphrasing pre-training method. Since large-scale corpora lack corresponding relation category annotations, and pre-training on specific relation categories can easily lead to model overfitting, a triplet extraction method is used. The predicates within the triplets are used as relation descriptions, resulting in a large number of diverse relation descriptions. Subsequently, based on the extracted triplets, corresponding paraphrased text is generated, and triplet-paraphrasing pre-training is performed accordingly. Specifically... Figure 2 As shown in (a) and (b).

[0054] 2.1 Large-scale triplet extraction

[0055] This section extracts a large number of triples from large-scale plain text. The extraction tool used is the OpenIE open-source tool. OpenIE is designed to annotate triples from plain text, including subject, verb, and object, all of which are derived from corresponding segments of the plain text. In linguistics, the verb expresses the property or relationship between the subject and object, and therefore can also be regarded as a specific relation description.

[0056] 2.2 Paraphrasing Text Generation

[0057] This section utilizes a generative pre-trained language model to generate corresponding text from the extracted triples, resulting in paraphrased sentences, i.e., paraphrased text, such as... Figure 2 As shown in (a). Specifically, each of the above triples is first concatenated with "[H]", "[R]", and "[T]" before the subject, predicate, and object, respectively, to obtain a new structured text. This structured text is then input into a pre-trained generator (i.e., a generative language model) to generate the corresponding paraphrased text. Subsequently, the generated paraphrased text and the extracted predicates are used to pre-train a multi-select matching network.

[0058] 2.3 Model Pre-training

[0059] For the large amount of paraphrased text and triples obtained above, a meta-learning approach is used to pre-train the multi-select matching network, such as... Figure 2 As shown in (b). Specifically, N triplet-restatement pairs are randomly sampled in each training round. The predicates in these N triplets are used as relation descriptions, thus forming a new extraction task containing N relation categories. These are then concatenated into a multiple-choice statement using a multiple-choice matching network. Subsequently, each restated text is concatenated with the multiple-choice statement and input into the multiple-choice matching network to obtain the corresponding extraction result. The loss function of the multiple-choice matching network is calculated based on the actual corresponding predicates.

[0060]

[0061] Where θ represents the model parameters to be learned, (x,y) represents the restatement-triplet pair, and I(.) is the indicator function, when y i I(.) is 1 when it corresponds to the predicate of the restated text x, and 0 otherwise. z is a multiple-choice statement containing all relational information. i Let P(y) represent the i-th relation category, N represent the number of relation categories mentioned above, and P(y) represent the number of relation categories mentioned above. i |x,z;θ) indicates that the current paraphrase instance x belongs to the relation y. i The probability of.

[0062] In addition, this invention utilizes a labeled relation extraction dataset for further original training and adds a "None of the above" category as an option for other relation category instances, thereby enhancing the robustness of the model.

[0063] 3. New task prediction based on online meta-learning

[0064] For each new relation extraction task, to fully utilize the limited number of labeled samples it may contain, this invention employs online learning to further generalize and train the model for each new task. Specifically, for each new task, there are N categories and K labeled samples that each category may contain. These N*K samples are fed into a multi-select matching network in mini-batches, and the corresponding loss function is calculated based on their prediction results to update the model. When predicting the next task, the model parameters are reset to their pre-online learning parameters, and then online learning for that task is performed again. Figure 2 As shown in (c), where D represents a large-scale pre-trained dataset constructed through triplet extraction and paraphrasing text generation, (x D ,y D ) represents an instance x contained in the pre-training dataset D. D Its corresponding relation description y D L D Let θ* represent the loss function of the multi-choice matching network trained on the pre-training dataset, and let θ* represent the model parameters of the multi-choice matching network. R ,y R ) represents an instance x in an online learning task. R And its corresponding relationship description y R Q R L represents the set of instances to be extracted from the online learning task. R f represents the training loss function of the multi-choice matching network on the labeled dataset for this task. θ f represents the mapping function represented by the multi-choice matching network during the pre-training phase. θ* This represents the mapping function used by the multiple-choice matching network during the online learning phase. This represents the update gradient of the loss function of the multi-choice matching network in dataset D during the pre-training phase. This represents the update gradient of the loss function of the multi-select matching network on dataset R during the online learning phase.

[0065] Example:

[0066] For the relation extraction task containing the following three relation categories: "employee", "employer" and "investor", with the instance to be classified as "Cook is the CEO of Apple", we will process it accordingly and extract the relation through a multi-select matching network.

[0067] Implementation:

[0068] (i) Describe all the relationships between the current task's objectives. Concatenate these into a multiple-choice statement:

[0069] [choice] Employee [choice] Employer [choice] Investor [choice] Other

[0070] (ii) Concatenate the instances to be classified with the multi-select statement:

[0071] [choice]Employees[choice]Employers[choice]Investors[choice]Others[sep][e1]Cook[ / e1]is[e2]Apple[ / e2]CEO

[0072] (iii) Input the results of the above processing into the multi-select matching network, and calculate the similarity between the instance to be classified and each category. The most similar relationship is the prediction result. In this embodiment, the prediction result is "employer".

[0073] Experimental data: Using the FewRel dataset, we tested the zero-shot relation extraction task, the few-shot relation extraction task, and the few-shot relation extraction task with rejection. The test results are shown in Table 1. The accuracy of our invention is higher than that of other comparative models in all tasks.

[0074] Table 1. Test Results

[0075]

[0076] Other embodiments of the method of the present invention include, but are not limited to, paraphrasing generation using the T5 model.

[0077] Other embodiments of the method of the present invention include, but are not limited to, using special characters such as “[cls]”, “[choice]”, “[sep]”, “[e1]”, and “[e2]” to process text in order to mark sentences, relationships and entities.

[0078] Based on the same inventive concept, another embodiment of the present invention provides a unified low-sample relation extraction device based on a multi-selection matching network employing the above-described method, comprising:

[0079] The model training module is used to train the multi-select matching network;

[0080] The relation extraction module is used to concatenate all relation descriptions of the current task into a multiple-choice statement, merge the instance to be classified with the multiple-choice statement, and input them into the encoder of the trained multiple-choice matching network. The representation of the instance to be classified after being encoded by the encoder is matched with the representation of each relation in the multiple-choice statement, and the relation with the highest similarity is taken as the prediction result, that is, the relation extraction result is obtained.

[0081] Based on the same inventive concept, another embodiment of the present invention provides an electronic device (computer, server, smartphone, etc.) including a memory and a processor, wherein the memory stores a computer program configured to be executed by the processor, and the computer program includes instructions for performing the steps of the method of the present invention.

[0082] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk), which stores a computer program that, when executed by a computer, implements the various steps of the method of the present invention.

[0083] The embodiments described above are merely illustrative of implementation methods of the present invention, and while the descriptions are detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A unified low-sample relation extraction method based on a multiple-choice matching network, characterized by, The method comprises the following steps: Splice all the relationship descriptions of the current task into a multiple-choice sentence; Merge the to-be-classified instance and the multiple-choice sentence, and input them into the encoder of the multiple-choice matching network; Match the representation of the to-be-classified instance after encoding by the encoder with the representation of each relationship in the multiple-choice sentence, take the relationship with the highest similarity as the prediction result, and obtain the relationship extraction result; The multiple-choice matching network is trained by the following steps based on the triple-rephrase pre-training method: a) In large-scale open domain text, (subject, predicate, object) triples are extracted by an OpenIE tool, and each part of the triple is a text segment in the original text; b) A large number of triples extracted are generated into corresponding rephrased texts containing triple semantics by a pre-trained generative language model; c) The predicate in the triple is extracted as the relationship description of the corresponding generated rephrased text, and a plurality of negative example relationships are sampled for each instance by a negative sampling strategy, relationship prediction and extraction are performed in a multiple-choice matching manner, and a loss function of relationship extraction is calculated; a large number of rephrased texts and triples are used for pre-training of the multiple-choice matching network, N triples are randomly sampled in each round of training, the predicates in the N triples are taken as relationship descriptions, a new extraction task containing N relationship categories is constructed, a multiple-choice sentence is spliced according to the multiple-choice matching network, then each rephrased text is spliced with the multiple-choice sentence and input into the multiple-choice matching network to obtain the corresponding extraction result, and the loss function of the multiple-choice matching network is calculated according to the actual corresponding predicate: wherein, is a model parameter to be learned, is a triple-paraphrase pair, is an indicator function, when is the paraphrase text x corresponding to the predicate is 1, otherwise 0; is a multi-choice sentence containing all the relation information; denotes the i th relation category, N denotes the number of relation categories mentioned above, denotes the probability that the current paraphrase instance x belongs to the relation . d) For each new task containing a small amount of labeled samples, the loss function of the relationship extraction result output by the multiple-choice matching network is calculated by using the labels of the small amount of labeled samples, and the model of the multiple-choice matching network is updated through online learning; The processing process of the multiple-choice matching network comprises: For any new relationship extraction task, first, all target relationships are segmented and concatenated by the special identifier "[choice]", which is used as a multiple-choice sentence for each query instance; for each to-be-classified query instance, the head entity and the tail entity are inserted into the two sides with "[e1] [ / e1]" and "[e2] [ / e2]" respectively to distinguish them from other characters, and the processed instance is spliced after the multiple-choice sentence; The processed instance is input into the encoder module of the multiple-choice matching network for encoding; after encoding, the representations of the head and tail entities are averaged to obtain the representation of the entire instance: wherein and are the head and tail entity encoded representation vectors, respectively; The representation of the special identifier "[choice]" in the multiple-choice sentence is taken as the representation of each candidate relationship, and the similarity between the candidate relationship representation and the instance representation is calculated: where D(.) is a distance function, is the representation of the candidate relation and computes the probability of each relation class by similarity. in For model parameters, N For the number of relation categories, This is a multi-select statement that includes all relational information. Indicates the first i Relationship categories, Indicates the current paraphrase instance x Belongs to Relationship The probability of the relationship is calculated; finally, the relationship with the highest similarity is selected as the prediction result, and the relationship extraction result is output: 。 2. The method of claim 1, wherein, The encoder comprises a Bert model and a RoBERTa model.

3. The method of claim 1, wherein, The large-scale open domain text comprises text contained in WikiPedia articles.

4. The method of claim 1, wherein, The OpenIE tool comprises an open-source OpenIE 5.0 tool.

5. The method of claim 1, wherein, The generative language model comprises a T5 model and a BART model.

6. A unified low sample relation extraction device based on multiple choice matching network using the method of any one of claims 1-5. It comprises: A model training module configured to train the multiple-choice matching network. The relationship extraction module is configured to splice all relationship descriptions of the current task into a multiple-choice sentence, combine the instance to be classified with the multiple-choice sentence, input the combined result into an encoder of a trained multiple-choice matching network, match a representation of the instance to be classified after encoding by the encoder with representations of each relationship in the multiple-choice sentence, take a relationship with the largest similarity as a prediction result, and obtain a relationship extraction result.

7. An electronic device, comprising: The computer readable storage medium stores a computer program, and the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method in any one of claims 1-5.

Images