Relation extraction method and system in low-resource scenarios based on neural network

Abstract

This invention provides a method and system for relation extraction in low-resource scenarios based on neural networks, comprising: step S1: constructing a neural network model based on a dual-tower structure; step S2: training the dual-tower structure neural network model; step S3: using the trained dual-tower structure neural network model to infer the relationships between entities in a sentence. This invention provides a method for relation extraction in low-resource scenarios and improves the accuracy of relation extraction in such scenarios.

Description

Technical Field

[0001] This invention relates to the field of relation discrimination technology in deep learning and natural language processing. Specifically, it relates to a relation extraction method and system based on neural networks in low-resource scenarios. More specifically, it relates to a method based on a dual-tower model to improve the accuracy of relation extraction in low-resource scenarios. Background Technology

[0002] Deep learning is a new field that has evolved from machine learning and artificial intelligence networks. It's a method for constructing mathematical models to fit data, but because it involves more than one layer of neural network, it's called deep learning. Deep learning refers to multi-layered artificial neural networks and the methods for training them. A single-layer neural network takes a large number of matrix numbers as input, applies weights through non-linear activation methods, and then generates another dataset as output.

[0003] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close relationship with linguistics, but also important differences. NLP does not simply study natural language in general; rather, it focuses on developing computer systems, especially the software systems, that can effectively achieve natural language communication. Therefore, it is a part of computer science.

[0004] Patent document CN113901230A (application number: 202111079384.7) discloses a relation extraction method for neural networks. Based on the relation extraction model of word dependency information type mapping memory neural network, it further models the context information by utilizing the context-related words and the dependency relationship types between words, and distinguishes and models the context features.

[0005] Low-Resource Relation Extraction. Relation extraction is a task in information extraction that aims to find the relationship between two entities (entity pairs) in a sentence. "Low-resource" refers to the situation in deep learning where there is insufficient training data. In real life, it is difficult to obtain large amounts of training data; often only a small amount is available for training, and the model performs poorly in such cases. The same applies to relation extraction; low-resource relation extraction is performed when only a small amount of training data is available. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for relation extraction in low-resource scenarios based on neural networks.

[0007] According to the present invention, a relation extraction method based on neural networks in low-resource scenarios includes:

[0008] Step S1: Construct a neural network model based on a dual-tower structure;

[0009] Step S2: Train the dual-tower structure neural network model;

[0010] Step S3: The corpus is used to infer the relationships between entities in the sentence using a trained dual-tower neural network model.

[0011] Preferably, the neural network model with the dual-tower structure includes: a first BERT encoder and a second BERT encoder;

[0012] The first BERT encoder uses function mapping to embed sentences and entities from the input dataset;

[0013] The second BERT encoder uses function mapping to embed the relationships in the input dataset.

[0014] Preferably, step S2 employs:

[0015] Step S2.1: Use the abundant relation extraction corpus to pre-train the dual-tower structure neural network model to obtain the pre-trained dual-tower structure neural network model;

[0016] Step S2.2: Use low-resource relation extraction corpora from different domains to train the pre-trained dual-tower structure neural network model to obtain the trained dual-tower structure neural network model.

[0017] Preferably, step S3 employs the following methods:

[0018] Step S3.1: Divide the statement into sentences, entities within sentences, and candidate relations between entities;

[0019] Step S3.2: Input the sentence and the entities in the sentence into the trained first BERT encoder to obtain the embedded representation of the sentence;

[0020] Step S3.3: Input the candidate relations between entities into the trained second BERT encoder to obtain the embedded representation of the relation name;

[0021] Step S3.4: Perform a dot product operation on the embedded representation of the statement and the embedded representation of the relation name to obtain matrix M;

[0022] Step S3.5: Perform a Softmax operation on matrix M to obtain matrix P;

[0023] Step S3.6: The relationship corresponding to the dimension with the largest value in matrix P is the predicted relationship.

[0024] According to the present invention, a relation extraction system based on a neural network for low-resource scenarios includes:

[0025] Module M1: Constructs a neural network model based on a dual-tower structure;

[0026] Module M2: Trains a dual-tower neural network model;

[0027] Module M3: The corpus uses a trained dual-tower neural network model to infer the relationships between entities in a sentence.

[0028] Preferably, the neural network model with the dual-tower structure includes: a first BERT encoder and a second BERT encoder;

[0029] The first BERT encoder uses function mapping to embed sentences and entities from the input dataset;

[0030] The second BERT encoder uses function mapping to embed the relationships in the input dataset.

[0031] Preferably, the module M2 adopts:

[0032] Module M2.1: Pre-trains the dual-tower structure neural network model using a resource-rich relation extraction corpus to obtain the pre-trained dual-tower structure neural network model;

[0033] Module M2.2: Use low-resource relation extraction corpora from different domains to train a pre-trained dual-tower neural network model to obtain a trained dual-tower neural network model.

[0034] Preferably, the module M3 adopts:

[0035] Module M3.1: Divides statements into sentences, entities within sentences, and candidate relations between entities;

[0036] Module M3.2: Input the sentence and entities in the sentence into the trained first BERT encoder to obtain the embedded representation of the sentence;

[0037] Module M3.3: Input the candidate relations between entities into the trained second BERT encoder to obtain the embedded representation of the relation name;

[0038] Module M3.4: Performs a dot product operation on the embedded representations of statements and the embedded representations of relation names to obtain matrix M;

[0039] Module M3.5: Performs a Softmax operation on matrix M to obtain matrix P;

[0040] Module M3.6: The relationship corresponding to the dimension with the largest value in matrix P is the predicted relationship.

[0041] According to the present invention, a computer-readable storage medium storing a computer program is provided, wherein the computer program, when executed by a processor, implements the steps of the method described above.

[0042] An electronic device according to the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the relation extraction method based on neural networks in low-resource scenarios described above.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. This invention performs better than other models in low-resource scenarios without introducing any additional information;

[0045] 2. This invention provides a method for relation extraction in low-resource scenarios and improves the accuracy of relation extraction in low-resource scenarios.

[0046] 3. Experiments show that, without introducing any additional information, this invention achieves a 10%-20% accuracy improvement over BERT-PAIR on the FewRel2.0 dataset in low-resource scenarios. Attached Figure Description

[0047] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0048] Figure 1 This is a flowchart of a relation extraction method for low-resource scenarios based on neural networks.

[0049] Figure 2 The screenshot shows the detailed algorithm of the relation extraction model proposed in this invention.

[0050] Figure 3 This is a structural diagram of the neural network model proposed in this invention.

[0051] Figure 4 This is a diagram illustrating the effect of the neural network model proposed in this invention. Detailed Implementation

[0052] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0053] This invention provides a relation extraction system and method for low-resource scenarios based on neural networks. It utilizes a neural network model with a dual-tower structure to infer relationships between entities in a sentence. The dual-tower neural network model consists of two pre-trained language models (BERT model in experiments). One language model embeds the sentence, while the other embeds relation names. The two embedded representations are then multiplied by a dot product to obtain the probabilities of each relation, thereby inferring the relationships between entities in the sentence. Training requires two rounds of pre-tuning and fine-tuning; prediction is performed using only the input data.

[0054] Example 1

[0055] According to the present invention, a relation extraction method based on neural networks in low-resource scenarios includes:

[0056] Step S1: Construct a neural network model based on a dual-tower structure;

[0057] Step S2: Train the dual-tower structure neural network model;

[0058] Step S3: The corpus is used to infer the relationships between entities in the sentence using a trained dual-tower neural network model.

[0059] Specifically, such as Figure 3 As shown, the neural network model of the dual-tower structure includes: a first BERT encoder and a second BERT encoder;

[0060] The first BERT encoder uses function mapping to embed sentences and entities from the input dataset;

[0061] The second BERT encoder uses function mapping to embed the relationships in the input dataset.

[0062] Specifically, step S2 employs the following:

[0063] Step S2.1: Use the abundant relation extraction corpus to pre-train the dual-tower structure neural network model to obtain the pre-trained dual-tower structure neural network model;

[0064] Step S2.2: Use low-resource relation extraction corpora from different domains to train the pre-trained dual-tower structure neural network model to obtain the trained dual-tower structure neural network model.

[0065] Specifically, step S3 employs the following:

[0066] Step S3.1: Divide the statement into sentences, entities within sentences, and candidate relations between entities;

[0067] Step S3.2: Input the sentence and the entities in the sentence into the trained first BERT encoder to obtain the embedded representation of the sentence;

[0068] Step S3.3: Input the candidate relations between entities into the trained second BERT encoder to obtain the embedded representation of the relation name;

[0069] Step S3.4: Perform a dot product operation on the embedded representation of the statement and the embedded representation of the relation name to obtain matrix M;

[0070] Step S3.5: Perform a Softmax operation on matrix M to obtain matrix P;

[0071] Step S3.6: The relationship corresponding to the dimension with the largest value in matrix P is the predicted relationship.

[0072] According to the present invention, a relation extraction system based on a neural network for low-resource scenarios includes:

[0073] Module M1: Constructs a neural network model based on a dual-tower structure;

[0074] Module M2: Trains a dual-tower neural network model;

[0075] Module M3: The corpus uses a trained dual-tower neural network model to infer the relationships between entities in a sentence.

[0076] Specifically, the neural network model with the dual-tower structure includes: a first BERT encoder and a second BERT encoder;

[0077] The first BERT encoder uses function mapping to embed sentences and entities from the input dataset;

[0078] The second BERT encoder uses function mapping to embed the relationships in the input dataset.

[0079] Specifically, module M2 adopts:

[0080] Module M2.1: Pre-trains the dual-tower structure neural network model using a resource-rich relation extraction corpus to obtain the pre-trained dual-tower structure neural network model;

[0081] Module M2.2: Use low-resource relation extraction corpora from different domains to train a pre-trained dual-tower neural network model to obtain a trained dual-tower neural network model.

[0082] Specifically, module M3 adopts:

[0083] Module M3.1: Divides statements into sentences, entities within sentences, and candidate relations between entities;

[0084] Module M3.2: Input the sentence and entities in the sentence into the trained first BERT encoder to obtain the embedded representation of the sentence;

[0085] Module M3.3: Input the candidate relations between entities into the trained second BERT encoder to obtain the embedded representation of the relation name;

[0086] Module M3.4: Performs a dot product operation on the embedded representations of statements and the embedded representations of relation names to obtain matrix M;

[0087] Module M3.5: Performs a Softmax operation on matrix M to obtain matrix P;

[0088] Module M3.6: The relationship corresponding to the dimension with the largest value in matrix P is the predicted relationship.

[0089] According to the present invention, a computer-readable storage medium storing a computer program is provided, wherein the computer program, when executed by a processor, implements the steps of the method described above.

[0090] An electronic device according to the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the relation extraction method based on neural networks in low-resource scenarios described above.

[0091] Example 2

[0092] Example 2 is a preferred example of Example 1.

[0093] Implemented on the FewRel 2.0 dataset. The comparison model is BERT-PAIR, a relation extraction model introduced with FewRel 2.0. The evaluation metric is accuracy.

[0094] like Figure 1-2 As shown, the twin-tower model of the present invention includes the following steps:

[0095] Step 1: The model needs to be pre-tuned first. The training data provided by FewRel2.0 is processed. The input data consists of three types: sentences, entities in the sentences (there are usually two entities, referred to as entity 1 and entity 2), and candidate relationships between these entities. The entities in the sentences are concatenated into entity pairs by English semicolons, i.e. entity 1 + ';' + entity 2.

[0096] Step 2: Input the sentence as Segment1 and the entity pair as Segment2 into the pre-trained language model 1 to obtain Embedding1. Then, directly input all candidate relation names into the pre-trained language model 2 to obtain Embedding2.

[0097] Step 3: Perform a dot product operation on Embedding1 and Embedding2 to obtain matrix M.

[0098] Step 4: Perform a Softmax operation on matrix M to obtain matrix P.

[0099] Step 5: Calculate the cross-entropy L between matrix P and the one-hot vector corresponding to the true relationship, and use L as the loss function.

[0100] Step 6: Update the model parameters in the direction of reducing loss L.

[0101] Step 7: After completing the above steps, the FewRel 2.0 test set provides training data for low-resource learning. Use this as training data for fine-tuning. Repeat steps 1-6.

[0102] Step 8: When the model makes predictions, the input data is replaced with the prediction data. Repeat steps 1-4. The largest correspondence in P is the relationship predicted by the model.

[0103] Experiments show that, without introducing additional information, this invention achieves a 10%-20% accuracy improvement over BERT-PAIR on the FewRel2.0 dataset in low-resource scenarios. Figure 4 As shown.

[0104] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0105] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A relation extraction method based on neural networks in low-resource scenarios, characterized in that, include: Step S1: Construct a neural network model based on a dual-tower structure; Step S2: Train the dual-tower structure neural network model; Step S3: The corpus is used to infer the relationships between entities in the sentence; The neural network model with the dual-tower structure includes: a first BERT encoder and a second BERT encoder; The first BERT encoder uses function mapping to embed sentences and entities from the input dataset; The second BERT encoder uses function mapping to embed the relationships in the input dataset; Step S2 employs the following: Step S2.1: Use the abundant relation extraction corpus to pre-train the dual-tower structure neural network model to obtain the pre-trained dual-tower structure neural network model; Step S2.2: Use low-resource relation extraction corpora from different domains to train the pre-trained dual-tower structure neural network model to obtain the trained dual-tower structure neural network model.

2. The relation extraction method based on neural networks in low-resource scenarios according to claim 1, characterized in that, Step S3 employs the following: Step S3.1: Divide the statement into sentences, entities within sentences, and candidate relations between entities; Step S3.2: Input the sentence and the entities in the sentence into the trained first BERT encoder to obtain the embedded representation of the sentence; Step S3.3: Input the candidate relations between entities into the trained second BERT encoder to obtain the embedded representation of the relation name; Step S3.4: Perform a dot product operation on the embedded representation of the statement and the embedded representation of the relation name to obtain matrix M; Step S3.5: Perform a Softmax operation on matrix M to obtain matrix P; Step S3.6: The relationship corresponding to the dimension with the largest value in matrix P is the predicted relationship.

3. A relation extraction system based on neural networks for low-resource scenarios, characterized in that, include: Module M1: Constructs a neural network model based on a dual-tower structure; Module M2: Trains a dual-tower neural network model; Module M3: The corpus is used to infer the relationships between entities in sentences using a trained dual-tower neural network model; The neural network model with the dual-tower structure includes: a first BERT encoder and a second BERT encoder; The first BERT encoder uses function mapping to embed sentences and entities from the input dataset; The second BERT encoder uses function mapping to embed the relationships in the input dataset; The module M2 adopts: Module M2.1: Pre-trains the dual-tower structure neural network model using a resource-rich relation extraction corpus to obtain the pre-trained dual-tower structure neural network model; Module M2.2: Use low-resource relation extraction corpora from different domains to train a pre-trained dual-tower neural network model to obtain a trained dual-tower neural network model.

4. The relation extraction system based on neural networks in low-resource scenarios according to claim 3, characterized in that, The module M3 adopts: Module M3.1: Divides statements into sentences, entities within sentences, and candidate relations between entities; Module M3.2: Input the sentence and entities in the sentence into the trained first BERT encoder to obtain the embedded representation of the sentence; Module M3.3: Input the candidate relations between entities into the trained second BERT encoder to obtain the embedded representation of the relation name; Module M3.4: Performs a dot product operation on the embedded representations of statements and the embedded representations of relation names to obtain matrix M; Module M3.5: Performs a Softmax operation on matrix M to obtain matrix P; Module M3.6: The relationship corresponding to the dimension with the largest value in matrix P is the predicted relationship.

5. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 2.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed by the processor, it implements the steps of the relation extraction method based on neural networks in low-resource scenarios as described in any one of claims 1 to 2.

Images