Document relation extraction method based on graph convolution network and relation evidence mutual guidance

Abstract

The application discloses a method for extracting a document relationship based on a graph convolution network and mutual guidance of relationship evidence, and belongs to the technical field of computer natural language processing. The application is aimed at a document-level text, constructs a document graph containing entity mention and entity relationship nodes, and learns relationship representation by using a graph convolution neural network. Meanwhile, the relationship representation is used to guide an evidence set extraction network to extract a corresponding evidence set, and finally, the evidence set representation is integrated into the relationship representation for relationship discrimination. The application is trained by combining a relationship extraction and an evidence set extraction two-channel task. The relationship extraction provides specific relationship information for the evidence set extraction, and the evidence set extraction provides basis information for the relationship extraction, so that the effect of the document relationship extraction is improved. Meanwhile, the relationship nodes are innovatively added in the document graph to promote message transmission between entities, which is beneficial to fully learning semantic representation by the model.

Description

TECHNICAL FIELD

[0001] The application relates to a passage relation extraction method based on a graph convolution network and relation evidence mutual guidance, and belongs to the technical field of computer natural language processing. BACKGROUND

[0002] Relation extraction is a key task in the field of information extraction of natural language processing, and aims to determine the relationship between two entities according to context information. Relation extraction plays a key role in the application fields of constructing large-scale knowledge graph, intelligent question answering, natural language understanding and the like.

[0003] At present, most of the researches on relation extraction methods are concentrated on simple relation extraction at the sentence level, and are divided into supervised learning, semi-supervised learning and unsupervised learning methods according to the amount of labeled data.

[0004] Under a large amount of labeled data, the popular method of supervised learning is a method based on a capsule network. The method is combined with an attention routing algorithm, and effectively solves the multi-label problem in relation extraction. Due to the excellent performance of the pre-training language model in various tasks, the method based on the pre-training language model is very popular. A team led by Chen Danqi uses a labeled seed data sharing position information to make the model predict multiple relationships in a sentence only once, greatly improving the efficiency of relation extraction.

[0005] In reality, the labeled data of the vertical field is often very small. In order to utilize a large amount of unlabeled data, many semi-supervised methods have emerged, such as Bootstrapping which learns to mine unlabeled data from labeled seed data, researchers introduce the teacher-student model framework into the supervised relation extraction task, the student model learns robust knowledge representation from unlabeled data, and the teacher model guides the student model to learn by using labeled relation data. Remote supervision can obtain a large amount of low-quality labeled data based on the assumption that unstructured sentences can represent the existence of a certain relationship between two entities in the knowledge base. Many scholars have carried out research on relation extraction in the direction of remote supervision and noise reduction, such as using reinforcement learning and adversarial learning to reduce noise and improve the robustness of the model.

[0006] The sentence-level relation extraction can only infer the entity relation appearing in a sentence, and the sentence-level relation extraction cannot recall the entity pair appearing in multiple sentences. However, in actual application, the entity relation appearing in multiple sentences accounts for a large proportion, and the document-level relation extraction solves the problem of cross-sentence entity pair relation reasoning. For the entity appearing in multiple sentences, the document-level relation extraction can infer the relation between the two entity pairs through more complex reasoning skills such as logical reasoning, coreference reasoning and common sense reasoning according to the related sentences of the entity pair. Therefore, modeling the complex relation between multiple entities in the document becomes a key problem of the document relation extraction. Most researchers model the semantic information of the entity in the document by using a document graph, and encode the entity node by using a graph convolutional neural network. Different sequence neural networks are also used to model the entity, sentence and document levels, so as to perform document relation extraction. In order to solve the coreference reasoning problem, some people add reference information to the mention to solve the coreference reasoning problem. In order to solve the multi-hop reasoning problem, some people construct an entity graph and introduce the multi-hop path information into the document relation reasoning, thereby improving the logical reasoning effect of the model.

[0007] Although most people construct a document graph by taking the mention, sentence, entity and document as nodes, no one has added a relation node to build a bridge for information transmission between different entities, and the evidence set of the relation is also extracted at the same time. On the one hand, the detailed document information of the evidence set is used to improve the effect of relation extraction, and on the other hand, the relation node of the document graph is used to guide the extraction of the evidence set, so that the document-level relation extraction becomes interpretable. SUMMARY

[0008] The purpose of the present application is to solve the technical problems in the prior art that the document relation extraction model does not fully model the relation between different entities, the evidence set information is not fully utilized and the like, and creatively proposes a document relation extraction method based on graph convolutional network and mutual guidance of relation evidence. The method of the present application adopts the mutual guidance mode of graph convolutional network and relation evidence, can effectively model the relation between different entities, improve the relation extraction effect, and increase the interpretability of the document relation extraction model.

[0009] The innovation of the present application lies in that:

[0010] For the document-level text, a document graph containing entity mention and entity relation nodes is constructed, and a graph convolutional neural network is used to learn the relation representation. At the same time, the relation representation is used to guide the evidence set extraction network to extract the corresponding evidence set, and finally the evidence set representation is integrated into the relation representation for relation discrimination.

[0011] The present application trains the two-channel tasks of relation extraction and evidence set extraction jointly. The relation extraction provides specific relation information for the evidence set extraction, and the evidence set extraction provides basis information for the relation extraction, thereby improving the effect of document relation extraction and increasing the interpretability of the model. At the same time, the relation nodes are innovatively added in the document graph to promote the message passing between entities and facilitate the model to fully learn semantic representations.

[0012] For the convenience of understanding, first, the professional terms involved in the present application are explained.

[0013] Mention: refers to the segment of an entity appearing in a document. It can appear in the form of a noun or a pronoun.

[0014] Document graph: some elements in the document are selected as nodes, and edges are constructed between the nodes through the relationship between the nodes to form an undirected graph.

[0015] Representation: refers to a vector representing the semantics of a text.

[0016] BERT model: full name Bidirectional Encoder Representation from Transformers, which is a pre-trained language representation model.

[0017] [CLS]: a special identifier in the head of the sentence in the BERT model input, representing the semantics of the whole sentence.

[0018] Average pooling: a vector set Do Operation, where x i represents a vector, and n represents the cardinality of the vector set.

[0019] Coreference: refers to different words pointing to the same entity.

[0020] Graph convolutional neural network (GCN): a convolution operation is extended from traditional data (image or grid) to graph data. The key is to learn a function f to generate the representation of node v i by aggregating its own features X i and neighbor features X j , where j∈N(v i ), N represents the set of adjacent nodes of v i .

[0021] Evidence set: a set of sentences proving the entity relationship.

[0022] The present application is realized by the following technical solutions.

[0023] A document relation extraction method based on graph convolution network and mutual guidance of relationship evidence, comprising the following steps:

[0024] Step 1: Divide the document into several sentences by one-sentence ending punctuation marks (such as period, question mark, exclamation mark).

[0025] Step 2: Use the pre-trained language model BERT as the encoder. Encode each sentence with BERT to obtain word representation, sentence representation, mention representation, entity representation and relationship representation.

[0026] Among them, the word representation and the sentence representation are obtained directly through the BERT output vector, which corresponds to the output vector of each word and the [CLS] output vector respectively.

[0027] The mention representation is obtained by performing average pooling operation on the word vectors contained in the mention.

[0028] The entity representation integrates different mention context information, entity category information and mention coreference information. Among them, the entity category representation is obtained by converting the entity category information (such as PER, LOC, ORG, etc.) into entity category representation through an entity category mapping matrix; the mention coreference representation is obtained by converting the entity id corresponding to the mention into mention coreference representation through an entity id mapping matrix; and the entity representation is obtained by performing average pooling on all mention representations of the same entity appearing in the document, and then concatenating the entity category representation and the mention coreference representation.

[0029] The relationship representation is obtained by selecting two entities in the document as the head entity and the tail entity respectively, and then concatenating the representation vectors of the two entities.

[0030] Step 3: Construct a document graph.

[0031] Among them, the nodes include two types: mention nodes and relationship nodes.

[0032] The edges include three types: edges between mentions of the same entity, edges between mentions of different entities in the same sentence, and edges between mentions and relationships.

[0033] Step 4: Use the graph convolutional neural network (GCN) to iterate n layers on the document graph. Since each layer contains different semantic information, the n-layer hidden vectors are concatenated as the final representation of each node.

[0034] Step 5: Extract the evidence set of the relationship according to the relationship representation.

[0035] Specifically, step 5 can include the following steps:

[0036] Step 5-1: Fuse the sentence representation and the relationship representation through the conditional fusion function (Condition Layer Normalization) to obtain the sentence representation that integrates specific relationship information.

[0037] Step 5-2: Perform binary classification on each sentence representation to determine whether the sentence belongs to the evidence set of the relation, thereby obtaining the evidence set under the specific relation.

[0038] Step 6: Weighted summation of sentence representations in the evidence set to obtain the evidence representation.

[0039] Step 7: Concatenate the relation representation and evidence representation obtained through the graph convolutional neural network, and then pass them through a multi-label classifier to obtain the final entity pair relation.

[0040] Beneficial effects

[0041] The method of the present invention has the following advantages compared with the prior art:

[0042] 1. This invention models a document as a document graph and uses graph convolutional neural network encoding to fully learn the relationships between the mentions in the document. The relationship nodes are more conducive to modeling the relationship between two entities.

[0043] 2. In this invention, the evidence set and entity relationship are causal relationships. By exchanging information through two channels of evidence set mining and relationship extraction, the effectiveness of discourse relationship reasoning is improved, and the model is made interpretable. Attached Figure Description

[0044] Figure 1 This is an overall flowchart of the method of the present invention;

[0045] Figure 2 This is a model architecture diagram of the method of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0047] Example

[0048] like Figure 1 As shown, a method for extracting discourse relations based on mutual guidance between graph convolutional networks and relational evidence is presented, and the specific implementation process is as follows.

[0049] The dataset used in this example is the DocRED document extraction dataset constructed by Tsinghua University. @inproceedings{yao2019DocRED.

[0050] title={{DocRED}:A Large-Scale Document-Level Relation ExtractionDataset},

[0051] author = {Yao, Yuan and Ye, Deming and Li, Peng and Han, Xu and Lin, Yankai and Liu, Zhenghao and Liu, Zhiyuan and Huang, Lixin and Zhou, Jie and Sun, Maosong},

[0052] booktitle = {Proceedings of ACL 2019},

[0053] year = {2019}

[0054] }

[0055] This dataset covers the relationship between science, art, time, personal life, etc. 61.1% of the data needs to be extracted by reasoning, and the instances of identifying relationships by simple models only account for 38.9%. This dataset is suitable for the present application

[0056] Step 1: Extract the sample from the dataset. Due to the large number of elements in the set, only a part of it is taken here, and the purpose is to explain and illustrate the format of the dataset.

[0057]

[0058]

[0059] Among them, the vertexSet field is all the mentioned sets in the document, and the same mention pointing to the same entity is in a set. For example, “Nisei” appears in the 3rd sentence and the 10th sentence, but it is the same entity of type “MISC”. The title field is the title of the document. The sents field is the sentence set of the document, and each sentence has been segmented. The labels field is all the relationship sets appearing in the document, r, h, and t represent the relationship type, the relationship head node, and the relationship tail node, respectively, and evidence represents the evidence set needed to derive the relationship by reasoning.

[0060] Step 2: Train the model using the pytorch framework and transformers library. The system uses Ubuntu 20.04 and uses GPU model Geforce GTX 1080Ti*2.

[0061] Step 3: Use BERT to encode the document.

[0062] Step 4: Map each mention set in vertexSet to a unique entity id, map the mention type to a unique type id, and map the id to a feature vector using the Embedding layer in the torch library, thereby obtaining the coreference information representation and entity type representation.

[0063] Step 5: Average pooling on mentions to obtain entity representation, entity representation and coreference information representation, and entity type representation are concatenated to incorporate other information about the entity.

[0064] Step 6: Arbitrarily collocate head entities and tail entities by permutation and combination, and concatenate to obtain the relationship representation.

[0065] Step 7: As shown in Figure 2 , the document graph construction process.

[0066] Through the previous steps, we obtain the entity representations of "Nisei", "David Nutter", and "Fox Mulder", as well as the 6 types of relationship nodes obtained by permutation and combination of the three entities. An entity may contain multiple mentions, such as "Nisei" appearing in sentences 1, 3, 4, and 8. The four nodes are connected to each other. If different entity mentions appear in the same sentence, they are connected, such as "David Nutter" and "Nisei" appearing in the third sentence, with "Nisei" as the reference "it". Finally, the mentions are connected to the relationship nodes associated with them, such as the four connections between the four mentions of Nisei and R2.

[0067] Step 8: Use GCN to iterate three times on the document graph, and concatenate the node values of the three times to obtain the final node representation.

[0068] The specific operation is: where, represents the representation of node u in the l+1 layer of the graph convolution network, E represents the set of edges starting from node u, N e (u) represents the adjacent node of node u under the e-th edge, represents the training parameter of the l-th layer, is the adjacent node of the l-th layer node u under the e-th type edge, represents the training parameter of the l-th layer, RelU() is the activation function, g u is the final representation of node u, represents the representation of node u in the 0th, 1st, and 2nd layers of the graph convolution network, respectively.

[0069] Step 9: Obtain the sentence representation Doc = {h sent1 ,h sent2 ,…,hsentn} and the relationship representation g obtained by GCN ri Conditional fusion is done: where A represents a linear function, h sentn represents the nth sentence representation, h senti represents the ith sentence representation, μ is the mean of Doc, σ is the variance of Doc, Doc represents the set of document sentence representations, ∈ is a slight perturbation, A(), B() are linear transformation functions.

[0070] Step 10: Perform conditional fusion vector h' obtained in step 9 senti Binary classification prediction probability P that belongs to the evidence set senti : P senti = sigmoid(Wh' senti +b), sigmoid is the activation function, W, b are trainable parameters, and the loss function is L1 = -∑ senti∈Doc (y i logP senti +(1-y i )log(1-P senti )) senti , P i represents the probability that the ith sentence belongs to the evidence set, senti represents the ith sentence, y m is the label of the ith sentence, taking values {0, 1}. Thus, the evidence set Evidence = {evi1, evi2, …, evin} is obtained, evi m represents the mth evidence representation of the evidence set, and m is the cardinality of the evidence set.

[0071] Step 11: Pooling the evidence set to obtain evidence set representation information h evi : where m represents the cardinality of the evidence set, evi i represents the ith evidence representation of the evidence set, and Evidence represents the evidence set.

[0072] Step 12: According to the relationship representation and the evidence representation, predict the relationship type P(r|g ri , h evi ) = softmax(W[g ri ; h evi ]+b), where r represents the predicted relationship, softmax is the activation function, W, b are trainable parameters, and the loss function is r k is the possible relationship in the permutation combination of entity pairs in the document, represents the evidence set representation.

[0073] Step 13: During the model training, the total loss function is L = L1 + L2.

[0074] Step 14: In the prediction stage, according to the pre-set threshold θ = 0.5, if P(r|g ri ,h evi )>0.5, it is considered that the head and tail entities belong to the relationship r, otherwise there is no relationship. If the relationship evidence set is empty.

[0075] In order to illustrate the content and implementation method of the present application, the purpose of introducing details in the above examples is not to limit the scope of the claims, but to help understand the method described in the present application. Those skilled in the art should understand that various modifications, changes or replacements of the steps of the best embodiment are possible without departing from the spirit and scope of the present application and the appended claims. Therefore, the present application should not be limited to the content disclosed in the best embodiment and the accompanying drawings.

Claims

1. A method for extracting a document relationship based on a graph convolution network and relationship evidence mutual guidance, characterized in that, For the document-level text, a document graph containing entity mention and entity relationship nodes is constructed, graph convolutional neural network is used to learn relationship representation, meanwhile, relationship representation is used to guide evidence set extraction network to extract corresponding evidence set, finally, evidence set representation is fused into relationship representation for relationship discrimination; The method comprises the following steps: Step 1: end punctuation of a document with a sentence, and divide the document into several sentences; Step 2: using a pre-trained language model BERT as an encoder, encoding each sentence by BERT to obtain word representation, sentence representation, mention representation, entity representation and relationship representation; Wherein, the word representation and the sentence representation are directly obtained through the BERT output vector, and each word output vector and the [CLS] output vector correspond to the [CLS] output vector, and the [CLS] is a special identifier in the BERT model input, representing the semantics of the whole sentence; The mention representation is obtained by average pooling operation on the word vector contained in the mention; The entity representation fuses different mention context information, entity category information and mention coreference information; wherein, the entity category representation is obtained by converting the entity category information into the entity category representation through an entity category mapping matrix; the mention coreference representation is obtained by converting the entity id corresponding to the mention into the mention coreference representation through an entity id mapping matrix; and the entity representation is obtained by performing average pooling on all mention representations of the same entity in the document, and then splicing the entity category representation and the mention coreference representation; The relationship representation is obtained by selecting two entities in the document as head entity and tail entity, and then splicing the representation vectors of the two entities; Step 3: constructing a document graph; Wherein, the nodes include two types: mention nodes and relationship nodes; and the edges include three types: edges between mentions in the same entity, edges between mentions in the same sentence of different entities, and edges between mentions and relationships; Step 4: using a graph convolutional neural network to iterate the document graph for n layers, and splicing the n-layer hidden vectors as the final representation of each node; Step 5: extracting the evidence set of the relationship according to the relationship representation; Step 6: obtaining evidence representation by weighted sum of sentence representation in the evidence set; Step 7: splicing the relationship representation obtained by the graph convolutional neural network and the evidence representation, and obtaining the final entity pair relationship through a multi-label classifier.

2. The method of claim 1, wherein the method is based on a graph convolution network and relationship evidence mutual guidance. In step 4, the graph convolutional neural network is used to iterate the document graph for 3 times, and the final node representation is obtained by splicing the node values of the 3 times, and the specific operation is as follows: wherein, represents the representation of the l+1th layer node u of the graph convolution network, E represents the set of edges starting from the node u, N e (u) represents the adjacent node of the node u under the e-th edge, represents the training parameter of the lth layer, is the adjacent node of the lth layer node u under the e-th type edge, represents the training parameter of the lth layer, RelU() is an activation function, g u is the final representation of the node u, respectively represent the representation of the node u at the 0th, 1st and 2nd layers of the graph convolution network.

3. The method of claim 1, wherein the method is based on a graph convolution network and relationship evidence mutual guidance. Step 5 comprises the following steps: Step 5-1: fusing the sentence representation and the relationship representation through a conditional fusion function to obtain the sentence representation fused with specific relationship information; Step 5-2: performing binary classification on each sentence representation to determine whether the sentence belongs to the evidence set of the relationship, thereby obtaining the evidence set under the specific relationship.

4. The method of claim 1, wherein during model training, a total loss function is L = L1 + L2. BERT gets sentence representation Doc = {h sent1 ,h sent2 ,…,h sentn} and relationship representation g ri from graph convolutional neural network, and then condition fusion is performed: wherein A represents a linear function, h sentn represents the n-th sentence representation, h senti represents the i-th sentence representation, μ is the mean of Doc, σ is the variance of Doc, Doc represents a document sentence representation set, ∈ is a slight disturbance, and A(), B() are linear transformation functions; The conditional fusion vector h' senti The probability P that a binary classification prediction belongs to the evidence set senti : P senti = sigmoid(Wh' senti +b), sigmoid is an activation function, W and b are trainable parameters, and a loss function is L1 = -∑ senti∈Doc (y i logP senti +(1-y i )log(1-P senti )), P senti represents the probability that the ith sentence belongs to the evidence set, senti represents the ith sentence, y i is the label of the ith sentence, and takes values {0, 1}; thus obtaining the evidence set Evidence = {evi1, evi2, …, evi m}, evi m represents the mth evidence representation of the evidence set, and m is the cardinality of the evidence set; pooling the evidence set to obtain evidence set representation information h evi : wherein m represents the cardinality of the evidence set, evi i denotes the i-th evidence representation of the evidence set, and Evidence denotes the evidence set. According to the relation representation and the evidence representation, a relation type P(r|g ri , h evi ) = softmax(W[g ri ; h evi ]+b), wherein r represents a predicted relation, softmax is an activation function, W and b are trainable parameters, and a loss function is r k is a possible relationship in the arrangement combination of the entity pair in the document, denotes an evidence set representation; 5. The method of claim 4, wherein ​ In the prediction phase, according to the pre-set threshold θ = 0.5, if P(r|g ri ,h evi )>0.5, it is considered that the head and tail entities belong to the relationship r, otherwise there is no relationship; if the no-relationship evidence set is empty.

Images