Multi-task document-level relation extraction method and device based on support sentence prediction

Abstract

The present invention is a multi-task document-level relationship extraction method and device based on support sentence prediction, which belongs to the technical field of information extraction in computer natural language processing. The device of the present invention includes a document preprocessing module, a pre-trained language model, a graph network encoding module, a graph propagation module, a support sentence recognition module and a relationship classification module. The method of the present invention includes: dividing the document into sentences and preprocessing; encoding using a pre-trained language model in units of sentences; constructing a graph network according to the document structure and predetermined rules; performing convolution operations on the graph network; finding supporting sentences for entity pairs and obtaining relationship reasoning paths; classifying entity pairs; and pre-training a feedforward neural network for finding entity pairs and a fully connected neural network for relationship classification. The present invention improves the mining of entity relationships of documents in complex contexts and the extraction efficiency of document-level relationships, and can further help achieve the accuracy of various NLP tasks such as information extraction and automatic question answering.

Description

Technical Field

[0001] The present invention belongs to the technical field of information extraction in computer natural language processing, and specifically relates to a multi-task document-level relationship extraction method and device based on support sentence prediction. Background Art

[0002] Relation extraction (RE) is an important task in the field of natural language processing (NLP). Its goal is to identify entities and the relationships between entities from unstructured natural language texts, thereby constructing structured entity-relationship triples. For example, from the sentence "Tsinghua University is located in the neighborhood of Beijing", an entity-relationship triple (Tsinghua University, located, Beijing) can be extracted, where Tsinghua University and Beijing are called entities, and located is the relationship between them. Relation extraction is an important way to automatically acquire knowledge, a basic task for building and expanding knowledge graphs (KGs), and an important foundation for realizing various NLP tasks such as information extraction and automatic question answering.

[0003] Existing relation extraction work mainly focuses on sentence-level relation extraction, that is, relation extraction based on information within a sentence, and various neural network models are also good at encoding sentence-level semantic information. In actual scenarios, there are still a large number of entity relationships that are expressed through multiple sentences. Relation extraction involves multiple entities mentioned in the text and shows complex interrelationships. For example, entity relationship information is extracted from the following text: "Tsinghua University, located in Beijing, is a university directly under the Ministry of Education of the People's Republic of China. Its current president is Academician Qiu Yong." Among them, entity relationship triples such as (Tsinghua University, located in, Beijing) can be easily identified, but (Tsinghua University, president, Qiu Yong) is not easy to identify, because in order to know which school "Qiu Yong" is the president of, you need to first identify which entity the pronoun "its" refers to.

[0004] According to statistics of manually annotated data sampled from Wikipedia, at least 40% of entity relations must be jointly extracted from multiple sentences. In order to extract cross-sentence relations between multiple entities, it is necessary to read and reason about multiple sentences in the document, which is obviously beyond the capabilities of sentence-level relation extraction methods. Therefore, it is imperative to conduct research on document-level relation extraction.

[0005] Existing document-level relationship extraction models generally use words in the input text as nodes to construct a graph network, and then use graph neural network methods such as GNN (Graph Neural Network) on the graph network to propagate features. Finally, the vectors of the two entities are concatenated and the relationship is classified to obtain the relationship between the two entities in the document.

[0006] The research focus of document-level relationship extraction is how to propagate features and capture long-distance relationships between entities. There are many methods for building graph networks, which can be divided into two categories: homogeneous graphs and heterogeneous graphs. In homogeneous graphs, all edges are of the same type, and the same method is used to update node expressions; in heterogeneous graphs, there are multiple types of edges, and different edges use different update strategies when propagating features. Therefore, heterogeneous graphs have the ability to integrate a variety of different information.

[0007] The representative work of document-level relationship extraction by constructing heterogeneous graph networks is the EoG model proposed by Christopoulou et al. (reference [1]). This model defines three types of entities: entity (E), mention (M), and sentence (S). The three types of entities can be connected in pairs, so there are six types of edges in the graph (EE, EM, ES, MS, MM, SS). When constructing a graph network, two entities are not connected, that is, there will be no EE type edge in the graph. The EoG model propagates features through five types of edges other than EE edges and updates the edge status information. In the prediction stage, a path passing through multiple different nodes is constructed between two entities to model the relationship reasoning process. Then, the relationship representation between entities is obtained through the path propagation algorithm and the relationship classification is performed. The GCNN model proposed by Sahu et al. also uses the method of constructing a heterogeneous graph network to extract document-level relationships. Unlike the EoG model, GCNN constructs a separate graph for each type of edge. Each graph can be regarded as an independent layer. The node representation is iteratively updated in units of layers. After the iterative propagation process is completed, the results of each layer are superimposed and the features of the two entities are used for relationship classification.

[0008] The representative work of using homogeneous graph networks for document-level relation extraction is the LSR model proposed by Nan et al. (reference [2]). The LSR model does not distinguish between edge types. The nodes are fully connected, the graph structure is regarded as a latent variable, and the node representation is updated in an end-to-end manner. The model is based on structured attention and uses a variant of the matrix tree theorem to generate task-specific dependency structures to capture non-local interactions between entities. In the process of iteratively updating node representations, the model can dynamically build latent structures based on the previous iteration, so that the model can continuously capture complex interaction information within the document through iterations, improve information aggregation in the entire document, and thus better perform multi-hop relation reasoning. GP-GNN constructs a fully connected homogeneous graph network using entities in the text sequence, does not distinguish between edge types, and by designing a unique edge encoding representation, the model automatically learns edge parameters during the GNN iteration process.

[0009] In document-level relationship extraction, most methods construct graph networks to propagate features and establish connections between two entities that are not in the same sentence. Therefore, how to construct graphs and how to use graph neural networks to propagate features on graphs are one of the key contents of research related to document-level relationship extraction.

[0010] The concept of graph neural network was first proposed by Gori et al. in 2005. It is a neural network model specifically designed to process graph structured data. Using graphs can more accurately and flexibly model data in real-world applications, such as user-product interaction graphs in e-commerce, molecular graphs in chemistry, and drug side effect graphs in medicine. Therefore, researchers have designed a variety of graph neural network models, including graph convolutional networks (GCNs), graph attention networks (GATs), etc.

[0011] The input text of document-level tasks is generally much longer than that of sentence-level tasks, so deep learning models that are good at processing text sequences, such as RNN (recurrent neural network) and LSTM (long short-term memory network), will lose the ability to summarize the features of the beginning of the input text due to the long dependencies. Although LSTM has taken measures such as forget gates to alleviate this problem, the problem is still serious when the text is too long. Graph neural networks can establish connections between two entities that are very far apart in the text sequence by constructing graphs, thereby greatly alleviating the problems of sequence-based deep learning models. Therefore, graph neural networks have gained many applications in tasks with long input texts. For example: C-GCN is a context-based GCN model (reference [3]), which prunes the dependency tree of a sentence and only retains the dependency path part of the entity pair; the attention-based graph convolutional network (AGGCN) uses the GCN model on the dependency tree of a sentence (reference [4]), and adopts a multi-head attention mechanism to select edges (dependencies) in a soft-weighted manner; Zhang et al. use GCN to encode the hierarchical structure of relations and propose a coarse-to-fine-grained knowledge-aware attention mechanism to improve the effect of relation extraction (reference [5]).

[0012] The existing document-level relationship extraction models have the following shortcomings: 1. Since the scale of input words required for a single extraction process of a document-level relationship extraction task is much larger than that of a sentence-level relationship extraction task, some fragments of the input text have to be discarded in the process of building the graph network. This makes it impossible for the model to fully use the information in the input text when reasoning about the relationship between two entities; 2. Using words as nodes to construct a homogeneous graph ignores the complex relationships between words. For example, there are different relationships in a sentence, such as co-reference relationships, semantic dependency relationships, etc. Constructing a homogeneous graph cannot capture the complex relationships within these texts.

[0013] References:

[0014] [1]Fenia Christopoulou,Makoto Miwa,Sophia Ananiadou.(2019)Connectingthe Dots:Document-level Neural Relation Extraction with Edge-orientedGraphs.EMNLP.

[0015] [2]Guoshun Nan,Zhijiang Guo,Ivan Sekulic and Wei Lu.(2020)Reasoningwith Latent Structure Refinement for Document-Level Relation Extraction.ACL.

[0016] [3]Zhang Y,Qi P,Manning C D.Graph Convolution over Pruned DependencyTrees Improves Relation Extraction[A].Proceedings of the 2018Conference onEmpirical Methods in Natural Language Processing[C].Association forComputational Linguistics,2018:2205–2215.

[0017] [4]Guo Z,Zhang Y,Lu W.Attention Guided Graph Convolutional Networksfor Relation Extraction[A].Proceedings of the 57th Annual Meeting of theAssociation for Computational Linguistics[C].Association for ComputationalLinguistics,2019:241–251.

[0018] [5]Zhang N, Deng S, Sun Z, et al. Long-tail Relation Extraction via Knowledge Graph Embeddings and Graph Convolution Networks[A]. Proceedings of the 2019Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies[C]. Association for Computational Linguistics, 2019:3016–3025. Summary of the invention

[0019] In response to the above-mentioned relationship extraction problems of the prior art on document type data sets, the present invention proposes a multi-task document-level relationship extraction method and device based on supporting sentence prediction, which improves the mining of entity relationships of documents in complex contexts and improves the extraction efficiency of document-level relationships, so as to further help achieve the accuracy of various NLP tasks such as information extraction and automatic question answering.

[0020] The invention discloses a multi-task document-level relationship extraction device based on support sentence prediction, comprising a document preprocessing module, a pre-trained language model, a graph network coding module, a graph propagation module, a support sentence recognition module and a relationship classification module.

[0021] The document preprocessing module first divides the Chinese document to be processed into sentences, then cleans the sentences, and obtains the vocabulary corresponding to all sentences based on the Chinese vocabulary of the BERT series. The pre-trained language model encodes each sentence using the pre-trained language model and represents the word segments in the sentence as embedded vectors.

[0022] The graph network coding module constructs a graph network according to the structure of the document and the predetermined rules, and obtains the embedding vector of each node according to the text. The graph network contains four types of nodes: document nodes, sentence nodes, entity nodes and mention nodes, and the edges between the nodes are established according to the predetermined rules. The embedding vector of each node is obtained, specifically: the embedding vectors of all sentence word segmentations are obtained according to the pre-trained language model encoding, the embedding vectors of the sentence nodes and the mention nodes are obtained by averaging the embedding vectors of the word segmentations contained in the corresponding sentences, the embedding vector of the document node is obtained by averaging the embedding vectors of the sentence nodes contained therein, and the embedding vector of the entity node is obtained by averaging the embedding vectors of the mention nodes contained therein.

[0023] The graph propagation module uses a graph neural network to perform graph convolution operations on the constructed graph network.

[0024] The supporting sentence identification module obtains the combination of all entity pairs in the document, infers the relationship category between each entity pair, and identifies the supporting sentences of each entity pair; when identifying the supporting sentence for a certain entity pair, the embedding vectors of the two entities are first concatenated to obtain a query vector, and then the query vector is concatenated with each sentence vector and input into the first forward neural network. The first forward neural network outputs a probability value p, which is compared with the preset threshold θ for judging it as a supporting sentence. When p≥θ is satisfied, the sentence is judged to be a supporting sentence for the entity pair.

[0025] The relation classification module concatenates the embedding vectors of the two entities of the entity pair and the embedding vector of the supporting sentence, inputs the concatenated vectors into the second forward neural network, and outputs the relation type of the entity pair.

[0026] A multi-task document-level relationship extraction method based on supporting sentence prediction of the present invention comprises the following steps:

[0027] Step 1: Divide the Chinese document into sentences, clean the sentences, traverse all sentences based on the Chinese vocabulary of the BERT series, and obtain the vocabulary Vocabulary.

[0028] Step 2: Encode sentences using a pre-trained language model. The BERT series model is recommended.

[0029] Step 3: Based on the structure of the document, a graph network that can be convolved using a graph neural network is constructed according to predetermined rules. The graph network contains four types of nodes: entity, entity mention, sentence, and document.

[0030] When constructing the graph network, the embedding vector of each node is also obtained according to the text. Specifically, the embedding vectors of all sentence word segmentations are obtained according to the pre-trained language model encoding, the embedding vectors of sentence nodes and mention nodes are obtained by averaging the embedding vectors of the word segmentations contained in the corresponding sentences, the embedding vector of document nodes is obtained by averaging the embedding vectors of sentence nodes contained in the document, and the embedding vector of entity nodes is obtained by averaging the embedding vectors of the mention nodes contained in the document.

[0031] Step 4: Use graph neural network to perform convolution operations on the graph network. It is recommended to use the interface in the graph neural network framework DGL.

[0032] Step 5: Determine the relational reasoning path. First, obtain all possible combinations of entity pairs. Use the vectors of entity nodes and sentence nodes obtained in step 4 to determine whether the sentence is a supporting sentence for each entity pair.

[0033] For entity pairs (e h ,e t ) When identifying a supporting sentence, the embedding vectors of the two entities are first concatenated to obtain a query vector, and then the query vector is concatenated with each sentence vector and input into the first forward neural network. The first forward neural network outputs a probability value p, which is compared with the preset threshold θ for judging it as a supporting sentence. When p ≥ θ is satisfied, the sentence is judged to be an entity pair (e h ,e t ); h and t are both positive integers.

[0034] Step 6: classify the relationship between the entity pairs, concatenate the two vectors of the entity pairs and the vector of the supporting sentence, input the concatenated vector into the second forward neural network layer, and classify the relationship according to the pre-defined relationship type.

[0035] In the step 1, the Chinese document is divided into multiple sentences according to the separator and the preset maximum sentence length threshold, and the sentences are preprocessed to remove special symbols, where the special symbols refer to symbols that do not appear in the BERT vocabulary.

[0036] In step 3, edges are established between nodes in the graph network according to the following predetermined rules:

[0037] (1) If the mention M appears in the sentence S, connect the mention node M and the sentence node S;

[0038] (2) Connect sentence nodes to adjacent sentence nodes according to the order in which the sentences appear in the document;

[0039] (3) Each entity node is connected to all the mention nodes of the entity;

[0040] (4) Connect the document node to all sentence nodes;

[0041] (5) All mention nodes of the same entity are connected pairwise.

[0042] The advantages and positive effects of the present invention are as follows: the method and device of the present invention, for the document-level relationship extraction task, designs a multi-task joint model that combines the support sentence prediction task and the relationship classification task; when predicting the relationship between entities, a relationship reasoning path is constructed through entities and support sentences, and the interference of noise data on the model is reduced by combining the effective information in the support sentences, which helps the model understand the relationship dependencies in complex contexts, improves the extraction effect of the document-level relationship extraction model, and can provide more reliable and accurate basic data for the implementation of subsequent NLP tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of a document-level relationship extraction method implemented in an embodiment of the present invention;

[0044] Figure 2 is a schematic diagram of encoding a sentence using a BERT model in an embodiment of the present invention;

[0045] Figure 3 This is an example diagram of document-level relationship extraction according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0047] like Figure 1 As shown, a multi-task document-level relationship extraction method based on supporting sentence prediction implemented in an embodiment of the present invention includes the following 6 steps.

[0048] Step 1: Divide the Chinese document into multiple sentences and perform preprocessing to remove special symbols. Special symbols refer to symbols that do not appear in the BERT vocabulary.

[0049] Since the length of data that can be directly modeled is limited, it is impossible to process the content of the entire document at one time. Therefore, it is necessary to set a maximum sentence length threshold L and decompose the document D into N sentences S1, S2, ..., S with a maximum length not exceeding L. N N is a positive integer. In addition, considering that the semantics in a sentence are closely related to the context, the present invention does not directly use a window sliding clause with a size of L. The present invention first decomposes a document into multiple long sentences according to the separators (.;?!) in the Chinese corpus, and then performs a multi-sentence analysis on the decomposed sentences S i Make a judgment. i If the length of does not exceed the maximum threshold L, it is directly retained, otherwise it is truncated with a window of size L to obtain m clauses. Sentence segmentation is represented as follows:

[0050]

[0051] Among them, Truncate(S i ,L) means using a window of size L to analyze sentence S i Scroll capture.

[0052] After the sentence segmentation is completed, the data needs to be further cleaned to remove most of the noise data, such as web page tags, spaces, etc. At the same time, a vocabulary needs to be maintained. For Chinese data sets, it is recommended to use the open source Chinese vocabulary of the BERT series. These vocabulary already contain most Chinese words. For words that do not appear in the vocabulary, The present invention counts the number of times the word appears in the full text And give the threshold value O of the number of occurrences. If the number of occurrences is less than the threshold value O, it means that the word is not a keyword and you can use the vocabulary in the Vocabulary <unk>Unified representation. <unk>Indicates unknown words by <unk>It can indicate the situation that most of the vocabulary does not contain the target word (Out Of Vocabulary, OOV). If the number of occurrences is greater than or equal to the threshold value O, the word is expanded into the vocabulary. The cutoff value O is set based on experience and can be adjusted through experiments.

[0053] A complete process of data preprocessing in an embodiment of the present invention is as follows:

[0054] Input:D

[0055] Output:S1,S2,...,S N

[0056] Params: Vocabulary, L, O

[0057] Process:

[0058] –Step 1: Split D according to the separators {. ;? !} to get a series of long sentences S i ,i is a positive integer.

[0059] –Step 2: For S i Make the following judgment and get the result of sentence segmentation.

[0060]

[0061] –Step 3: S i For data cleaning, you can match and remove noise data based on regular expressions.

[0062] –Step 4: Update the vocabulary, for Statistics of Occuracnce(w i )

[0063]

[0064] Finish

[0065] Step 2: For each sentence S1, S2, ..., S N Use the pre-trained language model Encoder for encoding. The BERT series model is recommended for pre-trained language models.

[0066] After step 1, the cleaned and segmented sentences are obtained as the input data of the pre-trained language model. This step directly models the input data. The pre-trained language model is used to embed the data into words, and each word in the sentence is converted into a low-dimensional space word vector representation with distributed semantics. Let vector The length of is K.

[0067] The embodiment of the present invention uses the BERT-Pytorch version to illustrate sentence encoding. i Sentence S j , j = 1, 2, ... N, such as Figure 2 As shown in the figure, first fill the sentences whose length is less than L with special identifiers to make the length of all sentences uniform, which is L. Segment each sentence and then segment each word w t Converted into K-dimensional word embedding vector by BERT The entire sentence can be represented as an L×K matrix, realizing the modeling process of sentence representation.

[0068] A process of modeling a sentence in an embodiment of the present invention is as follows:

[0069] image-20210629153940847

[0070] #First, perform preliminary processing on the input sentence and add special symbols to mark the beginning and end positions.

[0071] words=[self.CLS_TOKEN]+words+[self.SEP_TOKEN]

[0072] #Load the BERT word segmenter and use the Chinese word segmenter for Chinese corpus.

[0073] self.bert_tokenizer=BertTokenizer.from_pretrained('bert-base-chinese')

[0074] #Get the result after BERT segmentation

[0075] words=self.bert_tokenizer.tokenize(”.join(words))

[0076] #Map the word segmentation result to the ID corresponding to the word in the vocabulary. If the word does not exist in the vocabulary, use <unk>ID

[0077] #PAD_TOKEN pads the sentence to the maximum length threshold L to ensure that the length of the input batch data is consistent.

[0078] feature=self.bert_tokenizer.convert_tokens_to_ids(sent+[self.PAD_TOKEN for_in range(max_sent_len-len(sent))])

[0079] #Load the BERT pre-trained language model

[0080] self.BertModel=BertModel.from_pretrained('bert-base-chinese')

[0081] #Convert the input words into low-dimensional word vectors to represent the semantic information of each word in the vector space.

[0082] outputs=self.BertModel(input_ids=ii, token_type_ids=tti, attention_mask=am)

[0083] In the embodiment of the present invention, an English data set is used, and each sentence is divided into L words according to the sentence length L. Specifically, in actual use, the words are segmented and encoded according to the actual input sentences.

[0084] Step 3: Based on the structure of the document and in accordance with predetermined rules, a graph network is constructed that can be used for convolution operations using a graph neural network. The graph network contains four types of nodes: entities, entity mentions, sentences, and documents.

[0085] First, the method of the present invention abstracts the following four node types from a document, which are document nodes, sentence nodes, entity nodes and mention nodes from top to bottom.

[0086] Document node: represents the abstract representation of the content of the entire document, denoted by D;

[0087] Sentence node: represents the abstract representation of the meaning of a sentence in the document, denoted by S;

[0088] Entity node: represents the abstract representation of a certain entity type in the document, denoted by E;

[0089] Mention node: an abstract representation of the specific description object meaning of a certain entity type in a sentence in a document, denoted by M.

[0090] The four types of nodes are connected by edges defined by certain rules. The specific description and rules for establishing edges are as follows:

[0091] (1) Sentence-Mention edge (SM-edge): If a mention M appears in a sentence S, then the mention node and the sentence node are connected.

[0092] (2) Sentence-Sentence edge (SS-edge): A sentence node is connected to its adjacent sentence nodes (previous sentence and next sentence). Since sentences appear in order in a document, in order to preserve this structural information, they are also connected in the order of their appearance.

[0093] (3) Entity-Mention edge (EM-edge): Each entity node is directly connected to all its mention nodes.

[0094] (4) Document-Sentence edge (DS-edge): Document information comes from all sentence information, so the document node is connected to all sentence nodes.

[0095] (5) Mention-Mention edge (MM-edge): All mention nodes of the same entity are directly connected to each other.

[0096] Then, the symbolic description of the structural type in the method of the present invention is: given a document containing N sentences Each sentence Indicates that a sentence consists of M words. The set of all entities in the document D The entity is included in the text. For entity e i There are Q specific mentions in total, and P represents the number of entities. The method of the present invention needs to predict any two entities (e i ,e j ) between the belonging relationship category r ij , where r ij ∈R, R is a predefined set of relations.

[0097] The present invention uses the word vector obtained in step 2 to model the nodes defined above. For each sentence The word embedding of each word is obtained through Encoder, as follows:

[0098]

[0099] in, Representation word w M word embedding.

[0100] Mention node: If the entity mentions the segment in the sentence is s[i:j], that is, the i-th word to the j-th word in the sentence s, the present invention takes the average value of all word embeddings contained in this segment as the embedding vector of the mention It is expressed as:

[0101]

[0102] Entity node: For an entity, if it has a total of Q mentions in the text Get the embedding vector for each mention, and take the average of the embedding vectors of all mentioned nodes as the embedding vector of the entity.

[0103] Sentence node: For a sentence node, the present invention uses the average of the word vectors of all words contained in the sentence as the sentence embedding vector, that is,

[0104] Document node: Each document consists of N sentences. The present invention uses the average embedding vector of all sentences obtained in the previous step as the document’s vector embedding

[0105] So far, the present invention has abstracted four types of nodes from the structure of the document and established a heterogeneous graph G for each document based on the given rules. At the same time, the present invention provides a process for modeling the four types of nodes. Through the above process, the present invention transforms the document-level relationship extraction task into an entity relationship path reasoning problem based on a heterogeneous graph. Figure 3 As shown, a graph network construction method of the present invention is used to construct a graph for a specific example.

[0106] Step 4: Use the R-GCN neural network to perform graph convolution operations. The present invention recommends using the graph neural network framework in DGL to perform graph convolution.

[0107] On the constructed heterogeneous graph, the present invention needs to fuse the information of all defined nodes so that each node can learn the semantic information contained in other nodes. The present invention uses graph neural networks to directly model the graph to propagate information. Taking the R-GCN model implemented in the DGL deep learning library as an example, R-GCN is a graph neural network designed specifically for heterogeneous graphs. It allows nodes to fuse information from neighboring nodes through convolution operations. Unlike other graph neural networks, R-GCN takes into account the type information of nodes and edges when performing feature fusion.

[0108] Using the R-GCN neural network to perform graph convolution operations, the calculation formula for fusing neighbor node information is as follows:

[0109]

[0110] in They represent the embedding vectors of node i at the l+1th layer and the lth layer respectively. represents all neighboring nodes of node i and edge relationship r, is the number of neighboring nodes. R represents the edge relationship set. σ(·) is the activation function, for example, the activation function is the Relu function. The dimension of the first layer is d n The trainable two-dimensional matrix parameters of . is the embedding vector of node v at layer l.

[0111] The present invention concatenates the vector representations of multiple layers of R-GCN as the final output vector of all nodes. N is the number of layers of R-GCN.

[0112]

[0113] in, Represents the initial vector of input node i.

[0114] Step 5: Determine the relational reasoning path. First, obtain all possible combinations of entity pairs. Use the vectors of entity nodes and sentence nodes obtained in step 4 to determine whether the sentence is a supporting sentence for each entity pair.

[0115] After sufficient information propagation and information fusion through the graph neural network in step 4, each node in the constructed heterogeneous graph can learn the information in other nodes. i ,e j ) ij To make inferences, the present invention needs to establish i To entity node e j The relational reasoning path.

[0116] The construction of the relational reasoning path is a major innovation of the present invention. Before the construction of the relational reasoning path, the supporting sentences of the entity pairs must be identified first. Figure 3 The supporting sentence recognition module in is mainly responsible for the prediction of supporting sentences. Supporting sentences refer to sentences that contain information useful for judging the relationship between entities. By screening out supporting sentences of entity pairs, most of the content that is useless for relationship classification can be effectively filtered out, achieving a good denoising effect. In the method of the present invention, supporting sentence recognition and relationship classification are two subtasks that are combined together to construct a multi-task joint model. During the model training process, the training of the two subtasks is performed simultaneously.

[0117] Specifically, for entity e i To entity e j The present invention first uses the entity embedding vectors of two entities Concatenate to get a query vector Then the query vector is compared with all the sentence vectors After concatenation, the output is sent to the forward neural network. The output of the neural network is used to calculate a probability value p∈[0,1] using the sigmoid(·) function. The threshold for judging a sentence as a support sentence is set to θ=0.5. If the probability value p≥θ, the sentence is considered to be a predicted entity pair (e i ,e j ) relationship.

[0118] The algorithm flow of this part is as follows:

[0119] Input: entity pair (e i ,e j ), output threshold θ = 0.5, all sentence vectors

[0120] Output: True or False, indicating whether the sentence is a supporting sentence or not

[0121] Process

[0122] Step 1: Get the entity vector representation of the entity pair

[0123] Step 2: Concatenate to get the query vector

[0124] Step 3: For each sentence vector After concatenation, we get the input vector of the neural network

[0125]

[0126] Step 4: E i Input into the forward neural network, and get a probability value p through the sigmoid(·) function i

[0127] Step 5: Set the probability value p i Compare with the output threshold θ = 0.5 to get the output result

[0128]

[0129] Step 6: Classify the relationship between entity pairs. Concatenate the two vectors of the entity pairs and the vector of the supporting sentence. Input the concatenated vector into a feed-forward neural network layer to classify the relationship according to the pre-defined relationship type.

[0130] In this step, the present invention classifies the relationship between entity pairs. First, the present invention needs to construct a new entity-sentence graph to realize the relationship reasoning between entity relationships. The entity-sentence graph is the entity pair (e h ,e t ). The subgraph only contains entity nodes and supporting sentence nodes and two types of edges, as follows:

[0131] Entity nodes: two entity nodes that need to be predicted for relationship h ,e t .

[0132] Support sentence node: In step 5, for the entity pair (e h ,e t ) is judged as True.

[0133] Sentence-Sentence edge: connects sentence nodes in the order in which they appear in the text.

[0134] Entity-Sentence edge: If the entity's corresponding mention occurs in the sentence, the entity is connected to the sentence.

[0135] The present invention uses the node vector obtained after the graph neural network propagation in step 4 as the initial vector of each node in the subgraph. Then, the entity node is combined with the supporting sentence node to construct the relational reasoning path. The relational reasoning path is encoded into a vector of fixed dimension through LSTM. Finally, a forward neural network is input for classification.

[0136] For encoding the relational reasoning path, the present invention attempts two methods:

[0137] (1) Pooling method. After connecting the vectors of each node in the relational reasoning path, a fixed-dimensional vector p is obtained through pooling. h,t , which is expressed as follows:

[0138] p h,t =pooling[e h ;s1;...;s i ;e t ]

[0139] (2) LSTM method. Use an LSTM network to encode the relational reasoning path and obtain the vector p h,t as follows:

[0140] p h,t =LSTM[e h ;s1;...;s i ;e t ]

[0141] Among them, s1,s2,...s i is an entity pair (e h ,e t )’s supporting sentence. h ;s1;...;s i ;e t ] is a relational reasoning path constructed by combining entity nodes with supporting sentence nodes. The relational reasoning path between two entities is to take the two entities as the starting point and the end point, and then connect the supporting sentences in sequence between the starting point and the end point.

[0142] Finally, the present invention uses a forward neural network to perform feature compression on the vector of the relational reasoning path and converts it into a multi-label binary classification problem. For the relation type r∈R, the present invention uses the sigmoid(·) function to calculate the probability value p of the entity pair belonging to this relation. r ,as follows:

[0143] p r =sigmoid(FFNN(p h,t ))

[0144] FFNN represents a feed-forward neural network. R is a set of predefined relationship types.

[0145] The present invention sets the threshold for determining the entity pair relationship type as θ r =0.5, if the probability value p r ≥θ r , then the relationship type of the entity pair is considered to be r.

[0146] Correspondingly, the present invention provides a multi-task document-level relationship extraction device based on supporting sentence prediction, including: a document preprocessing module, a pre-trained language model, a graph network coding module, a graph propagation module, a supporting sentence recognition module and a relationship classification module.

[0147] The document preprocessing module first divides the Chinese document to be processed into sentences, then cleans the sentences, and obtains the vocabulary corresponding to all sentences based on the Chinese vocabulary of the BERT series. The specific implementation of sentence division and cleaning in the document preprocessing module is as described in step 1 above, and will not be repeated here.

[0148] The pre-trained language model encodes the sentence and represents the words in the sentence as embedding vectors.

[0149] The graph network coding module constructs a graph network according to the structure of the document and predetermined rules, and obtains the embedding vector of each node in the graph network according to the text. The graph network contains four types of nodes: document nodes, sentence nodes, entity nodes and mention nodes. The edge types include sentence-mention, sentence-sentence, entity-mention, document-sentence, and mention-mention. The graph network coding module obtains the embedding vectors of all sentence word segments according to the pre-trained language model encoding. The embedding vectors of sentence nodes and mention nodes are obtained by averaging the embedding vectors of the word segments contained in the corresponding sentences. The embedding vector of the document node is obtained by averaging the embedding vectors of the sentence nodes contained therein, and the embedding vector of the entity node is obtained by averaging the embedding vectors of the mention nodes contained therein. The implementation of the specific graph network coding module function is described in step 3 above.

[0150] The graph propagation module uses a graph neural network to perform graph convolution operations on the constructed graph network. The present invention preferentially uses the R-GCN neural network for graph convolution operations. During the graph convolution operation, the input node is fused with neighbor node information at each layer, and then the vector representations of all layers of R-GCN are concatenated as the final output vector of the node.

[0151] The supporting sentence identification module obtains the combination of all entity pairs in the document, infers the relationship category between each entity pair, and identifies the supporting sentence of each entity pair. As described in step 5 above, when the supporting sentence identification module identifies the supporting sentence for a certain entity pair, it first concatenates the embedding vectors corresponding to the two entities to obtain a query vector, and then concatenates the query vector with each sentence vector and inputs it into the first forward neural network. The forward neural network outputs a probability value p, which is compared with the preset threshold value θ for judging it as a supporting sentence. When p≥θ is satisfied, the sentence is judged to be a supporting sentence for the entity pair.

[0152] The relationship classification module classifies the relationship between the entity pairs. The relationship classification module concatenates the embedding vectors of the two entities of the entity pair and the embedding vector of the supporting sentence, inputs the concatenated vectors into the second forward neural network, and outputs the relationship type. The specific function of the relationship classification module is as described in step 6 above.

[0153] Experiments were conducted on the method and device implemented in the present invention. The dataset used was DocRED. DocRED is a manually annotated large-scale document-level relationship extraction dataset, which is constructed based on Wikipedia and Wikidata articles. In addition to annotated entities, mentions, and relationships, the dataset also annotates supporting sentences. The dataset contains 5053 documents, 132375 entities, and 56354 relationship triples. The experiment divided the data into three parts: training set / validation set / test set, which were used to train and verify the model effect, and F1 was used as the main indicator for judging the model effect.

[0154] The relation extraction task is essentially a classification task, and common evaluation indicators include accuracy and recall.

[0155] Accuracy: Based on the predicted results, how many samples are actually positive among the samples predicted to be positive. That is, the more positive samples are, the better. The accuracy is calculated as follows:

[0156]

[0157] Recall rate: How many positive samples are predicted correctly for the original positive samples. The Recall rate is calculated as follows:

[0158]

[0159] Among them, TP (True Positive) means that the predicted result is the positive class, which is actually a sample of the positive class; FP (False Positive) means that the predicted result is the positive class, but it is actually a sample of the negative class; FN (False negative) means that the predicted result is the negative class, but it is actually a sample of the positive class; TN (True negative) means that the predicted result is the negative class, but it is actually a sample of the negative class.

[0160] The present invention hopes that the model will show good accuracy and recall at the same time, but accuracy and recall are negatively correlated, so the present invention uses the F1 value to balance the calculation of accuracy and recall. Therefore, the present invention selects the F1 value as an indicator for evaluating performance on the test set. F1 is calculated from accuracy (Precision) and recall (Recall), which is the harmonic mean of the two. The larger the F1 value, the better the model effect.

[0161]

[0162] The widely used indicator F1 in relation extraction tasks is used in the experiment of the present invention. However, some relation facts exist in both the training set (train) and the test set (dev), so the model may remember their relations during training and obtain better performance on the test set in an undesirable way, thereby introducing evaluation bias. However, the overlap of relation facts between the training set and the test set is inevitable because many common relation facts may be shared in different documents. Therefore, the experiment of the present invention also uses F1 that excludes the relation facts shared by the training set (train) and the test set (dev), denoted as IgnF1, as a supplementary evaluation indicator.

[0163] The experimental environment is set up as follows:

[0164] Experimental environment Version Ubuntu (Linux) 18.10 Graphics card configuration GeForce RTX 3070 single card The experiment mainly depends on the library Version Number Pytorch 1.0.1 DGL v0.3 NumPy 1.13.3 Matplotlib 3.4.2 transformer 3.4.0 Neural Networks category Graph Neural Networks R-GCN Pre-trained models Roberta Main hyperparameters of the experiment Parameter size Learning rate 0.001 Weight decay 0.01 Maximum gradient pruning (max_grad_norm) 1.0 Epochs 30

[0165] The experimental results and analysis are as follows:

[0166] Table 1: Comparison of results with baseline

[0167] Model Dev Test Ign F1 F1 Ign F1 F1 Sequence-based models CNN 41.58 43.45 40.33 42.26 LSTM 48.44 50.68 47.71 50.07 BiLSTM 48.87 50.94 48.78 51.06 Context-Aware 48.94 51.09 48.4 50.7 HIN 51.06 52.95 51.15 53.3 Graph-based model GAT 45.17 51.44 47.36 49.51 GCNN 46.22 51.52 49.59 51.62 EoG 45.94 52.15 49.48 51.82 AGGCN 46.29 52.47 48.98 51.45 LSR+BERT-base 52.43 59 52.15 54.18 GAIN+BERT-base 59.14 61.22 59 61.24 GAIN+BERT-large 60.87 63.09 60.31 62.76 Pre-trained basic model BERT-base - 54.16 - 53.2 RoBERTa-base 53.85 56.05 53.52 55.77 CorefBERT-base 55.32 57.51 54.54 56.96 CorefBERT-large 56.73 58.88 56.48 58.7 Method of the present invention ERNN+BERT-base 59.66 61.67 59.34 61.81 ERNN+BERT-large 61.22 63.37 60.58 62.92

[0168] Table 2: Relationships between sentences and across sentences

[0169] Model Dev Dev Dev Intra-F1 Inter-F1 Infer-F1 CNN 51.87 37.58 37.11 LSTM 56.57 41.47 39.03 BiLSTM 57.05 43.49 38.73 Context-Aware 56.74 42.26 39.73 BERT-base 61.61 47.15 39.62 LSR+BERT-base 65.26 52.05 - GAIN+BERT-base 67.1 53.9 46.89 ERNN+BERT-base 67.27 54.36 47.12

[0170] Table 1 shows the comparison between the model of the present invention and the previous existing models (the models used in this article are all models that have achieved SOTA effects before, collectively referred to as SOTA models here), and Table 2 shows the comparison between the model of the present invention and the previous models in terms of inter and intra relationship extraction. By analyzing the results in the two tables, the present invention draws the following conclusions:

[0171] (1) Comparing the results in Table 1, it is found that the model with graph + pre-training structure of the method of the present invention achieves the best effect, and the graph-based model has the worst effect. The present invention analyzes that this is because the graph-based model is not as good as the sequence model and the pre-training model in extracting text content features, but the use of graph neural network is helpful to infer the information between two entities that are far apart in the input text.

[0172] (2) The model of the present invention achieved the best results. After analysis, this is because the present invention used supporting sentences and more important information when constructing the reasoning path. Therefore, compared with models such as EoG and GAIN that construct reasoning paths for relationship classification, the model of the present invention has better results.

[0173] (3) Table 2 compares the extraction effects of the model of the present invention and other models on different types of relations. The present invention outperforms the previous SOTA model on all three types of relations, with a more obvious improvement in infer-F1, which proves that the way the present invention constructs the reasoning path is reasonable. In addition, the model of the present invention does not significantly improve the extraction of intra relations, which may be because the role of supporting sentences is not particularly obvious for relations that do not require complex reasoning to be classified.

[0174] Except for the technical features described in the specification, all of them are known technologies to those skilled in the art. The present invention omits the description of known components and known technologies to avoid redundancy and unnecessary limitation of the present invention. The implementation methods described in the above embodiments do not represent all implementation methods consistent with the present application. On the basis of the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative labor are still within the protection scope of the present invention.< / unk> < / unk> < / unk> < / unk>

Claims

1. A multi-task document-level relation extraction method based on support sentence prediction, characterized in that: include: Step 1: Divide the Chinese document into multiple sentences, clean the sentences, and obtain the vocabulary corresponding to all sentences based on the Chinese vocabulary of the BERT series; Step 2: Encode the sentences using the pre-trained language model and represent the words in the sentences as embedding vectors. Step 3, constructing a graph network according to the structure of the document and the predetermined rules; The graph network contains four types of nodes: document nodes, sentence nodes, entity nodes, and mention nodes; edges are established between nodes according to the following predetermined rules: (1) If the mention M appears in the sentence S, connect the mention node M and the sentence node S; (2) Connect sentence nodes to adjacent sentence nodes according to the order in which the sentences appear in the document; (3) Each entity node is connected to all the mention nodes of the entity; (4) Connect the document node to all sentence nodes; (5) All mention nodes of the same entity are connected pairwise; Obtain the embedding vector of each node according to the text, specifically: obtain the embedding vectors of all sentence word segments according to the pre-trained language model encoding, the embedding vectors of sentence nodes and mention nodes are obtained by averaging the embedding vectors of the word segments contained in the corresponding sentences, the embedding vector of the document node is obtained by averaging the embedding vectors of the sentence nodes contained in it, and the embedding vector of the entity node is obtained by averaging the embedding vectors of the mention nodes contained in it; Step 4: Use the graph neural network to perform convolution operations on the constructed graph network; Step 5: Obtain the combination of all entity pairs in the document, infer the relationship category between each entity pair, and identify the supporting sentences of each entity pair; For entity pairs (e h ,e t ) When identifying a supporting sentence, the embedding vectors of the two entities are first concatenated to obtain a query vector, and then the query vector is concatenated with each sentence vector and input into the first forward neural network. The first forward neural network outputs a probability value p, which is compared with the preset threshold θ for judging it as a supporting sentence. When p ≥ θ is satisfied, the sentence is judged to be an entity pair (e h ,e t )’s supporting sentence; h, t are both positive integers; Step 6, classifying the relationship between the entity pairs, including: concatenating the embedding vectors of the two entities of the entity pair and the embedding vector of the supporting sentence, inputting the concatenated vectors into the second forward neural network, and outputting the relationship type of the entity pair.

2. The method according to claim 1, characterized in that In the step 1, the document is divided into multiple sentences by setting a maximum sentence length threshold L, first decomposing the document into multiple sentences according to the separators in the Chinese corpus, and then comparing each decomposed sentence to see if the length of the sentence exceeds L. If not, the sentence is retained directly; otherwise, the sentence is rolled and truncated using a window of length L.

3. The method according to claim 1 or 2, characterized in that: In the step 1, a vocabulary Vocabulary is obtained based on the Chinese vocabulary of the BERT series. For words in the sentence that do not appear in the Chinese vocabulary, the number of times the word appears in the full text is counted and compared with the set occurrence count threshold O. If it is less than O, the unknown words in the vocabulary Vocabulary are used for unified representation. Otherwise, the word is expanded into the vocabulary Vocabulary.

4. The method according to claim 1, characterized in that In the step 3, the text is vectorized to obtain a node vector, including: assuming that a Chinese document contains N sentences and the number of entities in the document is P; obtaining a word embedding vector for each sentence segment by a pre-trained language model; the embedding vector of each mention is the average of all word embeddings contained in the fragment of the mention in the sentence; for a certain entity, if the entity has a total of Q mentions in the text, the embedding vectors of the Q mention nodes are averaged to obtain the embedding vector of the entity; the embedding vector of the sentence is obtained by averaging the word embeddings of all the segmentations contained in the sentence; the embedding vector of the document is obtained by averaging the embedding vectors of N sentence nodes.

5. The method according to claim 1, characterized in that In step 4, the R-GCN neural network is used to perform a convolution operation. During the convolution operation, the neighbor node information is fused at each layer of the input node, and then the vector representations of all layers of the R-GCN are concatenated as the final output vector of the node.

6. The method according to claim 1, characterized in that In the step 5, the threshold θ for judging a sentence as a support sentence is preset to be 0.

5.

7. The method according to claim 1, characterized in that In step 6, for the entity pair (e h ,e t ), obtain a subgraph from the graph network obtained in step 3, which only contains entity node e h ,e t and supporting sentence nodes, as well as the corresponding sentence-sentence and entity-sentence edges; the sentence-sentence edges in the subgraph connect the sentence nodes in the order in which the sentences appear in the document; for the entity-sentence edges in the subgraph, if the corresponding mention of the entity appears in the sentence, the entity is connected to the sentence.

8. The method according to claim 1 or 7, characterized in that: In the step 6, the embedding vector of the entity and the embedding vector of the supporting sentence are obtained after the graph convolution operation in step 4; the entity node is combined with the supporting sentence node to construct a relational reasoning path; the relational reasoning path is encoded into a vector of fixed dimension through pooling or LSTM network, and input into the second forward neural network; the second forward neural network inputs the probability value of the entity pair belonging to the relation type r, r∈R, R is a predefined relation type set; Select the probability value greater than or equal to the preset threshold θ r The relationship type of the entity pair.

9. A multi-task document-level relationship extraction device based on support sentence prediction, characterized in that: The device comprises a document preprocessing module, a pretrained language model, a graph network coding module, a graph propagation module, a support sentence recognition module and a relationship classification module; wherein, The document preprocessing module first divides the Chinese document to be processed into sentences, then cleans the sentences, and obtains the vocabulary corresponding to all sentences based on the Chinese vocabulary of the BERT series; The pre-trained language model encodes each sentence using the pre-trained language model and represents the words in the sentence as an embedding vector; The graph network encoding module constructs a graph network according to the structure of the document and the predetermined rules, and obtains the embedding vector of each node according to the text; wherein the graph network includes four types of nodes: document nodes, sentence nodes, entity nodes and mention nodes, and the edges between the nodes are established according to the predetermined rules; the embedding vector of each node is obtained, specifically: the embedding vectors of all sentence word segments are obtained according to the pre-trained language model encoding, the embedding vectors of the sentence nodes and the mention nodes are obtained by averaging the embedding vectors of the word segments contained in the corresponding sentences, the embedding vector of the document node is obtained by averaging the embedding vectors of the sentence nodes contained therein, and the embedding vector of the entity node is obtained by averaging the embedding vectors of the mention nodes contained therein; The graph propagation module uses a graph neural network to perform graph convolution operations on the constructed graph network; The supporting sentence identification module obtains the combination of all entity pairs in the document, infers the relationship category between each entity pair, and identifies the supporting sentences of each entity pair; when identifying the supporting sentence for a certain entity pair, the embedding vectors of the two entities are first concatenated to obtain a query vector, and then the query vector is concatenated with each sentence vector and input into the first forward neural network. The first forward neural network outputs a probability value p, which is compared with the preset threshold value θ for judging it as a supporting sentence. When p≥θ is satisfied, the sentence is judged to be a supporting sentence for the entity pair; The relation classification module concatenates the embedding vectors of the two entities of the entity pair and the embedding vector of the supporting sentence, inputs the concatenated vectors into the second forward neural network, and outputs the relation type of the entity pair.

10. The device according to claim 9, characterized in that The graph network coding module establishes edges between nodes in the graph network according to the following rules: (1) If the mention M appears in the sentence S, connect the mention node M and the sentence node S; (2) Connect sentence nodes to adjacent sentence nodes according to the order in which the sentences appear in the document; (3) Each entity node is connected to all the mention nodes of the entity; (4) Connect the document node to all sentence nodes; (5) All mention nodes of the same entity are connected pairwise.

Images