A document-level relation extraction method based on selective attention and path reasoning

Abstract

The application discloses a document-level relation extraction method based on selective attention and path reasoning and belongs to the technical field of information extraction. The application comprises the following steps: inputting the whole document; hierarchically constructing intra-sentence relation graphs and inter-sentence relation graphs, obtaining vector representations of entity mentions and sentences by using a graph neural network; aggregating the two subgraphs to form a document graph, capturing local and non-local information interactions in the document by using a recurrent neural network; screening out sentences related to entity pairs by using selective attention, generating a document subgraph and taking the document subgraph as the range of relation extraction; and explicitly modeling reasoning chains among multiple entities in the document subgraph to predict the probability of each relation in a given relation path. The application can automatically extract relation facts and supporting evidence supporting the existence of the relation facts from a document to discover more accurate inter-sentence relations and supporting evidence.

Description

Technical Field

[0001] This invention belongs to the field of information extraction technology, specifically relating to a document-level relation extraction method based on selective attention and path reasoning. Background Technology

[0002] Relation extraction aims to detect semantic relationships between entities in text and plays an important role in many natural language processing applications such as knowledge base construction and automatic question answering.

[0003] Traditional relation extraction methods primarily focus on the sentence level, requiring a sentence to contain two entities. However, most relations between entities are implied in multiple sentences within a document or even across multiple documents. Therefore, it is necessary to advance relation extraction from the sentence level to the document level. Document-level relation extraction requires reading and reasoning across multiple sentences in a document, aiming to extract all possible relational facts from the document and provide supporting evidence.

[0004] Existing document-level relation extraction methods, in terms of document modeling, use static graphs to represent the structure of the entire document, which cannot capture the complex interactions in the document; in terms of relation extraction scope selection, they use sentence information of the entire document for relation extraction, introducing irrelevant sentences and bringing noise; in terms of relation reasoning, they do not explicitly model the reasoning chain, resulting in a lack of interpretability of the extraction results.

[0005] A search revealed patent application number 202210386134.6, filed on April 13, 2022, which discloses a document-level relation extraction method based on a heterogeneous graph attention network. This application combines nodes and edges in the document graph during processing at the graph representation layer, and uses a heterogeneous graph attention network to process both, capturing complex interaction information between mentioned nodes, entity nodes, and sentence nodes in the document graph. This provides more useful information for relation extraction, thereby improving the accuracy of relation extraction. However, this application uses all entities in the document for relation extraction, which introduces noise. Furthermore, the method lacks a reasoning path; the relation prediction between two entities relies solely on the entity's encoded representation, resulting in a lack of interpretability in the extraction results. Summary of the Invention

[0006] 1. The technical problem that the invention aims to solve

[0007] To overcome the problems existing in the prior art, the present invention provides a document-level relation extraction method based on selective attention and path reasoning. This method uses selective attention and path reasoning to discover more accurate inter-sentence relations and supporting evidence, thereby improving the accuracy of extraction and the interpretability of the extraction results.

[0008] 2. Technical Solution

[0009] To achieve the above objectives, the technical solution provided by the present invention is as follows:

[0010] The present invention provides a document-level relation extraction method based on selective attention and path reasoning, comprising the following steps:

[0011] S1: Input the entire document;

[0012] S2: Construct intra-sentence relation graphs and inter-sentence relation graphs in a hierarchical manner, and use graph neural networks to obtain vector representations of entity references and sentences themselves;

[0013] S3: Aggregate the two subgraphs above to form a document graph. Use a recurrent neural network to capture the interaction between local and non-local information in the document graph and output the final representation of each sentence.

[0014] S4: Use selective attention to filter out sentences related to entity pairs, generate a document subgraph and use it as the scope for relation extraction;

[0015] S5: Explicitly model inference chains between multiple entities in a document subgraph to predict the probability of each relation in a given relation path;

[0016] S6: Extract the relational facts contained in the document, as well as the supporting evidence that supports the existence of these relational facts.

[0017] The main difference between the training and testing phases of this invention lies in the input document level. When training the model, the input documents contain relational facts and supporting evidence; when testing the model, the input documents do not contain the above information, and the model needs to predict the relational facts and supporting evidence in the documents.

[0018] 3. Beneficial effects

[0019] Compared with existing known technologies, the technical solution provided by this invention has the following significant advantages:

[0020] (1) The present invention provides a document-level relation extraction method based on selective attention and path reasoning. This method addresses the problem that existing methods use static graphs to represent document structures and cannot capture complex interactions in documents. It utilizes hierarchical heterogeneous graph neural networks and recurrent neural networks to model documents and capture complex interactions in documents.

[0021] (2) The present invention provides a document-level relation extraction method based on selective attention and path reasoning. In view of the problem that existing methods use all sentences of a document as the scope of relation extraction, irrelevant sentences introduce noise, selective attention is used to filter out sentences related to entity pairs to form a document subgraph, thereby reducing the noise impact of irrelevant sentences.

[0022] (3) The document-level relation extraction method based on selective attention and path reasoning of the present invention addresses the problem that existing methods do not explicitly model the reasoning chain, resulting in a lack of interpretability in the inferred results. By explicitly modeling the reasoning chain among multiple entities in the subgraph through path reasoning, the relationship between multiple entities is inferred and corresponding supporting data is provided, increasing the interpretability of the extraction results. In addition, the selective attention and path reasoning of the present invention can discover more accurate inter-sentence relationships and supporting evidence. Attached Figure Description

[0023] Figure 1 This is a flowchart of a document-level relation extraction method based on selective attention and path reasoning according to the present invention.

[0024] Figure 2 This is a sample input document for the present invention. Detailed Implementation

[0025] To further understand the content of this invention, a detailed description of the invention will be provided in conjunction with the accompanying drawings and embodiments.

[0026] Example 1

[0027] Reference Figure 1 This embodiment of a document-level relation extraction method based on selective attention and path reasoning includes the following steps:

[0028] S1: Input the entire document:

[0029] like Figure 2 As shown, a document includes a document title, sentences, entity references, entity types, relational facts, and corresponding supporting evidence. Entity references in the document are represented by "vertexSet", relations by "labels", titles by "title", and sentences by "sents".

[0030] S2: Construct hierarchical intra-sentence relation graphs and inter-sentence relation graphs, and use graph neural networks to obtain vector representations of entity references and sentences themselves:

[0031] S2-1: Construct an intra-sentence relation graph consisting of entity references for each sentence, and output the vector representations corresponding to all entity references. The specific steps are as follows:

[0032] S2-1-1: Obtain the contextual representation of each word in the sentence relative to its referential pair. This representation is obtained by concatenating the word, coreference, entity type, sentence number, and word position embeddings in the sentence. The calculation method is as follows:

[0033]

[0034] Where w k c k , t k and n k They represent the word x k Word, coreference, entity type, and sentence number embedding, Represents the word x k Relative to the pair of references (h) i ,h j The positional embeddings are described below:

[0035] • Word embedding: Each word is embedded through a word embedding matrix. Mapping to d w A dimensional vector, where |V w | represents the size of the dictionary, d w It is the dimension of word embedding.

[0036] • Coreference embedding: Referents corresponding to the same entity are assigned the same entity ID, which is determined by the order in which the entities appear in the document. Each entity ID is identified through a coreference embedding matrix. Mapping to d c A dimensional vector, where |V c | represents the number of entities, d c It is the dimension of co-referential embedding.

[0037] • Entity type embedding: Each type label is embedded through an entity type embedding matrix. Mapping to d t A dimensional vector, where |V t | represents the number of entity types, d t It is the dimension of entity type embedding.

[0038] • Sentence number embedding: To facilitate the integration of intra-sentence and inter-sentence relationship graphs, a sentence number embedding is added to each word to indicate which sentence the word belongs to. For example, if a word is in the first sentence of a document, its sentence number embedding is set to zero.

[0039] • Word position embedding: Each word in a sentence is labeled as belonging to the first reference of a reference pair, the second reference, or not belonging to either reference. Each position label is generated through a position embedding matrix. Mapped to A dimensional vector, where It is the dimension of word position embedding.

[0040] S2-1-2: The sequence of contextual representations of words relative to reference pairs is input into an encoder containing a bidirectional long short-term memory network and a multilayer perceptron to capture the representations of reference pairs. The computation is as follows:

[0041]

[0042] in It is the sequence of contextual representations of words in a sentence relative to their referential pairs, where C is the number of words in the sentence. It refers to the edge representation of symmetric pairs.

[0043] S2-1-3: The edge representations of the referential pairs are input into a multi-layer recurrent neural network, which learns the vector representations of the referential nodes layer by layer. The calculation method is as follows:

[0044]

[0045] in W is the hidden vector of the referential node i in the sentence at the nth layer, σ is the non-linear activation function, N(i) is the set of neighboring nodes of the referential node i, and W is the hidden vector of the referential node i at the nth layer. h This represents the trainable parameters, and the superscript (n) indicates the layer number of the recurrent neural network.

[0046] S2-2: Construct a graph of relationships between sentences in a document and output the vector representation of each sentence. The specific steps are as follows:

[0047] S2-2-1: Obtain the contextual representation of each sentence relative to the sentence pair. This representation is obtained by concatenating the sentence embeddings and relative sentence position embeddings in the document, and is calculated as follows:

[0048]

[0049] in Representative sentence s l Sentence embedding. Representative sentence s l Relative positional embeddings relative to sentence pairs. These embeddings are described below:

[0050] • Sentence embedding: It is represented by the average of the word embeddings in the sentence.

[0051] • Sentence relative position embedding: Each position tag is embedded through a position embedding matrix Mapped to A dimensional vector, where It is the dimension of the relative position embedding of sentences.

[0052] S2-2-2: The sequence of sentence-to-sentence context representations is input into an encoder containing a bidirectional long short-term memory network and a multilayer perceptron to capture the representations of sentence pairs. The computation is as follows:

[0053] η i,j =MLP(BiLSTM(D) i,j ))

[0054] in It is the sequence of sentence representations relative to sentence pairs, where L is the number of sentences in the document, and η is the number of sentences in the document. i,j It represents the edge of a sentence pair.

[0055] S2-2-3: The edge representations of sentence pairs are input into a multi-layer recurrent neural network, which learns the vector representations of sentence nodes layer by layer. The calculation method is as follows:

[0056]

[0057] in W is the hidden vector of sentence node i at level n. The initial vector representation of the sentence is set to zero. g This represents the trainable parameters.

[0058] S3: Aggregate the two subgraphs to form a document graph. Use a recurrent neural network to capture the interaction between local and non-local information in the document graph and output the final representation of each sentence. This representation consists of the vector representation of each referent in the sentence and the vector representation of the sentence itself.

[0059] Document graphs capture various types of dependencies through different types of nodes and edges. Nodes consist of three parts: referents, entities, and sentences. The five types of edges are shown in Table 1.

[0060] Table 1

[0061]

[0062] S3-1: By establishing a connection edge between each referent and the sentence using the sentence number to which it belongs, the intra-sentence relation graph and the inter-sentence relation graph are aggregated to generate a document graph, whose embedded representation is as follows:

[0063]

[0064]

[0065] in The final representation of sentence l in the nth iteration is represented by a vector representation of each referent in the sentence. and sentence self-vector representation Composition, where M represents the number of referential terms in the sentence.

[0066] S3-2: Introduce a recurrent neural network to model the state transition process. During each state transition, each node simultaneously exchanges information with all its graph neighbors.

[0067] By default, intra-sentence relation graphs and inter-sentence relation graphs only exchange information among neighboring nodes. To accelerate information exchange, this embodiment increases the size of the neighbor window in each state transition to allow communication among more nodes. For the k-th state transition, the size of the neighbor window can be expanded sequentially by k words. As the iterations increase, the referential node and sentence node become richer and richer in contextual information.

[0068] S3-3: Different layers of the document graph can represent features at different levels of abstraction in a sentence, and all levels of abstraction are crucial to the final representation of the sentence. To cover features at all levels, the representations at each level of the sentence are connected to form the final representation H of sentence l. l The calculation method is as follows:

[0069]

[0070] S4: Based on the final representation of each sentence and the target entity pair calculated in S3, selective attention is used to filter out sentences related to the entity pair, and these related sentences are aggregated into a document subgraph as the scope for target entity pair relation extraction:

[0071] S4-1: Calculate the relevance of each sentence in the document to the entity pair, and then select the m most relevant sentences from high to low relevance based on the relevance results. The calculation method is as follows:

[0072]

[0073] in This is called a query-based function, used to score the degree of matching between sentences and entity pairs. Its calculation method is as follows:

[0074]

[0075] A is a weighted diagonal matrix. ε i,j It is with entity pair (e) i ,e j The associated query vector is the entity pair (e) obtained after performing average pooling. i ,e j The representation of ) is calculated as follows:

[0076] ε i,j =MeanPooling([e i ;e j ])

[0077] For those with multiple mentions Entity e i ,in It is entity ei The number of times it is mentioned is obtained globally through logsumexp pooling.

[0078]

[0079] S4-2: Aggregate m sentences into a document subgraph as the scope for relation extraction, and calculate the score of the document subgraph under entity pairs and the document graph. The calculation method is as follows: The conditional probability of the document subgraph is calculated as follows:

[0080]

[0081] Among them o i,j It is an entity pair (e) i ,e j The output vector representation of the subgraph corresponding to ) is calculated as follows:

[0082] o i,j =ε i,j g' i,j +d i,j

[0083] Where, d i,j It is the deviation vector, g' i,j Indicates that in a given pair of entities (e) i ,e j When ), the weighted sum of all sentence vectors on the subgraph is calculated as follows:

[0084]

[0085] S5: Explicitly model inference chains between multiple entities in a document subgraph to predict the probability of each relation in a given relation path:

[0086] S5-1: Construct the direct and indirect paths between entity pairs. A direct path indicates that the entity pair is in the same sentence, while an indirect path indicates that the entity pair is not in the same sentence and requires combining their intermediate paths. The direct path is calculated as follows:

[0087] φ i,j =χ i,j

[0088]

[0089] The indirect path is the indirect relationship information obtained from the subgraph G′ using the modified bilinear transformation, and it is calculated as follows:

[0090] f(χ i,k ,χ k,j )=σ(χ i,k ⊙(W r χk,j ))

[0091] Where χ i,j Represents node e i and e j The path representation between them, where ⊙ represents element-wise multiplication, K represents the hierarchy of the path reasoning module, and W r Let represent a trainable weight matrix, and σ represent the sigmoid non-linear activation function.

[0092] S5-2: Combine direct and indirect paths to form a unified path representation, calculated as follows:

[0093] r i,j =αφ i,j +(1-α)∑ k≠i,j f(χ i,k ,χ k,j )

[0094] Where α∈[0,1] is used to describe the relative weights between direct and indirect paths. If the direct path provides a very reliable prediction, there is no need to focus on the information of the indirect path.

[0095] S5-3: Using the combined path representation, predict the probability that there is a certain relationship between entity pairs. The calculation method is as follows:

[0096] P(r i,j |e i ,e j ,G′,θ)=σ(Linear(r i,j ))

[0097] Where P(r) i,j |e i ,e j (e, G′, θ) represents the entity pair (e) on the subgraph G′. i ,e j The relation type is r i,j The probability of.

[0098] S6: Extract the relational facts contained in the document, and the supporting evidence to support the existence of these relational facts:

[0099] Entity pairs in the document (e i ,e j The relationship between the probability P(r) and the relationship between the two probabilities i,j |e i ,e j The calculation method for G,θ is as follows:

[0100] P(r i,j |e i ,e j,G,θ)=∑ G ′P(r i,j |e i ,e j ,G′,θ)P(G′|e i ,e j ,G,θ).

[0101] Using the above calculation formula, entity pairs (e) in the document can be predicted. i ,e j Possible relationships. If an entity pair has only a single relationship, take the relationship with the highest probability as the relationship and derive the relationship fact (e). i ,r i,j ,e j If an entity pair has multiple relationships, a global threshold can be set using the training set. Relationships with probabilities exceeding this threshold can be considered as relational facts. Simultaneously, sentences containing entities appearing on this subgraph are marked as supporting evidence.

[0102] The present invention and its embodiments have been described above illustratively. This description is not restrictive, and the figures shown are only one embodiment of the present invention; the actual structure is not limited thereto. Therefore, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the present invention, such designs should fall within the protection scope of the present invention.

Claims

1. A document-level relation extraction method based on selective attention and path reasoning, characterized in that: The steps include the following: S1: Input the entire document; S2: Construct hierarchical intra-sentence relation graphs and inter-sentence relation graphs, and use graph neural networks to obtain vector representations of entity references and sentences themselves; the construction steps are as follows: S2-1: Construct an intra-sentence relation graph consisting of entity references for each sentence, and output the vector representations corresponding to all entity references; specifically: S2-1-1: Obtain the contextual representation of each word in the sentence relative to the reference pair, which is obtained by concatenating the word, core reference, entity type, sentence number, and word position embeddings in the sentence; S2-1-2: Input the sequence of contextual representations of words relative to reference pairs into an encoder containing a bidirectional long short-term memory network and a multilayer perceptron to capture the representations of reference pairs; S2-1-3: Input the edge representation of the reference pair into a multi-layer recurrent neural network, and learn the vector representation of the reference node layer by layer; S2-2: Construct an inter-sentence relationship graph consisting of sentences in the document, and output the vector representation of each sentence itself; specifically: S2-2-1: Obtain the contextual representation of each sentence relative to the sentence pair, which is obtained by concatenating the sentence embeddings and sentence relative position embeddings in the document; S2-2-2: Input the sequence of contextual representations of sentences relative to sentence pairs into an encoder containing a bidirectional long short-term memory network and a multilayer perceptron to capture the representations of sentence pairs; S2-2-3: Input the edge representation of the sentence pair into a multi-layer recurrent neural network, and learn the vector representation of the sentence node layer by layer; S3: Aggregate the above intra-sentence relationship graph and inter-sentence relationship graph to form a document graph. Use a recurrent neural network to capture the local and non-local information interactions in the document graph and output the final representation of each sentence. S4: Use selective attention to filter out sentences related to entity pairs, generate a document subgraph and use it as the scope for relation extraction; S5: Explicitly model inference chains between multiple entities in a document subgraph to predict the probability of each relation in a given relation path; the construction steps are as follows: S5-1: Construct direct and indirect paths between entity pairs. A direct path indicates that the entity pair is in the same sentence; an indirect path indicates that the entity pair is not in the same sentence. Use the modified bilinear transformation from the subgraph. Indirect relationship information obtained; S5-2: Combine direct and indirect paths to form a unified path representation; S5-3: Using the combined path representation, predict the probability that there is a certain relationship between entity pairs; S6: Extract the relational facts contained in the document, as well as the supporting evidence that supports the existence of these relational facts.

2. The document-level relation extraction method based on selective attention and path reasoning according to claim 1, characterized in that: The document in step S1 includes a document title, sentences, entity references, entity types, relational facts, and corresponding supporting evidence.

3. The document-level relation extraction method based on selective attention and path reasoning according to claim 2, characterized in that: The construction steps of step S3 are as follows: S3-1: By establishing a connection edge between each referential item and the sentence through the sentence number to which the referential item belongs, the intra-sentence relation graph and the inter-sentence relation graph can be aggregated to generate a document graph; S3-2: Introduce a recurrent neural network to model the state transition process; during each state transition, each node simultaneously exchanges information with all its graph neighbors; S3-3: Different layers of the document graph represent features at different levels of abstraction in a sentence. All levels of abstraction are crucial to the final representation of the sentence. To cover features at all levels, the representations at each layer of the sentence are connected to form the sentence. The final expression .

4. The document-level relation extraction method based on selective attention and path reasoning according to claim 3, characterized in that: In step S3-2, by default, the intra-sentence relationship graph and inter-sentence relationship graph only exchange information between neighboring nodes. To accelerate information exchange, the size of the neighbor window is increased in each state transition to allow communication between more nodes; for the first... During the next state transition, the size of the neighbor windows expands sequentially. One word; As iterations increase, the referential node and sentence node become richer and richer in contextual information.

5. The document-level relation extraction method based on selective attention and path reasoning according to claim 4, characterized in that: The construction steps in step S4 are as follows: S4-1: Calculate the relevance of each sentence in the document to the entity pair, and then select the most relevant sentences from high to low based on the relevance results. One sentence; S4-2: Will Sentences are aggregated into a document subgraph to serve as the scope for relation extraction.

Images