Document-level relation triple extraction method

Abstract

The application provides a document-level relation triple extraction method, comprising: mining prior knowledge in a document containing training data and constructing different kinds of prior graph structures to represent the internal connection between different prior knowledge; fusing each kind of prior graph structure and a relation pre-training model through a multi-channel model, calculating the weight of each prior graph structure after encoding the information in the document and the prior graph structure by using the relation pre-training model, and performing relation extraction and reasoning between entities in the document through the comprehensive information after weight aggregation to obtain corresponding relation triples. Through the comprehensive information after weight aggregation, the relation extraction and reasoning between entities in the document are performed, which helps to improve the related information discrimination ability of the model in the complex environment at the document level, and further improves the extraction performance of the relation triple which needs multi-step reasoning, so as to improve the result of the document-level relation triple extraction.

Description

Technical Field

[0001] This invention relates to the field of document knowledge extraction technology, and in particular to a method for extracting document-level relation triples. Background Technology

[0002] In recent years, with the rapid development of internet information technology, news and social networking websites generate massive amounts of new data every day. This data contains a wide variety of content, much of which is highly valuable and plays a crucial role in people's lives. To extract and effectively utilize this valuable information, the concept of knowledge graphs has been proposed. In a knowledge graph, proper nouns such as names of people and places in the massive dataset are represented as entities, and the relationships between any two entities are represented as relations. Thus, massive amounts of data are represented as triples of entities and relations (entity 1, relation, entity 2). Although existing knowledge graphs already contain hundreds of millions of data points, the information on the internet is constantly increasing daily, and the information in knowledge graphs needs to be improved accordingly.

[0003] To automate the improvement of knowledge graphs, researchers have conducted various technological studies, one of which is entity relation extraction (ERP). The main task of ERP is to identify entities in a document and determine the semantic relationships between them. ERP can transform unstructured documents into structured documents, helping people find the information they need more quickly and conveniently.

[0004] In recent years, most scholars have pointed out that the multi-step reasoning ability of a model is key to completing document-level relation extraction tasks, and two-stage models based on neural networks are currently the most effective method. Therefore, many works have attempted to model the reasoning process using graphs. Sahu et al. used coreference links to build dependency graphs, while Peng, Quirk, Song et al. used dependency graphs to capture key features of a document. De utilized coreference information to construct a document-level graph structure with entities as nodes. Ultimately, these graph structures have all been used in research using the two-stage model paradigm. Regarding the two-stage model paradigm, the method adopted by Christopoulou and Zeng et al. in their 2019 and 2020 research serves as an example. This method first uses a Transformer-based pre-trained language model to encode the document, and then uses rule-based methods such as syntax trees to establish preliminary connections between entities, constructing a graph structure with entities as nodes for a document. However, this two-stage approach not only separates the document encoding process from the graph structure construction process but also causes heterogeneity between document encoding models and graph structure encoding models. Meanwhile, in previous work, graph structures generally only contain entity nodes, while neglecting pronouns and keywords that contain important relational information. This may lead to gaps in the reasoning logic chain, thus limiting the model's reasoning ability. Summary of the Invention

[0005] The purpose of this invention is to provide a method for extracting document-level relation triples, which helps to improve the model's ability to identify relevant information in complex document-level environments, thereby improving the extraction performance of relation triples that require multi-step reasoning, and thus improving the results of document-level relation triple extraction.

[0006] To achieve the above objectives, the present invention provides a method for extracting document-level relation triples, comprising:

[0007] Prior knowledge in documents containing training data is mined and different types of prior graph structures are constructed to represent the intrinsic connections between different prior knowledge. The prior knowledge includes natural language grammatical features, sentence structure features, pronoun reference features, and keyword features that directly indicate relationships.

[0008] Various types of prior graph structures and relation pre-trained models are fused through a multi-channel model. After the relation pre-trained model encodes the document and the information in the prior graph structure, the weights of each prior graph structure are adaptively calculated for specific sentences.

[0009] By using the comprehensive information after weight aggregation, the document is used to extract and infer the relationships between entities, resulting in corresponding relation triples.

[0010] Optionally, the sentence structure features are calculated using the natural language processing tool Spacy, the pronoun reference features are obtained by matching prior rules, and the keywords of the direct indicator relation are obtained based on the shortest dependency path in the syntax tree.

[0011] Optionally, the steps for constructing the prior graph structure specifically include:

[0012] Find all entity words, pronouns, and relational keywords in the document;

[0013] Entity nodes, pronoun nodes, and keyword nodes are incorporated into the graph to strengthen the prior knowledge, and multiple different types of prior graph structures are constructed to address the diverse nature of prior knowledge.

[0014] In the prior graph structure, bidirectional edges are constructed between nodes corresponding to the words from the same sentence, between nodes of the pronoun and the entity word referred to by the pronoun, and between nodes of different names corresponding to the same entity word.

[0015] Optionally, the step of fusing various types of prior graph structures with the relation pre-trained model through a multi-channel model, encoding the document using the relation pre-trained model and the prior knowledge in the prior graph structures, and then adaptively calculating the weights of each prior graph structure for specific statements specifically includes:

[0016] Establish a relational pre-training model based on neural networks;

[0017] The prior knowledge is pre-trained using the relation pre-training model, and the representation of each relation is initialized to obtain the relation encoder;

[0018] The document is input into the relation encoder, which adaptively calculates the weights of each prior graph structure to obtain the embedded matrix after document encoding.

[0019] Optionally, in each round of training of the relation pre-trained model, the representation of the relation can be optimized by gradient descent.

[0020] Optionally, the prior knowledge is pre-trained using the relation pre-training model, and the representation of each relation is initialized. The pre-training loss function is defined as follows:

[0021]

[0022] in, It is a set of relations. It is a collection of documents, e s and e o These are the representations of the head entity and the tail entity, respectively, r ir is a binary variable that records whether a relationship exists between the head and tail entities. i A value of 1 indicates the existence of a relation, r i A value of 0 indicates that there is no relationship.

[0023] Optionally, the step of inputting the document into the relation encoder and adaptively calculating the weights of each prior graph structure to obtain the embedding matrix after document encoding specifically includes:

[0024] The document is input into the relation encoder to obtain the initial embedding matrix of the document, and the initial embedding matrix is ​​mapped to a query embedding matrix, a key-value embedding matrix, and a value embedding matrix;

[0025] Calculate the attention score for each prior graph structure based on the three matrices mentioned above;

[0026] Construct an aggregator based on the attention scores of each of the aforementioned prior graph structures;

[0027] The aggregator aggregates the attention scores of all prior graph structures to obtain the final attention score of the document, and the embedding matrix of the document after encoding is obtained based on the final attention score of the document.

[0028] Optionally, the document is input into the relation encoder to obtain the initial embedding matrix of the document as follows:

[0029]

[0030] Among them, Q l K l V l These correspond to the query embedding matrix, key-value embedding matrix, and value embedding matrix, respectively. This represents the embedding matrix of the document as input to the l-th layer of the relational encoder. These are the trainable model parameters in layer l.

[0031] Optionally, the initial embedding matrix is ​​mapped to a query embedding matrix, a key-value embedding matrix, and a value embedding matrix, and the attention score for each prior graph structure is calculated based on the above three matrices, using the following formula:

[0032]

[0033] in, It is the attention score for the structure of the i-th prior graph. A represents the trainable model parameters in layer l. i It is the binary adjacency matrix of the i-th prior graph structure.

[0034] Optionally, an aggregator is constructed based on the attention scores of each of the aforementioned prior graph structures, calculated as follows:

[0035]

[0036] in, is a trainable parameter that controls the degree of influence of different prior graph structures on the final relation encoder. M is the number of prior graphs, and l represents the number of attention layers.

[0037] Optionally, the attention scores of all prior graph structures are aggregated by the aggregator to obtain the final attention score of the document, and the embedding matrix of the document after encoding is obtained based on the final attention score of the document. The specific calculation is as follows:

[0038]

[0039] in, This represents the first embedding matrix output in the middle of the l-th layer of the relation encoder's encoding section;

[0040] By inputting the first embedding matrix into the forward propagation network, the document-encoded embedding matrix can be obtained:

[0041]

[0042] Where ReLU represents the activation function. and These are all weight parameters in a forward propagation network.

[0043] Optionally, the step of extracting and reasoning about relationships between entities in the document using the comprehensive information after weight aggregation to obtain the corresponding relation triples specifically includes:

[0044] Input the document into the embedded matrix after document encoding to obtain the feature vectors of all entities;

[0045] The feature vectors of the entities are passed through their respective linear layers, and the results encoded by the linear layers are then input into a bilinear classifier to determine whether a relationship exists between the entities. The specific calculation is as follows:

[0046]

[0047]

[0048] Among them, W s and W o These are the weight parameters of the linear layer corresponding to the feature vectors of the i-th and j-th entities, respectively, W. r These are the weight parameters of the bilinear layer, ei ,e j Representing the i-th entity and the j-th entity respectively. These represent the feature vectors corresponding to the i-th and j-th entities in the embedding matrix after document encoding, respectively.

[0049] In the document-level relation triple extraction method provided by this invention, the relationship between entities in the document is extracted and reasoned through the comprehensive information after weight aggregation. This helps to improve the model's ability to identify relevant information in complex document-level environments, thereby improving the extraction performance of relation triples that require multi-step reasoning, and thus improving the results of document-level relation triple extraction. Attached Figure Description

[0050] Those skilled in the art will understand that the accompanying drawings are provided to better understand the invention and do not constitute any limitation on the scope of the invention. Wherein:

[0051] Figure 1 This is a flowchart illustrating the steps of the document-level relation triple extraction method provided in this embodiment;

[0052] Figure 2 The flowchart shows the document-level relation triple extraction method provided in this embodiment. Detailed Implementation

[0053] To make the objectives, advantages, and features of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the drawings are all in a very simplified form and are not drawn to scale, and are only used to facilitate and clearly illustrate the objectives of the embodiments of the present invention. Furthermore, the structures shown in the drawings are often part of the actual structures. In particular, different figures may emphasize different aspects and sometimes use different scales. It should also be understood that, unless specifically stated or indicated, the terms "first," "second," "third," etc., in the specification are only used to distinguish the various components, elements, steps, etc., in the specification, and are not used to indicate the logical or sequential relationships between the various components, elements, steps, etc.

[0054] Please refer to Figure 1 , Figure 1 This embodiment provides a method for extracting document-level relation triples, including the following steps:

[0055] S1. Discover the prior knowledge in the documents containing training data and construct different kinds of prior graph structures to represent the intrinsic relationship between different prior knowledge. The prior knowledge includes natural language grammar features, sentence structure features, pronoun reference features and keyword features that directly indicate the relationship.

[0056] S2. The prior graph structures of various types are fused with the relation pre-trained model through a multi-channel model. After encoding the document and the information in the prior graph structure using the relation pre-trained model, the weights of each prior graph structure are adaptively calculated for specific sentences.

[0057] S3. Using the comprehensive information after weight aggregation, extract and reason about the relationships between entities in the document to obtain the corresponding relation triples.

[0058] First, step S1 is executed to mine prior knowledge in the documents containing training data and construct different types of prior graph structures to represent the intrinsic relationships between different prior knowledge. The prior knowledge includes natural language grammatical features, sentence structure features, pronoun reference features, and keyword features of direct indicative relationships. Specifically, the sentence structure features can be calculated using the natural language processing tool Spacy, the pronoun reference features can be obtained through prior rule matching, and the keywords of direct indicative relationships can be derived from the shortest dependency path in the syntax tree.

[0059] In this embodiment, the steps for constructing the prior graph structure specifically include:

[0060] First, find all entity words, pronouns, and relational keywords in the document;

[0061] Entity nodes, pronoun nodes, and keyword nodes are incorporated into the graph to strengthen the prior knowledge, and multiple different types of prior graph structures are constructed to address the diverse nature of prior knowledge.

[0062] In the prior graph structure, bidirectional edges are constructed between nodes corresponding to the words from the same sentence, between nodes of the pronoun and the entity word referred to by the pronoun, and between nodes of different names corresponding to the same entity word.

[0063] There are two rules to determine which entity a pronoun refers to: first, the entity the pronoun refers to must precede the pronoun; second, the type of the entity must conform to the referential category of the pronoun itself. A bidirectional edge must exist between the pronoun and the entity it refers to. If it is a possessive pronoun, then the first noun appearing after the pronoun should replace the pronoun, and then a bidirectional edge should be added to the prior graph structure.

[0064] Then, step S2 is executed, which involves fusing the various types of prior graph structures with the relation pre-trained model through a multi-channel model. After encoding the document using the relation pre-trained model and the prior knowledge in the prior graph structures, the weights of each prior graph structure are adaptively calculated for specific statements. This step specifically includes:

[0065] Establish a relational pre-training model based on neural networks;

[0066] The prior knowledge is pre-trained using the relation pre-training model, and the representation of each relation is initialized to obtain the relation encoder;

[0067] The document is input into the relation encoder, which adaptively calculates the weights of each prior graph structure to obtain the embedded matrix after document encoding.

[0068] Preferably, in each round of training of the relation pre-trained model, the representation of the relation can be optimized by gradient descent, so that the parameters of the model can be optimized by gradient descent, and the weights of each prior graph structure can also be optimized to the most appropriate proportion.

[0069] The prior knowledge is pre-trained using the aforementioned relation pre-training model, and the representation of each relation is initialized. The pre-training loss function is defined as follows:

[0070]

[0071] in, It is a set of relations. It is a collection of documents, e s and e o These are the representations of the head entity and the tail entity, respectively, r i r is a binary variable that records whether a relationship exists between the head and tail entities. i A value of 1 indicates the existence of a relation, r i A value of 0 indicates that there is no relationship.

[0072] In this embodiment, the steps of inputting the document into the relation encoder and adaptively calculating the weights of each prior graph structure to obtain the embedded matrix after document encoding specifically include:

[0073] The document is input into the relation encoder to obtain the initial embedding matrix of the document, and the initial embedding matrix is ​​mapped to a query embedding matrix, a key-value embedding matrix, and a value embedding matrix;

[0074] Calculate the attention score for each prior graph structure based on the three matrices mentioned above;

[0075] Construct an aggregator based on the attention scores of each of the aforementioned prior graph structures;

[0076] The aggregator aggregates the attention scores of all prior graph structures to obtain the final attention score of the document, and the embedding matrix of the document after encoding is obtained based on the final attention score of the document.

[0077] For specific details, please refer to... Figure 2 , Figure 2 This is a flowchart of the document-level relation triple extraction method provided in this embodiment. The document is input into the relation encoder to obtain an initial embedding matrix of the document, and the initial embedding matrix is ​​mapped to a query embedding matrix, a key-value embedding matrix, and a value embedding matrix, as detailed below:

[0078]

[0079] Among them, Q l K l V l These correspond to the query embedding matrix, key-value embedding matrix, and value embedding matrix, respectively. This represents the embedding matrix of the document as input to the l-th layer of the relational encoder. These are the trainable model parameters in layer l.

[0080] The attention score for each prior graph structure is calculated based on the three matrices mentioned above, using the following formula:

[0081]

[0082] in, It is the attention score for the structure of the i-th prior graph. A represents the trainable model parameters in layer l. i It is the binary adjacency matrix of the i-th prior graph structure.

[0083] Then, an aggregator is constructed based on the attention scores of each of the aforementioned prior graph structures, calculated using the following formula:

[0084]

[0085] in, These are trainable parameters that control the degree to which different prior graph structures influence the final relation encoder. M is the number of prior graphs, and l represents the number of attention layers. In this embodiment, the encoding part of the relation encoder consists of twelve layers. In the construction of the aggregator in each layer, the weights are continuously recalculated as the layer number increases, ultimately obtaining the most suitable values.

[0086] Then, the attention scores of all prior graph structures are aggregated by the aggregator to obtain the final attention score of the document. The embedding matrix of the document after encoding is obtained based on the final attention score of the document. The specific calculation is as follows:

[0087]

[0088] in, This represents the first embedding matrix output in the middle of the l-th layer of the relation encoder's encoding section;

[0089] By inputting the first embedding matrix into the forward propagation network, the document-encoded embedding matrix can be obtained:

[0090]

[0091] Where ReLU represents the activation function. and These are all weight parameters in a forward propagation network.

[0092] Finally, step S3, which involves extracting and reasoning about relationships between entities in the document using the comprehensive information after weight aggregation, to obtain the corresponding relation triples, specifically includes:

[0093] Input the document into the embedded matrix after document encoding to obtain the feature vectors of all entities;

[0094] The feature vectors of the entities are passed through their respective linear layers, and the results encoded by the linear layers are then input into a bilinear classifier to determine whether a relationship exists between the entities. The specific calculation is as follows:

[0095]

[0096]

[0097] Among them, W s and W o These are the weight parameters of the linear layer corresponding to the feature vectors of the i-th and j-th entities, respectively, W. r These are the weight parameters of the bilinear layer, e i ,e j Representing the i-th entity and the j-th entity respectively. These represent the feature vectors corresponding to the i-th and j-th entities in the embedding matrix after document encoding, respectively.

[0098] In summary, this invention provides a method for extracting document-level relation triples. By using comprehensive information after weight aggregation to extract and infer relationships between entities in a document, it helps improve the model's ability to identify relevant information in complex document-level environments, thereby improving the extraction performance of relation triples that require multi-step reasoning, and thus improving the results of document-level relation triple extraction.

[0099] Furthermore, it should be understood that although the present invention has been disclosed above with reference to preferred embodiments, these embodiments are not intended to limit the present invention. For any person skilled in the art, many possible variations and modifications can be made to the technical solutions of the present invention based on the disclosed technical content, or equivalent embodiments can be modified accordingly, without departing from the scope of the present invention. Therefore, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention, without departing from the content of the present invention, shall still fall within the scope of protection of the present invention.

Claims

1. A method for extracting document-level relation triplets, characterized in that, include: Prior knowledge in documents containing training data is mined and different types of prior graph structures are constructed to represent the intrinsic connections between different prior knowledge. The prior knowledge includes natural language grammatical features, sentence structure features, pronoun reference features, and keyword features that directly indicate relationships. Various types of prior graph structures and relation pre-trained models are fused through a multi-channel model. After encoding the document and the information in the prior graph structure using the relation pre-trained model, the weights of each prior graph structure are adaptively calculated for specific sentences. The document is used to extract and infer relationships between entities by using the comprehensive information after weight aggregation, and the corresponding relation triples are obtained. The steps of fusing various types of prior graph structures with relation pre-trained models through a multi-channel model, encoding documents using the relation pre-trained models and prior knowledge in the prior graph structures, and adaptively calculating the weights of each prior graph structure for specific statements specifically include: Establish a relational pre-training model based on neural networks; The prior knowledge is pre-trained using the relation pre-training model, and the representation of each relation is initialized to obtain the relation encoder; The document is input into the relation encoder, which adaptively calculates the weights of each prior graph structure to obtain the embedded matrix of the encoded document. This step specifically includes: The document is input into the relation encoder to obtain the initial embedding matrix of the document, and the initial embedding matrix is ​​mapped to a query embedding matrix, a key-value embedding matrix, and a value embedding matrix; Calculate the attention score for each prior graph structure based on the three matrices mentioned above; Construct an aggregator based on the attention scores of each of the aforementioned prior graph structures; The aggregator aggregates the attention scores of all prior graph structures to obtain the final attention score of the document, and the embedding matrix of the document after encoding is obtained based on the final attention score of the document.

2. The method of claim 1, wherein, The sentence structure features are calculated using the natural language processing tool Spacy, the pronoun reference features are obtained by matching prior rules, and the keywords of direct indicator relations are obtained based on the shortest dependency path in the syntax tree.

3. The method of claim 1, wherein, The steps for constructing the prior graph structure specifically include: Find all entity words, pronouns, and relational keywords in the document; Entity nodes, pronoun nodes, and keyword nodes are incorporated into the graph to strengthen the prior knowledge, and multiple different types of prior graph structures are constructed to address the diverse nature of prior knowledge. In the prior graph structure, bidirectional edges are constructed between nodes corresponding to the words from the same sentence, between nodes of the pronoun and the entity word referred to by the pronoun, and between nodes of different names corresponding to the same entity word.

4. The method of claim 1, wherein, In each round of training of the pre-trained relation model, the representation of the relation can be optimized using gradient descent.

5. The method of claim 4, wherein, The prior knowledge is pre-trained using the aforementioned relation pre-training model, and the representation of each relation is initialized. The pre-training loss function is defined as follows: in, It is a set of relations. It is a collection of documents. and These are the representations of the head entity and the tail entity, respectively. It is a binary variable that records whether a relationship exists between the beginning and end entities. A value of 1 indicates the existence of a relationship. A value of 0 indicates that there is no relationship.

6. The method of claim 1, wherein, The document is input into the relation encoder to obtain the initial embedding matrix of the document as follows: in, , , These correspond to the query embedding matrix, key-value embedding matrix, and value embedding matrix, respectively. The document represents the encoding section of the relation encoder. The embedding matrix of the layer input, They are the first Trainable model parameters in the layer.

7. The method of claim 6, wherein, The initial embedding matrix is ​​mapped to a query embedding matrix, a key-value embedding matrix, and a value embedding matrix. Attention scores for each prior graph structure are then calculated based on these three matrices, using the following formula: in, It is the first Zhang's attention score for the priori graph structure It is the first Trainable model parameters in the layer It is the first The binary adjacency matrix of Zhang's prior graph structure.

8. The method of claim 7, wherein, An aggregator is constructed based on the attention scores of each of the aforementioned prior graph structures, calculated using the following formula: in, These are trainable parameters that control the degree to which different prior graph structures influence the final relation encoder. M is the number of prior graphs. This represents the number of attention layers.

9. The method of claim 8, wherein, The aggregator aggregates the attention scores of all prior graph structures to obtain the final attention score of the document, and then obtains the embedded matrix of the document after encoding based on the final attention score. The specific calculation is as follows: wherein, representing the document in the encoding portion of the relational encoder the first embedding matrix output in the intermediate layer; By inputting the first embedding matrix into the forward propagation network, the document-encoded embedding matrix can be obtained: Where ReLU represents the activation function. , , and These are all weight parameters in a forward propagation network.

10. The method of claim 9, wherein, The specific steps of extracting and reasoning about relationships between entities in the document using the comprehensive information after weight aggregation to obtain the corresponding relation triples include: Input the document into the embedded matrix after document encoding to obtain the feature vectors of all entities; The feature vectors of the entities are passed through their respective linear layers, and the results encoded by the linear layers are then input into a bilinear classifier to determine whether a relationship exists between the entities. The specific calculation is as follows: in, as well as They are the first The entity and the first The weight parameters of the linear layer corresponding to the feature vectors of each entity. These are the weight parameters of the bilinear layer. Representing the first The entity and the first One entity, , Representing the first The entity and the first The feature vectors corresponding to each entity in the embedding matrix after document encoding.

Images