A "table-graph" two-stage-based document-level entity relationship joint extraction method and device

Abstract

The application relates to a document-level entity relationship joint extraction method and device based on a 'table-graph' two-stage method. The method comprises an encoding stage and a decoding stage; the encoding stage comprises the following steps: after word segmentation is performed on a text to be processed, a trained sequence labeling model is inputted to perform reference extraction; the reference is inputted into a trained 'table-graph' two-stage model to predict a co-reference score and a relationship score; the decoding stage comprises the following steps: a hierarchical clustering method is used to perform co-reference resolution decoding by using the co-reference score and the relationship score; and for an entity cluster obtained by the hierarchical clustering method, a mode voting method is used to perform relationship extraction decoding. The targeted design of the application in the encoding and decoding stages can effectively alleviate the error accumulation problem between the co-reference resolution and the relationship extraction, promote the semantic exchange between different sub-tasks, and improve the information extraction performance.

Description

Technical Field

[0001] This invention relates to the field of natural language processing, specifically to a method and apparatus for jointly extracting entities and relations from long texts at the document level using a neural network model. Background Technology

[0002] Currently, with the explosive growth of information in the internet age, massive amounts of text data have emerged. How to automatically extract valuable structured data from this vast amount of raw text data is a crucial issue in data analysis and information extraction. Among these, extracting high-quality information from long texts is more relevant to real-world scenarios and also presents a greater challenge compared to extracting information from short texts.

[0003] In long texts at the document level, entities often have multiple referents. Therefore, end-to-end joint entity relation extraction can often be divided into three sub-techniques: referent extraction, which identifies the text corresponding to an entity from the text; coreference resolution, which establishes coreference relations for the extracted referents to form an entity set; and relation extraction, which determines the relationship between entity pairs in the entity set. Previous technical solutions mainly used neural networks and pre-trained language models as feature extractors, and trained and predicted each sub-task in a pipeline manner according to the above division to extract entities and relations. To share useful information across different subtasks, some technical solutions propose using multi-task models for joint modeling, sharing a basic feature extractor across all subtasks (Markus Eberts and Adrian Ulges. 2021. An end-to-end model for entity-level relation extraction using multi-instance learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3650–3660, Online. Association for Computational Linguistics.). However, these solutions still cannot overcome the error propagation problem in pipelined models. During decoding, errors from previous subtasks can trigger cascading errors in subsequent subtasks, significantly impacting the model's accuracy. Summary of the Invention

[0004] To overcome the error accumulation problem in existing technical solutions, this invention provides a two-stage "table-graph" generation method and apparatus for document-level entity and relationship joint extraction, which can effectively extract structured information, including entities and relationships, from long texts.

[0005] The technical solution adopted in this invention is as follows:

[0006] A document-level entity relation joint extraction method based on a two-stage "table-graph" approach, comprising an encoding stage and a decoding stage;

[0007] The encoding stage includes:

[0008] After the text to be processed is segmented into words, it is input into the trained sequence labeling model for semantic extraction.

[0009] The two-stage "table-graph" model, which is trained using the input, is used to predict the coreference score and relation score.

[0010] The decoding stage includes:

[0011] Using coreference scores and relation scores, hierarchical clustering is employed to perform coreference resolution and decoding.

[0012] For entity clusters obtained by hierarchical clustering, the mode voting method is used for relation extraction and decoding.

[0013] Furthermore, the sequence labeling model is used to refer to the extraction subtask and is trained according to the BIO labeling format to predict the BIO category of the text.

[0014] Furthermore, the "table-graph" two-stage model is used for the coreference resolution subtask and the relation extraction subtask, treating the references as nodes. The coreference resolution subtask predicts whether the references point to the same entity between reference pairs, i.e., a 0 / 1 score; the relation extraction subtask predicts the relationship between the entities corresponding to the references between reference pairs, i.e., a multi-classification task.

[0015] Furthermore, the "table-graph" two-stage model includes:

[0016] In the table stage, the BERT model is first used to extract features from the original text to obtain the representation of the pronouns. Then, a biaffine transformation is used to predict the co-reference score and relation score for each pronoun pair.

[0017] In the graph phase, coreference fractions and relational fractions are used as edge weights on a dynamic graph with referents as nodes to construct corresponding coreference edges and relational edges; then, grammatical edges are statically constructed based on the grammatical structure between referents; for the original representations of the three types of edges and referents, the node representations are encoded using a relational graph neural network R-GCN.

[0018] Using the node representations obtained from the relational graph neural network R-GCN, we then use biaffine transformation to predict the final coreference score and relational score.

[0019] Furthermore, when calculating the distance between different clusters, the hierarchical clustering method considers two parts: one part is the coreference score predicted at the end of the encoding stage, and the other part is the penalty term calculated by the relation score; the weighted sum of the two parts is used as the inter-cluster distance.

[0020] Furthermore, the Hamming distance between the referential relation vectors is used as the penalty term.

[0021] Furthermore, the method of using the majority voting method for relation extraction and decoding includes: given a target entity pair, checking the relation prediction results of all referential pairs between the entity pairs, and taking the relation with more than half of the votes as the predicted relation between the entity pairs.

[0022] A document-level entity relation joint extraction system based on a two-stage "table-graph" approach, comprising an encoding module and a decoding module;

[0023] After the encoding module segments the text to be processed into words, it inputs the trained sequence labeling model to extract the references, and then inputs the references into the trained "table-graph" two-stage model to predict the co-reference score and relation score.

[0024] The decoding stage utilizes coreference scores and relation scores, employs hierarchical clustering to perform coreference resolution decoding, and uses the mode voting method to extract and decode relations for entity clusters obtained by the hierarchical clustering method.

[0025] The beneficial effects of this invention are as follows:

[0026] The targeted design of this invention in both encoding and decoding (a two-stage "table-graph" model and a hierarchical clustering method) can effectively alleviate the error accumulation problem between the two subtasks of coreference resolution and relation extraction, and promote semantic exchange between different subtasks, thereby improving information extraction performance. Attached Figure Description

[0027] Figure 1 This is a task illustration diagram of the present invention. Document-level relationship extraction requires extracting references, entities, and relationships from long texts.

[0028] Figure 2 This is a flowchart of the coding part of the present invention, which mainly shows the coding process of the "table-graph" model. Detailed Implementation

[0029] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0030] The method of this invention can be mainly divided into two stages: encoding and decoding. The main steps of the encoding stage include:

[0031] (1) Use a tokenizer to segment the text and convert it into characters in a dictionary;

[0032] (2) A sequence labeling model was constructed specifically for the subtask of referential extraction, and trained according to the labeling format B (begin), I (inside), O (outside). The sequence labeling model used BERT as the feature extractor and used linear layers to predict the sequence category of characters (i.e., BIO sequence). Here, referentials represent continuous text segments that refer to specific entity concepts; referentials can be obtained from BIO sequences. For example, starting from each B label position, the continuous text segment in the form of (BI-…-I)-O is the referential.

[0033] (3) Jointly modeling the coreference resolution subtask and the relation extraction subtask: Treating references as nodes, the coreference resolution subtask needs to predict whether reference pairs point to the same entity, i.e., a 0 / 1 score; considering relation extraction at the reference level, the relation extraction subtask can be seen as predicting the relationship between the entities corresponding to reference pairs, i.e., a multi-classification task. Therefore, under this unified modeling, the same two-stage "table-graph" model can be used to encode the embedding representations of these two tasks.

[0034] (4) In the table stage, 0 / 1 scores are predicted for both subtasks. First, the BERT model is used to extract features from the original text to obtain the representation of the referents. Then, a biaffine transformation is used to predict the coreference score and relation score for each referent pair (mi, mj).

[0035] Formula for calculating common fractions:

[0036]

[0037] Formula for calculating relational scores:

[0038]

[0039] in, This indicates a fraction that refers to the core reference of (mi, mj). This indicates the relational fraction referring to (mi, mj). This refers to the original characteristic of mi. W represents the original characteristics of mj.COREF,1 w COREF,2 b COREF W RE,1 w RE,2 b RE These are trainable parameters, T represents the transpose operation, and ⊕ represents the vector concatenation operation. Finally, the softmax function is used to normalize the scores.

[0040] The resulting n*n coreference scores are used as a coreference table, and the resulting n*n relation scores are used as a relation table. Here, a table is an n*n matrix, which can be viewed as the adjacency matrix corresponding to the graph with references as nodes.

[0041] (5) In the graph stage, the coreference score and relation score obtained in step (4) are used as edge weights on the dynamic graph with the referents as nodes to construct the corresponding coreference edges and relation edges. In addition, for the grammatical structure between referents, syntax edges are constructed statically (fixed after the referents are determined). The specific rule is: if two referents appear in the same sentence, the edge weight between them is 1, otherwise it is 0. For the original representations of the three types of edges and referents, the node representations are encoded using a relational graph convolutional network (R-GCN).

[0042] (6) Finally, using the node representation obtained by the relational graph neural network R-GCN, z in the calculation formula is replaced, and the biaffine transformation is used again to predict the final coreference score and relational score to complete the encoding.

[0043] The main steps in the decoding stage include:

[0044] (1) Reference extraction: The BIO sequence output by the sequence labeling model is restored to the predicted reference.

[0045] (2) Coreference resolution: Using the coreference scores and relation scores predicted during the encoding stage, the hierarchical agglomerative clustering (HAC) method is used to perform coreference resolution decoding.

[0046] When calculating the distance between different clusters, two parts are considered: one is the coreference score predicted at the end of the encoding stage, and the other is a penalty term calculated based on the relation score. Considering that during training, reference pairs pointing to the same entity have the same relation label, the Hamming distance between the relation vectors of the references is used as an additional penalty term, calculated as follows:

[0047]

[0048] Among them, |C x | represents the number of nodes in the current cluster x, |C y | represents the number of nodes in the current cluster y, σ represents the sigmoid function, and r i R represents the relation vector referring to i. j This represents the relation vector referring to j.

[0049] The relation vector for the referents (nodes) is constructed as follows: If there is a relation r between the predicted referents mi and mj, then Otherwise, it is 0. Let the relation vector referencing the pair (mi, mj) be denoted by . Then it refers to the relation vector of mi. Where n is the number of pronouns, n r For the type of relationship.

[0050] Finally, the weighted sum of these two parts is used as the inter-cluster distance.

[0051] (3) Relation extraction: Relation extraction and decoding are performed using a majority voting mechanism. Given a target entity pair, the relation prediction results of all referential pairs between the entity pairs are checked, and the relation that receives more than half of the votes is taken as the predicted relation between the entity pairs.

[0052] The application of this method involves two steps: the first step is to train an encoding model using a labeled text dataset and adjust the decoding hyperparameters on a validation set; the second step is to use the trained encoding model and decoding algorithm to identify entities and relationships in the target text. The first and second steps do not need to be performed consecutively; the first step can be performed alone to obtain a pre-trained model, and then the second step can be performed on new text for recognition.

[0053] This invention can be applied to the extraction of any type of relationship and entity. A specific embodiment is provided below. This embodiment is a two-stage "table-graph" generation method for document-level entity-relation joint extraction. Taking information extraction from encyclopedia text as an example, the aim is to automatically extract entities and relationships between entities from text data, such as... Figure 1 As shown.

[0054] The main steps of the training phase are as follows:

[0055] (1) First, it is necessary to construct the training data required for training the model. The training data needs to be labeled with the start and end range of references, coreference relationships, and relationships between entities.

[0056] (2) Segment the text data in the training data. In this embodiment, the word segmenter corresponding to the BERT model is used to perform word segmentation.

[0057] (3) Training the referential extraction submodule. The corresponding model is BERT as the feature extractor, which outputs the BIO category through a linear layer.

[0058] (4) Use the references, coreferences and relations labeled in the training data to construct labels, and train the coreference resolution submodule and the relation extraction submodule, i.e., the "table-graph" two-stage model.

[0059] (5) Finally, the model can output the 0 / 1 score of coreference resolution among all reference pairs and the relation score for each category.

[0060] The main steps in the testing phase are as follows:

[0061] (1) Segment the text data into words;

[0062] (2) Use the trained referential extraction submodule to predict the BIO category of the text and decode it into a referential range;

[0063] (3) Input the predicted references into the "table-graph" two-stage model to predict the coreference score and relation score;

[0064] (4) Use hierarchical clustering to perform coreference resolution decoding;

[0065] (5) For the entity clusters obtained by decoding, the relation extraction and decoding are performed using the mode voting method.

[0066] The DocRED dataset, constructed from Wikipedia text, was used for training with 3053 documents and for testing with 1000 documents. The results are shown in Table 1.

[0067] Table 1

[0068] Previous optimal solution 93.35 84.96 40.62 This patent solution 93.84 86.37 44.97

[0069] The experimental results of this example demonstrate that the method of this patent can bring about effective performance improvements and can better extract entities and relationships in combination compared with the currently commonly used models.

[0070] Another embodiment of the present invention provides a document-level entity relation joint extraction system based on a two-stage "table-graph" approach, including an encoding module and a decoding module:

[0071] After the encoding module segments the text to be processed into words, it inputs the trained sequence labeling model to extract the references, and then inputs the references into the trained "table-graph" two-stage model to predict the co-reference score and relation score.

[0072] The decoding stage utilizes coreference scores and relation scores, employs hierarchical clustering to perform coreference resolution decoding, and uses the mode voting method to extract and decode relations for entity clusters obtained by the hierarchical clustering method.

[0073] For the specific implementation process of each module, please refer to the description of the method of the present invention above.

[0074] Another embodiment of the present invention provides a computer device (computer, server, smartphone, etc.) including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the steps of the method of the present invention.

[0075] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk) storing a computer program that, when executed by a computer, implements the various steps of the method of the present invention.

[0076] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and to implement it accordingly. Those skilled in the art will understand that various substitutions, changes, and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the content disclosed in the embodiments of this specification; the scope of protection of the present invention is defined by the claims.

Claims

1. A document-level entity relation joint extraction method based on a two-stage "table-graph" approach, characterized in that... Includes the encoding and decoding stages; The encoding stage includes: After the text to be processed is segmented into words, it is input into the trained sequence labeling model for semantic extraction. The two-stage "table-graph" model, which is trained by referring to the input, is used to predict the coreference score and relation score. The decoding stage includes: Using coreference scores and relation scores, hierarchical clustering is employed to perform coreference resolution and decoding. For entity clusters obtained by hierarchical clustering, the mode voting method is used for relation extraction and decoding; The "table-graph" two-stage model is used for the coreference resolution subtask and the relation extraction subtask. The reference is regarded as a node. The coreference resolution subtask predicts whether the references point to the same entity between reference pairs, i.e., a 0 / 1 score. The relation extraction subtask predicts the relationship between the entities corresponding to the references between reference pairs, i.e., a multi-classification task. The "table-graph" two-stage model includes: In the table stage, the BERT model is first used to extract features from the original text to obtain the representation of the pronouns. Then, a biaffine transformation is used to predict the co-reference score and relation score for each pronoun pair. In the graph phase, coreference fractions and relational fractions are used as edge weights on a dynamic graph with referents as nodes to construct corresponding coreference edges and relational edges; then, grammatical edges are statically constructed based on the grammatical structure between referents; for the original representations of the three types of edges and referents, the node representations are encoded using a relational graph neural network R-GCN. Using the node representation obtained from the relational graph neural network R-GCN, the final coreference score and relation score are predicted again using biaffine transformation; The hierarchical clustering method considers two parts when calculating the distance between different clusters: one part is the coreference score predicted at the end of the encoding stage, and the other part is the penalty term calculated by relation score; the weighted sum of the two parts is used as the inter-cluster distance.

2. The method according to claim 1, characterized in that, The sequence labeling model is used to refer to the extraction subtask and is trained according to the BIO labeling format to predict the BIO category of the text.

3. The method according to claim 1, characterized in that, The Hamming distance between the referential relation vectors is used as the penalty term.

4. The method according to claim 1, characterized in that, The method of using the majority voting method for relation extraction and decoding includes: given a target entity pair, checking the relation prediction results of all reference pairs between the entity pairs, and taking the relation with more than half of the votes as the predicted relation between the entity pairs.

5. A document-level entity relation joint extraction system based on a two-stage "table-graph" approach, characterized in that... The system for performing the method according to any one of claims 1 to 4 includes an encoding module and a decoding module: After the encoding module segments the text to be processed into words, it inputs the trained sequence labeling model to extract the references, and then inputs the references into the trained "table-graph" two-stage model to predict the co-reference score and relation score. The decoding module uses coreference scores and relation scores to perform coreference resolution decoding using a hierarchical clustering method, and uses a mode voting method to extract and decode relations for entity clusters obtained by the hierarchical clustering method.

6. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer, implements the method according to any one of claims 1 to 4.

Images