Document level relationship extraction fusing enhanced entity and multi-level representation

Abstract

The invention relates to the technical field of document-level relation extraction, and discloses document-level relation extraction and preprocessing integrating an enhanced entity and multi-level representation, aiming at integrating a plurality of mentions and obtaining global representation of the entity, for a document D with the length of L, a given document # imgabs0 # representation, and xt represents a token at a position t (in natural language processing, the document is extracted, and the document is extracted and preprocessed. Token usually refers to a minimum semantic unit in a text, such as a word, a sub-word or a character, is used as a marking entity, a special mark * is used for marking the starting position and the ending position of the entity, and a context embedding vector H of a document D is obtained through a pre-trained BERT model. According to the document-level relation extraction of the fusion enhanced entity and the multi-level representation, the proposed MFDRE provides an effective solution for the long-distance dependence problem in the DocRE, the MFDRE obtains the global representation of the entity in the document by using a pre-training language model BERT, the local representation of the fusion entity pair and the supplementary representation of the entity pair under span expansion are fused, and the representation capability of the entity node is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of document-level relationship extraction, and in particular to document-level relationship extraction that integrates enhanced entities and multi-level representations. Background Art

[0002] Document-Level Relation Extraction (DocRE) is an important task in the field of natural language processing, which aims to extract and predict the relationship between entity pairs from the entire document. Unlike sentence-level relation extraction, document-level relation extraction needs to deal with long-distance dependencies across sentences, which makes the task more complex and challenging. In order to meet this challenge, the academic community has proposed a variety of methods and models in recent years, including the use of deep learning models such as GAT, ATLOP, KD-BERT, etc., as well as various enhanced models that integrate graph neural networks.

[0003] However, these DocRE methods still have shortcomings in dealing with multi-level relationships, especially in the use of entity span extension and mention. Span refers to a continuous subsequence related to an entity in the text, that is, the contextual information around the entity, and the existing methods are insufficient in span extension, which requires the use of span extension to obtain enhanced representation of entity pairs. As for mention, it refers to the specific reference to the entity in the document, recording the location and expression of the entity, often presented through nouns, pronouns or synonyms. For example, in Figure 1 In the DocRED dataset example, the mentions of the entity "Kungliga Hovkapelle" include "Kungliga Hovkapelle", "The Royal Court Orchestra" and "It". In the DocRE task, relations exist at multiple levels, in entity pairs, mention pairs and entity mention pairs, which require multi-level representation through heterogeneous graphs. Summary of the invention

[0004] In order to solve the problems raised in the above background technology, the present invention provides the following technical solution: document-level relationship extraction integrating enhanced entity and multi-level representation, comprising the following steps:

[0005] S1. Multi-level fusion document-level relation extraction model

[0006] S101, Pretreatment

[0007] The purpose of preprocessing is to integrate multiple mentions and obtain a global representation of the entity. For a document D of length L, given a document Indicates that x tRepresents the token at position t (in natural language processing, a token usually refers to the smallest semantic unit in a text, such as a word, subword, or character). To mark an entity, a special marker "*" is used to indicate the start and end positions of the entity. The context embedding vector H of document D is obtained through the pre-trained BERT model, as shown in formula (1):

[0008] H = BERT([x1,x2,…x L ])=[h1,h2,…h L ] (1)

[0009] Where H = [h1,h2,…h L ] is the hidden state representation of the token;

[0010] The entity e i All mentions of are embedded as Where m j Indicates the location of the mention, since each entity e i May appear multiple times in a document, so each entity is represented as the set of all mentions In order to obtain the global table of entities i As shown, using the logsumexp pooling method, by mentioning the number Weighted pooling of all mention embeddings This captures the contextual information of entities at different locations. See formula (2) for details:

[0011]

[0012] in, It is entity e i Pooling features;

[0013] S102. Enhanced entity representation

[0014] The enhanced entity representation module includes entity pair local representation, entity pair supplementary representation and fused entity representation.

[0015] The n entities in the document can be combined to form n(n-1) entity pairs, each of which has a corresponding local representation. Given a pre-trained multi-head attention matrix A ijk ∈R H×L×L , where H represents the number of attention heads, L represents the document length, and the matrix A ijk represents the attention of the i-th attention head from tokenj to tokenk;

[0016] First, the attention of the "*" symbol is regarded as the mention-level attention, and then the mention attentions of the same entity are averaged to get the entity-level attention Indicates the attention of the i-th entity to all tokens;

[0017] Next, for a given entity pair (e s ,e o ), locate the head entity e through entity-level attention s and tail entity e o All important local representations are obtained, and the local embedding vector C is obtained using formula (3-5) (s,o) :

[0018]

[0019] c (s,o) =H T (q (s,o) / 1 T q (s,o) ) (5)

[0020] Among them, A (s,o) Represents an entity pair (e s ,e o )’s aggregate attention, q (s,o) ∈R l represents its average attention weight, H is the context embedding vector in formula (1);

[0021] Then, the local embedding vector c (s,o) With entity e s The global representation of Combined, a personalized entity representation is provided for each entity pair, as shown in formula (6):

[0022]

[0023] in, Represents entity e s The local representation of W s , W c1 is the weight parameter;

[0024] According to formula (7), the entity e is obtained in the same way o Local representation of

[0025]

[0026] Among them, W o , W c2 is the weight parameter;

[0027] Sentence-level relation extraction has limited effect in DocRE tasks because entities are usually scattered across multiple sentences or paragraphs in a document and have various forms of expression. Traditional sentence-level methods tend to bias towards common categories and ignore rare categories, thus affecting the accuracy and generalization ability of the model.

[0028] To solve this problem, the span extension strategy is adopted, which aims to balance the data distribution by introducing more negative samples and enhance the model's ability to identify potential entities. The span in DocRE refers to the interval of entities at different positions in the document, including the start and end positions. When extending, the initial range of the entity is extended by 3 tokens to the front and back sides. For example, in the sentence "Beijing is the capital of China", the span extension strategy can identify multiple fine-grained entities, such as "Beijing", "China", "China", "China's" and "capital".

[0029] Then, according to formula (8), the entity after span expansion Combine them in pairs to form an entity pair set P, with a set length of n(n-1);

[0030] P=[(e1,e2),(e1,e3),…(e n-1 ,e n )] (8)

[0031] Then, based on the set P, the index of the related sentences is extracted for each entity pair, and these sentences are concatenated to generate new sentences through string concatenation. For example, entity 1 comes from sentence 1 and sentence 2, and entity 2 comes from sentence 3. After concatenation, the new sentence consists of sentence 1, sentence 2 and sentence 3. According to formula (9), a new sentence set S is created, which contains the related sentences of each entity pair, and the length of the set is n(n-1);

[0032] S=(s1,s2,…s n×(n-1) ) (9)

[0033] According to formula (10), BERT is used to process the sentence set S to obtain a complementary representation set B of entity pairs, with a set length of n(n-1);

[0034] B=BERT(S)=[b1,b2,…b n×(n-1) ] (10)

[0035] According to formula (11-13), the head entity representation v is generated by weighted linear fusion of the local representation and the supplementary representation of the entity pair s , the tail entity represents v o and enhanced entity representation v e ;

[0036] v s =Wz z s +W b b so (11)

[0037] v o =W z z o +W b b so (12)

[0038] v e = {v s ,v o} (13)

[0039] Among them, z s and z o They represent the local representation of the head entity and the tail entity respectively, and b so represents the complementary representation of entity pairs, W z and W b are the weight parameters of local representation and supplementary representation respectively;

[0040] S103, multi-level representation

[0041] In DocRE, entity relations exist not only between explicitly annotated entity pairs, but also between mention pairs in the document. However, it is difficult for sequence models to handle the complex relations between entity pairs and mention pairs at the same time. Therefore, this paper proposes a multi-level representation method based on heterogeneous graphs to capture these two types of relations.

[0042] The heterogeneous graph consists of four types of nodes: entity nodes, mention nodes, sentence nodes, and document nodes. Each type of node represents the structure of a document at a different level. Through the interaction of nodes at each level, the relationship features in the document can be effectively extracted.

[0043] Entity nodes represent specific entities in a document, such as people, places, and events. As the basic unit in the DocRE task, they reflect the multidimensional characteristics of entities in various contexts. According to formula (14), entity node h E The enhanced entity representation v corresponding to step S102 e ;

[0044] h E =v e (14)

[0045] The mention node reflects the contextual relationship of the entity through the connection with the entity node. Formula (15) shows that the mention node H is the hidden state of all tokens in the entity after BERT processing J Mean pooling, J represents the total number of mentions of the entity in the document;

[0046]

[0047] The sentence node represents each sentence in the document. By connecting the entity node and the mention node, it reflects the distribution and relationship description of the entity mentions in the sentence. According to formula (16), the sentence node H is the hidden state of all tokens in the sentence after BERT processing K Mean pooling, K represents the total number of sentences in the document;

[0048]

[0049] The document node represents the global context and topic information of the entire document. As the root node of the heterogeneous graph, it helps understand the long-distance relationship across sentences by connecting with other nodes. According to formula (17), the document node h D Represents the hidden state H of all tokens in the document after BERT processing N Mean pooling, N represents the total number of tokens in the document;

[0050] h D =Average(H N ) (17)

[0051] The edge representation of the heterogeneous graph fully considers the characteristics of the DocRE task. Sentence-level relation extraction assumes that the relation only exists in a single sentence, and entity pairs directly reflect the relation. However, in DocRE, entity relations often span multiple sentences or even the entire document. Relation extraction based solely on entity pairs has limitations. In addition, the same entity may have multiple mentions, and it is difficult to determine which mentions are related. Relying solely on mentions to mark relations may oversimplify the relation representation.

[0052] To bridge the gap between entity pairs and mention pairs and fully capture relational facts, multi-level representation divides DocRE's relation representation into two levels: direct relation representation and indirect relation representation. Direct relation representation explicitly marks relations through entity pairs, while indirect relation representation infers implicit relations between entities through mention pairs and entity mention pairs.

[0053] The target relationship of an entity pair is expressed as I1 = (e s ,e o ), if the relationship exists between entity pairs, entity e s and e o Constitute an entity pair, and the target relationship of the mention pair is expressed as I2 = (m s ,m o ), if the relationship exists in the mention pair, the mentions from different entities can be combined in pairs to form a mention pair to represent the relationship, for example, entity e s =[m1,m2,m3] and eo = [m4,m5,m6] each contains 3 mentions forming 9 mention pairs:

[0054] (m1,m4),(m1,m5),(m1,m6),(m2,m4),(m2,m5),(m2,m6),(m3,m4),(m3,m5),(m3,m6);

[0055] The target relation of an entity mention pair is represented as I3 = (e s ,m o ) / (e o ,m s ), if the relation exists in the entity mention pair, entity e s =[m1,m2,m3] and e o = [m4,m5,m6] contains 6 entity mention pairs: (e1,m4), (e1,m5), (e1,m6), (e2,m1)(e2,m2), (e2,m3);

[0056] Based on the above three relationship representations, six edge types are constructed in the heterogeneous graph: sentence-document edge connects all sentence nodes and document nodes, indicating the affiliation between sentences and documents; sentence-sentence edge connects all sentence nodes in the document, indicating the information flow between sentences; sentence-entity edge connects sentences containing entities and entity nodes, indicating the existence of entities in sentences; sentence-mention edge connects mention nodes and corresponding sentence nodes, indicating the role of mentions in sentences; entity-mention edge connects entity nodes and mention nodes related to entities, indicating entity mention pairs in documents; mention-mention edge connects two mention nodes under the same entity pair, reflecting the potential entity pair relationship;

[0057] Through the above node design and edge representation, the heterogeneous graph realizes the multi-dimensional semantic integration of words, sentences, and documents, which not only captures the implicit relationship between entities, but also reveals the overall semantic connection of documents;

[0058] S104, Feature Propagation

[0059] Traditional graph convolutional networks (GCNs) are only applicable to homogeneous graphs, that is, all edges in the graph are of the same type. In the DocRE task, the graph structure is usually heterogeneous, containing multiple types of nodes and edges. The relational graph convolutional network (R-GCN) effectively handles multiple relationships in the graph by defining an independent weight matrix for each relationship type. In R-GCN, the features of a node are updated by the features of neighboring nodes and the influence of different relationship types. The core idea of ​​R-GCN is to capture the complex relationships between nodes by distinguishing different types of edges.

[0060] Assume that the graph after integration in step S103 is a directed graph G = {V, E}, where the node set V includes four types: entity nodes, mention nodes, sentence nodes, and document nodes; E represents a directed edge set, and the edge has a type, which is defined as the relationship r from node i to node j, expressed as (i, r, j). The update formula of R-GCN for the node can be expressed as formula (18):

[0061]

[0062] Among them, u represents the current layer number, f i u+1 represents the updated features of node i at layer u+1, f i u represents the characteristics of node i, represents the characteristics of neighbor node j; R is a relationship set, r is a specific relationship type; N r (i) represents the set of neighbors connected to node i through relationship r; is the weight matrix of the corresponding relation r in the uth layer, is the linear transformation matrix of node i’s feature, c i,r is the normalization coefficient, which usually represents the number of neighbors of node i on relation r, and σ represents the sigmoid activation function;

[0063] In order to fully extract and fuse graph features at different levels, this paper uses a stacked multi-layer relational graph convolutional network for feature propagation. The node vector output by each layer of R-GCN represents the representation of the node at that layer, including the node's own attributes and the global information of multi-hop neighbors. As the number of layers increases, the node vector gradually expands from local to global graph features. After completing multi-layer propagation, the node vectors output by each layer of R-GCN are concatenated. The representation of the nth layer of R-GCN is: Where N represents the Nth node;

[0064] The feature propagation of stacked multi-layer relational graph convolutional networks integrates the features of each layer of graphs, retaining the original attributes and neighborhood structure expression of shallow nodes, and incorporating deep-level long-range dependencies and global pattern learning, so that the final vector of the node covers multi-level topological information and relational features, showing powerful expression capabilities;

[0065] S105. Relationship Classification

[0066] For the entity pair (e s ,e o ), the target relationship represents I so To connect I1, I2 and I3 in step S103, as shown in formula (19):

[0067] I so =[I1,I2,I3] (19)

[0068] Next, as shown in formula (20), under the sigmoid activation function, the graph propagation feature F of stacked multi-layer R-GCN is used n , through the fully connected feedforward neural network (FFNN) to the entity pair (e s ,e o )’s target relation representation I so Make predictions and output relationship categories;

[0069] y=σ(FFNN(I so |F n )) (20).

[0070] S2. Experimental Results and Analysis

[0071] S201, Dataset, Experimental Settings and Evaluation Metrics

[0072] This paper first uses DocRED as the experimental dataset. DocRED is a large-scale manually annotated DocRE dataset, which is built on the basis of Wikipedia and Wikidata using a remote supervision method. DocRED has two notable features: First, it contains 96 different relationship types, 132,375 entities, 56,354 annotated relationship facts, and covers 5,053 Wikipedia documents. It is currently the DocRE dataset with the largest data volume. Second, at least 40.7% of relationship facts are scattered in multiple sentences. Relying on a single sentence is not enough to determine the relationship within the document, and contextual understanding across sentences is required. The statistical information about DocRED in the training set, development set, and test set is shown in Table 1.

[0073] Table 1 DocRED dataset statistics

[0074]

[0075] DocRED-E2E is an end-to-end document-level information extraction dataset based on DocRED. It contains 96 different relation types, and each document contains an average of 19.4 entities and 12.5 facts. In DocRED-E2E, the relation classification task requires the model to be able to extract entity relations spanning multiple sentences from the document. In addition, the category balance problem is another challenge. The model needs to effectively deal with the distribution differences between common and rare relation types.

[0076] The experimental configuration information is shown in Table 2, including hardware facilities and software environment such as CPU model, GPU model, memory capacity, video memory capacity, operating system version, programming language, and deep learning framework version. The parameter settings of the training model are shown in Table 3, including maximum sequence length, batch size, number of labels, learning rate, warm-up rate, learning rate, number of iterations, dropout rate, maximum gradient norm, etc.

[0077] In terms of evaluation indicators, this paper uses F1 value and Ign F1 value as the core indicators for evaluating model performance. F1 value comprehensively measures precision and recall, and is defined as the harmonic mean of the two, which fully reflects the accuracy and coverage of the model in relational instance recognition. Ign F1 value eliminates repeated relational instances in the training set and development / test set, avoids the calculation of F1 scores for repeated triples in the training data, and reduces the interference of data redundancy on model evaluation.

[0078] Table 2 Experimental environment setup

[0079]

[0080]

[0081] Table 3 Model training parameters

[0082]

[0083] The calculation of precision, recall and F1 value is shown in formula (21-23), where a true positive (TP) indicates that the model accurately predicts that there is actually a relationship, a true negative (TN) indicates that the model accurately predicts that there is actually no relationship, a false positive (FP) indicates that the model incorrectly predicts that there is a relationship but there is actually no relationship, and a false negative (FN) indicates that the model incorrectly predicts that there is no relationship but there is actually a relationship.

[0084]

[0085] S202, comparative experiment

[0086] As shown in Table 4, on the DocRED dataset, RoBERTa-large (a variant of BERT, containing about 340M parameters, 24 encoder layers, each encoder layer containing 16 attention heads and 1024 hidden neurons) is used as the pre-trained language model, and MFDRE is compared with a variety of latest models. The main comparison models and their comparison results are as follows:

[0087] (1) DREEAM: It introduces a local context embedding mechanism to capture the dependencies between entities and uses evidence information as a supervisory signal to further optimize the performance of DocRE.

[0088] (2) KD: Through axial attention mechanism, adaptive focal loss and knowledge distillation, it effectively alleviates the problems of category imbalance and missing entity interaction information.

[0089] (3) SSAN: A structured self-attention network is introduced in the encoding layer, and two transformation modules are used to integrate the structural information of the entity into the attention mechanism to achieve more accurate structured relationship modeling.

[0090] (4) SAIS: Improve the accuracy of the model by using methods such as explicit modeling of key information sources, evidence retrieval, and data enhancement.

[0091] (5) Eider: Improves the model's efficiency in utilizing evidence sentences through lightweight model design and joint relation and evidence sentence extraction.

[0092] (6) DocuNet: Innovatively regarded the relationship extraction task as a semantic segmentation problem, using the U-shaped segmentation module and balanced Softmax optimization strategy, and achieved remarkable results.

[0093] (7) CGM2IR: Improving the integration of entity mentions in the DocRE task using context-guided mention integration and entity pair reasoning.

[0094] (8) ATLOP: Proposes adaptive threshold and local context pooling technology to solve the problem of complex relationship extraction of multiple entities and multiple labels.

[0095] (9) Coref: Enhances the model’s understanding of reference information by introducing a pre-training task of reference resolution.

[0096] In the overall comparison, MFDRE's F1 score is 67.58 and Ign F1 is 65.53, showing its strong advantage over DocRE. MFDRE effectively captures the complex relationships in documents by fusing local and complementary representations of entity pairs and implementing multi-level representation and feature propagation on heterogeneous graphs, and ultimately performs well in F1 and Ign F1 indicators, outperforming some other models.

[0097] Table 4 Comparative experiments of MFDRE and other models (RoBERTa-large)

[0098] Table4 Comparative experimental results between MFDRE and other models (RoBERTa-large)

[0099]

[0100]

[0101] As shown in Table 5, on the DocRED dataset, BERT-base (a variant of BERT, containing about 110M parameters, 12 encoder layers, each encoder layer containing 12 attention heads and 768 hidden neurons) is used as the pre-trained language model. MFDRE is compared with a variety of latest models. The comparison models and their comparison results are as follows:

[0102] (1) SagDRE: By constructing a sequence-aware document graph, path encoding, and adaptive margin loss, it improves the ability to handle long-distance dependencies and sequential information while solving the multi-label problem.

[0103] (2) SIEF: emphasizes the importance of sentences, and the model can focus on

[0104] In the extraction of evidence sentences, the robustness and interpretability are improved.

[0105] (3) SIRE: By distinguishing the relationships within and between sentences and combining the self-attention mechanism for logical reasoning, the model’s relationship recognition accuracy and reasoning ability are effectively improved.

[0106] (4) DRN: We use heterogeneous graphs to explicitly model the DocRE process and combine it with contextual information to improve the extraction performance in complex scenarios.

[0107] (5) GAIN: By combining a graph neural network with a pre-trained language model, a document-level graph structure is constructed to effectively capture multiple relationships between entities and improve the extraction effect.

[0108] (6) JEREX: It is the first to propose an entity-level relationship extraction model that combines multi-task learning and multi-instance learning. It does not rely on mention-level annotations and achieves end-to-end extraction.

[0109] (7) CFER: By combining the global context with the target entity path information, a document-level graph model is constructed to effectively improve the comprehensiveness of entity representation.

[0110] (8) HeterGSAN: We use a reconstructor to focus on the learning of related entity pairs and explicitly model the path dependencies of entity pairs to further improve the accuracy of relation extraction.

[0111] (9) LSR: Through multi-hop reasoning mechanism and structural induction learning technology, the model can perform deep reasoning on the learned document structure.

[0112] Among these comparison models, MFDRE demonstrates its unique advantages. By combining enhanced entity representation with multi-level representation methods, MFDRE can effectively handle complex relational structures in documents. In addition, its feature propagation mechanism significantly improves the expressiveness of entity representation, making it reach 63.21 and 61.14 in F1 and Ign F1 indicators respectively, which is better than most existing models.

[0113] Table 5 Comparative experiments between MFDRE and other models (BERT-base)

[0114] Table5 Comparative experimental results between MFDRE and other models (BERT-base)

[0115]

[0116] As shown in Table 6, MFDRE is compared with a variety of state-of-the-art models on the DocRED-E2E dataset. The comparison models and their comparison results are as follows:

[0117] (1) JEREX: It treats a document as a collection of multiple instances, each of which contains an entity and its corresponding text fragment. It can effectively handle multiple contexts in which an entity appears in a document.

[0118] (2) KBIE: Incorporates external knowledge such as Wikipedia text and Wikidata knowledge graph into the model to provide richer semantic information.

[0119] (3) REXEL: It uses a two-stage relation classification method, first performing coarse-grained relation classification, and then performing fine-grained relation classification on high-probability entity pairs.

[0120] Compared with these comparison models, MFDRE significantly improves the expressiveness of entity representation by fusing local representation of entity pairs with supplementary representation of entity pairs under span expansion. Moreover, it effectively solves the problem of long-distance dependency across sentences in DocRE through multi-level representation, making it reach 40.96 on F1.

[0121] Table 6 Comparative experiments between MFDRE and other models (DocRED-E2E)

[0122] Table6 Comparative experimental results between MFDRE and other

[0123] models(DocRED-E2E)

[0124]

[0125] S203, ablation experiment

[0126] As shown in Table 7, the enhanced entity representation and multi-level representation (RoBERTa-large) are removed on DocRED and ablation experiments are performed.

[0127] When the enhanced entity representation is removed from MFDRE, F1 drops slightly to 67.02, and Ign F1 also drops slightly to 65.09, which shows that the enhanced entity representation has a certain positive impact on the performance of MFDRE, but the impact is relatively small. When the multi-level representation is removed, F1 drops to 66.95 and Ign F1 drops to 64.92. Compared with the complete MFDRE, both indicators have dropped significantly, especially Ign F1, which drops by 0.61, which shows that the multi-level representation has a more significant effect on improving the performance of MFDRE.

[0128] Table 7 Ablation experiments on DocRED with removing enhanced entity representation and multi-level representation

[0129] (RoBERTa-large)

[0130] Table7 Ablation experiments on DocRED for removing enhanced solid representations and multilevel representations, respectively

[0131] (RoBERTa-large)

[0132]

[0133] As shown in Table 8, the ablation experiment results on DocRED (BERTa-base) show that enhanced entity representation and multi-level representation have a significant effect on improving the performance of the model. When the enhanced entity representation is removed, the F1 value of the model drops from 63.21 to 62.76, and the Ign F1 value drops from 61.14 to 60.04; when the multi-level representation is removed, the F1 value of the model increases slightly to 62.55, but the Ign F1 value drops to 59.19. These results further verify the effectiveness of MFDRE, that is, the performance of DocRE can be effectively improved by introducing enhanced entity representation and multi-level representation.

[0134] As shown in Table 9, ablation experiments on the DocRED-E2E dataset show that enhanced entity representation and multi-level representation have a significant impact on model performance. First, the base model MFDRE achieves an F1 value of 40.96. When the enhanced entity representation is removed, the F1 value of the model drops to 38.89, showing the importance of entity representation for capturing relationships in documents. In addition, if the multi-level representation is removed, the F1 value of the model drops to 39.33, indicating that the multi-level representation helps improve the accuracy of the model in identifying complex relationships. Overall, both components contribute to the model performance, and their synergy can further improve the performance of the model.

[0135] Table 8 Ablation experiments on DocRED with enhanced entity representation and multi-level representation removed

[0136] (BERT-base)

[0137] Table8 Ablation experiments on DocRED for removing enhanced solid representations and multilevel representations,

[0138] respectively(BERT-base)

[0139]

[0140] Table 9 Ablation experiments on DocRED-E2E for removing enhanced solid representations and multilevel representations, respectively,

[0141]

[0142] Compared with the prior art, the present invention provides document-level relationship extraction that integrates enhanced entities and multi-level representations, and has the following beneficial effects:

[0143] 1. The fusion enhances the extraction of document-level relations between entities and multi-level representations. The proposed MFDRE provides an effective solution to the long-distance dependency problem in DocRE. MFDRE uses the pre-trained language model BERT to obtain the global representation of entities in the document, and integrates the local representation of entity pairs with the supplementary representation of entity pairs under span expansion, significantly improving the representation ability of entity nodes.

[0144] 2. This fusion of enhanced entity and multi-level representation of document-level relation extraction solves the problem of long-distance dependencies across sentences through feature propagation based on heterogeneous graph-based multi-level representation and stacked multi-layer relational graph convolutional networks. The experimental results of MFDRE on the datasets DocRED and DocRED-E2E show that its performance is better than the existing methods, with F1 values ​​of 63.21 and 40.96 respectively. The ablation experiment further verifies the necessity of each module, especially the importance of enhanced entity representation and multi-level representation in relation classification. MFDRE provides new research ideas and technical paths for the future of DocRE, and also lays the foundation for complex relation extraction tasks. Future research can further explore the performance of the model on larger-scale datasets and its application potential in other natural language processing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0145] Figure 1 This is a sample diagram of the DocRED dataset;

[0146] Figure 2 Schematic diagram of the multi-level fusion document-level relation extraction model (MFDRE). DETAILED DESCRIPTION

[0147] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0148] See also Figure 1-2 The present invention provides a technical solution: document-level relationship extraction integrating enhanced entity and multi-level representation, comprising the following steps:

[0149] S1. Multi-level fusion document-level relation extraction model

[0150] S101, Pretreatment

[0151] The purpose of preprocessing is to integrate multiple mentions and obtain a global representation of the entity. For a document D of length L, given a document Indicates that x t Represents the token at position t (in natural language processing, a token usually refers to the smallest semantic unit in a text, such as a word, subword, or character). To mark an entity, a special marker "*" is used to indicate the start and end positions of the entity. The context embedding vector H of document D is obtained through the pre-trained BERT model, as shown in formula (1):

[0152] H = BERT([x1,x2,…x L ])=[h1,h2,…h L ] (1)

[0153] Where H = [h1,h2,…h L ] is the hidden state representation of the token;

[0154] The entity e i All mentions of are embedded as Where m j Indicates the location of the mention, since each entity e i May appear multiple times in a document, so each entity is represented as the set of all mentions In order to obtain the global table of entities i As shown, using the logsumexp pooling method, by mentioning the number Weighted pooling of all mention embeddings This captures the contextual information of entities at different locations. See formula (2) for details:

[0155]

[0156] in, It is entity e i Pooling features;

[0157] S102. Enhanced entity representation

[0158] The enhanced entity representation module includes entity pair local representation, entity pair supplementary representation and fused entity representation.

[0159] The n entities in the document can be combined to form n(n-1) entity pairs, each of which has a corresponding local representation. Given a pre-trained multi-head attention matrix A ijk ∈R H×L×L , where H represents the number of attention heads, L represents the document length, and the matrix A ijk represents the attention of the i-th attention head from tokenj to tokenk;

[0160] First, the attention of the "*" symbol is regarded as the mention-level attention, and then the mention attentions of the same entity are averaged to get the entity-level attention Indicates the attention of the i-th entity to all tokens;

[0161] Next, for a given entity pair (e s ,e o ), locate the head entity e through entity-level attention s and tail entity e o The local representation is important, and the local embedding vector C is obtained using formula (3-5) (s,o) :

[0162]

[0163] c (s,o) =H T (q (s,o) / 1 T q (s,o) ) (5)

[0164] Among them, A (s,o) Represents an entity pair (e s ,e o )’s aggregate attention, q (s,o) ∈R l represents its average attention weight, H is the context embedding vector in formula (1);

[0165] Then, the local embedding vector c (s,o) With entity e s The global representation of Combined, a personalized entity representation is provided for each entity pair, as shown in formula (6):

[0166]

[0167] in, Represents entity e s The local representation of W s , W c1 is the weight parameter;

[0168] According to formula (7), the entity e is obtained in the same way o Local representation of

[0169]

[0170] Among them, W o , W c2 is the weight parameter;

[0171] Sentence-level relation extraction has limited effect in DocRE tasks because entities are usually scattered across multiple sentences or paragraphs in a document and have various forms of expression. Traditional sentence-level methods tend to bias towards common categories and ignore rare categories, thus affecting the accuracy and generalization ability of the model.

[0172] To solve this problem, the span extension strategy is adopted, which aims to balance the data distribution by introducing more negative samples and enhance the model's ability to identify potential entities. The span in DocRE refers to the interval of entities at different positions in the document, including the start and end positions. When extending, the initial range of the entity is extended by 3 tokens to the front and back sides. For example, in the sentence "Beijing is the capital of China", the span extension strategy can identify multiple fine-grained entities, such as "Beijing", "China", "China", "China's" and "capital".

[0173] Then, according to formula (8), the entity after span expansion Combine them in pairs to form an entity pair set P, with a set length of n(n-1);

[0174] P=[(e1,e2),(e1,e3),…(e n-1 ,e n )] (8)

[0175] Then, based on the set P, the index of the related sentences is extracted for each entity pair, and these sentences are concatenated to generate new sentences through string concatenation. For example, entity 1 comes from sentence 1 and sentence 2, and entity 2 comes from sentence 3. After concatenation, the new sentence consists of sentence 1, sentence 2 and sentence 3. According to formula (9), a new sentence set S is created, which contains the related sentences of each entity pair, and the length of the set is n(n-1);

[0176] S=(s1,s2,…s n×(n-1) ) (9)

[0177] According to formula (10), BERT is used to process the sentence set S to obtain a complementary representation set B of entity pairs, with a set length of n(n-1);

[0178] B=BERT(S)=[b1,b2,…b n×(n-1) ] (10)

[0179] According to formula (11-13), the head entity representation v is generated by weighted linear fusion of the local representation and the supplementary representation of the entity pair s , the tail entity represents v o and enhanced entity representation v e ;

[0180] vs =W z z s +W b b so (11)

[0181] v o =W z z o +W b b so (12)

[0182] v e = {v s ,v o} (13)

[0183] Among them, z s and z o They represent the local representation of the head entity and the tail entity respectively, and b so represents the complementary representation of entity pairs, W z and W b are the weight parameters of local representation and supplementary representation respectively;

[0184] S103, multi-level representation

[0185] In DocRE, entity relations exist not only between explicitly annotated entity pairs, but also between mention pairs in the document. However, it is difficult for sequence models to handle the complex relations between entity pairs and mention pairs at the same time. Therefore, this paper proposes a multi-level representation method based on heterogeneous graphs to capture these two types of relations.

[0186] The heterogeneous graph consists of four types of nodes: entity nodes, mention nodes, sentence nodes, and document nodes. Each type of node represents the structure of a document at a different level. Through the interaction of nodes at each level, the relationship features in the document can be effectively extracted.

[0187] Entity nodes represent specific entities in a document, such as people, places, and events. As the basic unit in the DocRE task, they reflect the multidimensional characteristics of entities in various contexts. According to formula (14), entity node h E The enhanced entity representation v corresponding to step S102 e ;

[0188] h E =v e (14)

[0189] The mention node reflects the contextual relationship of the entity through the connection with the entity node. Formula (15) shows that the mention node H is the hidden state of all tokens in the entity after BERT processing J Mean pooling, J represents the total number of mentions of the entity in the document;

[0190]

[0191] The sentence node represents each sentence in the document. By connecting the entity node and the mention node, it reflects the distribution and relationship description of the entity mentions in the sentence. According to formula (16), the sentence node H is the hidden state of all tokens in the sentence after BERT processing K Mean pooling, K represents the total number of sentences in the document;

[0192]

[0193] The document node represents the global context and topic information of the entire document. As the root node of the heterogeneous graph, it helps understand the long-distance relationship across sentences by connecting with other nodes. According to formula (17), the document node h D Represents the hidden state H of all tokens in the document after BERT processing N Mean pooling, N represents the total number of tokens in the document;

[0194] h D =Average(H N ) (17)

[0195] The edge representation of the heterogeneous graph fully considers the characteristics of the DocRE task. Sentence-level relation extraction assumes that the relation only exists in a single sentence, and entity pairs directly reflect the relation. However, in DocRE, entity relations often span multiple sentences or even the entire document. Relation extraction based solely on entity pairs has limitations. In addition, the same entity may have multiple mentions, and it is difficult to determine which mentions are related. Relying solely on mentions to mark relations may oversimplify the relation representation.

[0196] To bridge the gap between entity pairs and mention pairs and fully capture relational facts, multi-level representation divides DocRE's relation representation into two levels: direct relation representation and indirect relation representation. Direct relation representation explicitly marks relations through entity pairs, while indirect relation representation infers implicit relations between entities through mention pairs and entity mention pairs.

[0197] The target relationship of an entity pair is expressed as I1 = (e s ,e o ), if the relationship exists between entity pairs, entity e s and e o Constitute an entity pair, and the target relationship of the mention pair is expressed as I2 = (m s ,m o ), if the relationship exists in the mention pair, the mentions from different entities can be combined in pairs to form a mention pair to represent the relationship, for example, entity e s=[m1,m2,m3] and e o = [m4,m5,m6] each contains 3 mentions forming 9 mention pairs:

[0198] (m1,m4),(m1,m5),(m1,m6),(m2,m4),(m2,m5),(m2,m6),(m3,m4),(m3,m5),(m3,m6);

[0199] The target relation of an entity mention pair is represented as I3 = (e s ,m o ) / (e o ,m s ), if the relation exists in the entity mention pair, entity e s =[m1,m2,m3] and e o = [m4,m5,m6] contains 6 entity mention pairs: (e1,m4), (e1,m5), (e1,m6), (e2,m1)(e2,m2), (e2,m3);

[0200] Based on the above three relationship representations, six edge types are constructed in the heterogeneous graph: sentence-document edge connects all sentence nodes and document nodes, indicating the affiliation between sentences and documents; sentence-sentence edge connects all sentence nodes in the document, indicating the information flow between sentences; sentence-entity edge connects sentences containing entities and entity nodes, indicating the existence of entities in sentences; sentence-mention edge connects mention nodes and corresponding sentence nodes, indicating the role of mentions in sentences; entity-mention edge connects entity nodes and mention nodes related to entities, indicating entity mention pairs in documents; mention-mention edge connects two mention nodes under the same entity pair, reflecting the potential entity pair relationship;

[0201] Through the above node design and edge representation, the heterogeneous graph realizes the multi-dimensional semantic integration of words, sentences, and documents, which not only captures the implicit relationship between entities, but also reveals the overall semantic connection of documents;

[0202] S104, Feature Propagation

[0203] Traditional graph convolutional networks (GCNs) are only applicable to homogeneous graphs, that is, all edges in the graph are of the same type. In the DocRE task, the graph structure is usually heterogeneous, containing multiple types of nodes and edges. The relational graph convolutional network (R-GCN) effectively handles multiple relationships in the graph by defining an independent weight matrix for each relationship type. In R-GCN, the features of a node are updated by the features of neighboring nodes and the influence of different relationship types. The core idea of ​​R-GCN is to capture the complex relationships between nodes by distinguishing different types of edges.

[0204] Assume that the graph after integration in step S103 is a directed graph G = {V, E}, where the node set V includes four types: entity nodes, mention nodes, sentence nodes, and document nodes; E represents a directed edge set, and the edge has a type, which is defined as the relationship r from node i to node j, expressed as (i, r, j). The update formula of R-GCN for the node can be expressed as formula (18):

[0205]

[0206] Among them, u represents the current layer number, f i u+1 represents the updated features of node i at layer u+1, f i u represents the characteristics of node i, represents the characteristics of neighbor node j; R is a relationship set, r is a specific relationship type; N r (i) represents the set of neighbors connected to node i through relationship r; is the weight matrix of the corresponding relation r in the uth layer, is the linear transformation matrix of node i’s feature, c i,r is the normalization coefficient, which usually represents the number of neighbors of node i on relation r, and σ represents the sigmoid activation function;

[0207] In order to fully extract and fuse graph features at different levels, this paper uses a stacked multi-layer relational graph convolutional network for feature propagation. The node vector output by each layer of R-GCN represents the representation of the node at that layer, including the node's own attributes and the global information of multi-hop neighbors. As the number of layers increases, the node vector gradually expands from local to global graph features. After completing multi-layer propagation, the node vectors output by each layer of R-GCN are concatenated. The representation of the nth layer of R-GCN is: Where N represents the Nth node;

[0208] The feature propagation of stacked multi-layer relational graph convolutional networks integrates the features of each layer of graphs, retaining the original attributes and neighborhood structure expression of shallow nodes, and incorporating deep-level long-range dependencies and global pattern learning, so that the final vector of the node covers multi-level topological information and relational features, showing powerful expression capabilities;

[0209] S105. Relationship Classification

[0210] For the entity pair (e s ,e o ), the target relationship represents I so To connect I1, I2 and I3 in step S103, as shown in formula (19):

[0211] I so=[I1,I2,I3] (19)

[0212] Next, as shown in formula (20), under the sigmoid activation function, the graph propagation feature F of stacked multi-layer R-GCN is used n , through the fully connected feedforward neural network (FFNN) to the entity pair (e s ,e o )’s target relation representation I so Make predictions and output relationship categories;

[0213] y=σ(FFNN(I so |F n )) (20).

[0214] S2. Experimental Results and Analysis

[0215] S201, Dataset, Experimental Settings and Evaluation Metrics

[0216] This paper first uses DocRED as the experimental dataset. DocRED is a large-scale manually annotated DocRE dataset, which is built on the basis of Wikipedia and Wikidata using a remote supervision method. DocRED has two notable features: First, it contains 96 different relationship types, 132,375 entities, 56,354 annotated relationship facts, and covers 5,053 Wikipedia documents. It is currently the DocRE dataset with the largest data volume. Second, at least 40.7% of relationship facts are scattered in multiple sentences. Relying on a single sentence is not enough to determine the relationship within the document, and contextual understanding across sentences is required. The statistical information about DocRED in the training set, development set, and test set is shown in Table 1.

[0217] Table 1 DocRED dataset statistics

[0218]

[0219] DocRED-E2E is an end-to-end document-level information extraction dataset based on DocRED. It contains 96 different relation types, and each document contains an average of 19.4 entities and 12.5 facts. In DocRED-E2E, the relation classification task requires the model to be able to extract entity relations spanning multiple sentences from the document. In addition, the category balance problem is another challenge. The model needs to effectively deal with the distribution differences between common and rare relation types.

[0220] The experimental configuration information is shown in Table 2, including hardware facilities and software environment such as CPU model, GPU model, memory capacity, video memory capacity, operating system version, programming language, and deep learning framework version. The parameter settings of the training model are shown in Table 3, including maximum sequence length, batch size, number of labels, learning rate, warm-up rate, learning rate, number of iterations, dropout rate, maximum gradient norm, etc.

[0221] In terms of evaluation indicators, this paper uses F1 value and Ign F1 value as the core indicators for evaluating model performance. F1 value comprehensively measures precision and recall, and is defined as the harmonic mean of the two, which fully reflects the accuracy and coverage of the model in relational instance recognition. Ign F1 value eliminates repeated relational instances in the training set and development / test set, avoids the calculation of F1 scores for repeated triples in the training data, and reduces the interference of data redundancy on model evaluation.

[0222] Table 2 Experimental environment setup

[0223]

[0224] Table 3 Model training parameters

[0225]

[0226] The calculation of precision, recall and F1 value is shown in formula (21-23), where a true positive (TP) indicates that the model accurately predicts that there is actually a relationship, a true negative (TN) indicates that the model accurately predicts that there is actually no relationship, a false positive (FP) indicates that the model incorrectly predicts that there is a relationship but there is actually no relationship, and a false negative (FN) indicates that the model incorrectly predicts that there is no relationship but there is actually a relationship.

[0227]

[0228] S202, comparative experiment

[0229] As shown in Table 4, on the DocRED dataset, RoBERTa-large (a variant of BERT, containing about 340M parameters, 24 encoder layers, each encoder layer containing 16 attention heads and 1024 hidden neurons) is used as the pre-trained language model, and MFDRE is compared with a variety of latest models. The main comparison models and their comparison results are as follows:

[0230] (1) DREEAM: It introduces a local context embedding mechanism to capture the dependencies between entities and uses evidence information as a supervisory signal to further optimize the performance of DocRE.

[0231] (2) KD: Through axial attention mechanism, adaptive focal loss and knowledge distillation, it effectively alleviates the problems of category imbalance and missing entity interaction information.

[0232] (3) SSAN: A structured self-attention network is introduced in the encoding layer, and two transformation modules are used to integrate the structural information of the entity into the attention mechanism to achieve more accurate structured relationship modeling.

[0233] (4) SAIS: Improve the accuracy of the model by using methods such as explicit modeling of key information sources, evidence retrieval, and data enhancement.

[0234] (5) Eider: Improves the model's efficiency in utilizing evidence sentences through lightweight model design and joint relation and evidence sentence extraction.

[0235] (6) DocuNet: Innovatively regarded the relationship extraction task as a semantic segmentation problem, using the U-shaped segmentation module and balanced Softmax optimization strategy, and achieved remarkable results.

[0236] (7) CGM2IR: Improving the integration of entity mentions in the DocRE task using context-guided mention integration and entity pair reasoning.

[0237] (8) ATLOP: Proposes adaptive threshold and local context pooling technology to solve the problem of complex relationship extraction of multiple entities and multiple labels.

[0238] (9) Coref: Enhances the model’s understanding of reference information by introducing a pre-training task of reference resolution.

[0239] In the overall comparison, MFDRE's F1 score is 67.58 and Ign F1 is 65.53, showing its strong advantage over DocRE. MFDRE effectively captures the complex relationships in documents by fusing local and complementary representations of entity pairs and implementing multi-level representation and feature propagation on heterogeneous graphs, and ultimately performs well in F1 and Ign F1 indicators, outperforming some other models.

[0240] Table 4 Comparative experiments of MFDRE and other models (RoBERTa-large)

[0241] Table4 Comparative experimental results between MFDRE and other models (RoBERTa-large)

[0242]

[0243]

[0244] As shown in Table 5, on the DocRED dataset, BERT-base (a variant of BERT, containing about 110M parameters, 12 encoder layers, each encoder layer containing 12 attention heads and 768 hidden neurons) is used as the pre-trained language model. MFDRE is compared with a variety of latest models. The comparison models and their comparison results are as follows:

[0245] (1) SagDRE: By constructing a sequence-aware document graph, path encoding, and adaptive margin loss, it improves the ability to handle long-distance dependencies and sequential information while solving the multi-label problem.

[0246] (2) SIEF: emphasizes the importance of sentences, and the model can focus on

[0247] In the extraction of evidence sentences, the robustness and interpretability are improved.

[0248] (3) SIRE: By distinguishing the relationships within and between sentences and combining the self-attention mechanism for logical reasoning, the model’s relationship recognition accuracy and reasoning ability are effectively improved.

[0249] (4) DRN: We use heterogeneous graphs to explicitly model the DocRE process and combine it with contextual information to improve the extraction performance in complex scenarios.

[0250] (5) GAIN: By combining a graph neural network with a pre-trained language model, a document-level graph structure is constructed to effectively capture multiple relationships between entities and improve the extraction effect.

[0251] (6) JEREX: It is the first to propose an entity-level relationship extraction model that combines multi-task learning and multi-instance learning. It does not rely on mention-level annotations and achieves end-to-end extraction.

[0252] (7) CFER: By combining the global context with the target entity path information, a document-level graph model is constructed to effectively improve the comprehensiveness of entity representation.

[0253] (8) HeterGSAN: We use a reconstructor to focus on the learning of related entity pairs and explicitly model the path dependencies of entity pairs to further improve the accuracy of relation extraction.

[0254] (9) LSR: Through multi-hop reasoning mechanism and structural induction learning technology, the model can perform deep reasoning on the learned document structure.

[0255] Among these comparison models, MFDRE demonstrates its unique advantages. By combining enhanced entity representation with multi-level representation methods, MFDRE can effectively handle complex relational structures in documents. In addition, its feature propagation mechanism significantly improves the expressiveness of entity representation, making it reach 63.21 and 61.14 in F1 and Ign F1 indicators respectively, which is better than most existing models.

[0256] Table 5 Comparative experiments between MFDRE and other models (BERT-base)

[0257] Table5 Comparative experimental results between MFDRE and other

[0258] models(BERT-base)

[0259]

[0260] As shown in Table 6, MFDRE is compared with a variety of state-of-the-art models on the DocRED-E2E dataset. The comparison models and their comparison results are as follows:

[0261] (1) JEREX: It treats a document as a collection of multiple instances, each of which contains an entity and its corresponding text fragment. It can effectively handle multiple contexts in which an entity appears in a document.

[0262] (2) KBIE: Incorporates external knowledge such as Wikipedia text and Wikidata knowledge graph into the model to provide richer semantic information.

[0263] (3) REXEL: It uses a two-stage relation classification method, first performing coarse-grained relation classification, and then performing fine-grained relation classification on high-probability entity pairs.

[0264] Compared with these comparison models, MFDRE significantly improves the expressiveness of entity representation by fusing local representation of entity pairs with supplementary representation of entity pairs under span expansion. Moreover, it effectively solves the problem of long-distance dependency across sentences in DocRE through multi-level representation, making it reach 40.96 on F1.

[0265] Table 6 Comparative experiments between MFDRE and other models (DocRED-E2E)

[0266] Table6 Comparative experimental results between MFDRE and other models (DocRED-E2E)

[0267]

[0268]

[0269] S203, ablation experiment

[0270] As shown in Table 7, the enhanced entity representation and multi-level representation (RoBERTa-large) are removed on DocRED and ablation experiments are performed.

[0271] When the enhanced entity representation is removed from MFDRE, F1 drops slightly to 67.02, and Ign F1 also drops slightly to 65.09, which shows that the enhanced entity representation has a certain positive impact on the performance of MFDRE, but the impact is relatively small. When the multi-level representation is removed, F1 drops to 66.95 and Ign F1 drops to 64.92. Compared with the complete MFDRE, both indicators have dropped significantly, especially Ign F1, which drops by 0.61, which shows that the multi-level representation has a more significant effect on improving the performance of MFDRE.

[0272] Table 7 Ablation experiments on DocRED with removing enhanced entity representation and multi-level representation

[0273] (RoBERTa-large)

[0274] Table7 Ablation experiments on DocRED for removing enhanced solid representations and multilevel representations, respectively

[0275] (RoBERTa-large)

[0276]

[0277] As shown in Table 8, the ablation experiment results on DocRED (BERTa-base) show that enhanced entity representation and multi-level representation have a significant effect on improving the performance of the model. When the enhanced entity representation is removed, the F1 value of the model drops from 63.21 to 62.76, and the Ign F1 value drops from 61.14 to 60.04; when the multi-level representation is removed, the F1 value of the model increases slightly to 62.55, but the Ign F1 value drops to 59.19. These results further verify the effectiveness of MFDRE, that is, the performance of DocRE can be effectively improved by introducing enhanced entity representation and multi-level representation.

[0278] As shown in Table 9, ablation experiments on the DocRED-E2E dataset show that enhanced entity representation and multi-level representation have a significant impact on model performance. First, the base model MFDRE achieves an F1 value of 40.96. When the enhanced entity representation is removed, the F1 value of the model drops to 38.89, showing the importance of entity representation for capturing relationships in documents. In addition, if the multi-level representation is removed, the F1 value of the model drops to 39.33, indicating that the multi-level representation helps improve the accuracy of the model in identifying complex relationships. Overall, both components contribute to the model performance, and their synergy can further improve the performance of the model.

[0279] Table 8 Ablation experiments on DocRED with enhanced entity representation and multi-level representation removed

[0280] (BERT-base)

[0281] Table8 Ablation experiments on DocRED for removing enhanced solid representations and multilevel representations,

[0282] respectively(BERT-base)

[0283]

[0284] Table 9 Ablation experiments on DocRED-E2E for removing enhanced solid representations and multilevel representations, respectively,

[0285]

[0286] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. Document-level relation extraction that integrates enhanced entities and multi-level representations, characterized by: The following steps are involved: S1. Multi-level fusion document-level relation extraction model S101, Pretreatment The purpose of preprocessing is to integrate multiple mentions and obtain a global representation of the entity. For a document D of length L, given a document Indicates that x t Represents the token at position t (in natural language processing, a token usually refers to the smallest semantic unit in a text, such as a word, subword, or character). To mark an entity, a special marker "*" is used to indicate the start and end positions of the entity. The context embedding vector H of document D is obtained through the pre-trained BERT model, as shown in formula (1): H=BERT([x1,x2,…x L ])=[h1,h2,…h L ] (1); The entity e i All mentions of are embedded as Where m j Indicates the location of the mention, since each entity e i May appear multiple times in a document, so each entity is represented as the set of all mentions In order to obtain the global table of entities i As shown, using the logsumexp pooling method, by mentioning the number Weighted pooling of all mention embeddings This captures the contextual information of entities at different locations. See formula (2) for details: S102. Enhanced entity representation The enhanced entity representation module includes entity pair local representation, entity pair supplementary representation and fused entity representation. The n entities in the document can be combined to form n(n-1) entity pairs, each of which has a corresponding local representation. Given a pre-trained multi-head attention matrix A ijk ∈R H×L×L , where H represents the number of attention heads, L represents the document length, and the matrix A ijk represents the attention of the i-th attention head from tokenj to tokenk; First, the attention of the "*" symbol is regarded as the mention-level attention, and then the mention attentions of the same entity are averaged to get the entity-level attention Indicates the attention of the i-th entity to all tokens; Next, for a given entity pair (e s ,e o ), locate the head entity e through entity-level attention s and tail entity e o All important local representations are obtained, and the local embedding vector C is obtained using formula (3-5) (s,o) : c (s,o) =H T (q (s,o) / 1 T q (s,o) ) (5) Then, the local embedding vector c (s,o) With entity e s The global representation of Combined, a personalized entity representation is provided for each entity pair, as shown in formula (6): According to formula (7), the entity e is obtained in the same way o Local representation of Sentence-level relation extraction has limited effect in DocRE tasks because entities are usually scattered across multiple sentences or paragraphs in a document and have various forms of expression. Traditional sentence-level methods tend to bias towards common categories and ignore rare categories, thus affecting the accuracy and generalization ability of the model. To solve this problem, the span extension strategy is adopted, which aims to balance the data distribution by introducing more negative samples and enhance the model's ability to identify potential entities. The span in DocRE refers to the interval of entities at different positions in the document, including the start and end positions. When extending, 3 tokens are extended to the front and back sides based on the initial range of the entity. For example, in the sentence "Beijing is the capital of China", the span extension strategy can identify multiple fine-grained entities, such as "Beijing", "China", "China", "China's" and "capital". Then, according to formula (8), the entity after span expansion Combine them in pairs to form an entity pair set P, with a set length of n(n-1); P=[(e1,e2),(e1,e3),…(e n-1 ,and n )] (8) Then, based on the set P, the index of the related sentences is extracted for each entity pair, and these sentences are concatenated to generate new sentences through string concatenation. For example, entity 1 comes from sentence 1 and sentence 2, and entity 2 comes from sentence 3. After concatenation, the new sentence consists of sentence 1, sentence 2 and sentence 3. According to formula (9), a new sentence set S is created, which contains the related sentences of each entity pair, and the length of the set is n(n-1); S=(s1,s2,…s n×(n-1) ) (9) According to formula (10), BERT is used to process the sentence set S to obtain a complementary representation set B of entity pairs, with a set length of n(n-1); B=BERT(S)=[b1,b2,…b n×(n-1) ] (10) According to formula (11-13), the head entity representation v is generated by weighted linear fusion of the local representation and the supplementary representation of the entity pair s , the tail entity represents v o and enhanced entity representation v e ; v s =W z z s +W b b so (11) v o =W z z o +W b b so (12) v e ={v s ,v o } (13) Among them, z s and z o They represent the local representation of the head entity and the tail entity respectively, and b so represents the complementary representation of entity pairs, W z and W b are the weight parameters of local representation and supplementary representation respectively; S103, multi-level representation In DocRE, entity relations exist not only between explicitly annotated entity pairs, but also between mention pairs in the document. However, it is difficult for sequence models to handle the complex relations between entity pairs and mention pairs at the same time. Therefore, this paper proposes a multi-level representation method based on heterogeneous graphs to capture these two types of relations. The heterogeneous graph consists of four types of nodes: entity nodes, mention nodes, sentence nodes, and document nodes. Each type of node represents the structure of a document at a different level. Through the interaction of nodes at each level, the relationship features in the document can be effectively extracted. Entity nodes represent specific entities in a document, such as people, places, and events. As the basic unit in the DocRE task, they reflect the multidimensional characteristics of entities in various contexts. According to formula (14), entity node h E The enhanced entity representation v corresponding to step S102 e ; h E =v e (14) The mention node reflects the contextual relationship of the entity through the connection with the entity node. Formula (15) shows that the mention node H is the hidden state of all tokens in the entity after BERT processing J Mean pooling, J represents the total number of mentions of the entity in the document; The sentence node represents each sentence in the document. By connecting the entity node and the mention node, it reflects the distribution and relationship description of the entity mentions in the sentence. According to formula (16), the sentence node H is the hidden state of all tokens in the sentence after BERT processing K Mean pooling, K represents the total number of sentences in the document; The document node represents the global context and topic information of the entire document. As the root node of the heterogeneous graph, it helps understand the long-distance relationship across sentences by connecting with other nodes. According to formula (17), the document node h D Represents the hidden state H of all tokens in the document after BERT processing N Mean pooling, N represents the total number of tokens in the document; h D =Average(H N ) (17) The edge representation of the heterogeneous graph fully considers the characteristics of the DocRE task. Sentence-level relation extraction assumes that the relation only exists in a single sentence, and entity pairs directly reflect the relation. However, in DocRE, entity relations often span multiple sentences or even the entire document. Relation extraction based solely on entity pairs has limitations. In addition, the same entity may have multiple mentions, and it is difficult to determine which mentions are related. Relying solely on mentions to mark relations may oversimplify the relation representation. To bridge the gap between entity pairs and mention pairs and fully capture relational facts, multi-level representation divides DocRE's relation representation into two levels: direct relation representation and indirect relation representation. Direct relation representation explicitly marks relations through entity pairs, while indirect relation representation infers implicit relations between entities through mention pairs and entity mention pairs. The target relationship of an entity pair is expressed as I1 = (e s ,e o ), if the relationship exists between entity pairs, entity e s and e o Constitute an entity pair, and the target relationship of the mention pair is expressed as I2 = (m s ,m o ), if the relationship exists in the mention pair, the mentions from different entities can be combined in pairs to form a mention pair to represent the relationship, for example, entity e s =[m1,m2,m3] and e o = [m4,m5,m6] each contains 3 mentions forming 9 mention pairs: (m1,m4),(m1,m5),(m1,m6),(m2,m4),(m2,m5),(m2,m6),(m3,m4),(m3,m5),(m3,m6); The target relation of an entity mention pair is represented as I3 = (e s ,m o ) / (e o ,m s ), if the relation exists in the entity mention pair, entity e s =[m1,m2,m3] and e o = [m4,m5,m6] contains 6 entity mention pairs: (e1,m4), (e1,m5), (e1,m6), (e2,m1)(e2,m2), (e2,m3); Based on the above three relationship representations, six edge types are constructed in the heterogeneous graph: sentence-document edge connects all sentence nodes and document nodes, indicating the affiliation between sentences and documents; sentence-sentence edge connects all sentence nodes in the document, indicating the information flow between sentences; sentence-entity edge connects sentences containing entities and entity nodes, indicating the existence of entities in sentences; sentence-mention edge connects mention nodes and corresponding sentence nodes, indicating the role of mentions in sentences; entity-mention edge connects entity nodes and mention nodes related to entities, indicating entity mention pairs in documents; mention-mention edge connects two mention nodes under the same entity pair, reflecting the potential entity pair relationship; Through the above node design and edge representation, the heterogeneous graph realizes the multi-dimensional semantic integration of words, sentences, and documents, which not only captures the implicit relationship between entities, but also reveals the overall semantic connection of documents; S104 Feature Propagation Traditional graph convolutional networks (GCNs) are only applicable to homogeneous graphs, that is, all edges in the graph are of the same type. In the DocRE task, the graph structure is usually heterogeneous, containing multiple types of nodes and edges. The relational graph convolutional network (R-GCN) effectively handles multiple relationships in the graph by defining an independent weight matrix for each relationship type. In R-GCN, the features of a node are updated by the features of neighboring nodes and the influence of different relationship types. The core idea of ​​R-GCN is to capture the complex relationships between nodes by distinguishing different types of edges. Assume that the graph after integration in step S103 is a directed graph G = {V, E}, where the node set V includes four types: entity nodes, mention nodes, sentence nodes, and document nodes; E represents a directed edge set, and the edge has a type, which is defined as the relationship r from node i to node j, expressed as (i, r, j). The update formula of R-GCN for the node can be expressed as formula (18): In order to fully extract and fuse graph features at different levels, this paper uses a stacked multi-layer relational graph convolutional network for feature propagation. The node vector output by each layer of R-GCN represents the representation of the node at that layer, including the node's own attributes and the global information of multi-hop neighbors. As the number of layers increases, the node vector gradually expands from local to global graph features. After completing multi-layer propagation, the node vectors output by each layer of R-GCN are concatenated. The representation of the nth layer of R-GCN is: Where N represents the Nth node; The feature propagation of stacked multi-layer relational graph convolutional networks integrates the features of each layer of graphs, retaining the original attributes and neighborhood structure expression of shallow nodes, and incorporating deep-level long-range dependencies and global pattern learning, so that the final vector of the node covers multi-level topological information and relational features, showing powerful expression capabilities; S105. Relationship Classification For the entity pair (e s ,e o ), the target relationship represents I so To connect I1, I2 and I3 in step S103, as shown in formula (19): I so =[I1,I2,I3] (19) Next, as shown in formula (20), under the sigmoid activation function, the graph propagation feature F of stacked multi-layer R-GCN is used n , through the fully connected feedforward neural network (FFNN) to the entity pair (e s ,e o )’s target relation representation I so Make predictions and output relationship categories. y=σ(FFNN(I so |F n )) (20).

2. The document-level relationship extraction of fusion-enhanced entity and multi-level representation according to claim 1 is characterized in that: In step S101, H = [h1, h2, ... h L ] is the hidden state representation of the token, It is entity e i Pooling features.

3. The document-level relationship extraction of fusion-enhanced entity and multi-level representation according to claim 1, characterized in that: In step S102, A (s,o) Represents an entity pair (e s ,e o )’s aggregate attention, q (s,o) ∈R l represents its average attention weight, H is the context embedding vector in formula (1), Represents entity e s The local representation of W s , W c1 is the weight parameter, W o , W c2 is the weight parameter.

4. The document-level relationship extraction of fusion-enhanced entity and multi-level representation according to claim 1, characterized in that: In step S104, u represents the current layer number, f i u+1 represents the updated features of node i at layer u+1, f i u represents the characteristics of node i, represents the characteristics of neighbor node j; R is a relationship set, r is a specific relationship type; N r (i) represents the set of neighbors connected to node i through relationship r; is the weight matrix of the corresponding relation r in the uth layer, is the linear transformation matrix of node i’s feature, c i,r is the normalization coefficient, which usually represents the number of neighbors of node i on relation r, and σ represents the sigmoid activation function.

Images