An enhanced model and construction method for document-level relation extraction

Abstract

The present invention discloses an enhanced model and construction method for document-level relation extraction, the method comprising: performing coreference resolution on key entity pronouns in the original document and integrating the result into a data set to obtain enhanced document data; annotating the enhanced document data to obtain a document encoding representation H and a sentence encoding representation s of the input text; n ; Use the attention matrix to encode the selected entity pairs (e h ,e t ) calculates the context encoding of the overall perception of the document #imgabs0# uses the entity-level co-occurrence matrix to encode the entities (e i ,e j ) Calculate and obtain the local perception context coding of the sentence #imgabs1#The overall perception context coding of the document and the local perception context coding of the sentence are calculated to obtain a comprehensive context coding; the comprehensive context coding is input into the text coding module to obtain the predicted entity pronoun relationship; the present invention improves the effect of predicting the expression relationship between sentences.

Description

Technical Field

[0001] The present invention relates to language modeling and relationship extraction systems, and is a research issue in the field of natural language processing and deep learning. In particular, it relates to an enhanced model and construction method for document-level relationship extraction. Background Art

[0002] Natural Language Processing (NLP) is an important research area in computer science and artificial intelligence, aiming to enable computers to understand, process, and generate human language. Document-level relation extraction is a fundamental and important task in NLP.

[0003] Document-level relation extraction refers to the process of automatically extracting relationships between entities from text documents. These entities can be people, places, organizations, or other named entities, and relationships are semantic associations between entities, such as affiliations. In recent years, with the development of deep learning, neural network-based methods have made significant progress in document-level relation extraction tasks. By using pre-trained language systems such as BERT[1], rich semantic representations can be learned and fine-tuned on large-scale data, improving the performance of relation extraction. In addition, there are also some studies based on technologies such as graph neural networks[2-4] that aim to further improve the accuracy and generalization ability of document-level relation extraction.

[0004] Document-level relationship extraction has important applications in many fields, such as information extraction, knowledge graph construction, public opinion analysis, and intelligent question-answering. By automatically extracting entity relationships from text, it helps people organize and understand information, aids decision-making and prediction, and thus promotes the development of scientific research, business, and society. With the continuous advancement of technology, document-level relationship extraction will continue to play a vital role in the field of NLP and provide a more intelligent and efficient text analysis tool for this invention.

[0005] Current research focuses on enhancing the semantic understanding capabilities of systems. Some studies based on graph neural networks (GNNs) construct graph structures to capture relevant information and complex interactions between multiple elements (e.g., mentions and entities) [2-4]. Meanwhile, some Transformer-based systems leverage the rich contextual information provided by pre-trained language systems to learn relevant features [5-7].

[0006] Modeling document-level relationship extraction using a Transformer-based system is a more popular and effective approach than graph-based systems. These studies implicitly capture long-range dependencies between words by using a pre-trained language system as an encoder for modeling. In this direction, Zhou et al. [5] proposed using a self-attention module to extract contextual information related to entity pairs; Zhang et al. [8] considered DocRE as a semantic segmentation task and adopted a U-net system to capture the dependencies between entities; Xu et al. [9] fused the dependencies between entities in the self-attention mechanism and the encoding stage to guide relationship prediction; in addition, Jiang et al.

[10] proposed a key instance classifier to identify key mention pairs. However, these systems have certain defects when facing pronouns referring to entities in documents, resulting in incomplete learning of entity information and biased relationship prediction results.

[0007] Huang et al.

[11] first pointed out through verification that more than 95% of entity pairs only need three evidence sentences at most to determine the relationship. Since then, some studies have focused on the extraction and utilization of evidence sentences. Huang et al.

[12] proposed to guide the document-level relationship extraction process by predicting evidence sentences; Xie et al. [6] jointly trained a relationship extraction system and an evidence extraction system; Xiao et al. [7] designed an evidence retrieval module to guide prediction. However, these works rely too much on the annotation of evidence sentences in the dataset during the extraction process of training evidence sentences. In fact, the labels of evidence sentences in the dataset are incomplete. Therefore, to some extent, the effectiveness of these systems is limited by incomplete data annotation.

[0008] [References]

[0009] [1]Devlin,J.,Chang,MW,Lee,K.,Toutanova,K.:Bert:Pre-training of deepbidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805(2018)

[0010] [2]Zeng,S.,Xu,R.,Chang,B.,Li,L.:Double graph based reasoning fordocumentlevel relation extraction.arXiv preprint arXiv:2009.13752(2020)

[0011] [3]Nan,G.,Guo,Z., I.,Lu,W.:Reasoning with latent structurerefinementfor document-level relation extraction.arXiv preprint arXiv:2005.06312(2020)

[0012] [4]Zhao,C.,Zeng,D.,Xu,L.,Dai,J.:Document-level relation extractionwith context guided mention integration and inter-pair reasoning.ArXiv abs / 2201.04826(2022)

[0013] [5]Zhou,W.,Huang,K.,Ma,T.,Huang,J.:Document-level relationextractionwith adaptive thresholding and localized context pooling.ArXiv abs / 2010.11304(2020)

[0014] [6]Xie,Y.,Shen,J.,Li,S.,Mao,Y.,Han,J.:Eider:Empowering document-levelrelation extraction with efficient evidence extraction and inference-stage fusion.In:Findings of the Association for Computational Linguistics:ACL2022.pp.257–268(2022)

[0015] [7]Xiao,Y.,Zhang,Z.,Mao,Y.,Yang,C.,Han,J.:Sais:supervising andaugmenting intermediate steps for document-level relation extraction.arXivpreprintarXiv:2109.12093(2021)

[0016] [8]Zhang,N.,Chen,X.,Xie,X.,Deng,S.,Tan,C.,Chen,M.,Huang,F.,Si,L.,Chen,H.:Document-level relation extraction as semantic segmentation.arXivpreprintarXiv:2106.03618(2021)

[0017] [9]Xu,B.,Wang,Q.,Lyu,Y.,Zhu,Y.,Mao,Z.:Entity structure within andthroughout:Modeling mention dependencies for document-level relationextraction.In:Proceedings of the AAAI conference on artificialintelligence.vol.35,pp.14149–14157(2021)

[0018]

[10] Jiang,F.,Niu,J.,Mo,S.,Fan,S.:Key mention pairs guided document-level relationextraction.In:Proceedings of the 29th International Conferenceon ComputationalLinguistics.pp.1904–1914(2022)

[0019]

[11] Huang,Q.,Zhu,S.,Feng,Y.,Ye,Y.,Lai,Y.,Zhao,D.:Three sentencesareall you need:Local path enhanced document relation extraction.arXivpreprintarXiv:2106.01793(2021)

[0020]

[12] Huang,K.,Qi,P.,Wang,G.,Ma,T.,Huang,J.:Entity and evidenceguideddocument-level relation extraction.In:Proceedings of the 6th Workshopon Representation Learning for NLP(RepL4NLP-2021).pp.307–315(2021) Summary of the Invention

[0021] The purpose of the present invention is to overcome the shortcomings of the existing technology. The present invention proposes a dual enhancement model of entity pronouns and evidence sentences for document-level relationship extraction. The model effectively utilizes the pronoun information referring to entities and extracts evidence sentences to improve the performance of the system in document-level relationship extraction tasks.

[0022] The purpose of the present invention is achieved through the following technical solutions:

[0023] An enhanced model for document-level relation extraction, comprising an entity pronoun enhancement module, a text encoding module, a document-aware context encoding module, a sentence-aware context encoding module, adaptive fusion, and a classifier; wherein:

[0024] The entity pronoun enhancement module obtains enhanced document data by performing coreference resolution on key entity pronouns in the original document and integrating the data set;

[0025] The text encoding module annotates the enhanced document data to obtain the document encoding representation H and sentence encoding representation s of the input text n ;

[0026] The document-aware context encoding module uses an attention matrix to encode the selected entity pairs (e h ,e t ) Calculate the context encoding of the overall perception of the document

[0027] The context encoding module for acquiring sentence perception uses the entity level co-occurrence matrix to encode the entities (e i ,e j ) Calculate the local perceptual context encoding of the sentence

[0028] The adaptive fusion module calculates the comprehensive context coding by combining the overall perceived context coding of the document and the local perceived context coding of the sentence according to the following formula:

[0029]

[0030] v (h,t) =gs (h,t) +(1-g)c (h,t)

[0031] Where: W is a trainable parameter, g is the weight representation calculated based on two types of context encoding,

[0032] The classifier inputs the comprehensive context encoding into the text encoding module according to the following formula to obtain the predicted entity pronoun relationship; namely:

[0033]

[0034]

[0035] in: is a trainable parameter, and is the final entity representation.

[0036] Furthermore, the text encoding module annotates the enhanced document data to obtain the document encoding representation H and sentence encoding representation s of the input text. n Process; including:

[0037] The document encoding representation is obtained by the following formula:

[0038] H=[h1,h2,…,h l ]=BERT([w1,w2,…,w l ])

[0039] in: represents the document encoding, l is the number of tokens, and d represents the dimension of the encoding; BERT is the language training unit;

[0040] By i The jth time and the code marked with "*" are entered as the mention code

[0041] The entity embedding representation is obtained by applying the LogSumExp pooling operation to all mentioned encodings

[0042] The entity encoding is obtained by the following formula:

[0043]

[0044] in: Indicates the j-th mention of the entity, and the code of the mention is represented by the code of the "*" before the mention;

[0045] The sentence encoding is obtained by the following formula:

[0046]

[0047] in Encode the tokens contained in the sentence.

[0048] Furthermore, the document-aware context encoding module uses an attention matrix to encode the selected entity pairs (e h ,e t ) Calculate the context encoding of the overall perception of the document Process; including:

[0049] Obtain multi-head attention A from the document encoding representation stage ijk , the attention value of the “*” before the mention to each token is taken as the mention attention score, and the mention attention is averaged to get the entity attention score A E ;

[0050] For a given entity pair (e h ,e t ), get the attention scores of these two entities from the multi-head attention and The entity pair (e h ,e t )Attention score for each token:

[0051]

[0052]

[0053]

[0054] in, This means that each token-entity pair (e h ,e t )’s importance;

[0055] According to the document encoding representation H, the document-aware context encoding based on entity pairs is obtained by interactive calculation according to the following formula:

[0056] c (h,t) =H T a (h,t) .

[0057] Furthermore, the context encoding module for acquiring sentence perception uses the entity level co-occurrence matrix to encode the entities (e i ,e j ) Calculate the local perceptual context encoding of the sentence Process; including:

[0058] Construct an entity-level co-occurrence matrix M, where M ij Stores the identified entity e i and e j The IDs of all evidence sentences of the relationship between them;

[0059] According to the sentence number label (sent_id) and relative position label (pos) of the entity mentioned in the dataset, ij Record explicit co-occurring sentence IDs in

[0060] Compare the elements in row i and row j one by one; if the elements in column k of both rows are not empty, it means entity e i and e j Respectively with entity e k There is an explicit co-occurrence relationship where entity e k is a bridging entity; ik and M jk The sentence ID is added to M ij In this case, M ij The set in contains the complete list of evidence sentences extracted from explicit and implicit co-occurring sentences;

[0061] For the entity pair (e h ,e t ), from the co-occurrence matrix M ht All their evidence sentences are obtained;

[0062] The sentence encodings of all evidence sentences are averaged and compared with the document-aware context encoding based on entity pairs. Interact to achieve sentence-aware context encoding

[0063]

[0064] in: M ht A list of all evidence sentences extracted by the heuristic rules.

[0065] The present invention is also implemented by the following technical solutions:

[0066] A method for constructing an enhanced model for document-level relation extraction.

[0067] Enhanced document data is obtained by performing coreference resolution on key entity pronouns in the original document and integrating them into the dataset;

[0068] Annotate the enhanced document data to obtain the document encoding representation H and sentence encoding representation s of the input text n ;

[0069] The attention matrix is ​​used to encode the selected entity pairs (e h ,e t ) Calculate the context encoding of the overall perception of the document

[0070] The entity-level co-occurrence matrix is ​​used to encode the entities (e i ,e j ) Calculate the local perceptual context encoding of the sentence

[0071] The comprehensive context coding is obtained by calculating the overall perceived context coding of the document and the local perceived context coding of the sentence according to the following formula;

[0072] The comprehensive context encoding is input into the text encoding module according to the following formula to obtain the predicted entity pronoun relationship; that is:

[0073]

[0074]

[0075] in: is a trainable parameter, and is the final entity representation.

[0076] Beneficial effects

[0077] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0078] 1. Compared to previous work that ignored the pronouns' reference to entities, resulting in incomplete entity information learning, this paper focuses on using coreference resolution to address the problem of missing pronoun information, paying special attention to the presence of pronouns to ensure the integrity of entity information. This method can obtain more comprehensive entity information.

[0079] 2. This invention uses heuristic rules to extract evidence sentences, without being restricted by manual annotation. This approach can better cope with incomplete evidence sentence annotations in datasets and is more flexible and scalable in practical applications. By introducing heuristic rules, this invention can automatically extract sentences with potential probative power from text, providing more reliable supporting evidence for document-level relationship extraction tasks.

[0080] 3. The present invention was experimentally verified on the Re-DocRED dataset, and the present invention showed excellent performance on this dataset. It is worth noting that the present invention achieved the most significant improvement in the extraction of entity pairs distributed in multiple sentences, which shows that the extraction and utilization of evidence sentences in the present invention are effective in cross-sentence reasoning. These experimental results emphasize the superiority of the method proposed in the present invention and prove the effectiveness of the design of the entity pronoun enhancement module and the sentence-aware context encoding module in the document-level relationship extraction task. By making full use of evidence sentences and cross-sentence associations, the present invention can more accurately capture the relationship between entities and achieve a higher level of reasoning ability.

[0081] 4. These findings are of great significance for promoting the research and practical application of document-level relationship extraction tasks, and provide strong guidance for further exploration and improvement of relationship extraction methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 Schematic diagram of the system of the present invention.

[0083] Figure 2 This is a schematic diagram of the process of extracting evidence sentences. DETAILED DESCRIPTION

[0084] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0085] The method flow of the present invention is as follows Figure 1 As shown in the figure. First, the original documents in the dataset are enhanced through coreference resolution and data automatic fusion algorithms. Second, the present invention uses a pre-training system to obtain entity encoding, sentence encoding, and document encoding. Then, based on the document encoding and attention mechanism, the document-aware context encoding of the specified entity pair is obtained. Then, evidence sentences are extracted using rules to obtain sentence-aware context encoding. Finally, the present invention fuses these two feature encodings and adds them to the entity encoding to perform relationship prediction.

[0086] First, to improve the integrity of entity-level information, the present invention introduces an entity pronoun enhancement module based on coreference resolution. In this module, the present invention utilizes a pre-trained large-scale system to perform coreference resolution on documents and designs an algorithm to automatically fuse the processed results with the original dataset. By using a mature pre-trained language system, the propagation of errors generated during the coreference resolution phase is avoided, and the designed automatic data fusion algorithm reduces the need for manual annotation.

[0087] Secondly, the present invention proposes a sentence-aware context encoding module. This module defines two types of evidence sentences and designs a set of heuristic rules to extract evidence sentences related to entity pairs. This approach avoids the tedious process of manually labeling evidence sentences while also extracting evidence sentences in accordance with practical logic, achieving the goal of relational reasoning. In this invention, by utilizing an attention mechanism, two different levels of feature encoding are obtained: document-aware context encoding and sentence-aware context encoding.

[0088] Finally, the present invention proposes an adaptive fusion module for filtering and fusing the two levels of features obtained above. This module can automatically adjust the weights and correlation levels of features according to task requirements to obtain a more accurate and consistent representation.

[0089] Through this process, the system can fully utilize coreference resolution and data fusion to enhance information learning about entity pronouns. Simultaneously, the system acquires encodings through a pre-trained system to capture semantic information. Feature extraction based on document encoding and attention mechanisms enables the present invention to find information related to entity pairs within documents. By extracting evidence sentences to obtain sentence-aware context encodings, it is possible to find more fine-grained information related to entity pairs at the sentence level.

[0090] 1. Entity Pronoun Enhancement Module

[0091] To ensure the accuracy and completeness of coreference resolution and avoid error propagation due to insufficient training data, this design uses the AllenNLP tool to implement coreference resolution. In practice, the results of coreference resolution are formatted differently from the dataset, but contain some of the same information. To address this issue, we designed an algorithm to automatically detect and fill in incomplete entity mentions in the dataset without the need for manual annotation.

[0092] The specific algorithm description and pseudo code are as follows:

[0093] Step 1: Select key entities. Since a document usually has only one most frequently mentioned key entity, we select the longest mention list from the results of coreference resolution as the "key entity" and then compare it with the mention list of the corresponding entity in the dataset.

[0094] Step 2: Matching and filling. Since both the results of coreference resolution and the mention list in the dataset contain the position information of the mentions, such as the sentence number label (sent_id) and the relative position label (pos), the present invention compares the results of coreference resolution with the dataset based on these labels to find the position of the "key entity" in the dataset and fill in the missing mention information.

[0095]

[0096] Through tools and automated algorithm processes, we can effectively update the dataset, supplement the pronoun annotations of entities, and improve the information of entities, thereby improving the performance of document-level relationship extraction tasks.

[0097] 2. Text encoding module

[0098] For a given document D, the present invention inserts a special marker “*” before the start position and after the end position of each mention. Then, the present invention inputs the entire document into the BERT system and obtains the corresponding document encoding representation, i.e.

[0099] H=[h1,h2,…,h l ]=BERT([w1,w2,…,w l ])

[0100] in Represents the document encoding, l is the number of tokens, and d represents the dimension of the encoding.

[0101] For entity e i The jth mention of the present invention uses the encoding of the start symbol "*" as the mention encoding Then, by applying the LogSumExp pooling operation on all mentioned encodings, we obtain the embedding representation of the entity

[0102]

[0103] For a sentence, the embedding representation of the sentence is obtained by performing an average pooling operation on all the token encodings it contains.

[0104]

[0105] 3. Obtaining a document-aware context encoding module

[0106] In the encoder of the previous text encoding module, the present invention can obtain a pre-trained multi-head attention matrix A, where A ijk Indicates the attention score from the jth token to the kth token in the ith attention head. This paper considers the attention score of the special symbol “*” before the mention as the mention-level attention, and averages the attention of all mentions of the entity to obtain the entity-level attention. The attention represents the degree of attention of the i-th entity to all tokens. For a given entity pair (e h ,e t ), calculate the importance of each token to them using the following formula:

[0107]

[0108]

[0109] a (h,t) =q (h,t) / 1 T q (h,t)

[0110] in, Represents each token-entity pair (e h ,e t ), and the document-aware context encoding based on entity pairs is obtained by interactively calculating it with the document encoding H

[0111] c (h,t) =H T a (h,t)

[0112] 4. Obtaining sentence-aware context encoding modules

[0113] Since the annotations of evidence sentences in the dataset are incomplete, this paper cannot directly use them to train the system. Therefore, designing a set of heuristic rules to extract evidence sentences from documents is a relatively ideal method. In these rules, based on objective facts and normal logic, this paper defines two types of co-occurrence situations that exist in the vast majority of entity pairs, and regards the corresponding sentences as evidence sentences:

[0114] (1) Explicit co-occurrence: If two entities appear in the same sentence, the present invention regards all sentences containing their co-occurrence as evidence sentences of explicit co-occurrence.

[0115] (2) Implicit co-occurrence: Assume that two entities each appear in the same sentence with a third entity (called a “bridge entity”). In this case, the present invention regards these sentences as evidence sentences for implicit co-occurrence between the two entities.

[0116] In order to extract evidence sentences based on this definition, the present invention designs extraction rules by constructing an entity-level co-occurrence matrix M, where M ij Stores the identified entity e i and e j First, we use the sentence number label (sent_id) and relative position label (pos) of the entity mentioned in the dataset to identify the ID of all evidence sentences between the entities. ij Record the explicit co-occurrence sentence ID in . Then, compare the elements in row i and row j one by one. If the elements in column k of both rows are not empty, then entity e i and ej Respectively with entity e k There is an explicit co-occurrence relationship where entity e k is the bridging entity. Then, ik and M jk The sentence ID is added to M ij At this time, M ij The set in contains explicit and implicit co-occurring sentences, which is the complete list of evidence sentences extracted by this invention. Taking an entity pair ("The X-Files", "Chris Carter") in the dataset as an example, the process of extracting its evidence sentences is as follows: Figure 2 shown.

[0117] So far, for entity pairs (e h ,e t ), from the co-occurrence matrix M ht Then, according to the IDs of these sentences, we find the corresponding sentence encodings, average the encodings of these sentences, and compare them with the document-aware context encoding based on entity pairs. Interact to achieve sentence-aware context encoding

[0118]

[0119] in M ht A list of all evidence sentences extracted by the above heuristic rules, where ⊙ represents an element-wise product operation.

[0120] 5. Adaptive fusion module and classification module

[0121] The adaptive fusion module can retain relevant and useful information at two different levels and filter out redundant and repeated information to avoid introducing noise. By fusing document-aware context encoding and sentence-aware context encoding, a comprehensive context encoding is obtained:

[0122]

[0123] v (h,t) =gs (h,t) +(1-g)c (h,t)

[0124] Where: W is a trainable parameter, Encode the context v (h,t) Added to the entity encoding obtained by the text encoding module, we get two enhanced entity representations in the entity pair:

[0125]

[0126]

[0127] in: is a trainable parameter, and is the final entity representation. In order to predict a given entity pair (e h ,e t ) relations, the present invention follows the practice of previous work. The encoding is divided into k equal-sized groups using a group bilinear method, and bilinearity is applied within the group to calculate the probability of each relation:

[0128]

[0129]

[0130]

[0131] in is the system parameter, P(r|e h ,e t ) is a relation r is an entity pair (e h ,e t ) is the probability of the correct relationship.

[0132] 6. Loss function of system training

[0133] To address the multi-label and sample imbalance issues, this paper uses an adaptive threshold loss as the classification loss in the training model. Specifically, a special relation category TH is introduced. This threshold category learns a threshold for entity dependencies, such that the probability of the positive class is higher than TH and the probability of the negative class is lower than TH.

[0134]

[0135]

[0136] The proposed system uses the Re-DocRED dataset as experimental material. This dataset is a revised version of the classic DocRED dataset for document-level relationship extraction. DocRED is a large-scale, manually annotated dataset extracted from Wikipedia articles. However, it contains a large number of false negative instances. Tan et al. re-annotated it and released Re-DocRED. Detailed information about the dataset is shown in Table 1. For system training, the dataset is divided into training, validation, and test sets.

[0137] Table 1 Dataset information statistics

[0138]

[0139] In the experiment, six metrics were used for evaluation: (1) F1 is a weighted average of the precision and recall of the results. (2) Ign-F1 measures the F1 score when the relation facts shared by the training set and the development / test set are excluded. (3) Freq-F1 is calculated by considering only the ten most common relation type instances in the training set. (4) LT-F1 is calculated by considering only the relation type instances with a long tail distribution in the dataset. (5) Intra-F1 calculates the F1 value of entity pair-relation triples in a sentence. (6) Inter-F1 calculates the F1 value of relation triples across sentences.

[0140] Table 2 shows the performance of our system compared to other systems on the dataset. The performance evaluation results show that the proposed system achieves superior performance across different evaluations on the same dataset. Particularly noteworthy is that our system outperforms the well-known classic baseline method, ATLOP, in Ign-F1 / F1 metrics, achieving improvements of 0.09 and 0.14 percentage points respectively. Furthermore, our system significantly improves inter-sentence relationships, achieving a 0.29 point improvement in the Inter-F1 metric.

[0141] Table 2 Evaluation comparison results based on datasets

[0142]

[0143]

[0144] This paper hypothesizes that reasoning about relationships between sentences presents a bottleneck in locating relevant context within documents. Through entity pronoun enhancement, the system is able to account for the presence of pronoun information within sentences and construct multi-hop connection paths between entities using two types of evidence sentences extracted by the sentence-aware context encoding module. This enables the system to better reason about relationships between entity pairs across sentences, thereby improving system performance.

[0145] In summary, the present invention proposes a dual enhancement system, which aims to improve the document-level relationship extraction task by effectively utilizing entity pronoun information and accurately extracting evidence sentences. The system consists of two key modules: an entity pronoun enhancement module and a sentence-aware context encoding module. The entity pronoun enhancement module updates the dataset through coreference resolution and data fusion algorithms to effectively utilize entity pronoun information. This module can improve the quality and richness of the dataset. The sentence-aware context encoding module uses heuristic rules to extract evidence sentences that conform to actual logic, which play a key role in predicting relationships. The design of this module aims to highlight the most important information in relationship prediction. Experimental results show that the system significantly outperforms classical language systems and some other Transformer-based systems in performance, especially in inferring relationships between sentences.

[0146] The present invention is not limited to the embodiments described above. The above description of the specific embodiments is intended to describe and illustrate the technical solutions of the present invention. The above specific embodiments are merely illustrative and not restrictive. Without departing from the scope of the present invention and the scope of protection of the claims, those skilled in the art may make various specific modifications based on the teachings of the present invention, all of which fall within the scope of protection of the present invention.

Claims

1. An enhanced model for document-level relationship extraction, characterized by: The model includes an entity pronoun enhancement module, a text encoding module, a document-aware context encoding module, a sentence-aware context encoding module, an adaptive fusion and a classifier; wherein: The entity pronoun enhancement module obtains enhanced document data by performing coreference resolution on key entity pronouns in the original document and integrating the data set; The text encoding module annotates the enhanced document data to obtain the document encoding representation H and sentence encoding representation s of the input text n ; The document-aware context encoding module uses an attention matrix to encode the selected entity pairs (e h ,e t ) Calculate the context encoding of the overall perception of the document The context encoding module for acquiring sentence perception uses the entity level co-occurrence matrix to encode the entities (e i ,e j ) Calculate the local perceptual context encoding of the sentence The adaptive fusion module calculates the comprehensive context coding by combining the overall perceived context coding of the document and the local perceived context coding of the sentence according to the following formula: v (h,t) =gs (h,t) +(1-g)c (h,t) Where: W is a trainable parameter, g is the weight representation calculated based on two types of context encoding, The classifier inputs the comprehensive context encoding into the text encoding module according to the following formula to obtain the predicted entity pronoun relationship; namely: in: is a trainable parameter, and is the final entity representation.

2. The enhanced model for document-level relationship extraction according to claim 1, characterized in that: The text encoding module annotates the enhanced document data to obtain the document encoding representation H and sentence encoding representation s of the input text n Process; including: The document encoding representation is obtained by the following formula: H=[h1,h2,…,h l ]=BERT([w1,w2,…,w l ]) in: represents the document encoding, l is the number of tokens, and d represents the dimension of the encoding; BERT is the language training unit; By i The jth time and the code marked with "*" are entered as the mention code The entity embedding representation is obtained by applying the LogSumExp pooling operation to all mentioned encodings The entity encoding is obtained by the following formula: in: Indicates the j-th mention code of the entity, and the code with the "*" before the mention code is used to indicate the mention code; The sentence encoding is obtained by the following formula: in Encode the tokens contained in the sentence.

3. The enhanced model for document-level relationship extraction according to claim 1, characterized in that: The document-aware context encoding module uses an attention matrix to encode the selected entity pairs (e h ,e t ) Calculate the context encoding of the overall perception of the document process; include: Obtain multi-head attention A from the document encoding representation stage ijk , the attention value of the "*" before the mention to each token is taken as the mention attention score, and the mention attention is averaged to get the entity attention score A E ; For a given entity pair (e h ,e t ), get the attention scores of these two entities from the multi-head attention and The entity pair (e h ,e t )Attention score for each token: a (h,t) =q (h,t) / 1Tq (h,t) in, This means that each token-entity pair (e h ,e t )’s importance; According to the document encoding representation H, the document-aware context encoding based on entity pairs is obtained by interactive calculation according to the following formula: c (h,t) =H T a (h,t) 。 4. The enhanced model for document-level relationship extraction according to claim 1, characterized in that: The context encoding module for acquiring sentence perception uses the entity level co-occurrence matrix to encode the entities (e i ,e j ) Calculate the local perceptual context encoding of the sentence Process; including: Construct an entity-level co-occurrence matrix M, where M ij Stores the identified entity e i and e j The IDs of all evidence sentences of the relationship between entities; the sentence number label (sent_id) and relative position label (pos) mentioned by the entity in the dataset are listed in M ij Record explicit co-occurring sentence IDs in Compare the elements in row i and row j one by one; if the elements in column k of both rows are not empty, it means entity e i and e j Respectively with entity e k There is an explicit co-occurrence relationship where entity e k is a bridging entity; ik and M jk The sentence ID is added to M ij In this case, M ij The set in contains the complete list of evidence sentences extracted from explicit and implicit co-occurring sentences; For the entity pair (e h ,e t ), from the co-occurrence matrix M ht All their evidence sentences are obtained; The sentence encodings of all evidence sentences are averaged and compared with the document-aware context encoding based on entity pairs. Interact to achieve sentence-aware context encoding in: M ht A list of all evidence sentences extracted by the heuristic rules.

5. A method for constructing an enhanced model for document-level relationship extraction, characterized in that: Enhanced document data is obtained by performing coreference resolution on key entity pronouns in the original document and integrating them into the dataset; Annotate the enhanced document data to obtain the document encoding representation H and sentence encoding representation s of the input text n ; The attention matrix is ​​used to encode the selected entity pairs (e h ,e t ) Calculate the context encoding of the overall perception of the document The entity-level co-occurrence matrix is ​​used to encode the entities (e i ,e j ) Calculate the local perceptual context encoding of the sentence The comprehensive context coding is obtained by calculating the overall perceived context coding of the document and the local perceived context coding of the sentence according to the following formula; The comprehensive context encoding is input into the text encoding module according to the following formula to obtain the predicted entity pronoun relationship; that is: in: is a trainable parameter, and It is the final entity representation.

Images