Document-level relation extraction method based on fine-grained global dependency information

Abstract

The invention discloses a document-level relation extraction method based on fine-grained global dependency information, which relates to the technical field of information data processing, and comprises the following steps: S1, a document coding layer: marking the position of an entity appearing in a text by adopting an entity marking method in order to realize accurate representation of entity mentions crossing a plurality of continuous words; specifically, for a given document D = {t1, t2,..., tN}, where N represents the number of words in the document, a special mark "*" is explicitly inserted before and after each entity mention. Then, the marked document is input into a pre-training language model to obtain a representation vector of context perception: H = {h1, h2,..., hN} = Encoder ([t1, t2,..., tN]) # (1) for any entity mention mij, embedding of a mark '*' in front of the entity mention is adopted as a vector of the entity mention to represent # imgabs0 #;

Description

Technical Field

[0001] The present invention relates to the technical field of information data processing, and in particular to a document-level relationship extraction method based on fine-grained global dependency information. Background Art

[0002] Early research in the field of relation extraction focused on extracting entity pair relationships from a single sentence, which usually assumes that the head and tail entities are in the same sentence. However, this assumption greatly limits the application scenarios and scope of the model. In real application scenarios, it is usually necessary to simultaneously use multi-sentence (cross-sentence) associations to achieve joint reasoning to accurately extract entity relationship triplets. Therefore, researchers proposed two methods: a Transformer-based method and a Graph Neural Network (GNN)-based method.

[0003] GNN can model the dependencies between entities or mentions and has strong reasoning ability. By taking entities as nodes and building edges using heuristic rules and semantic dependencies, GNN can be used for reasoning on the graph (such as LSR). Furthermore, some studies (such as GAIN, DRN) further consider the information at the mention level and the entity level. First, a document graph at the mention level is constructed to capture the interaction information between different mentions, and then a document graph at the entity level is constructed to infer the relationship between entities.

[0004] However, due to the advantages of Transformer in implicitly modeling long-distance dependencies, some studies directly apply pre-trained language models to document-level relation extraction (e.g., SSAN, ATLOP). The attention module in the Transformer architecture is used to predict the relationship between entities and further classify the relationship. In addition, in order to deal with the problem of multiple entities in documents, a local context pooling method is proposed. The information related to the target entity pair is located through the attention matrix of the pre-trained language model, and a convolutional neural network (CNN) is introduced to capture the dependencies between entity pairs (e.g., DUN). However, the CNN structure has limitations in modeling all two-hop reasoning path information corresponding to the current element. Some studies have introduced an axial attention mechanism to enable the current entity pair to explicitly pay attention to its horizontal and vertical entity pairs, thereby optimizing the information aggregation process. Although the above methods have achieved certain performance improvements in the field of document-level relation extraction, they still have limitations. In addition, different mention pairs usually point to different contextual information, which leads to the possibility that a single entity pair may show multiple relationships. Therefore, there is a difference between the representation of entity pair relationships and the representation of mention pair relationships. Summary of the invention

[0005] The purpose of the present invention is to provide a document-level relationship extraction method based on fine-grained global dependency information to solve the problems raised in the above background technology.

[0006] By adopting the above technical solution, more sophisticated global dependency information modeling can be achieved.

[0007] In view of the above problems, the technical solution proposed by the present invention is:

[0008] A document-level relation extraction method based on fine-grained global dependency information includes the following steps: S1, document encoding layer: In order to achieve accurate representation of entity mentions across multiple consecutive words, an entity tagging method is used to mark the location where the entity appears in the text. Specifically, for a given document D = {t1, t2, ..., t N}, where N represents the number of words in the document, and a special token "*" is explicitly inserted before and after each entity mention. The tokenized document is then fed into the pre-trained language model to obtain a context-aware representation vector:

[0009] H={h1,h2,...,h N}=Encoder([t1,t2,...,t N ])#(1)

[0010] For any entity mention m ij , using the embedding of the token “*” preceding the entity mention as its vector representation Furthermore, since the same entity may have multiple mentions with the same semantics but involving different contexts, in order to obtain entity e i The corresponding representation The logsumexp function is used to aggregate all mention representations contained in this entity:

[0011]

[0012] Among them, k represents entity e i The corresponding number of mentions, logsumexp represents a smoothing operation of max pooling, which can effectively accumulate weak signals from different mention representations, thereby improving the performance of the model.

[0013] Next, for each mention pair (m s ,m o ), first mention the head to m s and the last mention m o Send it to the linear layer and the nonlinear activation function layer to get the corresponding representation v s and v o ; Next, v s and vo Concatenate them and feed them into the feed-forward neural network layer as input to obtain the corresponding mention pair relationship representation U s,o :

[0014] U s,o =FFNN([v s ,v o ])#(3)

[0015]

[0016]

[0017] Among them, U s,o Indicates the reference to (m s ,m o ). By repeating the above steps, a mention pair relationship representation matrix can be constructed. Among them, N m represents the number of mentions in the document and d represents the dimension of the hidden layer.

[0018] S2, mention reasoning layer: In view of the problem that the original mention pair relationship representation only focuses on its own semantic information and ignores the global dependency information between different mention pairs, a corresponding mention pair reasoning layer is proposed to capture the global dependency information. The specific mention pair reasoning layer structure is composed of L stacked graph attention blocks. In each graph attention block, each mention pair in the document is regarded as a node in the graph. These nodes are initialized with the U matrix representation obtained by the above process, and then relevant information is aggregated from neighboring nodes in the graph attention network. In fact, this graph attention block is a variant of the multi-head attention mechanism. The calculation process of the original attention mechanism is:

[0019]

[0020]

[0021] Among them, k represents the total number of multi-head attention heads, Concat(·) represents vector concatenation,

[0022] represents the learnable parameters of the i-th head.

[0023] However, not all node information in the graph is equally important to the current node. Therefore, a pruning and distance bias strategy is proposed to dynamically adjust the weights of the edges between different neighbor nodes to effectively reduce the noise problem introduced in the learning of node representation. Next, the pruning strategy is first introduced, and then the proposed distance bias strategy is explained. Finally, a hierarchical connection operation is introduced to improve the representation ability and training effect of the model.

[0024] S3, pruning strategy: In the process of modeling the global dependency between entity pairs, not all semantic information of mention pairs can help infer the relationship between the current entity pairs. Based on this phenomenon, it can be inferred that if the mention pair (m s ,e b ) exists and (e b ,m o ) exists, then mentioning (m s ,m o ) may also have a certain correlation. Therefore, the current node t i =(m s ,m o ) The most relevant node set P i is defined as the mention pairs in the document that have overlapping mentions with the node, where P i It means searching for all possible intermediate entities e from the first mentioned node to the last mentioned node. b The reference pair node set that is composed of bridge nodes:

[0025]

[0026] Among them, m b ,m b 'Represents entity e b Mention expressions in different sentences, N b Indicates that the document contains entity e s and entity e o Further considering the directionality of the relationship instance, update the set P i That is, P i Indicated by m s The first mentioned pair, with m o The last mentioned pair, with m s The last mentioned pair and the m o The set union of the mention pairs mentioned by the head:

[0027]

[0028] Finally, node t i The corresponding mask matrix M is:

[0029]

[0030] Next, use the mask matrix M obtained above to optimize the current node t i In the process of aggregating neighbor node information, it aims to make the current node focus more on the logical reasoning information related to the current mention when aggregating information, thereby effectively reducing the noise interference caused by irrelevant nodes. Specifically, the original attention mechanism is modified to formula (11):

[0031]

[0032] Where M represents the attention mask matrix. During the attention calculation process, when a position in M ​​is set to 0, the attention score will approach 0.

[0033] S4, distance bias strategy: The adjacency matrix designed above makes the current mention of node t i It can explicitly focus on other nodes related to it in the document. However, it is also necessary to consider the impact of the position relationship between nodes on the relationship extraction results. Based on this, the distance bias matrix is ​​introduced so that the node t i When aggregating neighbor node information, the relative position relationship between different nodes in the graph can be considered more accurately.

[0034] Specifically, referring to t i =(m s ,m o ) as an example, its neighbor node set is P i , let P i One of the nodes in is t j , then the corresponding distance bias matrix B is:

[0035]

[0036] Among them, Dis(i,j) represents the mention of node t i and t j The length of the word interval between non-repeated mentions; represents the distance mapping function, which maps the word interval length between two mentions into different groups. In addition, the position relationship between nodes is directional, which means that even if the distance is the same, different node position relationships may have different effects on logical reasoning. Therefore, when constructing the distance bias matrix, different values ​​are assigned to the same distance but different node position relationships.

[0037] Finally, the calculation formula of the corresponding attention mechanism is modified to formula (13):

[0038]

[0039] In order to dig deeper into the information, residual connection, normalization layer and feedforward neural network layer are introduced as transition modules to enhance the representation ability of the model and improve the training effect, namely:

[0040]

[0041]

[0042] Among them, LN(·) represents the layer normalization operation, and FFN(·) represents the mapping operation of the feedforward neural network.

[0043] S5, Mention Pair Aggregation Layer: Given the differences in contextual information involved in different mention pairs, entity pairs may show different relationships in different contexts. To solve this problem, a relative distance-aware mention pair relationship representation aggregation layer is constructed. This layer considers the relative distance between mentions, allowing the model to specifically aggregate mention pair representations that contribute significantly to relationship inference.

[0044] Specifically, in the process of inferring entity pair relationships, annotators usually first focus on the contextual information between two entity mentions that appear adjacently. Based on this intuitive inference, we first calculate the mention pair (m i ,m j )’s relative distance representation:

[0045] D i,j =F(Dis(m i ,m j ))#(16)

[0046] Among them, it means to mention m i and mentioned m j The length of the word interval between them; then it represents the distance mapping function shown, which maps the distance between two mentions into different groups.

[0047] Next, formula (17) is used to calculate the contribution α of the contextual information contained in different mention pairs of the target entity pair in the relationship inference process. For the target entity pair, the higher the α value, the more effective the semantic information of the relationship instance of the mention pair can be in supporting the inference of the entity pair relationship.

[0048] α=softmax(FFN(U',Emb(D)))#(17)

[0049] Finally, the relation representations of different mentions belonging to the same entity pair are weighted summed to obtain the entity pair relation representation G enhanced with global dependency information. h,t :

[0050]

[0051] Among them, α ij Indicates that the aggregation belongs to the same entity pair (e h ,e t ) in different mentions, the mention pair relation instance (m i ,m j ) weight coefficient, W pRepresents the learnable network parameters. S6. Relation classification layer: In the relationship classification layer, for a given relationship instance (e h ,e t ), firstly, the representations of the head entity and the tail entity obtained in Section 2.1 are respectively combined with the entity pair relationship representation G obtained in the above reasoning layer with global dependency enhancement h,t Then, the concatenated vector is sent to the nonlinear activation layer to obtain the updated entity representation z h and z t Finally, a bilinear function is used to calculate the entity pair (e h ,e t ) is the score of relation r. The specific calculation process is as follows:

[0052]

[0053]

[0054]

[0055] Traditional multi-label classification methods usually use a global threshold as a fixed boundary for determining entity pair relationships, which ignores the problem that the confidence of the prediction results of the model for different entity pairs and different relationship types varies. Therefore, referring to the approach of Wang et al.

[12] , an adaptive hinge balance loss function is used as the classification loss function to guide the model to learn a more accurate decision boundary. Specifically, for entity pairs (e h ,e t ), the relationship between them is defined as the positive class set A non-existent relationship is defined as a negative class set The threshold class is defined as TH. Then, the adaptive hinge balance loss function aims to promote the output score s of the positive class r The output fraction s of the class above the threshold TH , while ensuring that the output score s of the negative class r The output fraction s of the class below the threshold TH .

[0056]

[0057]

[0058]

[0059] Where σ represents the sigmoid function; d r Represents the score s of the entity pair relationship instance for r relationship category r To the decision boundary s TH If the distance between rIf it is greater than 0, it means that the entity pair relationship is correctly classified; r represents the adjustment parameter; m represents a constant. r When it is greater than m, the entity pair relationship will not be penalized by the loss function. By adjusting this parameter, the impact of simple samples and most negative samples on the model during training can be effectively reduced.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] This model introduces mention-pair information into the modeling process of global dependency information, constructs a global dependency graph at the mention-pair level, and uses the graph attention mechanism to fully learn the semantic feature representation of the mention-pair nodes. In order to alleviate the noise problem in the process of aggregating information by the graph attention mechanism, a pruning strategy and a distance bias strategy are further proposed to adjust the edge weights of neighbor nodes. Among them, the core idea of ​​the pruning strategy is to screen neighbor nodes and only retain nodes that make important contributions to the current node relationship reasoning. The distance bias strategy dynamically adjusts the edge weights based on the positional relationship between nodes. In addition, in order to effectively aggregate different mention-pair relationship representations of the same entity pair, an attention mechanism based on relative distance perception is introduced to achieve more refined global dependency information modeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 A schematic diagram of the overall algorithm framework of FGGD of a document-level relationship extraction method based on fine-grained global dependency information provided by the present invention;

[0063] Figure 2 A reasoning layer structure diagram of a document-level relationship extraction method based on fine-grained global dependency information provided by the present invention. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] See also Figure 1-Figure 2 The present invention provides a technical solution: a document-level relationship extraction method based on fine-grained global dependency information, comprising the following steps:

[0066] S1, document encoding layer: In order to accurately represent entity mentions across multiple consecutive words, an entity tagging method is used to mark the location where the entity appears in the text. Specifically, for a given document D = {t1, t2, …, t N}, where N represents the number of words in the document, and a special token "*" is explicitly inserted before and after each entity mention. The tokenized document is then fed into the pre-trained language model to obtain a context-aware representation vector:

[0067] H={h1,h2,...,h N}=Encoder([t1,t2,...,t N ])#(1)

[0068] For any entity mention m ij , using the embedding of the token “*” preceding the entity mention as its vector representation Furthermore, since the same entity may have multiple mentions with the same semantics but involving different contexts, in order to obtain entity e i The corresponding representation The logsumexp function is used to aggregate all mention representations contained in this entity:

[0069]

[0070] Among them, k represents entity e i The corresponding number of mentions, logsumexp represents a smoothing operation of max pooling, which can effectively accumulate weak signals from different mention representations, thereby improving the performance of the model.

[0071] Next, for each mention pair (m s ,m o ), first mention the head to m s and the last mention m o Send it to the linear layer and the nonlinear activation function layer to get the corresponding representation v s and v o ; Next, v s and v o Concatenate them and feed them into the feed-forward neural network layer as input to obtain the corresponding mention pair relationship representation U s,o :

[0072] U s,o =FFNN([v s ,v o ])#(3)

[0073]

[0074]

[0075] Among them, U s,o Indicates the reference to (m s ,m o). By repeating the above steps, a mention pair relationship representation matrix can be constructed. Among them, N m represents the number of mentions in the document and d represents the dimension of the hidden layer.

[0076] S2, mention reasoning layer: In view of the problem that the original mention pair relationship representation only focuses on its own semantic information and ignores the global dependency information between different mention pairs, a corresponding mention pair reasoning layer is proposed to capture the global dependency information. The specific mention pair reasoning layer structure is composed of L stacked graph attention blocks. In each graph attention block, each mention pair in the document is regarded as a node in the graph. These nodes are initialized with the U matrix representation obtained by the above process, and then relevant information is aggregated from neighboring nodes in the graph attention network. In fact, this graph attention block is a variant of the multi-head attention mechanism. The calculation process of the original attention mechanism is:

[0077]

[0078]

[0079] Among them, k represents the total number of multi-head attention heads, Concat(·) represents vector concatenation,

[0080] represents the learnable parameters of the i-th head.

[0081] However, not all node information in the graph is equally important to the current node. Therefore, a pruning and distance bias strategy is proposed to dynamically adjust the weights of the edges between different neighbor nodes to effectively reduce the noise problem introduced in the learning of node representation. Next, the pruning strategy is first introduced, and then the proposed distance bias strategy is explained. Finally, a hierarchical connection operation is introduced to improve the representation ability and training effect of the model.

[0082] S3, pruning strategy: In the process of modeling the global dependency between entity pairs, not all semantic information of mention pairs can help infer the relationship between the current entity pairs. Based on this phenomenon, it can be inferred that if the mention pair (m s ,e b ) exists and (e b ,m o ) exists, then mentioning (m s ,m o ) may also have a certain correlation. Therefore, the current node t i =(m s ,m o ) The most relevant node set P i is defined as the mention pairs in the document that have overlapping mentions with the node, where P iIt means searching for all possible intermediate entities e from the first mentioned node to the last mentioned node. b The reference pair node set that is composed of bridge nodes:

[0083]

[0084] Among them, m b ,m b 'Represents entity e b Mention expressions in different sentences, N b Indicates that the document contains entity e s and entity e o Further considering the directionality of the relationship instance, update the set P i That is, P i Indicated by m s The first mentioned pair, with m o The last mentioned pair, with m s The last mentioned pair and the m o The set union of the mention pairs mentioned by the head:

[0085]

[0086] Finally, node t i The corresponding mask matrix M is:

[0087]

[0088] Next, use the mask matrix M obtained above to optimize the current node t i In the process of aggregating neighbor node information, it aims to make the current node focus more on the logical reasoning information related to the current mention when aggregating information, thereby effectively reducing the noise interference caused by irrelevant nodes. Specifically, the original attention mechanism is modified to formula (11):

[0089]

[0090] Where M represents the attention mask matrix. During the attention calculation process, when a position in M ​​is set to 0, the attention score will approach 0.

[0091] S4, distance bias strategy: The adjacency matrix designed above makes the current mention of node t i It can explicitly focus on other nodes related to it in the document. However, it is also necessary to consider the impact of the position relationship between nodes on the relationship extraction results. Based on this, the distance bias matrix is ​​introduced so that the node t i When aggregating neighbor node information, the relative position relationship between different nodes in the graph can be considered more accurately.

[0092] Specifically, referring to t i =(m s ,m o ) as an example, its neighbor node set is P i , let P i One of the nodes in is t j , then the corresponding distance bias matrix B is:

[0093]

[0094] Among them, Dis(i,j) represents the mention of node t i and t j The length of the word interval between non-repeated mentions; represents the distance mapping function, which maps the word interval length between two mentions into different groups. In addition, the position relationship between nodes is directional, which means that even if the distance is the same, different node position relationships may have different effects on logical reasoning. Therefore, when constructing the distance bias matrix, different values ​​are assigned to the same distance but different node position relationships.

[0095] Finally, the calculation formula of the corresponding attention mechanism is modified to formula (13):

[0096]

[0097] In order to dig deeper into the information, residual connection, normalization layer and feedforward neural network layer are introduced as transition modules to enhance the representation ability of the model and improve the training effect, namely:

[0098]

[0099]

[0100] Among them, LN(·) represents the layer normalization operation, and FFN(·) represents the mapping operation of the feedforward neural network.

[0101] S5, Mention Pair Aggregation Layer: Given the differences in contextual information involved in different mention pairs, entity pairs may show different relationships in different contexts. To solve this problem, a relative distance-aware mention pair relationship representation aggregation layer is constructed. This layer considers the relative distance between mentions, allowing the model to specifically aggregate mention pair representations that contribute significantly to relationship inference.

[0102] Specifically, in the process of inferring entity pair relationships, annotators usually first focus on the contextual information between two entity mentions that appear adjacently. Based on this intuitive inference, we first calculate the mention pair (m i ,m j)’s relative distance representation:

[0103] D i,j =F(Dis(m i ,m j ))#(16)

[0104] Among them, it means to mention m i and mentioned m j The length of the word interval between them; then it represents the distance mapping function shown, which maps the distance between two mentions into different groups.

[0105] Next, formula (17) is used to calculate the contribution α of the contextual information contained in different mention pairs of the target entity pair in the relationship inference process. For the target entity pair, the higher the α value, the more effective the semantic information of the relationship instance of the mention pair can be in supporting the inference of the entity pair relationship.

[0106] α=softmax(FFN(U′,Emb(D)))#(17)

[0107] Finally, the relation representations of different mentions belonging to the same entity pair are weighted summed to obtain the entity pair relation representation G enhanced with global dependency information. h,t :

[0108]

[0109] Among them, α ij Indicates that the aggregation belongs to the same entity pair (e h ,e t ) in different mentions, the mention pair relation instance (m i ,m j ) weight coefficient, W p Represents the learnable network parameters. S6. Relation classification layer: In the relationship classification layer, for a given relationship instance (e h ,e t ), firstly, the representations of the head entity and the tail entity obtained in Section 2.1 are respectively combined with the entity pair relationship representation G obtained in the above reasoning layer with global dependency enhancement h,t Then, the concatenated vector is sent to the nonlinear activation layer to obtain the updated entity representation z h and z t Finally, a bilinear function is used to calculate the entity pair (e h ,e t ) is the score of relation r. The specific calculation process is as follows:

[0110]

[0111]

[0112]

[0113] Traditional multi-label classification methods usually use a global threshold as a fixed boundary for determining entity pair relationships, which ignores the problem that the confidence of the prediction results of the model for different entity pairs and different relationship types varies. Therefore, referring to the approach of Wang et al.

[12] , an adaptive hinge balance loss function is used as the classification loss function to guide the model to learn a more accurate decision boundary. Specifically, for entity pairs (e h ,e t ), the relationship between them is defined as the positive class set A non-existent relationship is defined as a negative class set The threshold class is defined as TH. Then, the adaptive hinge balance loss function aims to promote the output score s of the positive class r The output fraction s of the class above the threshold TH , while ensuring that the output score s of the negative class r The output fraction s of the class below the threshold TH .

[0114]

[0115]

[0116]

[0117] Where σ represents the sigmoid function; d r Represents the score s of the entity pair relationship instance for r relationship category r To the decision boundary s TH If the distance between r If it is greater than 0, it means that the entity pair relationship is correctly classified; r represents the adjustment parameter; m represents a constant. r When it is greater than m, the entity pair relationship will not be penalized by the loss function. By adjusting this parameter, the influence of simple samples and most negative samples on the model during training can be effectively reduced. In view of the problem that previous methods do not sufficiently represent learning in the process of logical reasoning modeling, a document-level relationship extraction model based on fine-grained global dependency information is proposed to achieve more sophisticated global dependency information modeling.

[0118] In order to evaluate the effectiveness of the scheme proposed in this invention, we mainly conduct experimental verification through experiments on real-world datasets.

[0119] Embodiment 1

[0120] Relation classification performance comparison:

[0121] Experimental setup: Following previous work, we use two re-annotated DocRED datasets: Re-DocRED and DocGNRE. Table 2 provides a comparative statistical analysis of the original DocRED dataset and the two re-annotated datasets. As can be seen from Table 2, the re-annotated datasets significantly supplement the missing relation instances in the original dataset. Among them, Re-DocRED is a re-annotated version of the DocRED dataset.

[0122]

[0123]

[0124] Table 1: Comparison of experimental results on the VQA-CPv2 dataset

[0125] Embodiment 2

[0126] On the other hand, DocGNRE also originates from the re-annotation of the DocRED dataset, which uses a large-scale language model to expand the entity-relation triples of the tail category. For the training set of DocGNRE, the more GPT version is used.

[0127] According to the comparative experimental results given in Table 2, it can be concluded that compared with the baseline models, the proposed model FGGD achieves the best performance; compared with models that do not consider the global dependency between different entity pairs, such as ATLOP, SSAN, etc., the DocuNet and KD-DocRE models achieve better performance, indicating that modeling the global dependency information between entity pairs helps to improve the performance of the model in document-level relationship extraction tasks; compared with the DocuNet model and the KD-DocRE model, the proposed FGGD model still has better performance, indicating that when modeling the global dependency between entity pairs, the effectiveness of fully considering the semantic differences between relationship instances of different mention pairs can help the model obtain more fine-grained logical reasoning information, thereby improving the accuracy of model relationship reasoning.

[0128]

[0129] Table 2: Experimental results on Re-DocRED and DocGNRE datasets

[0130] Embodiment 3

[0131] Comparative experiment on intra-sentence / inter-sentence relation extraction:

[0132] In order to verify the ability of the proposed FGGD model in solving cross-sentence relation instances, the Intra F1 and Inter F1 values ​​of different variant models on the Re-DocRED test set are compared, where ATLOP represents the baseline model without considering logical reasoning; DocuNet and KD-DocRE represent models that consider logical reasoning, and corresponding reasoning modules are designed in the method, but both of them model the global dependency information at the entity pair level.

[0133] Model IntraF1 InterF1 ATLOP 76.37 71.29 DocuNet 77.09 72.17 KD-DocRE 77.85 72.89 FWf 79.50 73.85 w / oMPR 77.68 73.03 w / oMPA 78.62 73.32

[0134] Table 3: Experimental comparison of the Intra F1 and Inter F1 values ​​of the model on the Re-DocRED test set with the most advanced models: The experimental results of multi-attribute sentiment control are shown in Table 3, and we have the following observations: the performance of all models in processing inter-sentence relationship instances is lower than that in processing intra-sentence relationship instances. This is because inter-sentence relationship instances need to use multi-sentence (cross-sentence) associations to achieve joint reasoning to accurately extract the corresponding relationship types between entities; compared with the baseline model that does not explicitly model the logical reasoning phenomenon in the document-level relationship extraction task, it has achieved significant improvement in the Inter F1 indicator, and its results fully demonstrate the importance of modeling the global dependency information between entity pairs; compared with the DocuNet and KD-DocRE models, the Inter F1 indicator has improved by 1.68% and 0.96% respectively, which proves the effectiveness of considering the differences in semantic expression of relationship representations of different mention pairs in the process of modeling global dependency information between different entity pairs.

Claims

1. A document-level relationship extraction method based on fine-grained global dependency information, characterized in that: The following steps are involved: S1, document encoding layer: In order to accurately represent entity mentions across multiple consecutive words, an entity tagging method is used to mark the location of the entity in the text. Specifically, for a given document D = {t1, t2, …, t N }, where N represents the number of words in the document, and a special token "*" is explicitly inserted before and after each entity mention. The tokenized document is then fed into the pre-trained language model to obtain a context-aware representation vector: H={h1,h2,...,h N }=Encoder([t1,t2,...,t N ])#(1) For any entity mention m ij , using the embedding of the token "*" preceding the entity mention as its vector representation Furthermore, since the same entity may have multiple mentions with the same semantics but involving different contexts, in order to obtain entity e i The corresponding representation The logsumexp function is used to aggregate all mention representations contained in this entity: Next, for each mention pair (m s ,m o ), first mention the head to m s and the last mention m o Send it to the linear layer and the nonlinear activation function layer to get the corresponding representation v s and v o ; Next, v s and v o Concatenate them and feed them into the feed-forward neural network layer as input to obtain the corresponding mention pair relationship representation U s,o : U s,o =FFNN([v s ,v o ])#(3) S2. Mention reasoning layer: In order to solve the problem that the original mention pair relationship representation only focuses on its own semantic information and ignores the global dependency information between different mention pairs, a corresponding mention pair reasoning layer is proposed to capture the global dependency information. The reasoning layer structure is composed of L graph attention blocks stacked together. In each graph attention block, each mention pair in the document is regarded as a node in the graph. These nodes are initialized by the U matrix representation obtained by the above process, and then the relevant information is aggregated from the neighboring nodes in the graph attention network. In fact, the graph attention block is a variant of the multi-head attention mechanism. The calculation process of the original attention mechanism is: However, not all node information in the graph is equally important to the current node. Therefore, a pruning and distance bias strategy is proposed to dynamically adjust the weights of the edges between different neighbor nodes to effectively reduce the noise problem introduced in the learning of node representation. Next, the pruning strategy is first introduced, followed by the proposed distance bias strategy. Finally, the hierarchical connection operation is introduced to improve the representation ability and training effect of the model. S3, pruning strategy: In the process of modeling the global dependency between entity pairs, not all semantic information of the mention pair can help infer the relationship between the current entity pair. Based on this phenomenon, it can be inferred that if the mention pair (m s ,e b ) exists and (e b ,m o ) exists, then mentioning (m s ,m o ) may also have a certain correlation, so the current node t i =(m s ,m o ) The most relevant node set P i is defined as the mention pairs in the document that have overlapping mentions with the node, where P i It means searching for all possible intermediate entities e from the first mentioned node to the last mentioned node. b The reference pair node set that is composed of bridge nodes: Further considering the directionality of the relationship instance, update the set P i , that is, P i Indicated by m s The first mentioned pair, with m o The last mentioned pair, with m s The last mentioned pair and the m o The set union of the mention pairs mentioned by the head: Finally, node t i The corresponding mask matrix M is: Next, use the mask matrix M obtained above to optimize the current node t i In the process of aggregating neighbor node information, it aims to make the current node focus more on the logical reasoning information related to the current mention when aggregating information, thereby effectively reducing the noise interference caused by irrelevant nodes. Specifically, the original attention mechanism is modified to formula (11): S4, distance bias strategy: The adjacency matrix designed above makes the current mention of node t i It can explicitly focus on other nodes related to it in the document. However, it is also necessary to consider the impact of the position relationship between nodes on the relationship extraction results. Based on this, the distance bias matrix is ​​introduced so that node t i When aggregating neighbor node information, the relative position relationship between different nodes in the graph can be considered more accurately. Specifically: To mention t i =(m s ,m o ) as an example, its neighbor node set is P i , let P i One of the nodes in is t j , then the corresponding distance bias matrix B is: In addition, the positional relationship between nodes is directional, which means that even if the distance is the same, different node position relationships may have different effects on logical reasoning. Therefore, when constructing the distance bias matrix, different values ​​are assigned to the same distance but different node position relationships. Finally, the calculation formula of the corresponding attention mechanism is modified to formula (13): In order to dig deeper into the information, residual connection, normalization layer and feedforward neural network layer are introduced as transition modules to enhance the representation ability of the model and improve the training effect, namely: Among them, LN(·) represents the layer normalization operation, and FFN(·) represents the mapping operation of the feedforward neural network; S5. Mention pair aggregation layer: Given the differences in contextual information involved in different mention pairs, entity pairs may show different relationships in different contexts. To solve this problem, a relative distance-aware mention pair relationship representation aggregation layer is constructed. This layer considers the relative distance between mentions, allowing the model to specifically aggregate mention pair representations that contribute significantly to relationship inference. Specifically: In the process of inferring entity pair relationships, annotators usually first focus on the contextual information between two entity mentions that appear adjacently. Based on this intuitive inference, they first calculate the mention pair (m i ,m j )’s relative distance representation: D i,j =F(Dis(m i ,m j ))#(16) Next, formula (17) is used to calculate the contribution α of the contextual information contained in different mention pairs of the target entity pair in the relationship inference process. For the target entity pair, the higher the α value, the more effective the semantic information of the relationship instance of the mention pair can be in supporting the inference of the entity pair relationship. α=softmax(FFN(U',Emb(D)))#(17) Finally, the relation representations of different mentions belonging to the same entity pair are weighted summed to obtain the entity pair relation representation G enhanced with global dependency information. h,t : Among them, α ij Indicates that the aggregation belongs to the same entity pair (e h ,e t ) in different mentions, the mention pair relation instance (m i ,m j ) weight coefficient, W p represents the learnable network parameters; S6. Relation classification layer: In the relationship classification layer, for a given relationship instance (e h ,e t ), firstly, the representations of the head entity and the tail entity obtained in Section 2.1 are respectively combined with the entity pair relationship representation G obtained in the above reasoning layer with global dependency enhancement. h,t Then, the concatenated vector is sent to the nonlinear activation layer to obtain the updated entity representation z h and z t Finally, the bilinear function is used to calculate the entity pair (e h ,e t )For the score of relation r, the specific calculation process is as follows: An adaptive hinge balance loss function is used as the classification loss function to guide the model to learn a more accurate decision boundary. Specifically, for entity pairs (e h ,e t ), the relationship between them is defined as the positive class set P T , the non-existent relationship is defined as the negative class set N T , the threshold class is defined as TH, then the adaptive hinge balance loss function aims to promote the output score s of the positive class r The output fraction s of the class above the threshold TH , while ensuring that the output score s of the negative class r The output fraction s of the class below the threshold TH Among them, if d r If it is greater than 0, it means that the entity pair relationship is correctly classified; r represents the adjustment parameter; m represents a constant. r When it is greater than m, the entity pair relationship will not be penalized by the loss function.

2. According to claim 1, a document-level relationship extraction method based on fine-grained global dependency information is characterized in that: In S1, k represents entity e i The corresponding number of mentions, logsumexp represents a smoothing operation of max pooling, which can effectively accumulate weak signals from different mention representations, thereby improving the performance of the model.

3. According to claim 1, a document-level relationship extraction method based on fine-grained global dependency information is characterized in that: In S1, U s,o Indicates the reference to (m s ,m o ). By repeating the above steps, a mention pair relationship representation matrix can be constructed. Among them, N m represents the number of mentions in the document and d represents the dimension of the hidden layer.

4. According to claim 1, a document-level relationship extraction method based on fine-grained global dependency information is characterized in that: In S2, k represents the total number of multi-head attention heads, Concat(·) represents vector concatenation, represents the learnable parameters of the i-th head.

5. According to claim 1, a document-level relationship extraction method based on fine-grained global dependency information is characterized in that: In S3, m b ,m b 'Represents entity e b Mention expressions in different sentences, N b Indicates that the document contains entity e s and entity e o Entity nodes other than .

6. The document-level relationship extraction method based on fine-grained global dependency information according to claim 1 is characterized in that: In S3, M represents the attention mask matrix. During the attention calculation process, when a position in M ​​is set to 0, the attention score will approach 0.

7. The document-level relationship extraction method based on fine-grained global dependency information according to claim 1 is characterized in that: In S4, Dis(i,j) represents the mention of node t. i and t j The length of the word interval between non-repeated mentions; represents the distance mapping function, which maps the length of the word interval between two mentions into different groups.

8. The document-level relationship extraction method based on fine-grained global dependency information according to claim 1 is characterized in that: In S5, it means that m is mentioned i and mentioned m j The length of the word interval between them; then it represents the distance mapping function shown, which maps the distance between two mentions into different groups.

9. The document-level relationship extraction method based on fine-grained global dependency information according to claim 1 is characterized in that: In S6, σ represents the sigmoid function; d r Represents the score s of the entity pair relationship instance for r relationship category r To the decision boundary s TH The distance between.

Images