A document-level entity relation extraction method based on multi-scale and non-bridging reasoning

Abstract

The application provides a document-level entity relation extraction method based on multi-scale and non-bridging reasoning. The method realizes fine-grained and adaptive document-level relation extraction by constructing a multi-scale semantic space and a cross-axis attention mechanism. First, a pre-trained language model is used to encode the document to generate semantic representations of entity mentions and entities. Then, a multi-scale reasoning module is used to construct multi-granularity relation semantic representations through mention-to-mention, entity-to-mention and entity-to-entity. Next, the representations are enriched through context enhancement and axis attention mechanism. Then, the dependency relationship between bridging and non-bridging elements is fused through cross-axis multi-head attention. Finally, the model is optimized through a classification module and loss calculation.

Description

Technical Field

[0001] This invention relates to the field of natural language processing and information extraction technology, and in particular to a document-level entity relation extraction method based on multi-scale and non-bridging reasoning. Background Technology

[0002] Document-level relation extraction aims to identify semantic relationships between entities appearing in different sentences throughout a document. In practical applications, many relationships need to span multiple sentences and rely on reference resolution and document-level cues. Compared to sentence-level relation extraction, document-level relation extraction needs to address cross-sentence reference problems, capture long-distance dependencies, and integrate scattered evidence while filtering irrelevant context.

[0003] Currently, document-level relation extraction methods are mainly divided into two categories: graph-based methods and sequence-based methods. Graph-based methods construct document-level graph structures and use graph neural networks for reasoning; sequence-based methods directly use the Transformer architecture to capture long-distance dependencies. However, existing methods have the following limitations: (1) they mainly rely on single-granularity representations and cannot adapt to the different semantic granularity requirements of different relation triples; (2) although they effectively utilize bridging entities to capture dependencies, they ignore the potential reasoning contributions of non-bridging elements. These limitations result in limited performance of existing methods when dealing with complex cross-sentence relations and specialized domain texts. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention proposes a document-level entity relation extraction method based on multi-scale and non-bridging reasoning. The core of this invention lies in: constructing a multi-scale semantic space, enabling the model to flexibly select the semantic granularity most suitable for different relation triples; and effectively utilizing bridging and non-bridging elements through a cross-axis attention mechanism to improve the robustness of reasoning. Attached Figure Description

[0005] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings are briefly described below. The following drawings only show some exemplary embodiments of the present invention; those skilled in the art can obtain other forms of drawings based on these drawings without creative effort.

[0006] Figure 1 This is a schematic diagram of the overall framework of the method of this invention. The diagram shows the composition and data flow of the encoding module, the multi-scale reasoning module (including three sub-modules: mention-to-mention, entity-to-mention, and entity-to-entity), the context enhancement module, the axial attention module, the non-bridging reasoning module, and the classification module. The modules have clear connections, labeled in Chinese.

[0007] Figure 2Performance comparison chart of the method of this invention on the DocRED dataset. Clearly showing the F1 and IgnF1 scores compared with the baseline method, the chart is highly clear and labeled in Chinese.

[0008] Figure 3 Performance comparison charts of the proposed method on the CDR and GDA datasets. The charts clearly show the F1, intra-sentence F1, and inter-sentence F1 scores compared to baseline methods. The charts are highly clear and labeled in Chinese. Detailed Implementation

[0009] Example 1: Relation Extraction from General Domain Documents

[0010] Taking the DocRED dataset as an example, this dataset contains 5,053 documents extracted from Wikipedia and Wikidata, covering 97 relation types. This embodiment uses BERT-base as the pre-trained language model encoder.

[0011] Step 1: Document Encoding. For the input document, insert special markers "*" at the beginning and end of entity references. Encode using the BERT-base model to obtain a 768-dimensional contextual semantic embedding for each marker. For documents longer than 512 characters, an overlapping block strategy is used to average and merge the embeddings of overlapping parts.

[0012] Step 2: Entity Representation. For each entity in the document, collect all its mention embeddings. Aggregate these mention embeddings using logarithmic summation and exponential pooling:

[0013]

[0014] Obtain the global semantic representation of the entity. .

[0015] Step 3: Multi-scale reasoning

[0016] 1. Mention-to-mention reasoning: For entity pairs Calculate the relation representation of all its mention pairs. First, obtain the context relation matrix through label-level attention. Then for each mention Calculation relation representation:

[0017]

[0018] in

[0019]

[0020] Finally, the weights are used. Weighted aggregation of all reference pairs.

[0021] 2. Entity-based reasoning: Regarding entities... Entity representations are obtained by modeling the interactions between mentions within a two-layer graph attention network. The entity representation is adapted to the target mention through an attention mechanism. The computational relationship is represented as follows:

[0022]

[0023] in

[0024]

[0025] 3. Entity-to-entity reasoning: reasoning about entities and Enhanced entity embeddings are obtained through graph attention networks and logarithmic summation exponential pooling, respectively. and The computational relationship is represented as follows:

[0026]

[0027] in

[0028]

[0029] Step 4: Context-enhanced Aggregation Entities and All mentioned attention weights are used to calculate the local context vector. Enhance entity representation through linear transformations:

[0030]

[0031] Step 5: Axial attention enhancement on the enhanced solid representation and Apply self-attention along the height and width directions respectively:

[0032]

[0033]

[0034] Step Six: Non-bridging reasoning Decomposed into high-level features and width features Interactive fusion is achieved through a cross-axis multi-head attention mechanism (8 heads):

[0035]

[0036] Obtain the final relation representation .

[0037] Step 7: Relationship Classification A linear layer is used to map the relation to a 97-dimensional relation category space, and the probability of each relation category is calculated using the sigmoid function. An adaptive thresholding strategy is then employed to determine the final predicted relation.

[0038] Step 8: Loss Calculation and Training. An asymmetric polynomial loss function is used, and the hyperparameters are set as follows: The AdamW optimizer is used, and the encoder parameter learning rate is [missing information]. The learning rate for other modules is Train for 30 epochs.

[0039] Experimental Results: The experimental results of this embodiment on the DocRED dataset are shown in Figure 2. On the development set, the proposed method achieved an Ign F1 score of 60.52% and an F1 score of 62.41%; on the test set, it achieved an Ign F1 score of 60.74% and an F1 score of 62.36%. Compared with existing mainstream methods, this method achieves the best performance in all metrics. To further verify the performance of this method, a larger-scale RoBERTa-large encoder was used for experiments. The results show that the proposed method achieves an Ign F1 score of 62.31% and an F1 score of 64.37% on the test set, significantly outperforming the existing best methods CGM2IR (61.96% Ign F1, 63.89% F1) and ATLOP (61.39% Ign F1, 63.40% F1).

[0040] Example 2: Relation Extraction from Biomedical Documents

[0041] Taking the Chemical-Disease Response (CDR) and Gene-Disease Association (GDA) datasets as examples, these two datasets respectively contain the relationships between chemistry and disease, and genes and disease. This embodiment uses SciBERT-base as the pre-trained language model encoder.

[0042] The implementation steps are basically the same as in Example 1, with the main difference being:

[0043] 1. The encoder uses a SciBERT model pre-trained specifically for scientific texts;

[0044] 2. The relationship categories are 2 and 2 respectively;

[0045] 3. To address the characteristics of biomedical texts, the number of layers in the graph attention network and the number of heads in the multi-head attention mechanism were adjusted.

[0046] Experimental Results: The experimental results of this method on biomedical datasets are as follows: Figure 3As shown in the figure. On the CDR dataset, our method achieves an overall F1 score of 75.4%, with 81.4% for intra-sentence relations and 57.7% for cross-sentence relations. Compared to the state-of-the-art method CGM2IR (overall F1 73.8%), our method improves overall performance by 1.6 percentage points, especially in cross-sentence relation extraction by 2.6 percentage points, validating our advantages in long-distance dependency modeling. On the GDA dataset, our method achieves an overall F1 score of 86.7%, with 89.1% for intra-sentence relations and 64.5% for cross-sentence relations. Compared to CGM2IR (overall F1 84.7%), our method improves cross-sentence relation extraction by 5.5 percentage points, fully demonstrating the effectiveness of multi-scale reasoning and non-bridging reasoning mechanisms in complex document modeling.

[0047] This invention, through multi-scale semantic space and cross-axis attention mechanism, has the following beneficial effects:

[0048] 1. Multi-granularity adaptability: Through multi-scale reasoning at three levels—mention-to-mention, entity-to-mention, and entity-to-entity—it can adapt to the different semantic granularity requirements of different relation triples, thereby improving the accuracy of relation extraction.

[0049] 2. Utilization of non-bridging elements: By effectively integrating the dependencies between bridging and non-bridging elements through cross-axis attention mechanisms, more comprehensive contextual information can be captured, thereby improving the robustness of reasoning.

[0050] 3. Positive and negative sample imbalance processing: An asymmetric polynomial loss function is adopted, and the problem of positive and negative sample imbalance is effectively alleviated through an asymmetric focusing mechanism and polynomial coefficient adjustment.

[0051] 4. Strong domain generalization ability: It exhibits excellent performance in both general and biomedical fields, especially in cross-sentence relation extraction tasks, which demonstrates the wide applicability of the method.

[0052] 5. Optimized computational efficiency: By combining axial attention and graph attention networks, computational complexity is controlled while ensuring performance, making it suitable for processing long documents.

[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A document-level entity relation extraction method based on multi-scale and non-bridging reasoning, characterized in that, Includes the following steps: a. Encoding steps: The input document is encoded using a pre-trained language model. Special tags are inserted at the start and end positions of entity mentions to generate contextual semantic embeddings of all tags in the document. For each entity, logarithmic summation and exponential pooling are performed on the embeddings of all its mentions to obtain the global semantic representation of the entity. b. Multi-scale reasoning steps: Constructing a multi-granularity relational semantic representation that includes three levels: mention-to-mention, entity-to-mention, and entity-to-entity. i. Mention Pair Mention Inference Layer: For each mention pair in the entity pair, the context relation matrix is ​​obtained through label-level attention, the relation representation is calculated by combining the head and tail mention embeddings, and the final representation is obtained through weighted aggregation; ii. Entity Pair Mention Inference Layer: The interaction between mentions within an entity is modeled using a graph attention network, and the relationship representation is calculated by combining the entity representation and the target mention representation; iii. Entity-to-Entity Reasoning Layer: Based on entity embeddings and contextual information enhanced by graph attention networks, it computes direct relation representations between entity pairs; c. Context enhancement step: Aggregate the attention weights of all mentions of an entity, calculate the local context vectors between entity pairs, and enhance the entity representation through linear transformation; d. Axial attention enhancement steps: Apply a two-hop axial attention mechanism to the enhanced entity representation, model the interaction between the entity and other entities along the height and width directions respectively, and obtain the relation representation; e. Non-bridging reasoning steps: Through the cross-axis multi-head attention mechanism, interactive fusion is performed between the height and width features of the relation representation to capture the dependency relationship between bridging elements and non-bridging elements; f. Classification steps: Map the fused relation representations to the relation category space through a linear layer, and calculate the probability of each relation category using the sigmoid function; g. Loss calculation steps: The loss between the predicted value and the true value is calculated using an asymmetric polynomial loss function to alleviate the imbalance between positive and negative samples, and the model parameters are updated through backpropagation.

2. The document-level entity relation extraction method based on multi-scale and non-bridging reasoning according to claim 1, characterized in that, The specific implementation of the mention-to-mention-layer is as follows: for entity pairs... and The first mention pair, calculate the first The relationship between two mentions is: ,in , , They are respectively head and tail entity mentions and embeddings. The context relation matrix is ​​calculated through label-level attention; through weights We obtain the final representation by weighted aggregation of all mention pairs. .

3. The document-level entity relation extraction method based on multi-scale and non-bridging reasoning according to claim 1, characterized in that, The specific implementation of the entity in the inference layer is as follows: for the entity Entity representations are obtained by modeling the interactions between internal references using graph attention networks. Through attention weights Adapt entity representations to target mentions The computational relationship is represented as follows: ,in , , .

4. The document-level entity relation extraction method based on multi-scale and non-bridging reasoning according to claim 1, characterized in that, The specific implementation of the entity-to-entity inference layer is as follows: for entities and Enhanced entity embeddings are obtained through graph attention networks and logarithmic summation exponential pooling, respectively: , ; Calculation relation representation: ,in , .

5. The document-level entity relation extraction method based on multi-scale and non-bridging reasoning according to claim 1, characterized in that, The specific implementation of the context enhancement step is as follows: for entities All mentions are aggregated, and their attention weights are calculated: ; Calculate entity pairs Local context vector: , Enhanced entity representation: .

6. The document-level entity relation extraction method based on multi-scale and non-bridging reasoning according to claim 1, characterized in that, The specific implementation of the axial attention enhancement step is as follows: For the enhanced entity representation... and Apply self-attention along the height and width directions respectively: , ,in To learn the projection matrix Calculate the query, key, and value vector.

7. The document-level entity relation extraction method based on multi-scale and non-bridging reasoning according to claim 1, characterized in that, The specific implementation of the non-bridging inference step is as follows: the relational representation of the axial attention output is decomposed into height features. and width features Interactive fusion is achieved through a cross-axis multi-head attention mechanism: , The final output is: .

8. The document-level entity relation extraction method based on multi-scale and non-bridging reasoning according to claim 1, characterized in that, The asymmetric polynomial loss function used in the loss calculation step is: in and For asymmetric focusing hyperparameters, For the truncation probability, These are the polynomial coefficients.

9. The document-level entity relation extraction method based on multi-scale and non-bridging reasoning according to claim 1, characterized in that, It also includes model training and inference steps: during training, the AdamW optimizer is used, and the learning rate adopts linear warm-up and cosine decay scheduling; during inference, the document to be processed and entity information are input into the trained model, and the final relationship prediction is generated through the multi-scale and non-bridging inference process.

10. A document-level entity relation extraction system implementing the method as described in any one of claims 1-9, characterized in that, include: The encoding module is used to encode the input document to obtain a semantic representation; A multi-scale reasoning module is used to construct multi-granularity relational semantic representations; The context enhancement module is used to enhance the contextual information of entity representations; Axial attention module, used to model the interaction relationships between entities; The non-bridging inference module is used to merge the dependencies between bridged and non-bridging elements; The classification module is used to calculate the probability of relation categories; The training module is used to optimize model parameters.

Images