Document-level relation extraction method based on multi-level feature collaborative modeling
By combining local interactive convolution and global semantic attention mechanisms through the HCAN network, evidence is dynamically selected and the loss function is optimized. This solves the problems of insufficient local and global semantic modeling and class imbalance in document-level relation extraction, and achieves higher inference accuracy and interpretability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHONGQING UNIV OF POSTS & TELECOMM
- Filing Date
- 2025-12-08
- Publication Date
- 2026-05-12
AI Technical Summary
Existing document-level relation extraction techniques suffer from insufficient local and global semantic modeling, imbalanced relation categories, and inaccurate evidence selection in long documents, resulting in inference accuracy and interpretability.
A hierarchical multi-level feature modeling network (HCAN) is adopted, which combines local interactive convolution and global semantic attention mechanism to dynamically select relation evidence. A hybrid adaptive loss function and knowledge distillation framework are designed to improve cross-sentence reasoning ability and robustness.
It significantly improves cross-sentence semantic reasoning ability, alleviates the class imbalance problem, and enhances the interpretability and generalization ability of the model.
Smart Images
Figure CN122019776A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of Natural Language Processing (NLP), and more particularly to a feature modeling technique for document-level relation extraction tasks. Specifically, this invention proposes a document-level relation extraction method based on multi-level feature collaborative modeling, applicable to scenarios such as entity relation recognition in long texts, knowledge graph construction, information extraction, and question answering systems. Background Technology
[0002] Relation extraction is one of the core tasks in natural language processing, aiming to automatically identify semantic relationships between entity pairs from text. Document-level relation extraction (DocRE), compared to sentence-level tasks, requires cross-sentence reasoning across multiple sentences or even the entire document, placing higher demands on the model's global semantic modeling capabilities.
[0003] Existing methods can be broadly categorized into three types: sequence-based encoding methods, graph-based methods, and pre-trained language model-based methods. The first two types are effective for processing short texts, but they have significant limitations in complex scenarios such as long documents, cross-sentence relationships, and class imbalance. (1) Insufficient local and global semantic modeling: Single-sentence-level models have difficulty capturing cross-sentence dependencies between entities, resulting in decreased inference accuracy; (2) The distribution of relation categories is extremely unbalanced: a few high-frequency relations dominate the training process, while low-frequency or long-tail relations are easily ignored. (3) Static evidence selection mechanism: The model has difficulty dynamically focusing on key sentences in the document that support relational inference, resulting in poor interpretability; (4) Scarcity of labeled data: High-quality manual labeling is expensive, and weakly supervised or remotely supervised data often contains noise, resulting in insufficient model generalization ability.
[0004] Therefore, existing document-level relation extraction techniques urgently need to achieve breakthroughs in the fusion of local interaction features and global semantic information, adaptive optimization of difficult samples, and knowledge transfer under weak supervision conditions, so as to improve both the accuracy and interpretability of relation extraction. Summary of the Invention
[0005] The purpose of this invention is to overcome the problems of insufficient local semantic modeling, class imbalance, and inaccurate evidence selection in existing document-level relation extraction techniques, and to provide a document-level relation extraction method based on multi-level feature collaborative modeling. By introducing a Hierarchical Collaborative Attention Network (HCAN), combined with local interactive convolution and global semantic attention mechanisms, relation evidence sentences are dynamically selected, and a hybrid adaptive loss function and knowledge distillation framework are designed to improve the model's cross-sentence reasoning ability and robustness.
[0006] To achieve the above objectives, the method of the present invention includes the following steps: 1) Document representation and entity embedding generation: The input document is segmented into sentences and words and then fed into a pre-trained language model (such as BERT) to obtain the contextual representation of each word; and the contextual embedding of entities is generated through a dynamic weighted aggregation mechanism. 2) Local interactive convolution modeling: A local interaction convolution module is constructed to perform convolution operations on the neighborhood semantic matrix of entity pairs, extract local interaction features, and reflect fine-grained semantic relationships between entities. 3) Global semantic interaction modeling: The global semantic interaction module is designed to dynamically aggregate the most relevant global semantic information of entity pairs in the document through a co-attention mechanism, thereby realizing cross-sentence global dependency modeling. 4) Multi-level feature synergy fusion: By alternately combining local convolutions and global attention mechanisms across multiple stacked layers, multi-layer semantic fusion and feature enhancement are achieved, resulting in multi-scale feature representations. 5) Evidence selection mechanism: Dynamic attention scores are assigned to each sentence, and the set of sentences that best support entity pair relation inferences is selected as evidence to improve the interpretability of the reasoning. Hybrid adaptive loss optimization: The design incorporates a hybrid adaptive loss function (HAL) that includes dynamic perceptual adjustment and class boundary separation mechanisms to dynamically adjust the gradient weights of hard samples and enhance the discriminative power of relational class boundaries. 6) Knowledge distillation training: A teacher-student framework is adopted to train the teacher model on labeled data to generate pseudo-relation labels and evidence distributions, and guide the student model to jointly optimize relation classification and evidence selection tasks on remotely supervised data.
[0007] Compared with the prior art, the present invention has the following advantages and innovations: (1) Multi-level feature collaborative modeling mechanism: For the first time, multi-level collaborative fusion of local convolution and global attention is realized, which significantly improves the ability of cross-sentence semantic reasoning; (2) Dynamic evidence selection module: Automatically selects key sentences through attention guidance to achieve interpretability of relational reasoning; (3) Hybrid adaptive loss function: Introducing dynamic perception and boundary separation mechanism to effectively alleviate class imbalance problem and improve model robustness; (4) Knowledge distillation framework: Utilizing pseudo-supervisory signals for remote data optimization to enhance model generalization ability and stability. Attached Figure Description
[0008] Figure 1 This is a schematic diagram of the document-level relation extraction method based on multi-level feature collaborative modeling proposed in this invention. Detailed Implementation
[0009] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings, but should not be construed as limiting the scope of protection of the present invention.
[0010] like Figure 1 As shown, the overall architecture of this invention includes five core modules: document representation module, local interactive convolution module, global semantic interaction module, evidence selection module, and hybrid adaptive optimization module.
[0011] The input document is encoded by a pre-trained language model (such as BERT-base) to obtain a word-level context representation. For each entity e i Entity embeddings are generated by dynamically weighting and aggregating the representations of all its mentions: The resulting entity vector As input for subsequent feature modeling.
[0012] Constructing the feature matrix of entity pairs Extracting neighborhood semantic features through local interactive convolution operations: F′ ij =W*F ij Where * represents a convolution operation, and W is a learnable convolution kernel. This module captures local interaction patterns between entities, improving the ability to recognize short-range dependencies.
[0013] The correlation between entity pairs and the global context is calculated based on the co-attention mechanism. α=softmax((F′ ij W q (HW) k ) T ) H′=αH This allows for dynamic focusing on the sentences and words most relevant to relational inference, enabling long-distance semantic capture.
[0014] Local and global modules are alternately stacked in L layers to form a multi-level feature fusion structure: F (l) =Fusion(F (l-1) H ′(l-1) ) This mechanism can transmit information between different semantic levels, enabling hierarchical collaborative learning between local interactions and global context.
[0015] Assign evidence scores to each sentence: S i =softmax(W e h i ) A higher score indicates a greater contribution of the sentence to the current entity-to-relationship. This module utilizes pseudo-supervisory signals generated by the teacher model to constrain the evidence distribution, enhancing interpretability.
[0016] The HAL loss function of this invention is composed of a dynamically sensed adjustment term L. adaptive Separation item L from category boundary sep composition: L HAL =λ1L adaptive +λ2L sep The former dynamically adjusts the gradient attention based on the sample difficulty, while the latter optimizes the relationship to determine the boundary through learnable interval constraints, thereby improving robustness and generalization as a whole.
[0017] The teacher model is trained on a labeled dataset to generate pseudo-relation labels and sentence-level evidence distributions; the student model is optimized on remote supervised data using these as soft labels, with the loss defined as: This design fully leverages the complementarity between labeled and unlabeled data, improving the overall performance of relation classification and evidence selection.
[0018] Experiments on the publicly available DocRED dataset demonstrate that the HCAN model proposed in this invention improves the F1 and Ign_F1 scores by approximately 2 percentage points compared to mainstream methods, especially in scenarios involving cross-sentence relationships and long-distance dependencies.
[0019] In summary, the document-level relation extraction method based on multi-level feature collaborative modeling proposed in this invention has significant innovations in model structure, optimization strategy, and interpretability, and can be widely applied in fields such as knowledge graph construction, intelligent question answering, and semantic understanding.
Claims
1. A document-level relation extraction method based on multi-level feature collaborative modeling, characterized in that, include: (a) Document encoding steps: Input the target document and use a pre-trained language model to generate word-level representations containing document context information to capture long-distance dependencies and inter-sentence semantic structure. (b) Entity representation construction steps: Aggregate multiple mentions of entities in the document to obtain entity-level representations and form initial feature vectors for entity pairs. (c) Local interaction modeling steps: Based on the initial vector of entity pairs, neighborhood interaction information is extracted through the local interaction convolution module to represent the fine-grained interaction pattern of entity pairs within the local semantic range. (d) Global semantic modeling steps: Based on the correlation between entity pair features and the overall semantic representation of the document, focus on the global semantic information most relevant to the current entity pair through an attention mechanism to generate a global semantic interaction representation. (e) Multi-level feature fusion step: The local interactive convolutional module and the global semantic module are stacked in multiple levels, so that semantic information of different scales is gradually fused between layers to obtain deep, multi-scale entity pair semantic features. (f) Evidence selection step: Based on deep fusion features, the evidence weight of each sentence in the document is estimated to obtain the distribution of key sentences that support the inference of the relationship between the current entity pairs. (g) Relationship prediction step: Combine fusion features and evidence distribution to classify entity pairs into relationships and output the set of relationship categories to which they belong. (h) Hybrid adaptive loss optimization steps: By using a hybrid adaptive loss function with dynamic adjustment capabilities, the model can be made more sensitive and robust to difficult-to-classify relationships and low-frequency relationships. (i) Knowledge distillation step: Using a teacher-student framework, the student model is guided to improve its ability to infer long-distance and complex relationships through pseudo-relation labels and pseudo-evidence distribution generated by the teacher model.
2. The method according to claim 1, wherein, The local interaction convolution module is used to capture the neighborhood interaction features of entity pairs in a local range, so as to enhance the model's ability to recognize fine-grained semantic cues.
3. The method according to claim 1, wherein, The global semantic modeling step uses an attention mechanism to weighted aggregate the semantic information of the entire document in order to highlight important contexts related to entity relationships.
4. The method according to claim 1, wherein, The multi-level feature fusion step adopts an alternating stacking method, which gradually strengthens local features and global features in a multi-layer structure, thereby capturing semantic relationships across sentences and over long distances.
5. The method according to claim 1, wherein, The evidence selection step calculates the evidence importance score for each sentence based on the semantic representation of the sentence, thereby forming a sentence-level evidence distribution to assist in the relation inference results.
6. The method according to claim 1, wherein, The hybrid adaptive loss function includes: (a) Dynamic sample adjustment term, used to adaptively adjust the weights of hard-to-classify samples and long-tail category samples; (b) Inter-category boundary adjustment term, used to construct learnable separation boundaries between similar relationship categories, thereby improving the discriminative ability of relationship prediction.
7. The loss function according to claim 6, wherein, The inter-category boundary adjustment term dynamically adjusts the minimum separation interval between categories based on the confusion structure between relation categories, in order to reduce the relation prediction error rate.
8. The method according to claim 1, wherein, The knowledge distillation step guides the student model to learn the attention distribution of key sentences in the document by using the pseudo-evidence distribution provided by the teacher model, thereby improving the accuracy of evidence selection.
9. The method according to claim 1, wherein, The teacher model is trained on a small-scale set of real-labeled data, and its relation prediction results and evidence distribution are used as pseudo-supervisory signals for training the student model.
10. A system for performing the document-level relation extraction method according to any one of claims 1 to 9, comprising: (a) Document encoding module; (b) Entity aggregation module; (c) Entity pair building module; (d) Local interactive convolution module; (e) Global semantic interaction module; (f) Multi-level feature fusion module; (g) Evidence selection module; (h) Relationship prediction module; (i) Loss optimization module; (j) Knowledge distillation module; The above modules work together to execute the method described in claim 1.
11. The system according to claim 10, wherein, The evidence selection module utilizes sentence-level semantic representation and entity pair fusion features to collaboratively generate sentence importance scores, providing explanatory basis for relation classification.
12. A computer-readable storage medium having a program stored thereon that, when executed by a processor, causes the processor to perform the method according to any one of claims 1 to 9.