Evidence perception document level relation extraction method based on dynamic semantics and reinforcement learning
By combining a dynamic semantic enhancement module and a reinforcement learning strategy, the problems of insufficient task interaction and poor adaptability of static thresholds in document-level relation extraction are solved. This achieves synergistic optimization of relation classification and evidence prediction, improving the accuracy and generalization ability of document-level relation extraction.
Patent Information
- Application Number
- CN202511532267.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-24
- Publication Date
- 2026-02-10
AI Technical Summary
Existing technologies suffer from insufficient task interaction, lack of adaptability of static thresholds, and limited ability to model long documents in document-level relation extraction, resulting in insufficient accuracy and generalization ability of relation classification and evidence prediction.
This study employs a Dynamic Semantic Augmentation Module (DynaSpan) and a reinforcement learning strategy, utilizing an end-to-end joint learning mechanism to achieve synergistic optimization of relation classification and evidence prediction. Specific steps include data preprocessing, long text segmentation and encoding, entity pair feature extraction, dynamic semantic augmentation, feature fusion, relation classification, and reinforcement learning-based evidence retrieval and dynamic threshold optimization.
It improves the accuracy of relation extraction and the flexibility of evidence discovery, enhances the ability to model long documents, addresses the limitations of traditional methods in complex relation reasoning and evidence fusion, and strengthens the semantic interaction representation and generalization ability of the model.
Smart Images

Figure CN121503626A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of natural language processing and artificial intelligence, and in particular to an evidence-aware document-level relation extraction method based on dynamic semantics and reinforcement learning. Background Technology
[0002] Document-Level Relation Extraction (DocRE), a core research area in Natural Language Processing and Information Extraction, aims to accurately identify semantic relationships between all entity pairs from a complete document composed of multiple sentences, extracting them into structured triples (head entity, relation, and tail entity), and locating key evidence sentences supporting relation determination. Document-level relation extraction technology plays an irreplaceable role in applications such as knowledge graph construction, supporting intelligent question-answering systems, and improving text understanding capabilities. With the exponential growth of information volume, how to efficiently and accurately extract valuable relational information from massive amounts of documents has become a major challenge facing both academia and industry.
[0003] Currently, the mainstream techniques for document-level relation extraction primarily rely on pre-trained language models (such as BERT and RoBERTa) to obtain deep contextual representations of documents. These models, through pre-training on large-scale corpora, acquire rich linguistic knowledge and contextual dependencies, providing powerful feature representation capabilities for relation extraction. In practical implementation, bilinear classifiers or linear classifiers are typically used to predict the relationship type between entity pairs. For locating evidence sentences, most methods rely on attention mechanisms or other salient feature extraction techniques, binarizing sentence-level attention scores by setting a static threshold to identify evidence sentences supporting relation judgments.
[0004] In related technologies, CN119938927A proposes a document-level relation extraction method based on natural language processing. This method constructs a relatively complete document-level relation extraction framework, attempting to solve the problem of extracting entity relations from documents holistically. It utilizes natural language processing techniques to parse and analyze documents to identify entities and the relationships between them. However, when processing long documents, the accuracy of this method in capturing complex relationships between entities and locating evidence still needs improvement. Long documents contain a large number of sentences and entities, and the relationships between entities may span multiple sentences or even paragraphs. When handling such complex situations, this framework struggles to comprehensively and accurately identify all relationships and accurately locate evidence sentences.
[0005] CN114548117B proposes a causal relation extraction method based on BERT semantic enhancement. This method utilizes the BERT model to enhance semantic representation. The BERT model can capture contextual information and semantic features in text, providing a richer semantic foundation for causal relation extraction. This method achieves certain improvements in causal relation extraction tasks, as deep encoding of text using the BERT model enables a better understanding of the causal logic between entities in sentences. However, this method mainly focuses on causal relations, while the types of relations in documents are diverse, including not only causal relations but also parallel relations, subordinate relations, etc. This patented method lacks adaptability to extracting other types of relations in documents and cannot fully meet the needs of document-level relation extraction.
[0006] However, existing technologies still have many significant shortcomings. Regarding task interaction, relation classification and evidence prediction are often optimized as independent or loosely coupled subtasks, lacking an effective multi-task collaboration mechanism. This results in the semantic information and reasoning structure between the two tasks failing to fully enhance each other, thus limiting overall performance improvement. For example, when processing documents containing multiple entities and complex relationships, the relation classification task may not be able to fully utilize the information provided by the evidence prediction task, and vice versa.
[0007] The lack of adaptability of static thresholds is also a major problem with existing technologies. Attention distributions across different documents are highly heterogeneous, making it difficult for fixed global thresholds to adapt to variations in document types and relationship categories. For example, methods using fixed thresholds for evidence screening are limited in handling diverse interaction patterns between entities in long documents. Especially in cross-sentence and cross-paragraph relationship reasoning, this often leads to low evidence recall or precision, thus affecting the model's generalization ability.
[0008] Given the numerous challenges and shortcomings of existing technologies in document-level relation extraction, such as insufficient task interaction and poor adaptability of static thresholds, and the failure of the aforementioned patented methods to fully address these issues, the industry urgently needs a novel document-level relation extraction method that can adaptively adjust the evidence judgment threshold, deeply integrate relations and evidence semantic representations, and possess strong generalization capabilities. This invention presents an innovative solution to this need, achieving synergistic optimization of relation classification and evidence prediction by integrating a dynamic semantic enhancement module (DynaSpan) with reinforcement learning strategies. Summary of the Invention
[0009] The technical problem to be solved by this invention is to provide an evidence-aware document-level relation extraction method based on dynamic semantics and reinforcement learning, which solves the shortcomings of existing technologies in document-level relation extraction (DocRED) tasks, such as insufficient task interaction, lack of adaptability of static thresholds, and limited ability to model long documents, and achieves simultaneous improvement in relation classification and evidence discovery performance.
[0010] To achieve the above objectives, the present invention adopts the following technical solution: The evidence-aware document-level relation extraction method based on dynamic semantics and reinforcement learning is implemented as follows, with each step closely linked to form an end-to-end joint learning mechanism: Step 1: Data Preprocessing and Format Conversion Several DocRED (Document-Level Relation Extraction Dataset) datasets are processed, and training and validation sets are divided. Entities and their mentions in the data are traversed, and the sentence numbers and position ranges of the mentions are recorded. Entity markers are inserted to clarify entity boundaries. The head entity, tail entity, relation type, and a list of evidence sentences for each relation instance are extracted. The relation type is mapped to a predefined category ID to construct a multi-label classification label vector, and weight labels are generated for the evidence sentences. The sentences are converted into Token ID sequences using a token segmenter, and the [CLS] and [SEP] special symbols are added and truncated to a preset length to obtain a format suitable for model input.
[0011] Step 2: Long text segmentation and encoding To address the issue that long texts exceed the single-pass processing capacity of pre-trained Transformer models [such as BERT (Bidirectional Encoder Representations from Transformers) and RoBERTa (Robustly Optimized BERT Approach)], an overlapping segmentation method is employed to divide long sequences into multiple overlapping sub-segments. Each sub-segment is encoded separately using a pre-trained model to obtain hidden states and attention weights. During merging, the hidden states of the overlapping regions are weighted and averaged, and the attention weights are standardized. Then, a padding function is used to adjust the length to the original sequence length, ultimately obtaining a complete sequence representation at the token level, thus avoiding the loss of global structural information in long documents.
[0012] Step 3: Entity Pair Feature Extraction The encoded sequence is input into the get_hrt module to calculate the entity embedding and the joint attention distribution of entity pairs: if an entity consists of multiple token spans, the exponential average embedding (LSE, LogSumExp) is used to generate the overall entity representation and calculate the mean of attention weights; if it is a single token span entity, the sequence embedding and attention weights at the corresponding positions are directly extracted; for each candidate entity pair, the attention weights of the head entity and the tail entity are multiplied element by element and the mean is taken in the attention head dimension, and after normalization, the joint attention distribution of entity pairs is obtained; then the sequence embeddings are weighted and summed using this distribution to obtain the entity pair context vector, and finally the head entity vector, tail entity vector, entity pair context vector, joint attention distribution, and number of entity pairs in the batch are output.
[0013] Step 4: Entity Feature Assembly The head entity vector, tail entity vector, and entity pair context vector output by the get_hrt module are input into the forward_rel module. The head entity representation and tail entity representation are concatenated with the context representation along the feature dimension to enhance the initial interaction between entity features and relational context, laying the foundation for subsequent semantic enhancement.
[0014] Step 5: Dynamic Semantic Enhancement The concatenated head and tail entity representations are input into the Dynamic Semantic Enhancement Module (DynaSpan): This module contains a learnable parameter matrix new_tokens (which is optimized through training after initialization to learn specific semantic features) and a Feature Enhancement Unit (composed of a TokenLearner and a Transformer encoder). The Token Learner generates selection weights through a lightweight MLP (Multi-Layer Perceptron) to perform soft selection on the concatenated token sequence to achieve feature compression. The Transformer encoder captures the global dependencies of the compressed sequence based on a multi-head self-attention mechanism, combines a feedforward neural network to perform feature transformation, and adds positional encoding to preserve the sequence order, finally outputting the enhanced entity features.
[0015] Step 6: Feature Fusion The enhanced entity representation output by DynaSpan is weighted and averaged with the original entity representation output by the forward_rel module. Dynamic interactive features are incorporated while preserving the original semantic information to obtain the final entity representation, balancing feature integrity and interactivity.
[0016] Step 7: Relationship Classification The final entity representation is input into a bilinear classifier: first, the three-dimensional feature tensors of the head and tail entities are expanded in dimension and then multiplied element by element to generate bilinear interaction features; then, the interaction features are reshaped into two-dimensional tensors, and the relationship prediction logits between the head and tail entities are output through the classification layer to realize relationship type recognition.
[0017] Step 8: Reinforcement Learning Evidence Retrieval and Dynamic Threshold Optimization Input the entity pair joint attention distribution, batch entity pair quantity, and sentence position output by the get_hrt module into the policy gradient reinforcement learning Evidence Predictor With Policy Gradient policy network (including Policy Gradient Threshold Agent).
[0018] The policy network is a multi-layer fully connected network that outputs the probability distribution of three actions: "increase threshold, decrease threshold, and maintain threshold". It dynamically adjusts the evidence decision threshold based on the actions and constrains it within a reasonable range. Design a multi-objective reward function: (1) In the formula, For the total reward, The average attention score for predicting evidence (encouraging the selection of high-attention sentences). To penalize items where the predicted amount of evidence deviates from the ideal amount (to avoid too much / too little evidence). This is to encourage terms related to the variance of attention distribution (enhancing the diversity of evidence).
[0019] During training, the cumulative reward gradient of the discount is calculated, an entropy regularization term is added to prevent the policy from converging too early, and the network parameters are updated through the optimizer to achieve accurate evidence localization.
[0020] The evidence-aware document-level relation extraction method based on dynamic semantics and reinforcement learning provided by this invention has the following beneficial effects: 1. This invention effectively solves the problem of insufficient task interaction in document-level relation extraction tasks. Traditional methods treat relation classification and evidence prediction as independent or loosely coupled sub-tasks, failing to establish an effective multi-task collaboration mechanism. In contrast, this invention achieves effective collaboration between the two.
[0021] 2. This invention overcomes the problem of static thresholds lacking adaptability. Fixed global thresholds are difficult to adapt to changes in multiple document types and relationship categories, resulting in low evidence recall or precision. This invention adopts a dynamic strategy to improve adaptability.
[0022] 3. This invention improves the computational efficiency and semantic coherence issues of the traditional Transformer architecture when processing long documents, avoids the loss of global structural information, and enhances the ability to model long documents.
[0023] 4. This invention solves the representation bottleneck in existing methods for entity interaction modeling and enhances the interaction capabilities between entities through more refined modeling.
[0024] 5. This invention overcomes the error propagation problem in the evidence fusion process and improves the accuracy and reliability of evidence fusion.
[0025] 6. This invention has made a breakthrough in complex relationship reasoning, overcoming the limitations of traditional methods in handling complex relationships and improving the ability of relationship reasoning.
[0026] 7. This invention achieves more refined semantic interaction representation through a dynamic semantic enhancement module, thereby enhancing the model's ability to understand complex semantic relationships.
[0027] 8. This invention uses a strategy gradient network to dynamically optimize the evidence decision threshold, which solves the problem of traditional methods relying on fixed thresholds and improves the flexibility and accuracy of evidence screening.
[0028] 9. This invention designs a multi-objective optimization reward function that combines basic rewards, quantity penalty terms, and sparsity rewards, thereby improving the model convergence speed and accelerating the training process.
[0029] 10. This invention proposes an end-to-end joint training mechanism, which realizes effective collaboration between relation classification and evidence prediction tasks, and enhances the mutual reinforcement of semantic information and reasoning structure.
[0030] 11. This invention preprocesses DocRED data, including entity labeling, positive and negative example sampling, and annotation of evidence sentences, converting the raw data into a format more suitable for model processing and improving data quality.
[0031] 12. This invention performs segmentation processing on long texts by dividing long sequences into multiple sub-segments using an overlapping block method, encoding them separately, and then merging the results, effectively solving the problem of long text encoding.
[0032] 13. This invention utilizes the get_hrt module to obtain vector representations of the head entity, tail entity, and the context between them, and generates a joint attention distribution of entity pairs, providing rich semantic information for relation extraction.
[0033] 14. This invention puts the head entity, tail entity, and context vector into the forward_rel module for feature concatenation to generate enhanced entity features, which further improves the quality of feature representation.
[0034] 15. The present invention uses the Dyna Span dynamic semantic enhancement module to enhance the features of the concatenated head and tail entity representations, which significantly improves the semantic recognition capability and enhances the accuracy of relation extraction.
[0035] 16. This invention uses a policy gradient reinforcement learning network, Evidence Predictor With PG, to predict evidence and dynamically adjust thresholds, thereby achieving joint optimization of relation extraction and evidence discovery and improving overall performance. Attached Figure Description
[0036] The present invention will be further described below with reference to the accompanying drawings and embodiments: Figure 1 This is a schematic diagram of the visual process framework of the technical solution of the present invention. Detailed Implementation
[0037] The technical solutions of the present invention will be further described below with reference to the embodiments and accompanying drawings: Example 1 like Figure 1 As shown, this embodiment provides an evidence-aware document-level relation extraction method based on dynamic semantics and reinforcement learning, aiming to accurately identify semantic relationships between entity pairs in long texts and locate evidence sentences supporting these relationships. This method achieves joint optimization of relation extraction and evidence retrieval through a Dynamic Semantic Enhancement Module (DynaSpan) and a Policy Gradient Reinforcement Learning Network (PolicyGradient). The specific steps are as follows: Step 1: Data Preparation and Preprocessing Step 1.1, Data Collection: Collect N DocRED datasets as experimental data. This dataset contains a large number of documents, and each document contains multiple entities and the relationships between them.
[0038] Step 1.2, Data Labeling: For each data point, entity labeling, positive and negative example sampling, and evidence sentence labeling are performed. Specific steps include: Step 1.2.1: Traverse each entity and its mention in the document, record the sentence number (sent_id) and position range (pos) of the mention, and insert entity markers (e.g., ...) into the text. <entity>).
[0039] Step 1.2.2: Traverse each relation instance in the document, extracting the head entity (h), tail entity (t), relation type (r), and a list of evidence sentences. Map the relation type to a predefined category ID to construct a multi-label classification label vector (relation).
[0040] Step 1.2.3: Use a tokenizer to convert the sentence into a token ID sequence and add special symbols [CLS] and [SEP] to truncate the sequence to a maximum length of 1024 to improve computational efficiency.
[0041] Step 2, Long Text Encoding Step 2.1, Long Text Segmentation: For long texts with more than 512 tokens, the overlapping block method is used to segment them into multiple processable sub-segments. Specifically, two overlapping segments are created: the first segment contains the beginning part of the sequence (positions 0 to 512-len_end) and the end marker [SEP], and the second segment contains the start marker [CLS] and the end part of the sequence (positions l_i-512+len_start to l_i).
[0042] Step 2.2, Sub-segment Encoding: Input each sub-segment into the pre-trained Transformer model (such as BERT or RoBERTa) for encoding, obtain the hidden states and attention weights of the last three layers of the model, and calculate their average value as the final output of the sub-segment.
[0043] Step 2.3, Result Merging: The hidden states and attention weights in the overlapping regions are weighted and standardized to obtain a complete sequence representation. Specifically, the weighted average is performed using the following formula: (2) Standardize the attention weights: (3) Finally, the padding function (F.pad) is used to adjust each segment of the output to the original sequence length.
[0044] Step 3: Entity and Relationship Feature Extraction Step 3.1, Entity Embedding and Attention Calculation: Input the encoded training set into the get_hrt module to obtain the head entity vector, tail entity vector, entity pair context vector, and joint attention distribution of entity pairs. Specific steps include: Step 3.1.1: Traverse each entity in the document and calculate its entity embedding vector and attention weights. If an entity consists of multiple token spans, calculate its exponential average embedding as the overall representation and calculate the average of the corresponding attention weights.
[0045] Step 3.1.2: For each candidate entity pair, multiply the attention weight of the head entity and the attention weight of the tail entity element-wise, and take the average over the attention head dimension to obtain the joint attention weight. Normalize the joint attention weight so that its sum is 1.
[0046] Step 3.1.3: Use the normalized joint attention weights to perform a weighted summation of the embeddings of the entire sequence to obtain a vector representation of the entity in relation to the relevant context.
[0047] Step 3.2, Feature Concatenation: Input the head entity vector, tail entity vector, and entity pair context vector into the forward_rel module for feature concatenation. Specifically, the head entity representation and tail entity representation are concatenated with the context representation along the last dimension, and then linear transformation and nonlinear activation (such as the hyperbolic tangent function tanh) are performed through a fully connected layer.
[0048] Step 4: Dynamic Semantic Enhancement Feature Enhancement: The concatenated head and tail entity representations are input into the Dynamic Semantic Enhancement module DynaSpan for feature enhancement. This module contains a learnable parameter matrix new_tokens and a FeatureEnhancement Unit, which consists of a Token Learner and a Transformer encoder.
[0049] Step 4.1: Initialize the learnable parameter matrix new_tokens and use the Xavier uniform initialization method to initialize the parameters.
[0050] Step 4.2: Concatenate new_tokens with the input head and tail entity representations in the sequence dimension to form an enhanced token sequence.
[0051] Step 4.3: TokenLearner generates selection weights using a lightweight multilayer perceptron (MLP) to perform soft selection on the enhanced token sequence, achieving feature compression. Specifically, weighted feature fusion is performed using the following formula: (4) The Transformer encoder captures the global dependencies between tokens in the compressed token sequence based on a multi-head self-attention mechanism and performs feature transformation through a feedforward neural network. Simultaneously, positional encoding is added to preserve the sequence's order information.
[0052] Step 5: Feature Fusion and Relationship Classification Step 5.1, Feature Fusion: The refined features output by the DynaSpan module and the original enhanced features output by the forward_rel module are weighted and averaged to obtain the final entity representation. Specifically, the refined features and the original enhanced features are weighted and averaged at a ratio of 50% each, and the fused features are reshaped into a 3D tensor to extract fine-grained feature interaction patterns.
[0053] Step 5.2, Relationship Classification: Input the final entity representation into a bilinear classifier to obtain the relationship prediction logits between the head and tail entities. Specifically, the three-dimensional tensor is expanded in dimension and then multiplied element-wise to obtain bilinear interaction features, which are then reshaped into a two-dimensional tensor and input into the bilinear classification layer to generate the relationship classification logits.
[0054] Step 6: Evidence Retrieval and Threshold Adjustment Evidence Prediction: The joint attention distribution of entity pairs output from the get_hrt module, the number of entity pairs in each batch, and the sentence position are input into the policy gradient reinforcement learning Evidence Predictor WithPG policy network for evidence prediction and dynamic threshold adjustment. This module includes a Policy Gradient Threshold Agent, whose policy network is a multilayer perceptron (MLP). The specific steps are as follows: Step 6.1, State Feature Extraction: Extract five-dimensional feature vectors from document attention, including average attention, attention standard deviation, maximum attention value, current threshold, and normalized document length, as input to the policy network.
[0055] Step 6.2, Action Selection and Threshold Update: The policy network outputs the probability distribution of three actions: increasing the threshold, decreasing the threshold, and maintaining the threshold. The agent updates the current threshold according to the selected action and constrains the updated threshold within a preset reasonable range (e.g., [0.01, 0.99]).
[0056] Step 6.3, Reward Calculation: The reward function is designed as a sum of three parts: a base reward (the average attention score of the predicted evidence), a quantity penalty (penalizing deviations from the ideal quantity of predicted evidence), and a sparsity reward (encouraging variance in attention distribution). Specifically, the total reward... for: (1); in, To predict the average attention score of the evidence, To penalize terms where the number of predicted evidence deviates from the ideal number (e.g., 1 / 5 of the number of sentences in a document), This is a term to encourage the variance of attention distribution.
[0057] Step 6.4, Policy Update: During training, calculate the gradient of the cumulative reward with discount and add an entropy regularization term to prevent premature policy convergence. Update the network parameters using the Adam (Adaptive Moment Estimation) optimizer and prune the gradients to avoid gradient anomalies.
[0058] Step 7, Experiment and Evaluation Step 7.1, Experiment Setup: Conduct experiments on the DocRED dataset, using BERT or RoBERTa as the pre-trained Transformer model. Set appropriate hyperparameters, such as learning rate, batch size, and hidden layer dimension.
[0059] Step 7.2 Evaluation Metrics: Standard information extraction evaluation metrics, such as precision, recall, and F1 score, are used to evaluate the model's performance on relation extraction and evidence retrieval tasks.
[0060] Step 7.3, Experimental Results: Through comparative experiments, the significant improvement of the method of the present invention in terms of relation extraction accuracy and evidence retrieval robustness is verified.
[0061] Example 2 In another preferred embodiment, based on Embodiment 1, this embodiment provides an evidence-aware document-level relation extraction method based on dynamic semantics and reinforcement learning, such as... Figure 1 As shown in the visualized workflow framework of the technical solution, the core objective is to address the three major technical pain points mentioned in this invention: insufficient long text modeling capabilities, fragmented semantic interaction, and poor adaptability to static thresholds. It aims to achieve "end-to-end joint optimization of relationship classification and evidence retrieval," covering the entire logical chain from raw data input to final result output. Each module is closely integrated according to the technical details of this invention, forming a complete technical implementation path. The specific steps are as follows: The process begins with the raw DocRED data. It first enters the "Data Preprocessing and Format Conversion" module of the data source and initial processing layer. The core function of this module is to standardize the raw data, providing suitable input for subsequent modeling, corresponding to the core logic of the "Data Preprocessing" section of this invention. Specifically, it iterates through the entities and mentions in the vertexSet field of the data, inserting entity markers such as [ENT-H] / [ENT-T] to clarify the head / tail entity boundaries; simultaneously, it extracts the relation type of the labels field, maps it to predefined category IDs, and generates weight labels based on the evidence sentence list, constructing a label vector for multi-label classification; finally, it generates a Token ID sequence through a token segmenter, adds the special symbols [CLS] / [SEP], truncates or pads it according to the preset max_seq_length of this invention, and outputs model input features such as input_ids / entity_pos / labels. The processed data then flows to the long text encoder.
[0062] The core objective of the long text encoder is to address the "feature loss problem when traditional Transformers process long documents" as pointed out in this invention, and to achieve complete semantic modeling of long texts. This module includes three sub-steps: "long text segmentation → sub-segment encoding → merging sequence representations". When the length of the input sequence exceeds the single processing capacity of the pre-trained Transformer (such as BERT / RoBERTa), the long sequence is split into multiple sub-segments according to the overlapping block method of this invention with overlap_rate=0.3. The overlapping region is set to avoid the breakage of entity features across segments. Then, each sub-segment is input into the pre-trained Transformer, which outputs the hidden state (sequence semantic representation) and attention weights (inter-token correlation) of the sub-segment. Finally, the mask_guided_fusion function disclosed in this invention is called to fuse the hidden states of the overlapping regions according to the principle of "directly retaining non-overlapping areas and weighted averaging of overlapping areas", while standardizing the attention weights. The final output is a token-level sequence representation with the same length as the original text, which is then further input into the Get_HRT module.
[0063] The Get_HRT module extracts core features of entity pairs from the encoded sequence, providing foundational features for subsequent semantic enhancement and evidence retrieval, corresponding to the core design of the "Get_HRT module" in this invention. In terms of feature generation logic, for the head and tail entity vectors, if the entity spans multiple tokens, the "exponential average embedding (logsumexp)" method of this invention is used to calculate the overall representation; if it spans a single token, the sequence embedding at the corresponding position is directly extracted. The generation of the entity pair context vector requires first multiplying the head / tail entity attention weights of the candidate entity pair element-wise, taking the mean and normalizing it along the attention head dimension, and then using this "joint attention distribution" to weightedly sum the sequence embeddings to obtain the context semantic vector related to the entity pair. Simultaneously, the generated entity pair joint attention distribution records the token-level attention weights of the head / tail entity interactions, providing attention basis for subsequent evidence retrieval. From a data flow perspective, the head entity vector, tail entity vector, and entity pair context vector are input to the feature concatenation module, while the entity pair joint attention distribution flows to the Evidence Predictor WithPG policy network.
[0064] After receiving the three types of vectors output by the Get_HRT module, the feature concatenation module concatenates them along the feature dimension, integrating the entity's own semantic information with the contextual semantic information. Then, the concatenated features are input into the DynaSpan dynamic semantic enhancement module, thus entering the "semantic enhancement and relation classification branch". The purpose of this branch is to improve the entity's ability to express interaction features through dynamic semantic enhancement, so as to achieve accurate relation classification, which corresponds to the "DynaSpan module" and "bilinear classifier" sections of this invention. The DynaSpan module comprises a "feature enhancement unit Token Learner + Transformer". The learnable parameter matrix `new_tokens` is initialized according to the "relationship category prototype vector" method of this invention to learn the semantic features of specific relations. The Token Learner generates selection weights through a lightweight MLP and uses the sigmoid activation (threshold 0.2) set in this invention to perform soft selection on the concatenated features, achieving redundant feature compression. The Transformer encoder captures the global dependency of compressed features based on multi-head self-attention, combines a feedforward neural network for feature transformation, and incorporates the "semantic consistency loss (L_consist)" of this invention to prevent enhanced features from deviating from the original semantics. Finally, the enhanced features and the original concatenated features are weighted and averaged according to the ratio of this invention (e.g., 7:3) to output the final entity features. The final entity features are input into a bilinear classifier, first reshaped into a three-dimensional tensor according to the "feature grouping granularity" of this invention, then the tensor dimension expansion and element-wise multiplication capture fine-grained interaction features of the head / tail entities. After flattening the interaction features, they are input into a fully connected layer, outputting predicted logits corresponding to the predefined relation category. These logits ultimately point to the "relationship result output" stage.
[0065] Simultaneously, the "Evidence Retrieval and Threshold Optimization Branch" is launched. This branch aims to dynamically optimize the evidence decision threshold through reinforcement learning, addressing the "poor adaptability of static thresholds" issue in this invention and achieving accurate evidence sentence localization. First, the joint attention distribution of entity pairs output by the Get_HRT module is sliced according to "sentence position (sent_pos)," and the attention score for each sentence is summed. This score reflects the correlation between the sentence and the entity pair relationship. Then, the attention scores are input into the Evidence Predictor WithPG policy network. This network, according to this invention, expands the 8-dimensional state vector (including sentence attention mean / standard deviation, current threshold, relation category entropy, etc.) to comprehensively characterize the current decision-making environment, outputting the probability distribution of three actions: "increase threshold / decrease threshold / maintain threshold." The action execution step size and threshold range (e.g., [0.01, 0.99]) are preset according to this invention. In the Policy Gradient dynamic threshold adjustment stage, the "multi-objective dynamic weight reward function" of this invention is adopted. (1); in, , , The weights are determined by predicting the average attention score of the evidence, penalizing terms that deviate from the ideal number of predicted evidence, and encouraging terms that encourage the variance of the attention distribution; these weights are adaptively adjusted based on the entropy value of the current batch relationship distribution (high entropy relationship enhancement). Weighting, reduction The weights are calculated, and the gradient of the cumulative reward for discount is calculated. The "entropy regularization term" of this invention is added to prevent the strategy from converging too early. The network parameters are updated by the Adam optimizer and the gradient is pruned according to the threshold of this invention. Finally, according to the adjusted dynamic threshold, sentences with attention scores exceeding the threshold are selected as the final evidence sentences and output to the "evidence result output" stage.
[0066] From the perspective of overall logical closed loop and technical synergy, the Get_HRT module provides basic features for the two core branches, realizing feature sharing for "relationship classification and evidence retrieval"; the semantic enhancement of the DynaSpan module not only improves the accuracy of relationship classification but also provides a more reliable attention basis for evidence retrieval, while the reinforcement learning threshold optimization provides interpretable evidence support for the relationship classification results, forming a collaborative mechanism of "classification-evidence" two-way verification. Regarding the resolution of technical pain points, for long text modeling, the "overlapping block + mask fusion" approach overcomes the length limitation, combined with the DynaSpan module to enhance long-distance semantic interaction; for fragmented semantic interaction, the "feature concatenation + bilinear interaction" approach achieves deep interaction between entities and context, and between head and tail entities; for the defects of static thresholds, the "reinforcement learning dynamic threshold + adaptive reward" approach adapts to heterogeneous documents and relationship categories. The entire process, from raw data input to "relationship result + evidence sentence" output, has no redundant or broken links between modules, and is completely connected according to the technical details of this invention, ultimately achieving an end-to-end, deployable DocRE solution.
[0067] Example 3 In another preferred embodiment, based on embodiments 1 and 2, this embodiment provides an evidence-aware document-level relation extraction method based on dynamic semantics and reinforcement learning, the specific steps of which are as follows: Step 1: Process N DocRED datasets to obtain training and validation sets; perform entity labeling, positive and negative sample sampling, and evidence sentence annotation on the training and testing sets, converting the original DocRED data into a model input format suitable for relation classification and evidence extraction.
[0068] Specifically, it iterates through each entity and its mention in the vertexSet, records the sentence number (sent_id) and position range (pos) of each mention, inserts entity markers into the text to help the model identify entity boundaries, and generates the processed sentence sequence (sents), sentence map (sent_map), and sentence position list (sent_pos).
[0069] Iterate through each relation instance in `labels`, extracting the head entity (h), tail entity (t), relation type (r), and a list of evidence sentences. Map the relation type to a predefined category ID (using the `docred_rel2id` dictionary) to construct a multi-label classification label vector (relation). Generate a weight label (sent_evi) for each evidence sentence for subsequent evidence sentence prediction tasks.
[0070] The sentence is converted into a sequence of token IDs (input_ids) using a tokenizer, and special symbols ([CLS], [SEP]) are added. The sequence is truncated to the maximum length (max_seq_length=1024) to ensure computational efficiency.
[0071] The output consists of a feature list containing the input ID (input_ids), entity position (entity_pos), relation label (labels), entity pair list (hts), and sentence position list (sent_pos) for each document.
[0072] Step 2: Perform long text segmentation on the transformed training set, dividing it into multiple overlapping sub-segments. Encode each sub-segment separately and then merge the results to obtain a token-level sequence representation, thus achieving long text encoding.
[0073] Specifically, it receives a sequence of token IDs (input_ids) and a corresponding attention mask, defining start and end markers specific to the model type: for the BERT model, the start marker is the ID of the classification marker [CLS], and the end marker is the ID of the separator marker [SEP]. For the RoBERTa model, the start marker is the ID of [CLS], and the end marker is the IDs of both [SEP] markers.
[0074] When the length of the input sequence is less than or equal to 512 tokens (i.e., within the model's single-processing capacity), the module adopts a direct processing method. The entire input_ids and attention_mask are directly input into the pre-trained Transformer model (such as BERT or RoBERTa), the hidden states and attention weights of the last three layers of the model are obtained, and their average value is calculated as the final output.
[0075] When the input sequence length exceeds 512 tokens, the module initiates an intelligent segmentation mechanism: It uses overlapping block segmentation to divide the long sequence into multiple processable segments. For each sequence exceeding 512 tokens, two overlapping segments are created: the first segment contains the beginning of the sequence (positions 0 to 512 - len_end) and an end marker; the second segment contains the start marker and the end of the sequence (positions l_i - 512 + len_start to l_i). Each segment is then input into the Transformer model for independent encoding. The module uses a weighted average method to fuse the outputs of each segment into a complete sequence representation. Output fusion, hidden states of overlapping regions, and weighted average based on attention masks: (2) Attention fusion and standardization of attention weights for overlapping regions: (3) Finally, padding and alignment are performed using the padding function (F.pad) to adjust each segment of the output to the original sequence length, ensuring dimensional consistency.
[0076] The output, sequence_output, has dimensions (batch_size, doc_len, hidden_dim), and represents the final hidden state encoded by a pre-trained model (such as BERT), carrying deep semantic information for each token. The attention tensor, with dimensions (batch_size, num_attn_heads, doc_len, doc_len), represents the weight matrix of the pre-trained model's self-attention mechanism, reflecting the degree of attention between tokens.
[0077] Step 3: Put the encoded training set into the get_hrt module to obtain vector representations of the head entity, tail entity, and the context between them, and generate the joint attention distribution of entity pairs. This yields the head entity vector, tail entity vector, entity pair context vector, entity pair joint attention distribution, and the number of entity pairs in each batch.
[0078] Specifically: First, the size (n, h, _, c) of the attention tensor is obtained, and an empty list is initialized to temporarily store the head entity embedding (hss), tail entity embedding (tss), context embedding (rss), and joint attention (ht_atts) for each batch. Then, the module processes the data in the batch sample by sample.
[0079] The module calculates entity embeddings and entity attention. For the i-th sample in a batch, it iterates through the entity position information e of each sample in entity_pos[i]. If an entity consists of multiple token spans (i.e., len(e) > 1), the module calculates the exponential average embedding (logsumexp) of the entity as its overall representation and calculates the average value of its corresponding attention weights. This method effectively fuses information from multiple parts of an entity and maintains numerical stability. If an entity is represented by only one token span, the sequence embedding and attention weight at the corresponding offset position are directly extracted. To prevent index out-of-bounds errors, the module checks whether the position index plus the offset is still within the valid range. If it is out of bounds, it is padded with a zero vector.
[0080] Stack the embedding vectors and attention weights of all entities to obtain tensors entity_embs and entity_atts, with dimensions (num_entities_in_sample, hidden_dim) and (num_entities_in_sample, num_attn_heads, doc_len), respectively.
[0081] Entity pair feature extraction and interactive computation are performed. For each candidate entity pair to be examined in the sample (specified by hts[i]), the embedding vectors (hs, ts) and attention weights (h_att, t_att) of the corresponding head and tail entities are selected from entity_embs and entity_atts using torch.index_select based on the entity pair index. The attention weights h_att of the head entity and t_att of the tail entity are multiplied element-wise and averaged over the attention head dimension to obtain the joint attention weight ht_att, which reflects the common attention region of the entity pair. This weight is then normalized to make its sum equal to 1.
[0082] The normalized joint attention weights ht_att are used to perform a weighted summation on the embedding sequence_output[i] of the entire sequence (implemented by matrix multiplication via contract("ld, rl->rd", sequence_output[i], ht_att)) to obtain the vector representation rs representing the entity pair with its relevant context. The results for the current sample are stored: hs, ts, rs, and ht_att calculated for the current sample are added to temporary lists hss, tss, rss, and ht_atts, respectively. The number of candidate entity pairs in each batch is calculated and stored in the list rels_per_batch.
[0083] The results of all samples in the temporary list are concatenated along the 0th dimension (batch dimension) to return the final five output data items: head entity vector, tail entity vector, entity pair context vector, entity pair joint attention distribution, and number of entity pairs in each batch.
[0084] Step 4: Put the head entity embedding vector hs, the tail entity embedding vector ts, and the relation embedding vector rs into the forward_rel module, and concatenate the head entity representation and the tail entity representation with the context representation respectively.
[0085] Specifically, the module receives three input tensors: head entity embedding vector `hs`, tail entity embedding vector `ts`, and relation embedding vector `rs`. These tensors all have a dimension of (num_ent_pairs_all_batches, emb_size), where `num_ent_pairs_all_batches` represents the number of entity pairs in the current batch. To enhance the interaction between entity and relation features, the module concatenates `hs` and `rs` along the last dimension (`torch.cat([hs, rs], dim=-1)`), performs a linear transformation through a fully connected layer (`self.head_extractor`), and finally applies a non-linear mapping using the hyperbolic tangent activation function (`torch.tanh`) to obtain the enhanced head entity feature `hs`. The enhancement process for the tail entity feature `ts` is exactly the same, performed through `self.tail_extractor`.
[0086] Step 5: The concatenated head and tail entity representations are fed into the Dynamic Semantic Enhancement module DynaSpan for further feature enhancement, improving semantic recognition capabilities. The core of this module is the DynaSpan Transformer class, which contains a learnable parameter matrix `new_tokens` and a Feature Enhancement Unit (FMU), composed of a Token Learner and a Transformer encoder. The Token Learner generates selection weights using a lightweight MLP to perform soft selection on the input sequence, achieving feature compression. Subsequently, the Transformer encoder encodes the compressed feature sequence, capturing global dependencies, and finally outputs the enhanced representation.
[0087] Specifically, the concatenated head and tail entity representations are divided into multiple small batches (with a step size of 256). For each small batch, the head entity feature hs[batch_2:batch_2 + step] and the tail entity feature ts[batch_2:batch_2 + step] are expanded and concatenated in the second dimension to form a tensor input_tokens of shape (step, 2, emb_size), which serves as the input to the dynamic semantic enhancement module DynaSpan.
[0088] First, initialize a set of learnable token parameters as the starting point for semantic enhancement: (5) These learnable tokens are continuously optimized through backpropagation during training, ultimately learning vector representations of specific semantic features. The learnable tokens are then concatenated with the input token along the sequence dimension. (6) This operation creates an enhanced token sequence, providing a foundation for subsequent feature selection and semantic fusion.
[0089] The concatenated token sequence is intelligently compressed using a dynamic feature selector. First, the input tokens are standardized to stabilize the training process. Then, a multilayer perceptron (MLP) block maps the input features to a high-dimensional space. A softmax function is used to generate attention weights, indicating the importance of each token. Weighted feature fusion is achieved using the Einstein summation convention. (4) The compressed token sequence is fed into a semantic fusion unit, where deep semantic modeling is performed using a Transformer encoder. The Transformer encoder captures global dependencies between tokens based on a multi-head self-attention mechanism and performs feature transformation through a feed-forward network. Positional encoding is implicitly learned or incorporated through traditional sine / cosine functions to preserve the sequence's order information.
[0090] Finally, the dynamic semantic enhancement module DynaSpan outputs the encoded features. Its first dimension corresponds to the head entity, and its second dimension corresponds to the tail entity. These refined features are collected into lists hs2 and ts2 respectively. After all mini-batches are processed, the refined features are concatenated into complete tensors hs2 and ts2, with the same dimensions as the input hs and ts.
[0091] Step 6: Finally, the enhanced entity representation and the original entity representation are weighted and averaged to obtain the final entity representation that retains the original information while introducing interactive features.
[0092] Specifically, the module performs a weighted average fusion of the refined features (hs2, ts2) and the original enhanced features (hs, ts) (each accounting for 50%) to obtain the final feature representation. Next, the fused features are reshaped by the view into two 3D tensors, b1 (head entity) and b2 (tail entity), with dimensions (-1, self.emb_size / / self.block_size, self.block_size). This operation groups the feature vectors along the last dimension, aiming to extract finer-grained feature interaction patterns. self.block_size defines the granularity of the grouping.
[0093] By unsqueezeing b1 and b2 along a specific dimension and performing element-wise multiplication, the module computes the bilinear interaction feature bl2. This interaction feature is then reshaped into a two-dimensional tensor (-1, self.emb_size × self.block_size) for use by the classification layer.
[0094] Step 7: Put the final entity representation into a bilinear classifier to obtain the relationship prediction logits between the head and tail entities.
[0095] Specifically, the bilinear interaction feature bl2 is processed by a bilinear classification layer (self.bilinear) to generate the final relation classification logits, which have the dimension (num_ent_pairs_all_batches, num_rel_labels), where num_rel_labels represents the total number of predefined relation types. These logits can be used to calculate the loss or to obtain the probability distribution through a softmax function, thereby completing the relation classification.
[0096] Step 8: Next, evidence retrieval is performed. The entity pair joint attention distribution output by the get_hrt module, the number of entity pairs in each batch, and the sentence position are input to the policy gradient reinforcement learning Evidence Predictor WithPG policy network's predict_evidence function for evidence prediction and dynamic threshold adjustment. This module includes a PolicyGradient Threshold Agent. Its policy network is a multilayer perceptron (MLP). The reward function is designed as follows: (1) in, To obtain the average attention score for the predicted evidence, we directly take the average attention score of the sentence containing the predicted evidence. To penalize items where the number of predicted evidences deviates from the ideal number (e.g., 1 / 5 of the number of sentences in a document), the degree of deviation between the number of predicted evidences and the preset ideal number is calculated, and penalty items are set based on this degree of deviation. To incentivize terms that reflect the variance of the attention distribution, the dispersion of the attention distribution is calculated, and reward terms are set based on this dispersion; the total reward R is... , , The sum of these three factors. During training, the network parameters are updated by calculating the gradient of the cumulative reward with discounts and adding an entropy regularization term to prevent the policy from converging prematurely.
[0097] Specifically, the system comprises two core components: the Policy Gradient Threshold Agent and the Evidence Predictor WithPG. The following parameters need to be set during initialization: State dimension (state_dim): The default value is 5, which corresponds to the 5-dimensional feature vector extracted from document attention; Hidden layer dimension (hidden_dim): The size of the hidden layers in the policy network, which defaults to 64; Learning rate (lr): The learner rate, which defaults to 1e-3; Discount factor (gamma): The discount coefficient for cumulative rewards, the default value is 0.99.
[0098] Threshold range (evi_thresh_min and evi_thresh_max): The minimum and maximum values for threshold adjustment, with a default value of [0.01, 0.99].
[0099] The system first receives input data: document attention matrix (doc_attn), sentence position list (sent_pos), relation count list (batch_rel), and offset.
[0100] The evidence predictor calls the `forward_evi` method to calculate the attention score for each sentence. It slices the attention matrix according to sentence position, sums the attention values within each sentence, and obtains the sentence-level attention score (s_attn). The sentence attentions from different documents are concatenated into a tensor `s_attn` with the shape (number of documents × maximum number of sentences).
[0101] The threshold agent extracts state features (extract_state_features method): average attention (features[0]): the mean of attention for all sentences in the current document, attention standard deviation (features[1]): the dispersion of attention distribution, maximum attention value (features[2]): the highest attention score in the current document, current threshold (features[3]): the threshold currently set by the agent, document length normalization (features[4]): the normalized value of the total number of sentences in the document divided by 100. These 5 features together constitute the state vector, which serves as the input to the policy network.
[0102] Next, action selection and dynamic threshold adjustment are performed. The policy network (a 3-layer fully connected network) receives the state vector and outputs the probability distributions of three actions (increasing the threshold, decreasing the threshold, and maintaining the threshold). The agent selects an action through random sampling (using the `select_action` method) and saves the log probability of that action (for subsequent gradient calculation).
[0103] Update the threshold based on the action (update_threshold method): Action 0 (Increase threshold): Increase the current threshold by a fixed step (0.01). Action 1 (Reduce threshold): Reduce the current threshold by a fixed step (0.01). Action 2 (Keep the threshold): The threshold remains unchanged; The updated threshold will be cropped to the preset range [0.01, 0.99] to ensure the value is reasonable.
[0104] The updated threshold is used to binarize the attention of sentences in the current document: sentences with attention scores above the threshold are marked as evidence (evidence_mask).
[0105] The agent calculates the reward (compute_reward method), and the reward function consists of three parts: Basic reward: the average attention score of the evidence sentences (encouraging the selection of high-attention sentences); Quantity penalty: Based on the absolute difference between the number of evidences and the ideal number (1 / 5 of the total number of sentences in the document), multiply by a penalty coefficient of -0.1 (to avoid too much or too little evidence). Sparsity reward: The difference between the highest and lowest attention levels is multiplied by 0.05 (encourages dispersed attention distribution); The total reward is the sum of the three and is used to guide strategy learning.
[0106] After a batch is processed, the agent updates the network using the policy gradient method (update_policy method): Calculate the cumulative discount reward: Iterate backwards from the final reward and calculate the cumulative discount reward for each time step; Standardized rewards: The cumulative rewards are standardized by mean and variance to reduce training variability; Calculate the loss function: The loss includes policy gradient loss (negative log probability multiplied by reward) and entropy regularization term (encourages exploration, coefficient 0.01). Gradient clipping and optimization: Update parameters using the Adam optimizer, with a gradient clipping limit of 0.5 (to prevent gradient explosion); The system iteratively processes each document batch during training, dynamically adjusting the threshold and optimizing the strategy. Finally, it outputs the evidence prediction result (binary mask) and the current threshold. This method significantly improves the accuracy and robustness of evidence selection through a reinforcement learning adaptive threshold mechanism.
[0107] Example 4 In another preferred embodiment, based on embodiments 1 to 3, this embodiment provides an evidence-aware document-level relation extraction method based on dynamic semantics and reinforcement learning. The method is designed with experimental environment, data configuration, module parameters, and operation steps in mind to ensure reproducibility and practical applicability. The specific steps are as follows: 1. Experimental Environment Configuration To ensure efficient model training and inference, the following hardware and software environment is used: Hardware environment: CPU is Intel Xeon Gold 6348, GPU is NVIDIA RTX 3090 (24GB VRAM), RAM is 64GB, storage is 1TB SSD; Software environment: Operating system Ubuntu 20.04 LTS, deep learning framework PyTorch 1.10.0, Python 3.8, Transformers library 4.12.5, CUDA 11.3, CuDNN 8.2.1; Pre-trained model: RoBERTa-large was selected as the base encoder (768 hidden layer dimensions, 12 attention heads, and 24 encoder layers) to meet the needs of long text semantic modeling.
[0108] 2. Implementation steps and methods Step 1: Data Processing Implementation Step 1.1: Dataset Selection and Partitioning The DocRED public dataset was used as the experimental data, with a total sample size of N=10662 records (containing 181 relation types), which were divided into 7:3 groups: (1) Training set: 8000 documents, containing 68234 entity pairs and 24512 relational facts; (2) Validation set: 2,662 documents, including 18,356 entity pairs and 6,548 relational facts.
[0109] Step 2.2, Data Format Conversion Operation Step 2.2.1, Entity Tagging: Iterate through the vertexSet field of each document, record the sentence_id (sentence number) and pos (character position range) of the entity mention, and insert entity tags into the text (e.g., wrapping header entities with [ENT-H] and [ / ENT-H], and wrapping footer entities with [ENT-T] and [ / ENT-T]). Example: Original text "Apple Inc. is headquartered in California and was founded by Steve Jobs" → after the tag "[ENT-H]Apple Inc.[ / ENT-H] is headquartered in California and [ENT-T]Steve Jobs[ / ENT-T] was founded"; Step 2.2.2, Relationship and Evidence Labeling: Traverse the labels field to extract the head entity ID (h), tail entity ID (t), relationship type (r), and evidence sentence list. Map the relationship type to category ID using the doscript_rel2id dictionary (e.g., "Founder" is mapped to ID=12). Construct a multi-label vector (e.g., if the relationship contains "Founder", the 12th bit of the vector is 1, and the rest are 0). At the same time, generate weight labels for the evidence sentences (evidence sentence weight = 1, non-evidence sentence weight = 0). Step 2.2.3, Tokenization and Truncation: Use RoBERTa's built-in tokenizer to convert the sentence into a sequence of token IDs, add the special symbols [CLS] (sequence start) and [SEP] (sequence separator), and set: (7) The extremely long sequence is truncated to 1024 tokens. If it is insufficient, it is padded with [PAD] to finally generate the model input features (including input_ids, entity_pos, labels, hts, sent_pos).
[0110] Step 2: Implementation of Long Text Encoding Taking a long document containing 1200 tokens as an example, the process of segmentation and encoding is illustrated: Step 2.1, Long Text Segmentation Rules Because RoBERTa's maximum number of tokens that can be processed in a single run is 512, and its input format requires a start token of 1 [CLS] (i.e., len_start=1) and an end token of 2 [SEP] (i.e., len_end=2), the maximum number of usable raw tokens per segment is 512 - len_start - len_end = 509. To ensure that the model can process the entire text and preserve contextual information, an overlapping segmentation strategy is adopted to divide the long text into multiple independently coded segments. Therefore, the segmentation strategy is as follows: First paragraph: Take the original text range from position 0 to 508, model input: [CLS]+Tokens(0~508)+[SEP]+[SEP], so the total length is: 1+509+2=512.
[0111] Second segment: Take the starting position as 1×509-100=409, so the ending position is 2×509-1=1017. Model input: [CLS]+Tokens(409~1017)+[SEP]+[SEP], so the total length is 1+(1017-409+1)+2=512, which overlaps with the first segment: Tokens(409~508), for a total of 100 tokens.
[0112] Third paragraph: Starting position: 2×509-100=918, so ending position: original text ending position 1199 (because 3×509-1=1526 exceeds the original text length), model input: [CLS]+Tokens(918~1199)+[SEP]+[SEP], so the total length: 1+(1199-918+1)+2=285, overlapping with the second paragraph: Tokens(918~1017), a total of 100 tokens.
[0113] Using this strategy, we successfully segmented a 1200-Token document into three segments that met the model input length limit, and effectively maintained the semantic coherence of the context through overlapping regions.
[0114] Step 2.2, Encoding and Merging Step 2.2.1, Sub-segment Encoding: Input the two sub-segments into the RoBERTa model respectively, and obtain the hidden state (output1, output2, both with dimensions (1, 513, 768)) and attention weights (att1, att2, both with dimensions (1, 12, 513, 513)) for each segment. Step 2.2.2, Overlapping Region Fusion: Hidden state fusion: Calculated according to the document formula, mask1 and mask2 are attention masks for two segments (non-PAD positions are 1, otherwise 0): (2) Eliminate errors caused by repeated calculations in overlapping areas; Attention weight fusion: (3) Standardization maintains consistency in attention distribution; Step 2.2.3, Padding Alignment: Use the F.pad function to adjust the merged output to the original document length ((1, 1200, 768) hidden state, (1, 12, 1200, 1200) attention weight), and complete the long text encoding.
[0115] Step 3: Implementation of the get_hrt module Taking a document in the training set (containing 5 entities and 10 candidate entity pairs) as an example, the module operation is explained as follows: Step 3.1, Entity Embedding Computation Step 3.1.1, for Entity 1 (containing 3 token spans: Token 10-12, 25-27, 40-42): (1) Calculate the exponentially average embedding: (8) (2) Calculate the mean of attention weights: (9) Step 3.1.2, for Entity 2 (single token span: Token 50-50): (1) Direct extraction of embeddings: (10) (2) Directly extract attention weights: (11) Step 3.2: Joint attention to entity pairs and context vectors Taking entity pair (entity 1 - entity 2) as an example: Step 3.2.1, Joint Attention Calculation: (12) After multiplying element by element, take the mean along the attention head dimension (1, 1200), and then use formula (13): (13) Normalize to ensure the sum of weights is 1; Step 3.2.2, Context Vector Calculation: (14) That is, by using joint attention weighted summation sequence embedding, we can capture the relevant contextual information of entity pairs.
[0116] Step 3.3, Output Results After processing this batch (containing 8 documents), the output is: Head entity vector hs: Dimensions (80, 768) (8 documents × 10 entity pairs); Tail entity vector ts: dimension (80, 768); Entity-to-context vector rs: dimension (80, 768); Joint attention distribution of entity pairs ht_atts: dimension (80, 1200); The number of entity pairs per batch (rels_per_batch) is [10, 8, 12, 9, 7, 11, 8, 15] (the number of entity pairs in each of the 8 documents).
[0117] Step 5: Implementation of the Dynamic Semantic Enhancement Module (DynaSpan) Step 5.1, Module Parameter Settings new_tokens_size=16 (number of learnable tokens), input_dim=768 (consistent with entity embedding dimension); TokenLearner: 2-layer MLP (input 768 → hidden layer 64 → output 1, activation function ReLU); Transformer encoder: 2 layers, 8 attention heads, 2048 feedforward hidden layers, dropout=0.1, and position encoding uses sine and cosine encoding.
[0118] Step 5.2, Feature Enhancement Operation Taking a small batch in hs (step size 256, taking the first 256 entity pairs features) as an example: Step 5.2.1, Initialize the learnable token: (15) use initialization; Step 5.2.2, Sequence assembly: (16) (17) Step 5.2.3: Token Learner feature compression, generating selection weights via MLP: (18) Then, feature compression from 18 to 1 is achieved using formula (19): (19) Step 5.2.4, Transformer Encoding: Input the compressed feat (dimension (256, 1, 768)) into the Transformer encoder, and output the enhanced feature enhanced_feat (dimension (256, 1, 768)). After splitting, we get the head entity enhancement feature hs2 (dimension (256, 768)) and the tail entity enhancement feature ts2 (dimension (256, 768)).
[0119] Steps 6 and 7: Implementation of Feature Fusion and Relationship Classification Step 6: Feature Fusion Step 6.1, Weighted Average: (20) (twenty one) Each retains 50% of the original features and 50% of the enhanced features, with dimensions of (80, 768). Step 6.2, Feature Reshaping: Set block_size=16,
[0120]
[0121] Step 6.3, Bilinear Interaction: (twenty two) After expanding the dimensions, multiply element-wise, resulting in dimensions (80, 48, 16, 16), and then reshape it into... A two-dimensional tensor.
[0122] Step 7: Relationship Classification Step 7.1, Bilinear Classifier: Input The relation prediction logits (dimension (80, 96)) are output through a fully connected layer (12288→96, where 96 is the number of DocRED relation types). Step 7.2, Loss Calculation: Multi-label cross-entropy loss is used. (twenty three) in, The true relation label vector for entity pairs (dimension (80, 96)).
[0123] Step 8: Implementation of Reinforcement Learning Evidence Retrieval Step 8.1: Policy Gradient Threshold Agent Parameter Settings state_dim=5 (state feature dimension), hidden_dim=64 (policy network hidden layer), lr=1e-3 (Adam optimizer learning rate), gamma=0.99 (discount factor), evi_thresh_min=0.01, evi_thresh_max=0.99 (threshold range).
[0124] Step 8.2, Evidence Prediction and Threshold Optimization (Taking a document as an example) Step 8.2.1 Input data: 30 sentences in the document, sent_pos records the token range of each sentence, doc_attn is the joint attention of the entity pair in ht_atts (dimension (1, 1200)), batch_rel=10 (number of entity pairs in the document); Step 8.2.2, Sentence Attention Calculation: Slice doc_attn by sentence_pos, sum the token attention for each sentence to obtain s_attn (dimension (1, 30), such as [0.02, 0.05, 0.8, 0.03, ..., 0.01]). Step 8.2.3, State Feature Extraction: The initial threshold is 0.5, so features = [0.08, 0.15, 0.8, 0.5, 0.3]. Step 8.2.4, Action Selection and Threshold Update: The policy network outputs action probabilities [0.3, 0.2, 0.5] (increase, decrease, keep), and randomly samples to select the "keep" action, with the threshold remaining at 0.5; Step 8.2.5, Evidence Screening: Sentences with an attention value greater than 0.5 (such as sentence 3) are selected and marked as evidence. Step 8.2.6, Reward Calculation:
[0125] Ideal quantity = 30 × 1 / 5 = 6 ; ; Total Rewards ; Step 8.2.7, Strategy Update: Calculate the cumulative discount reward (batch reward traversed in reverse according to gamma=0.99), standardize it, multiply it by the negative log probability of the action, add an entropy regularization term with a coefficient of 0.01, update the network parameters through the Adam optimizer, and set the gradient clipping upper limit to 0.5.
[0126] 3. Verification of experimental results Table 1 shows the performance comparison between the method in this embodiment and the traditional method (RoBERTa + static threshold 0.3) on the DocRED validation set: Table 1. Performance comparison between the method in this embodiment and the traditional method.
[0127] The results show that the proposed method significantly improves the performance of relation extraction and evidence retrieval through dynamic semantic enhancement and reinforcement learning threshold optimization, especially in long document processing, thus verifying the effectiveness of the technical solution.
[0128] This embodiment is only an exemplary implementation. In actual applications, parameters (such as new_tokens_size, block_size, learning rate, etc.) can be adjusted according to the size of the dataset and hardware resources.
[0129] In the preferred embodiment, step 1 of processing DocRED data includes: traversing entities and mentions, recording sentence numbers and positions, and inserting entity tokens; traversing relation instances, extracting head and tail entities, relation types, and evidence sentences, mapping relation types to category IDs and constructing label vectors, and generating evidence sentence weight labels; segmenting words to obtain a Token ID sequence, adding [CLS] and [SEP], and truncating to a preset maximum length; the above settings ensure that the information such as entities and relation instances in DocRED data is processed comprehensively and in a standardized manner, providing high-quality and standardized input data for subsequent model training, which helps the model learn data features and patterns more accurately.
[0130] In the preferred embodiment, step 2, long text processing, includes: when the sequence exceeds the processing capacity of the pre-trained Transformer model, overlapping blocks are divided into multiple segments; during merging, the hidden states of the overlapping areas are weighted and the attention weights are standardized to fill the original sequence length; the model is BERT (start and end markers [CLS], [SEP]) or RoBERTa (start and end markers [CLS], [SEP]). These settings effectively improve the efficiency and accuracy of long text processing and reduce information loss. Overlapping blocks and weighted merging better capture local and global text features; using BERT or RoBERTa models fully utilizes their powerful language understanding capabilities, providing high-quality text representations for subsequent tasks.
[0131] In the preferred embodiment, the get_hrt module in step 3 processes the following: when there are multiple token spans for an entity, the exponential average embedding and the mean of the attention weights are calculated; when there is a single span, the corresponding sequence embedding and weights are taken; the attention weights of the first and last entities of the entity pair are multiplied element by element and the mean is taken as the joint attention weight, which is then normalized and weighted to sum the sequence embeddings to obtain the context vector; the above settings can effectively improve the accuracy and efficiency of entity relationship extraction; by processing multiple token spans through exponential average embedding, the complete semantic information of the entity can be fully captured; the calculation of the joint attention weight can accurately focus on the association of entity pairs, and the normalized weighted sum further strengthens the expression of contextual features.
[0132] In the preferred embodiment, the attention tensor processed by the get_hrt module in step 3 includes batch size, number of attention heads, and sequence length dimension. These settings enable the module to handle data of different scales more flexibly. The batch size can be adjusted according to the amount of data, the number of attention heads can affect the model's attention to different features, and the sequence length dimension adapts to input sequences of various lengths, thereby improving model performance and adaptability.
[0133] In the preferred embodiment, the head entity embedding vector, tail entity embedding vector, and relation embedding vector received by the forward_rel module in step 4 all have dimensions corresponding to the number of all entity pairs and the feature embedding dimension in the current batch. This setting ensures that the vector dimensions match and the computation is efficient when processing large-scale entity relation data. By setting the dimensions uniformly, the forward_rel module can process all entity pairs in the current batch in parallel, improving the model inference speed while maintaining the integrity of the feature information.
[0134] In the preferred embodiment, step 5 of the DynaSpan module includes a learnable parameter matrix `new_tokens` and a Feature Enhancement Unit composed of a Token Learner and a Transformer encoder. The DynaSpan module processes the following: initializing `new_tokens` and concatenating the head and tail entity representations of the input; the Token Learner uses a lightweight network to select weights to compress features; the Transformer encoder captures global dependencies of the tokens, performs feature transformation, and adds positional encoding to preserve sequence order. These settings enable the DynaSpan module to effectively extract and enhance the key information of the input features. Through the dynamic selection mechanism of the learnable parameter matrix and the Token Learner, combined with the global dependency capture capability of the Transformer encoder, the richness and accuracy of the feature representation are improved.
[0135] In the preferred embodiment, step 6 includes: refining the features output by DynaSpan using a weighted average and combining the original enhanced features output by forward_rel; reshaping and fusing the features into three-dimensional tensors corresponding to the head and tail entities respectively, and extracting feature interaction patterns in groups according to a preset granularity; the above settings can strengthen the semantic correlation between features, enabling the model to more accurately capture the complex relationships between the head and tail entities. The feature interaction patterns extracted through grouping can effectively filter noise information and improve the accuracy and robustness of entity relationship judgment.
[0136] In the preferred embodiment, the bilinear classifier processing in step 7 includes: expanding the three-dimensional tensor dimension of the head and tail entities and multiplying them element-wise to obtain bilinear interaction features; reshaping them into a two-dimensional tensor input classification layer to generate relation prediction logits for the number of corresponding entity pairs and the total number of relation types; the above settings can effectively capture complex interaction relationships between entities and improve the accuracy of relation prediction; through bilinear interaction features, different relation types can be distinguished more finely; at the same time, this processing method has high computational efficiency and is suitable for relation extraction tasks of large-scale knowledge graphs.
[0137] In the preferred embodiment, step 8 involves a policy gradient threshold agent and a reward function. The policy gradient threshold agent processing includes: the policy network is a multi-layer fully connected network that outputs three threshold action probabilities; updating the thresholds and constraining them within a reasonable range; calculating the cumulative reward gradient during training, adding an entropy regularization term, pruning the gradients, and updating the parameters. These settings effectively improve the learning efficiency and stability of the policy network. Outputting three threshold action probabilities allows for flexible handling of different scenarios, and the reasonable range constraint avoids abnormal decisions. The entropy regularization term increases exploratory power, and gradient pruning prevents over-updating, ensuring the effectiveness of parameter optimization.
[0138] In summary, the evidence-aware document-level relation extraction method based on dynamic semantics and reinforcement learning provided by this invention offers solutions to several key technical problems in the document-level relation extraction task.
[0139] First, addressing the issue of insufficient task interaction, this invention, for the first time, combines a dynamic semantic enhancement module with a reinforcement learning strategy. By dynamically adjusting the evidence judgment threshold, it achieves simultaneous improvement in the performance of relation extraction and evidence discovery, effectively establishing a multi-task collaborative mechanism. Second, to solve the problem of the lack of adaptability of static thresholds, a policy gradient network is introduced for evidence retrieval, dynamically optimizing the threshold and improving the flexibility and accuracy of evidence screening, overcoming the limitations of traditional fixed thresholds.
[0140] In processing long documents, this invention effectively solves the problems of computational efficiency and semantic coherence in the traditional Transformer architecture when processing long documents by using long text segmentation and merging strategies combined with a dynamic semantic enhancement module, thus ensuring the integrity of global structural information.
[0141] Meanwhile, addressing the representational bottlenecks and error propagation issues of existing methods in entity interaction modeling, evidence fusion, and complex relationship reasoning, the DynaSpan dynamic semantic enhancement module is proposed. This module achieves refined modeling of entity interactions through learnable parameter matrices and feature enhancement units, enhancing semantic recognition capabilities and opening new avenues for understanding complex semantic relationships.
[0142] Furthermore, this invention designs a multi-objective optimized reward function, combining basic rewards, quantity penalties, and sparsity rewards to guide policy learning. This unique design is significant in the field of document-level relation extraction. Simultaneously, an end-to-end joint training mechanism is proposed, co-optimizing relation classification and evidence prediction tasks. This enhances the mutual reinforcement of semantic information and reasoning structure, significantly improving the overall performance of the model. The feature compression and global dependency capture of the DynaSpan module, along with the intelligent application of the policy gradient network in evidence retrieval, together constitute an efficient framework for model training, accelerating model convergence and improving the system's performance in practical applications. Through this series of innovative designs, this invention forms a complete and efficient document-level relation extraction technology solution, injecting new vitality into the development of the natural language processing field.< / entity>
Claims
1. An evidence-aware document-level relation extraction method based on dynamic semantics and reinforcement learning, characterized in that, Includes the following steps: Step 1: Process several DocRED datasets to obtain training and validation sets, and convert them into a model input format suitable for relation classification and evidence extraction; Step 2: After segmentation and transformation, the training set consists of overlapping sub-segments, which are then encoded and merged to obtain a token-level sequence representation; Step 3: Input the get_hrt module to obtain the head entity vector, tail entity vector, entity pair context vector, joint attention distribution, and batch entity pair count; Step 4: Input the forward_rel module to concatenate the head and tail entity representations with the context representations respectively; Step 5: Input the DynaSpan dynamic semantic enhancement module to enhance features; Step 6: After weighted average enhancement, the final entity representation is obtained by combining it with the original entity representation; Step 7: Input the bilinear classifier to obtain the relation prediction logits; Step 8: Input the policy gradient reinforcement learning policy network to perform evidence prediction and dynamic threshold adjustment.
2. The evidence-aware document-level relation extraction method based on dynamic semantics and reinforcement learning according to claim 1, characterized in that, Step 1, which processes DocRED data, includes: traversing entities and mentions, recording sentence numbers and positions, and inserting entity tags; traversing relation instances, extracting head and tail entities, relation types, and evidence sentences, mapping relation types to category IDs and constructing tag vectors, and generating evidence sentence weight tags; segmenting words to obtain Token ID sequences, adding [CLS] and [SEP], and truncating to a preset maximum length.
3. The evidence-aware document-level relation extraction method based on dynamic semantics and reinforcement learning according to claim 1, characterized in that, Step 2, long text processing, includes: when the sequence exceeds the processing capacity of the pre-trained Transformer model, overlapping blocks are divided into multiple segments; during merging, the hidden states of the overlapping area are weighted and the attention weights are standardized to fill the original sequence length; the model is BERT or RoBERTa.
4. The evidence-aware document-level relation extraction method based on dynamic semantics and reinforcement learning according to claim 1, characterized in that, In step 3, the get_hrt module processes the following: when there are multiple token spans for an entity, the exponential average embedding and the mean of the attention weights are calculated; when there is a single span, the corresponding sequence embedding and weights are taken; the attention weights of the first and last entities of the entity pair are multiplied element by element and the mean is taken as the joint attention weight; after normalization, the sequence embeddings are weighted and summed to obtain the context vector.
5. The evidence-aware document-level relation extraction method based on dynamic semantics and reinforcement learning according to claim 1, characterized in that: In step 3, the attention tensor processed by the get_hrt module includes batch size, number of attention heads, and sequence length dimension; in step 4, the forward_rel module receives head entity embedding vector, tail entity embedding vector, and relation embedding vector, the dimensions of which correspond to the number of all entity pairs in the current batch and the feature embedding dimension.
6. The evidence-aware document-level relation extraction method based on dynamic semantics and reinforcement learning according to claim 1, characterized in that, In step 5, the DynaSpan module includes a learnable parameter matrix new_tokens and a Feature Enhancement Unit composed of a Token Learner and a Transformer encoder. The DynaSpan module processes the following: initializing new_tokens and concatenating the head and tail entity representations of the input; the Token Learner generates weighted features through a lightweight network; and the Transformer encoder captures global token dependencies, performs feature transformations, and adds positional encoding to preserve the sequence order.
7. The evidence-aware document-level relation extraction method based on dynamic semantics and reinforcement learning according to claim 1, characterized in that, Step 6 includes: weighted averaging of refined features output by DynaSpan and original enhanced features output by forward_rel; reshaping and fusing the features into three-dimensional tensors corresponding to the head and tail entities respectively, and extracting feature interaction modes by grouping according to preset granularity.
8. The evidence-aware document-level relation extraction method based on dynamic semantics and reinforcement learning according to claim 1, characterized in that, The bilinear classifier processing in step 7 includes: expanding the three-dimensional tensor dimension of the head and tail entities and multiplying them element by element to obtain bilinear interaction features; reshaping them into a two-dimensional tensor input classification layer to generate relation prediction logits for the number of corresponding entity pairs and the total number of relation types.
9. The evidence-aware document-level relation extraction method based on dynamic semantics and reinforcement learning according to claim 1, characterized in that, In step 8, the policy network includes a Policy Gradient Threshold Agent and a reward function. The Policy Gradient Threshold Agent processing includes: the policy network is a multi-layer fully connected network that outputs three threshold action probabilities; updating the thresholds and constraining them within a reasonable range; calculating the discount cumulative reward gradient during training, adding entropy regularization terms, pruning gradients, and updating parameters.
10. The evidence-aware document-level relation extraction method based on dynamic semantics and reinforcement learning according to claim 9, characterized in that, The reward function is designed as follows: (1); In the formula, For the total reward, To predict the average attention score of the evidence, To penalize terms where the predicted amount of evidence deviates from the ideal amount, This is a term to encourage the variance of attention distribution.
