A document-level relation extraction method based on dual-encoder fusion and bidirectional state modeling

Abstract

This invention discloses a document-level relation extraction method based on adaptive dual-encoder fusion and bidirectional state space modeling, belonging to the field of natural language processing and information extraction technology. It aims to address the problems of insufficient cross-sentence dependency modeling, inadequate semantic fusion flexibility, and lack of inference consistency in existing methods. The method includes: encoding the document using two pre-trained encoders with complementary attention patterns; dynamically adjusting the weights of the two encoders through a probability fusion mechanism based on Dirichlet distribution to achieve adaptive fusion of local semantics and long-range information; utilizing a Mamba-based bidirectional state flow module to perform forward and backward state recursion on the fused representation to enhance cross-sentence dependency modeling; and introducing a symmetric consistency loss to constrain the probability distribution difference between forward and backward predictions to improve inference consistency. This method can more fully capture long-range dependencies and cross-sentence entity interactions in the document, improving document-level relation extraction performance.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing and information extraction technology, specifically involving a document-level relation extraction method based on adaptive dual encoder fusion and bidirectional state space modeling. Background Technology

[0002] Relation extraction is a fundamental task in the field of information extraction, aiming to identify semantic relationships between entities in unstructured text. It plays a crucial supporting role in downstream applications such as knowledge graph construction and intelligent question answering. Early research focused primarily on sentence-level relation extraction, but over 40.7% of relational facts in real text require cross-sentence or even full-text context to be valid, thus giving rise to the research direction of document-level relation extraction. Around this direction, researchers have successively proposed sequence models based on Long Short-Term Memory (LSTM) networks and convolutional neural networks, pre-trained language models represented by BERT and Longformer, graph neural network methods for explicitly constructing entity interaction graphs, and hybrid reasoning paradigms combining first-order logic rules with neural networks. The research focus has also gradually shifted from static context encoding to more adaptive semantic interaction modeling such as gating mechanisms and relation awareness.

[0003] Despite the aforementioned methods continuously improving the performance of document-level relation extraction, multiple bottlenecks remain in long documents with complex semantics. Most methods rely on a single encoder structure, limited by the secondary complexity of dense self-attention and a finite context window, making it difficult to simultaneously consider fine-grained local semantics and long-range document-level dependencies. Existing multi-encoder fusion techniques mostly remain at the level of static concatenation or fixed weighting, unable to dynamically adjust the contributions of information from different sources based on the document context. Meanwhile, semantic propagation often follows a pre-defined unidirectional path, making it difficult to achieve bidirectional information flow and global verification across the entire document. Effective constraints on the sensitivity to the order of head and tail entities and the consistency of bidirectional predictions have not yet been established; directional bias causes the model to produce inconsistent relation judgments for the same entity pair under different orders. Therefore, there is an urgent need for a document-level relation extraction method that can balance local semantics and global long-range dependencies, support bidirectional information propagation, and impose consistency constraints on forward and reverse predictions. Summary of the Invention

[0004] Objective: This invention addresses the problems of insufficient cross-sentence dependency modeling, inadequate semantic fusion flexibility, and lack of reasoning consistency in document-level relation extraction. It proposes a document-level relation extraction method based on adaptive dual-encoder fusion and bidirectional state space modeling. This method achieves dynamic fusion of local semantics and long-range information through an adaptive dual-encoder module, realizes bidirectional information propagation and long-range dependency modeling across sentences through a bidirectional state flow module, and enhances the consistency and stability of reasoning through symmetric consistency constraints.

[0005] 1. A document-level relation extraction method based on adaptive dual encoder fusion and bidirectional state space modeling, wherein the method utilizes an adaptive dual encoder module, a bidirectional state flow module, and a symmetric consistency constraint module to perform relation extraction processing on entity pairs in a document, characterized by the following steps: (1) Take a document containing multiple named entities as input, the document being represented as ,in The total number of terms in the document; the document is input into two pre-trained encoders with complementary attention patterns for encoding. The first encoder uses a dense self-attention mechanism to obtain local semantic representation, and the second encoder uses a sparse global-local attention mechanism to obtain long-range semantic representation. (2) Adaptively fuse the two sets of sequence representations using the adaptive dual encoder module: concatenate the [CLS] vector representations of the first encoder and the second encoder, input a linear layer and a Softplus activation function to obtain the concentration parameter vector of the Dirichlet distribution; construct the Dirichlet distribution based on the concentration parameter vector, sample the weight vector from the distribution during the training phase, and use the expected value instead of sampling during the inference phase; perform a weighted summation of the weight vector and the sequence representations of the two encoders to obtain the fused representation, and then refine it through a single-layer Transformer encoder to obtain the refined representation; (3) The refined representation is modeled in a bidirectional state space using the bidirectional state flow module: the forward sequence segment and the backward sequence segment of the sequence are processed by the forward state space transformation operator and the backward state space transformation operator respectively to obtain the forward state representation and the backward state representation. After alignment, they are added element by element to obtain the bidirectional aggregated representation. The refined representation is fused with a learnable residual scaling parameter to obtain the enhanced context representation. In the recursive process of the state space transformation, a bounded pruning operation is applied to the state recursion coefficient vector, and the input feature sequence is stabilized and weighted by a stabilizing segmented accumulation operator to maintain the numerical stability of long sequence modeling. (4) Based on the enhanced context representation, construct two relation prediction branches, forward and reverse, using the symmetric consistency constraint module: for each entity pair, input the head entity and tail entity representation into a shared bilinear classifier to obtain the logical value of the forward prediction, and obtain the logical value of the reverse prediction by swapping the input positions of the head and tail entities, and then obtain the probability distribution of the forward and reverse predictions by the softmax function respectively; constrain the difference between the two prediction probability distributions by the symmetric consistency loss function, and perform weighted joint training with the main task loss function to obtain the document-level relation extraction model; (5) Input the document to be extracted into the document-level relation extraction model, and encode the representation through the adaptive dual encoder module and the bidirectional state flow module in sequence. Then, use the relation prediction branch to determine the relation type of each entity pair in the document and output the relation extraction result.

[0006] 2. The document-level relation extraction method according to claim 1, characterized in that, The first encoder is a BERT pre-trained language model, and the second encoder is a Longformer pre-trained language model. The two encoders share the same word-level input, ensuring that the two outputs are aligned at word positions. In the adaptive dual encoder module, the concentration parameter vector of the Dirichlet distribution... The specific calculations are as follows: in, and These are the [CLS] vector representations of the first encoder and the second encoder, respectively. Let b be the learnable weight matrix of the linear layer, and b be the bias term; let the weight vector be... Sequence representation with two encoders and The fusion obtained by weighted summation is represented as: The single-layer Transformer encoder is configured with 8 attention heads, a feedforward dimension of 2048, and a dropout rate of 0.1.

[0007] 3. The document-level relation extraction method according to claim 1, characterized in that, The bidirectional state space transformation operator is implemented based on the Mamba state space model. The forward state space transformation operator performs forward state recursion on the segment of the sequence from the 1st time step to the tth time step, and the backward state space transformation operator performs backward state recursion on the segment of the sequence from the tth time step to the Tth time step. The backward state representation has undergone time alignment processing to ensure that the forward and backward information can be fused at the same time step. In the bidirectional state flow module, the enhanced context representation... The specific calculations are as follows: in, Let the refined representation be the vector at time step t. This is a bidirectional aggregation representation. and These are the forward state representation and the backward state representation, respectively. The parameters are learnable scalar parameters; the bounded coefficient pruning operation and the stabilized piecewise accumulation operator are expressed as follows: ⊙ in, This is the vector of state recursion coefficients. and To crop the top and bottom boundaries, Will Each element is restricted to the interval stated above. For the stabilized piecewise accumulation operator, ⊙ denotes element-wise multiplication. Given the input feature sequence, This is the output feature sequence.

[0008] 4. The document-level relation extraction method according to claim 1, characterized in that, The main task loss function An adaptive threshold loss function is employed, which is suitable for the multi-label classification characteristics of document-level relation extraction. The forward and backward prediction branches share the parameters of the same bilinear classifier. The backward prediction is achieved by swapping the input positions of the head and tail entities in the shared bilinear classifier, while reusing the same context representation. The distinction between the forward and backward prediction branches operates at the entity pair classifier level, which is orthogonal to the bidirectional state space scan in the bidirectional state flow module. In the symmetric consistency constraint module, the specific calculation of the symmetric consistency loss function is as follows: in, and Entity pairs The positive prediction probability distribution and the negative prediction probability distribution, The symmetric consistency loss function is the set of entity pairs in the training batch. The relative preference distribution among candidates for lightweight regularization constraint relations; the objective function of the joint training is: in, This is a balancing coefficient used to control the relative importance between the main task objective and the symmetric consistency constraints.

[0009] 5. The document-level relation extraction method according to claim 1, characterized in that, The balance coefficient The value is 0.1; the learning rate is 2e-5; the training batch size is 4; and the warm-up ratio is 0.06. Beneficial effects

[0010] The model was trained and its performance was validated using three public document-level relation extraction datasets: DWIE, Re-DocRED, and HacRED. These datasets cover English news, Wikipedia, and Chinese domain texts, ensuring the authority of the training data and the generalizability of the method validation.

[0011] To address the practical problems of insufficient cross-sentence dependency modeling in long documents and the difficulty of a single encoder in taking into account both local semantics and long-range context, the method of this invention employs two complementary encoders, dense self-attention and sparse global-local attention, to process the same document in parallel. This method can effectively capture document-level long-range context information while preserving fine-grained local semantics, thereby improving the relation discrimination capability in complex semantic scenarios.

[0012] An adaptive dual encoder fusion module is introduced in the encoding stage. The representations of the two heterogeneous encoders are dynamically weighted by a probability fusion mechanism based on Dirichlet distribution. It can adaptively balance the contribution of local semantics and long-range information according to the document context. Compared with the traditional fixed weight or simple splicing fusion method, it can more flexibly adapt to different document instances and enhance the robustness of representation.

[0013] In the representation refinement stage, a bidirectional state flow module is introduced. Based on the bidirectional Mamba state space model, state recursion is performed on the forward and backward sequence segments, and then aligned and fused. This enables bidirectional dependency modeling of long sequences in linear complexity, thereby enhancing the ability to propagate information across sentences and capture relationships between distant entity pairs. Simultaneously, by constraining the state recursion process through bounded coefficient pruning and stable segmented accumulation strategies, the numerical overflow and information loss problems in long sequence state recursion are effectively mitigated, improving the model's training stability on long documents.

[0014] By applying L2 norm difference constraints to the probability distributions of the forward and reverse relationship prediction branches through the symmetric consistency constraint module as an auxiliary training objective, the directional bias caused by the sensitivity of the head and tail entity order can be suppressed without increasing inference overhead, thereby improving the prediction consistency and stability of the same entity pair under different input orders. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the overall structure of the document-level relation extraction method proposed in this invention. Detailed Implementation

[0016] Example: The document-level relation extraction method based on adaptive dual encoder fusion and bidirectional state space modeling provided by this invention will be described in detail using the DWIE dataset as an example. The specific operation is as follows: 1. Dataset Preparation This embodiment uses the DWIE dataset as the primary validation object. The original DWIE dataset contains 802 English news articles, labeled with 65 relation types. After removing incompletely labeled documents, 799 articles were actually used, divided into 602 training documents, 98 validation documents, and 99 test documents. The method of this invention was also validated on the Re-DocRED dataset (3053 training documents, 96 relation types) and the HacRED Chinese dataset (6231 training documents, 26 relation types).

[0017] 2. Document Preprocessing and Encoding In the data input phase, the document text is first segmented. Taking a document from the DWIE dataset as an example, let's assume that after segmentation, we get... A sequence of 100 words .

[0018] The word sequence is simultaneously input into two pre-trained encoders: the first encoder uses the BERT-base-uncased model, with a hidden layer dimension of [missing information]. The maximum input length is 512 words; the second encoder uses the longformer-base-4096 model, with hidden layer dimensions... The maximum input length is 4096 words. For the input of 450 words in this example, the two encoders output sequence representations of size 450×768. and The two encoders share the same lexical-level input, therefore and Strict alignment at word positions.

[0019] 3. Adaptive Dual Encoder Fusion Extracting the [CLS] vector representations of the two encoders and (All dimensions are 768), after concatenation, a 1536-dimensional vector is obtained. Input a weight matrix Bias Using linear layers and the Softplus activation function, the concentration parameter vector of the Dirichlet distribution is obtained. : A Dirichlet(α) distribution is constructed based on α, and weight vectors are sampled from this distribution during the training phase. (satisfy ); Use expected value during the inference phase Instead of sampling, to ensure stability, the weight vector is weighted and summed with the sequence representations of the two encoders to obtain a fused representation H of size 450×768. fusion representation Input a single-layer Transformer encoder The encoder was refined with 8 attention heads, a feedforward dimension of 2048, and a dropout rate of 0.1. A refined representation of size 450×768 was obtained through residual connections and layer normalization. : 4. Bidirectional state flow modeling The refined representation is 450×768 pixels. Input the forward state space transformation operator implemented based on the Nd-BiMamba2 bidirectional Mamba architecture respectively. and inverse state-space transformation operator The forward operator starts from the time step. The input segments are processed sequentially. The reverse operator starts from the time step. The input segments are processed sequentially. Output the forward state representation respectively. and reverse state representation (All output dimensions are 768): After flipping and aligning the reversed state representation along the time axis, it is added element-wise to the forward state representation to obtain a bidirectional aggregated representation of size 450×768. : Through learnable scalar parameters (Initialized to 0.1) Controls the contribution of the bidirectional state representation, and its relationship with the refined representation. Residual fusion is performed to obtain an enhanced post-context representation with a size of 450×768. : In the Mamba state recursion process, to prevent numerical overflow or information loss due to excessively large or small recursion coefficients in long sequences, the state recursion coefficient vector is modified accordingly. Apply a bounded pruning operation and stabilize the piecewise accumulation operator. For the input feature sequence Perform weighted processing: ⊙ Will Each element is limited to Within the range; A lower triangular mask matrix is ​​constructed for the recursive coefficient sequence, and the summation is performed along the time dimension. The numerical stability of the exponential operation is ensured by subtracting the maximum value of each segment, effectively reducing the risk of numerical overflow in long sequences. ⊙ represents element-wise multiplication.

[0020] 5. Relationship Prediction and Symmetric Consistency Constraints Based on enhanced context representation (Size 450×768), the token representations corresponding to all mention positions of each entity are aggregated into a 768-dimensional entity vector using LogSumExp pooling. For each candidate entity pair... The head and tail entity representations are input to a bilinear classifier that shares the same input, generating logical values ​​for relation prediction. (The DWIE dataset is 66-dimensional, containing 65 relation categories and 1 threshold class); the logistic value for inverse prediction is generated by swapping the input positions of the head and tail entities in the classifier. The classifier parameters are shared for both directions. The probability distributions for forward and backward predictions are obtained by applying the softmax function to the logistic values ​​respectively. The symmetric consistency loss function calculates the average of the squared L2 norm differences between the forward and backward predicted probability distributions of all entity pairs in the training batch: in This is the set of all candidate entity pairs in the current training batch.

[0021] 6. Joint Training The objective function of joint training combines the main task loss and the symmetric consistency loss in a weighted manner. Wherein the main task loss function An adaptive thresholding loss function is employed, suitable for the multi-label classification characteristics of document-level relation extraction; balancing coefficients are used. Set it to 0.1.

[0022] The model was implemented using PyTorch 1.13.1 and trained on an NVIDIA A800 GPU. The optimizer was AdamW, with a learning rate of 2e-5 and a training batch size of 4. A linear warm-up strategy was used for learning rate scheduling, with a warm-up ratio of 0.06. The model was trained for 300 epochs on the DWIE dataset and 100 epochs on the Re-DocRED and HacRED datasets.

[0023] 7. Analysis of Processing Results The method of this invention uses two evaluation metrics, F1 and Ign F1, to quantify the results. F1 comprehensively measures the overall predictive ability of the model, while Ign F1 filters out relation triples that have already appeared in the training set during calculation, thus more objectively evaluating the model's generalization ability.

[0024] The experimental results of the method of this invention on three datasets are as follows: 72.71% Ign F1 and 78.31% F1 on the DWIE test set; 75.22% Ign F1 and 76.19% F1 on the Re-DocRED test set; and 78.77% Ign F1 and 79.15% F1 on the HacRED test set. The experimental results show that the method proposed in this invention achieves superior performance on multiple document-level relation extraction benchmark datasets, demonstrating strong generalization ability and practical value. The document-level relation extraction method based on adaptive dual-encoder fusion and bidirectional state space modeling proposed in this invention exhibits high extraction performance. This is of great significance in the field of document-level relation extraction.

[0025] As described above, although the invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the invention itself. Various changes in form and detail may be made without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A document-level relation extraction method based on adaptive dual-encoder fusion and bidirectional state space modeling, the method uses an adaptive dual-encoder module, a bidirectional state flow module and a symmetric consistency constraint module to perform relation extraction processing on entity pairs in a document, characterized in that, The specific steps are as follows: (1) Take a document containing multiple named entities as input, the document being represented as ,in The total number of terms in the document; the document is input into two pre-trained encoders with complementary attention patterns for encoding. The first encoder uses a dense self-attention mechanism to obtain local semantic representation, and the second encoder uses a sparse global-local attention mechanism to obtain long-range semantic representation. (2) Adaptively fuse the two sets of sequence representations using the adaptive dual encoder module: concatenate the [CLS] vector representations of the first encoder and the second encoder, input a linear layer and a Softplus activation function to obtain the concentration parameter vector of the Dirichlet distribution; construct the Dirichlet distribution based on the concentration parameter vector, sample the weight vector from the distribution during the training phase, and use the expected value instead of sampling during the inference phase; The weight vector is weighted and summed with the sequence representations of the two encoders to obtain a fused representation, which is then refined by a single-layer Transformer encoder to obtain a refined representation. (3) The refined representation is modeled in a bidirectional state space using the bidirectional state flow module: the forward sequence segment and the backward sequence segment of the sequence are processed by the forward state space transformation operator and the backward state space transformation operator respectively to obtain the forward state representation and the backward state representation. After alignment, they are added element by element to obtain the bidirectional aggregated representation; the refined representation is fused with a learnable residual scaling parameter to obtain the enhanced context representation. During the recursive process of the state space transformation, a bounded pruning operation is applied to the state recursion coefficient vector, and a stabilizing piecewise accumulation operator is used to perform stabilizing weighting on the input feature sequence in order to maintain the numerical stability of long sequence modeling. (4) Based on the enhanced context representation, construct two relation prediction branches, forward and reverse, using the symmetric consistency constraint module: for each entity pair, input the head entity and tail entity representation into a shared bilinear classifier to obtain the logical value of the forward prediction, and obtain the logical value of the reverse prediction by swapping the input positions of the head and tail entities, and then obtain the probability distribution of the forward and reverse predictions by the softmax function respectively; constrain the difference between the two prediction probability distributions by the symmetric consistency loss function, and perform weighted joint training with the main task loss function to obtain the document-level relation extraction model; (5) Input the document to be extracted into the document-level relation extraction model, and encode the representation through the adaptive dual encoder module and the bidirectional state flow module in sequence. Then, use the relation prediction branch to determine the relation type of each entity pair in the document and output the relation extraction result.

2. The document-level relation extraction method according to claim 1, characterized in that, The first encoder is a BERT pre-trained language model, and the second encoder is a Longformer pre-trained language model. The two encoders share the same word-level input, ensuring that the two outputs are aligned at word positions. In the adaptive dual encoder module, the concentration parameter vector of the Dirichlet distribution... The specific calculations are as follows: α = S o f t p l u s W h B E R T h L o n g + b in, and These are the [CLS] vector representations of the first encoder and the second encoder, respectively. Let b be the learnable weight matrix of the linear layer, and b be the bias term; let the weight vector be... Sequence representation with two encoders and The fusion obtained by weighted summation is represented as: The single-layer Transformer encoder is configured with 8 attention heads, a feedforward dimension of 2048, and a dropout rate of 0.

1.

3. The document-level relation extraction method according to claim 1, characterized in that, The bidirectional state space transformation operator is implemented based on the Mamba state space model. The forward state space transformation operator performs forward state recursion on the sequence segment from the 1st time step to the tth time step, and the backward state space transformation operator performs backward state recursion on the sequence segment from the tth time step to the Tth time step. The backward state representation has undergone time alignment processing to ensure that forward and backward information can be fused at the same time step. In the bidirectional state flow module, the enhanced context representation... The specific calculations are as follows: in, Let the refined representation be the vector at time step t. This is a bidirectional aggregation representation. and These are the forward state representation and the backward state representation, respectively. The parameters are learnable scalar parameters; the bounded coefficient pruning operation and the stabilized piecewise accumulation operator are expressed as follows: ⊙ in, This is the vector of state recursion coefficients. and To crop the top and bottom boundaries, Will Each element is restricted to the interval stated above. For the stabilized piecewise accumulation operator, ⊙ denotes element-wise multiplication. Given the input feature sequence, This is the output feature sequence.

4. The document-level relation extraction method according to claim 1, characterized in that, The main task loss function An adaptive threshold loss function is employed, which is suitable for the multi-label classification characteristics of document-level relation extraction. The forward and backward prediction branches share the parameters of the same bilinear classifier. The backward prediction is achieved by swapping the input positions of the head and tail entities in the shared bilinear classifier, while reusing the same context representation. The distinction between the forward and backward prediction branches operates at the entity pair classifier level, which is orthogonal to the bidirectional state space scan in the bidirectional state flow module. In the symmetric consistency constraint module, the specific calculation of the symmetric consistency loss function is as follows: in, and Entity pairs The positive prediction probability distribution and the negative prediction probability distribution, The symmetric consistency loss function is the set of entity pairs in the training batch. The relative preference distribution among candidates for lightweight regularization constraint relations; the objective function of the joint training is: in, This is a balancing coefficient used to control the relative importance between the main task objective and the symmetric consistency constraints.

5. The document-level relation extraction method according to claim 1, characterized in that, The balance coefficient The value is 0.1; the learning rate is 2e-5; the training batch size is 4; and the warm-up ratio is 0.06.

Images