A document-level relation extraction method based on multi-task learning and knowledge distillation

Abstract

This invention discloses a document-level relation extraction method based on multi-task learning and knowledge distillation. The method includes: acquiring the document text to be extracted, inputting it into a trained document-level relation extraction model, and outputting the corresponding document-level text relations. The document-level relation extraction model includes a feature extraction module, a teacher model, and a student model. The training process of the model includes: pre-training the teacher model using manually labeled document data through multi-task learning; using the pre-trained teacher model to complete the evidence in document data lacking labeled evidence information; and training the student model using the evidence-completed document data. The trained document-level relation extraction model includes a feature extraction module and a trained student model. This invention can accurately extract contextual semantic information from documents, effectively alleviate data sparsity problems through knowledge distillation, achieve efficient extraction of document-level relations, and significantly improve the model's performance in complex contexts.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing technology, and in particular relates to a document-level relation extraction method based on multi-task learning and knowledge distillation. Background Technology

[0002] With the rapid development of artificial intelligence and natural language processing (NLP), relation extraction (RE) technology has been widely applied in downstream applications such as knowledge graph construction, question answering systems, and digital education. The purpose of relation extraction technology is to identify and extract relationships between entities from unstructured text for subsequent generation and storage of structured data. Currently, most relation extraction technologies have made significant progress in processing sentence-level entity relationship recognition. However, sentence-level relation extraction methods are usually limited to single-sentence relation judgments, and suffer from information loss and insufficient long-distance dependencies in relation recognition across multiple sentences or even entire documents, making it difficult to handle complex relationships in long texts.

[0003] To address the aforementioned issues, document-level relation extraction has become an important research direction in the field of relation extraction. This method utilizes full-text semantics and contextual information to achieve more comprehensive relation identification. Existing document-level relation extraction techniques can be categorized into sequence-based, graph-based, and pre-trained language model-based methods.

[0004] Sequence-based methods extract relationships from the sequential dependencies of textual information, making them suitable for shorter texts. However, they tend to lose semantic information across sentences when processing long texts with a large span.

[0005] Graph-based methods model entity relationships through graph structures and can uncover multi-level relationships, but they introduce high computational complexity in large-scale document construction.

[0006] Methods based on pre-trained language models rely on the contextual understanding capabilities of pre-trained models to effectively capture long-distance document-level dependencies, and have become the mainstream research direction in this field in recent years.

[0007] Chinese invention patent application CN118095278A discloses a document-level relation extraction method based on a pre-trained model, which uses a Transformer-based pre-trained model to capture long-distance dependencies and improve the accuracy of relation recognition. However, this method mainly focuses on single-task training and cannot effectively utilize the semantic information of different tasks for multi-task joint training.

[0008] Chinese invention patent application CN118798369A discloses a document-level relation extraction method based on graph structures, which constructs entity graphs for relation modeling and exhibits good performance in extracting relations from long texts. However, the high complexity of graph models in processing long documents leads to increased computational costs, making it unsuitable for quickly processing large-scale document data.

[0009] Chinese invention patent application CN115688901A discloses a multi-task learning method that applies knowledge distillation technology to text processing. It uses knowledge distillation technology to improve the generalization ability of the model, but the method does not combine document-level relation extraction tasks and cannot effectively solve the problem of long-distance dependencies in documents. Summary of the Invention

[0010] To address the aforementioned technical problems, this invention proposes a document-level relation extraction method based on multi-task learning and knowledge distillation. The method includes: acquiring the document text from which document-level relations to be extracted, inputting it into a trained document-level relation extraction model, and outputting the corresponding document-level text relations.

[0011] The document-level relation extraction model includes a feature extraction module, a teacher model, and a student model. The training process of this document-level relation extraction model includes:

[0012] The teacher model is obtained through multi-task learning pre-training using manually annotated document data;

[0013] Use a pre-trained teacher model to complete the evidence in document data that lacks labeled evidence information;

[0014] Train a student model using the document data with the evidence completed;

[0015] The trained document-level relation extraction model includes a feature extraction module and a trained student model.

[0016] The beneficial effects of this invention are as follows: First, by training the teacher model through multi-task learning, this invention enables semantic information sharing in entity recognition, evidence retrieval, and relation recognition tasks, thereby improving the accurate understanding of entity relationships in complex documents and effectively overcoming the shortcomings of existing models in handling long-distance dependencies. Second, by introducing knowledge distillation, this invention utilizes the teacher model to complete evidence on remote supervision data, alleviating the data sparsity problem. The student model is trained using the evidence-completed data, enabling it to maintain good generalization performance even under imbalanced data conditions. The performance of document-level relation extraction using the trained student model is significantly improved. Attached Figure Description

[0017] Figure 1 This is an overall flowchart of an embodiment of the present invention;

[0018] Figure 2 This is a flowchart of the multi-task learning process of the teacher model in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] This invention proposes a document-level relation extraction method based on multi-task learning and knowledge distillation, referring to... Figure 1 As shown, the method includes:

[0021] Obtain the document text from which document-level relations are to be extracted, input it into the trained document-level relation extraction model, and output the corresponding document-level text relations;

[0022] The document-level relation extraction model includes a feature extraction module, a teacher model, and a student model. The training process of this document-level relation extraction model includes:

[0023] S1: The teacher model is obtained through multi-task learning pre-training using manually labeled document data;

[0024] S2: Use a pre-trained teacher model to complete the evidence in document data that lacks labeled evidence information;

[0025] S3: Train the student model using the document data after evidence completion;

[0026] The trained document-level relation extraction model includes a feature extraction module and a trained student model.

[0027] Figure 1The entire process of this invention embodiment includes two stages: a training stage and a practical application stage. The document-level relation extraction model includes a feature extraction module, a teacher model, and a student model. The feature extraction module includes a BERT network and a Transformer encoder. Unlike traditional knowledge distillation models, this invention embodiment only uses the teacher model to transfer knowledge to the student model without compressing the model. Both the student model and the teacher model adopt a multi-task learning model with consistent structure and size, both including an evidence extraction module, an entity recognition module, and a relation extraction module. First, manually labeled document data is used to pre-train the teacher model through multi-task learning. The pre-trained teacher model is then used to complete the evidence in unlabeled document data (i.e., soft-labeled data, without evidence information, such as the remote supervision dataset: DocRED), resulting in evidence-completed document data (i.e., soft-labeled data). This evidence-completed document data is then used to train the student model. Finally, manually labeled document data, after feature extraction, is input into the trained student model for fine-tuning. In the practical application stage, the trained document-level relation extraction model includes a feature extraction module and the trained student model. Obtain the document text from which document-level relationships are to be extracted, extract features using the feature extraction module, input the extracted features into the trained student model, and output the corresponding document-level text relationships.

[0028] Figure 2 This is a flowchart of the multi-task learning process of the teacher model in an embodiment of the present invention. Figure 2 In this model, a teacher model is trained using manually annotated document data (also known as "golden data") through hard-shared multi-task learning. The specific process includes: inputting the manually annotated document data into BERT for encoding to obtain characters and entity types; inputting these characters and entity types into the Transformer encoder to further extract deeper character representations, generating mention representations based on these character representations, and then generating entity representations based on these mention representations; constructing entity pair representations based on the manually annotated entities to provide input for subsequent modules; inputting the entity representations into the entity recognition module to obtain the probability distribution of entity types and calculate the multi-class cross-entropy loss L. ET To measure the accuracy of identification, entity pair representations are input to the evidence extraction module to obtain the probability of the triplet evidence set for each entity pair, and the focal loss L of the binary cross-entropy is calculated. ER This is used to evaluate the effectiveness of evidence extraction. Entity pairs are input into the relationship recognition module to obtain the relationship probability of the entity pairs, and the classification cross-entropy loss L is calculated. RE The accuracy of relation identification is measured by the multi-class cross-entropy loss L. ET The focal loss L of binary cross-entropy ER And classification cross-entropy loss L REThe overall loss L(T) of the teacher model is obtained by weighted summation, and the teacher model is iteratively optimized by minimizing the overall loss L(T).

[0029] In some preferred embodiments, reference is made to Figure 1 , 2 As shown, both the teacher model and the student model are multi-task learning models, which include a feature extraction module, an entity recognition module, an evidence extraction module, and a relation extraction module.

[0030] In some preferred embodiments, reference is made to Figure 1 , 2 As shown, the feature extraction module includes a BERT network and a Transformer encoder. The BERT network is used to encode document and entity representations of the input document data, and the Transformer encoder is used to perform in-depth feature processing on the preliminary document and entity representations generated by the BERT network to obtain more accurate and richer feature representations.

[0031] Specifically, after preprocessing, the document data is input into the BERT network, which performs initial encoding. Through character-level and sentence-level encoding, BERT provides basic document and entity representations, which capture lexical and syntactic information within the document. The Transformer encoder then performs deeper feature processing based on these initial BERT representations. The Transformer encoder uses a self-attention mechanism to capture contextual and long-distance dependencies at different locations within the document. Multi-head attention layers weight entity information at different locations, thereby integrating contextual information and enhancing the correlation between entities. This allows the model to more comprehensively understand the relationships between entities at the document level, resulting in more accurate and richer feature representations. Specifically, this involves character representations, generating mention representations based on these character representations, generating entity representations based on the mention representations, and finally constructing entity pair representations based on entity annotations.

[0032] In some preferred embodiments, reference is made to Figure 1 , 2 As shown, a teacher model is obtained through multi-task learning pre-training using manually labeled document data. The specific process includes:

[0033] S101: Encode the manually annotated document data using the feature extraction module and extract character representations; generate mention representations based on character representations, generate entity representations based on mention representations; construct entity pair representations based on manually annotated entities.

[0034] Specifically, firstly, the manually annotated document data is preprocessed and then input into the BERT network. The BERT network encodes the text and extracts character-level representations. Next, the character representations are input into the Transformer encoder to further generate deeper contextual representations. Mention representations are generated from the character representations, and entity representations are then generated from the mention representations. Finally, entity pair representations are constructed based on the entity annotations, providing input for subsequent modules.

[0035] S102: Input the entity representation into the entity recognition module, classify each entity, and obtain the probability distribution of the entity type; based on this probability distribution, calculate the multi-class cross-entropy loss L. ET This is used to measure the accuracy of entity recognition results and help the model better distinguish between different types of entities.

[0036] S103: Input entity pair representations into the evidence extraction module, perform probability calculations on the triplet evidence set of entity pairs, extract evidence of relationship existence, and calculate the focal loss L of the binary cross-entropy based on the predicted evidence probability. ER The focal loss L of this binary cross-entropy is used. ER Optimize the model to ensure good performance in evidence extraction tasks. Focus loss L ER It can increase the model's focus on difficult-to-identify samples and minority categories, thereby improving the accuracy of evidence extraction.

[0037] S104: Input the entity pair representation into the relationship recognition module, classify the relationship of each entity pair, obtain the probability distribution of the relationship type, and calculate the classification cross-entropy loss L based on this probability distribution. RE Using the classification cross-entropy loss L RE The model's accuracy in relation identification tasks is optimized. This loss enables the model to accurately distinguish between various relation types between entity pairs.

[0038] S105: The multi-class cross-entropy loss L... ET The focal loss L of binary cross-entropy ER And classification cross-entropy loss L RE The overall loss L(T) of the teacher model is obtained by weighted summation. The teacher model is then iteratively optimized by minimizing the overall loss L(T) to achieve optimal performance in entity recognition, evidence extraction and relation recognition tasks simultaneously.

[0039] In some preferred embodiments, reference is made to Figure 1 , 2 As shown, in S101, the feature extraction module encodes the manually annotated document data, extracts character-level representations, and generates mention representations based on the character representations. Specifically:

[0040] S201: Preprocess the input document data, specifically including: summarizing the beginning and end of each sentence s in the document data d, inserting the classifier token "[CLS]" and the separator "[SEP]" at the beginning and end of each sentence respectively; and marking the entity with an entity tag "*" on each entity mention packet m, where "*" indicates the location of the entity mention.

[0041] Specifically: For each document data d, it is divided into multiple sentences s. A classifier token "[CLS]" is inserted at the beginning of each sentence and a separator "[SEP]" is inserted at the end to clarify sentence boundaries and provide global information. This allows the model to use the "[CLS]" embedding to represent the features of the entire sentence. For each entity mention package m in the document, an entity tag "*" is inserted at the entity location to identify the position of the entity mention. This tag helps the model to more accurately locate entities and pay attention to the context in which the entity is located.

[0042] S202: Input the preprocessed document data into BERT, and after encoding, output the characters and entity types.

[0043] S203: Input the characters and entity types output by BERT into the Transformer encoder. After encoding, entity representations and mention representations are obtained. Each entity representation is obtained by embedding the “*” mark before the entity mention and the “[CLS]” mark at the beginning of the sentence. Each mention representation is obtained by embedding the “*” mark in the sentence and the “[CLS]” mark at the beginning of the sentence.

[0044] Specifically, the characters and entity types output by BERT are input into the Transformer encoder, which extracts deep contextual representations through a self-attention mechanism, generating character-level representations, mention representations, and entity representations. The representation of each entity is obtained by combining the embeddings of the entity's mention marker "*" and the sentence-initial "[CLS]" marker. These embeddings summarize the entity's local information (provided by the "*" marker) and global contextual information (provided by the "[CLS]" marker). The representation of each mention includes the embedding of the mention's position marker "*" in the sentence and the sentence-initial "[CLS]" marker. The resulting mention representations thus include both the mention's local information within the sentence and information about the sentence as a whole.

[0045] Reference Figure 2 As shown, step S102 specifically includes:

[0046] This invention integrates the mention set M of entity e using the LogSumExp pooling function based on entity mention representation. e We obtain the embedded representation of entity e: Here, 'm' represents the embedding representation of the special character "*" at the start of an entity mention. This entity embedding 'e' gathers all mention information about the entity, providing a more comprehensive entity representation for the model. Since entity 'e' can appear as either the head or tail in an entity pair, two separate linear layers are used to distinguish the head entity embedding 'e'. h and tail entity embedded e t The specific formula is as follows:

[0047]

[0048] Among them, e h Indicates head entity embedding, e t Indicates tail entity embedding, These are the weight matrices for linear transformation, used respectively for calculating the embedding of head and tail entities. These are bias parameters, corresponding to the linear transformations of the head and tail entities, respectively. Therefore, input any entity representation e v (where v∈{h,t}, representing the head or tail), calculate the probability of the entity type.

[0049] In some preferred embodiments, the probability of the entity type is specifically calculated using the following formula:

[0050]

[0051] In the formula, W represents the entity type probability output by the entity recognition module, i.e., the probability estimate by which the model identifies an entity as belonging to a different type. c The weight matrix represents the entity type classification, used to map entity representations to the entity type space, where Wc∈R |C|×H R represents the real number space, |C| represents the number of entity types, i.e., the number of possible entity types, and H represents the entity representation e. v The dimension, e v e represents the representation vector of any entity (the representation of the head entity or the tail entity). v This is the context representation of the entity generated by the feature extraction module. `tanh(·)` represents the tanh activation function, used to map the input values ​​to the interval (-1, 1), increasing the model's non-linear expressive power. `δ(·)` represents the softmax function, used to map the output to the probability distribution of the entity type. c The bias term vector represents the entity type classification, used to adjust the bias for different types, and bc∈R. |C| b c ∈R |C| b c The dimension is consistent with the number of categories.

[0052] In entity recognition tasks, due to the imbalance of recognition categories, a multi-class cross-entropy loss function is used to calculate the loss value. The specific calculation formula is as follows:

[0053]

[0054] In the formula, This represents the entity recognition loss value for document d. It is obtained by summing the recognition losses of all entities in the document. This represents the probability that entity e is identified as category c, calculated by the entity recognition module. This is a label indicating that entity e belongs to category c. It is a binary indicator variable; it indicates whether entity e belongs to category c. otherwise e represents an entity in the current document d, E d Let d represent the set of all entities in document d, c represent one type of entity category, and C represent the set of all possible entity categories.

[0055] Reference Figure 2 As shown, step S103 specifically includes:

[0056] In this invention, an evidence retrieval task guides the attention computation in the Transformer's encoder to identify triples (e) in document d. h ,e t The relevant set of important evidence. Where V h,t,r Indicates the relationship with triples (e h ,e t A collection of sentences related to ,r).

[0057] To obtain better contextual semantics, character-level attention A is calculated using the equation H,A = Encoder(d), and a unique contextual embedding representation c containing the head and tail entities is computed. h,t The specific formula is as follows:

[0058]

[0059] Among them, c h,t This represents the context embedding representation containing the head and tail entities, ⊙ is the element-wise product, and A h This indicates all e in the entire document identified by the entity tag "*". h The average level of attention mentioned, A t All e in the entire document identified by the entity tag "*" tThe average attention weights mentioned are: H represents the document feature representation matrix, where each column corresponds to the embedding representation of a character, generated by the Transformer encoder; and T represents the matrix transpose operation, which transforms the column vectors of matrix H into row vectors.

[0060] Entity pair (e) h ,e t ) context embedding c h,t The collection of sentences in the document S d Input a set of bilinear layers, by c h,t Predict the sentence s∈S d It represents the probability of the entity against the set of triplet evidence.

[0061] In some preferred embodiments, the probability of the triplet evidence set of the entity pair is specifically represented as follows:

[0062]

[0063] in, This represents the probability of the triple evidence set for entity pairs, i.e., the probability that the model determines whether there is evidence for the entity pair (h,t,s). It is used in the evidence extraction module. h represents the head entity, which is the subject entity of the relation in the triple; t represents the tail entity, which is the object entity of the relation in the triple; s represents the statement or evidence sentence of the triple, i.e., the specific sentence that may contain the relation between h and t; σ(·) represents the sigmoid function, which maps the input values ​​to the range (0,1) and is used to output the probability value. Represents entity pairs (e) h ,e t The transpose of the context embedding at position k contains the specific location context information of the entity pair in the document. Let represent the weight matrix at position k, used for the bilinear transformation between entity features and sentence features, and b p This represents the bias term, and b p ∈R, s k This represents the feature representation of sentence s at position k.

[0064] The DocRED dataset also faces class imbalance issues in its evidence set. 97.1% of entity pairs lack valid relationships or supporting evidence, resulting in a severe imbalance between positive examples (supporting evidence for entity pairs) and negative examples (non-supporting evidence). DistilRE addresses this imbalance by using Focal Loss, which reduces the contribution of easily classified samples to the gradient, allowing the model to focus more on difficult-to-classify samples.

[0065] The focus loss function of the binary cross-entropy Specifically:

[0066]

[0067] in, Represents entity pairs (e) h ,e t Is the sentence s a positive example? If s∈v h,t (that is, sentence s belongs to entity pair (e)) h ,e t If the set of supporting evidence is ), then otherwise The model predicts that sentence s belongs to entity pair (e). h ,e t The probability of the set of supporting evidence, γ ER γ is a hyperparameter of the focus loss, controlling the degree of gradient descent for easily classifiable samples. A larger γ... ER The value makes the model pay more attention to samples that are difficult to classify. For class weights, the weights of positive and negative examples are... The frequency is inversely proportional to the value. This is used to balance class imbalances by assigning higher weights to rarer samples. h ∈ε d Represents the head entity e h The set of entities ε belonging to document d d e t ∈ε d Represents tail entity e t The set of entities ε belonging to document d d , s∈S d S represents the set of sentences S that belong to document d. d 。 ,v h,t Represents entity pairs (e) h ,e t The set of supporting evidence.

[0068] Reference Figure 2 As shown, S104 specifically includes:

[0069] For any entity pair (e h ,e t ), c h,t The encoding in the middle contains the relevant context of the entity pair, while e h and e t The entity type information is preserved within the layer. Therefore, it is obtained from c through two separate linear layer networks. h,t The contextual features required to extract the head and tail entities:

[0070]

[0071] In the formula, c′ h c′ represents the contextual feature representation of the head entity. t The contextual features representing the tail entity The weight matrix representing the contextual features of the head entity. The weight matrix representing the contextual features of the tail entity. Bias terms representing the contextual features of the head entity. This represents a bias term indicating the contextual features of the tail entity. c′ h Embedding c from the context of entity pairs through linear transformation h,t The extracted head entity features are used to capture the contextual information of the entity pair when it is used as a head entity. t Embedding c from the context of entity pairs through linear transformation h,t The extracted tail entity features are used to capture the contextual information of the entity pair when it is a tail entity. Used to extract contextual features c of entity pairs h,t Mapped to the head entity feature space, its dimension is H×H. Used to extract contextual features c of entity pairs h,t Mapped to the tail entity feature space, its dimension is also H×H. It is used to translate and adjust the contextual features of the head entity in a linear transformation, and its dimension is H. It is used to translate and adjust the contextual features of the tail entity in a linear transformation, and its dimension is H.

[0072] Represent the head and tail entities with contextual information, e h With type information, the head and tail entity representation e t Combined to generate new head and tail entity representations:

[0073] e″ h =tanh(e h +c′ h )

[0074] e″ t =tanh(e t +c′ t )

[0075] In the formula, e″ h This represents a new header entity representation, which incorporates the type information e of the header entity. h and context information c′ h The generated features. Through this combination, the new head entity representation e″ h It also preserves the type information and contextual features of the head entity, enabling the representation to more accurately reflect the entity's role in a specific relationship.t This represents a new tail entity representation, which incorporates the type information e of the tail entity. t and context information c′ t The generated features. The new tail entity representation is e″. t It also includes both the type information and contextual feature information of the tail entity. h This represents the header entity embedding, which includes type information representing the inherent attributes and category information of the header entity. This embedding is generated by the model during the encoding process based on the header entity type information. t This represents the tail entity embedding, which includes type information representing the inherent attributes and category information of the tail entity. This embedding is generated by the model during the encoding process based on the tail entity type information. h The contextual feature representation of the head entity is obtained by linear transformation from the context embedding c. h,t Extracted from c′, used to capture the contextual information of the head entity under a specific relationship. t The contextual feature representation of the tail entity is obtained by linear transformation from the context embedding c. h,t Extracting contextual information of the tail entity under specific relations is used to capture this information. `tanh(·)` represents the tanh activation function, used to perform a non-linear transformation on the result of adding two embeddings, mapping the result to the interval (-1, 1). The introduction of the tanh function increases the non-linear expressive power of the model, making the newly generated head and tail entity representations `e″` more flexible. h and e″ t It has greater discernment.

[0076] The new head entity is represented as e″ h The new tail entity is represented by e″ t Input a bilinear layer network to compute a new head-and-tail entity representation e″ h 、e″ t The probability r of the relationship between them.

[0077] In some preferred embodiments, the probability of the relationship between the entity pairs is specifically represented as follows:

[0078]

[0079] In the formula, This represents the probability of the relationship between entity pairs (h,t). This is the probability value predicted by the model that a specific relationship r exists between the head entity h and the tail entity t. h represents the head entity, i.e., the subject entity in the entity pair, t represents the tail entity, i.e., the object entity in the entity pair, and r represents the relationship type, i.e., the possible relationship categories between the head and tail entities. This represents the transpose of the context-type fusion feature representation of the head entity h at the k-th position. This includes the type information and contextual characteristics of the entity as a header entity. This represents the relation weight matrix at level k, used to calculate the relationship probability between the head entity and the tail entity. The dimension is R H / K×H / K H represents the total dimension of the hidden layer features, and K represents the number of layers in the relation prediction, i.e., the number of spatial dimensions into which the features are divided. This represents the context-type fusion feature representation of the new tail entity t at the k-th position. Including the type information and contextual characteristics of the entity as a tail entity, b r This represents the bias term in relation prediction, used to linearly adjust the output in relation prediction, and belongs to the real number space R.

[0080] Since only some entity pairs in the dataset have valid relationships, and multiple relationships can exist between a pair of entities simultaneously, a threshold relationship class TH is introduced according to the method of Zhou et al.

[24] to solve the problem of imbalanced classification of multiple labels. The loss value is calculated using an adaptive threshold loss, and the calculation formula is as follows:

[0081]

[0082] In the formula, The relationship recognition loss value for document d is calculated by summing the relationship recognition losses for each pair of entity pairs in the document. h Represents the head entity, e t Represents the tail entity. e h ∈ε d Represents the head entity e h The set of entities ε belonging to document d d e t ∈ε d Represents tail entity e t The set of entities ε belonging to document d d In each document d, they form an entity pair (e). h ,e t P h,t Represents entity pairs (e) h ,e t The set of positive relations (r∈P), i.e., the set of relations with valid relations, where r∈P h,t This indicates that relation r belongs to entity pair (e). h ,e t The set of positive relations P) h,t This means that the relationship is one of the target relationships that the model needs to correctly identify. h,t Represents entity pairs (e) h ,e tThe set of negative relations (i.e., relations without valid relationships) is used to introduce an auxiliary class in loss calculation to address the class imbalance problem. TH represents the threshold relation class, used to distinguish between valid and invalid relations. Introducing TH can more effectively solve the problem of relation label imbalance. Represents entity pairs (e) h ,e t The relation score on relation r, which is calculated by a bilinear layer network, is used to assess the probability that a specific relation r exists between entity pairs. Indicates the score for the relationship. The exponent value is used to calculate the normalized probability in softmax. This represents the normalized denominator, calculating whether relation r belongs to the set of positive relations P. h,t The probability, including the threshold relationship TH. The normalized denominator of the set of negative relations is used to calculate whether relation TH belongs to the set of negative relations N. h,t The probability of.

[0083] Finally, the total loss function in the multi-task learning pre-training process is:

[0084]

[0085] In the formula, l represents the total loss function during the multi-task learning pre-training process, which is obtained by summing the multi-task losses of all documents in the training set. D train This represents the training dataset, which is the collection of all documents used for training. This represents the loss value of document d in the relation recognition task, used to measure the model's performance on the relation recognition task. This represents the loss value of document d in the entity recognition task, used to evaluate the classification accuracy of the model in the entity recognition task. η represents the loss value of document d in the evidence extraction task, measuring the model's performance on this task. ET η ER The hyperparameters representing the task weights are used to adjust the contributions of entity recognition and evidence extraction tasks to the total loss. These weights enable the model to consider the importance of different tasks during multi-task learning.

[0086] In some preferred embodiments, refer to Figure 2 As shown, the overall loss of the teacher model during the training process is expressed as:

[0087] L(T)=L RE +αL ET +βL ER

[0088] Where L(T) represents the total loss of the teacher model during training, which is the weighted sum of the losses of different tasks in multi-task learning, used to guide the teacher model to optimize across all tasks. ET L represents the multi-class cross-entropy loss, used to measure the classification accuracy of the teacher model in entity recognition tasks. ET This loss is used to supervise the entity recognition module, optimizing the model's ability to recognize different entity types by minimizing this loss.

[0089] L RE L represents the classification cross-entropy loss, used to measure the classification accuracy of the teacher model in relation recognition tasks. RE Used to supervise the relation identification module, this loss is minimized to optimize the model's prediction of entity pair relation types, enabling the model to better distinguish different relation categories. ER The focus loss, representing the binary cross-entropy, is used to measure the teacher model's performance on the evidence extraction task. Focus loss is a weighted binary cross-entropy loss used to mitigate class imbalance, especially when there are few positive examples (supporting evidence). It improves the model's evidence extraction ability by increasing attention to hard-to-classify samples. α represents the first hyperparameter, and β represents the second hyperparameter, used to balance the contributions of entity recognition, relation recognition, and evidence extraction tasks to the overall loss, enabling the model to allocate appropriate weights according to the importance of each task in multi-task learning.

[0090] Reference Figure 1 As shown, this teacher model learns various semantic knowledge through multi-task learning. During model training, this invention applies the trained teacher model to unlabeled document data (such as the remote supervision dataset DocRED), using the teacher model's inference to supplement the incomplete data in the document data, thereby obtaining complete document data.

[0091] This invention trains a student model, denoted as Stu, using complete document data (such as the remote supervision dataset DocRED). The objective function of the student model is consistent with that of the teacher model.

[0092] L(Stu)=L RE +γL ET +θL ER

[0093] Where L(Stu) represents the overall loss function of the student model Stu during training, which is obtained by weighted summation of the losses for different tasks. L is the loss function for different tasks. task =Loss task (PreStu task Labels task), task∈{ET,ER,RE}, corresponding to entity recognition task (ET), evidence extraction task (ER), and relation recognition task (RE) respectively, PreStu task Labels represent the student model's predictions on a specific task (such as entity recognition, evidence extraction, or relation recognition). task The labels represent manually annotated tags used for supervised training to guide the model in learning the target task. γ and θ represent hyperparameters that balance task weights, used to adjust the contributions of entity recognition and evidence extraction tasks to the overall loss, enabling the model to balance the importance of different tasks in multi-task learning. Through remote supervised learning, the student model acquired preliminary relation extraction capabilities.

[0094] After remote supervised training, this invention uses manually labeled data to further fine-tune the student model to improve its performance and generalization ability, enabling it to adapt to complex contexts in real-world scenarios. The fine-tuning loss function is defined as L. finetune The specific formula is similar to that used in the training phase:

[0095] L finetune (Stu)=L RE +γL ET +θL ER

[0096] Where L represents finetune (Stu) The overall loss function during the fine-tuning phase, further optimized using manually labeled data to adapt the student model to more complex real-world scenarios. Other parameters L RE L ET L ER The meanings of γ and θ are consistent with those in the training phase, used to balance the contribution of each task's loss to the overall loss. After fine-tuning, the final output relation extraction model is obtained, which is the final student model after remote supervised training and fine-tuning optimization.

[0097] In some preferred embodiments, reference is made to Figure 1 As shown, the process of using the trained teacher model to complete unlabeled document data includes:

[0098] S301 Initialization: Obtain document data containing unlabeled evidence, which includes entity pairs and corresponding relation tags, but lacks explicit evidence sentence information. This unlabeled evidence document data is typically automatically generated from an external knowledge base and lacks the precision of manual annotation. Perform initialization processing on this unlabeled evidence document data.

[0099] S302 Feature Extraction and Preliminary Encoding: Each document in the unlabeled evidence document data is input into the feature extraction module of the teacher model. This module performs preliminary encoding on the document data through the BERT network and Transformer encoder to generate document-level and entity-level contextual representations.

[0100] S303 Entity Pair Context Feature Generation: Generates each entity pair based on document-level and entity-level context representations (e h ,e t The context embedding of entity pairs includes global information about entity pairs in the document.

[0101] S304 Evidence Candidate Sentence Filtering: Filter sentences in each document to construct an evidence candidate sentence set Sd. Each sentence s in the evidence candidate set Sd includes contextual information related to the entity pair (eh,et).

[0102] S305 Evidence Probability Calculation: Each sentence s in the candidate evidence sentence set Sd is combined with the entity pair context embedding, and the probability of each sentence s becoming evidence is calculated using the evidence extraction module of the teacher model. This probability is obtained through the following formula:

[0103]

[0104] Here, σ represents the sigmoid function, used to convert evidence scores into probabilities. Sentences with higher probabilities among the candidate sentences are considered strong evidence.

[0105] S306 Evidence Completion: For each sentence in the candidate evidence sentence set, according to probability... The data is sorted, and the sentence with the highest probability is selected as the evidence sentence for the entity pair. This information is then added to the document data that does not have evidence.

[0106] S307: Through steps S301-S306, generate document data containing evidence information. Each data record in this data includes both entity pairs and their relational tags, as well as evidence sentences completed by the teacher model.

[0107] Experimental data and evaluation:

[0108] This invention was tested on the DocRED dataset. Compared to other datasets, DocRED has a much richer and larger dataset in terms of the number of documents, words, sentences, entities, and especially relation types, relation instances, and relation facts. The large-scale DocRED dataset can better demonstrate the model's ability to extract document-level relations. The statistical results of the dataset are shown in Table 1:

[0109] Table 1 Statistics of the DocRED dataset

[0110] Table1 The statistics of the DocRED datase

[0111]

[0112] DocRED covers a wide range of entity types, including people (18.5%), places (30.9%), institutions (14.4%), time (15.8%), and quantities (5.1%), as well as other miscellaneous entity names (15.2%), such as events, works of art, laws, etc., with each entity annotated an average of 1.34 times. DocRED includes 96 frequent relation types from Wikidata, covering a broader range of relations related to science (33.3%), art (11.5%), time (8.3%), personal life (4.2%), and more, meaning that relational facts are not limited to any particular domain. Furthermore, relation types are organized in a well-defined hierarchy and taxonomy, providing rich information for document-level relation extraction systems.

[0113] Following the benchmark settings established by Yao et al., two benchmark settings were designed for supervised and remotely supervised scenarios, respectively. For both settings, the data were randomly divided into training, development, and test sets, and the model was evaluated on high-quality human-annotated datasets. Specifically, the development and test sets were derived from human-annotated datasets in both scenarios, each containing 1000 documents. The training set consisted of 3053 documents from the human-annotated dataset in the supervised scenario, and 101873 documents from remotely supervised datasets in the remotely supervised scenario.

[0114] DocRED's evaluation metrics are F1 score and IgnF1 score. The IgnF1 score, proposed by Yao et al., excludes the fact that the training set, validation set and test set share a relationship, making the performance metric more objective.

[0115] To verify the performance of DistilRE, it was compared with the following benchmark models for document-level relation extraction:

[0116] CNN: Based on Convolutional Neural Network (CNN) training document data;

[0117] BiLSTM: Training document data based on a bidirectional long short-term memory network (BiLSTM);

[0118] ATLOP: A document-level relation extraction task proposed by Zhou et al. to solve multi-label and multi-entity problems using adaptive thresholds and localized context pools;

[0119] SAIS: A document-level relation extraction model proposed by Xiao et al. that solves the problem of uninterpretable model predictions by introducing evidence data augmentation and ensemble reasoning methods;

[0120] EIDER: Xie et al. proposed an enhanced document-level relation extraction model for DocRE by effectively extracting evidence and fusing evidence extracted through reasoning;

[0121] KD-DocRE: Tan et al. used adaptive focus loss and knowledge distillation techniques to enhance the learning of inter-entity dependencies through an axial attention module, thus solving the problem of document class imbalance.

[0122] SRLR: Proposed by Huang et al., it captures multi-level relational representations at the mention level in a document by separating relational representations and logical reasoning, and uses evidence sentences for complex reasoning;

[0123] PSD: Proposed by Wang et al., this model utilizes online self-knowledge distillation to gradually soften hard labels and improves document-level relation extraction performance through richer supervision signals. Since PSD is a general framework, this paper selects DocuNet+PSD as the benchmark model for comparison, based on the description in the original paper. The results are shown in Table 2.

[0124] Table 2. Experimental results on the DocRED dataset (in %)

[0125] Table2 Experiment results on DocRED(unit:%)

[0126]

[0127]

[0128] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include ROM, RAM, disk, or optical disk, etc.

[0129] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A document-level relation extraction method based on multi-task learning and knowledge distillation, characterized in that, include: Obtain the document text from which document-level relations are to be extracted, input it into the trained document-level relation extraction model, and output the corresponding document-level text relations; The document-level relation extraction model includes a feature extraction module, a teacher model, and a student model. The feature extraction module includes a BERT network and a Transformer encoder. The BERT network is used to encode document and entity representations of the input document data. The Transformer encoder is used to perform in-depth feature processing on the preliminary document and entity representations generated by the BERT network to obtain more accurate and richer feature representations. The training process of the document-level relation extraction model includes: The teacher model is obtained through multi-task learning pre-training using manually labeled document data. The process includes: S101: Encode the manually annotated document data using the feature extraction module and extract character representations; generate mention representations based on character representations, generate entity representations based on mention representations; construct entity pair representations based on manually annotated entities; S102: Input the entity representation into the entity recognition module, classify each entity to obtain the probability distribution of the entity type, and calculate the multi-class cross-entropy loss L based on this probability distribution. ET ; S103: Input entity pair representations into the evidence extraction module, perform probability calculations on the triplet evidence set of entity pairs, extract evidence of relationship existence, and calculate the focal loss L of the binary cross-entropy based on the predicted evidence probability. ER ; S104: Input the entity pair representation into the relationship recognition module, classify the relationship of each entity pair, obtain the probability distribution of the relationship type, and calculate the classification cross-entropy loss L based on this probability distribution. RE ; S105: The multi-class cross-entropy loss L... ET The focal loss L of binary cross-entropy ER And classification cross-entropy loss L RE The teacher model is iteratively optimized by combining weighted sums to form the overall loss L(T) of the teacher model and minimizing the overall loss L(T). The process of using a pre-trained teacher model to complete unlabeled evidence data includes: S301: Obtain document data without labeled evidence, which contains entity pairs and corresponding relational labels, but lacks explicit evidence sentence information; S302: Input each document in the unlabeled evidence document data into the feature extraction module of the teacher model. This module performs preliminary encoding on the document data through the BERT network and Transformer encoder to generate document-level and entity-level contextual representations. S303: Generate a context embedding for each entity pair based on document-level and entity-level context representations, which includes global information about the entity pair in the document; S304: Filter the sentences in each document to construct a set of candidate evidence sentences S d The evidence candidate set S d Each sentence s in the document includes contextual information related to the entity pair; S305: Set the candidate evidence sentences S d Each sentence s in the model is combined with an entity pair context embedding, and the probability of each sentence s serving as evidence is calculated using the evidence extraction module of the teacher model. S306: For each sentence in the candidate evidence sentence set, sort it according to probability. Sort the data, select the sentence with the highest probability as the evidence sentence for the entity pair, and add this information to the document data that is not labeled with evidence; S307: Through steps S301-S306, document data containing evidence information is generated. Each data record in this data includes entity pairs and their relation labels, as well as evidence sentences completed by the teacher model. Train a student model using the document data with the evidence completed; The trained document-level relation extraction model includes a feature extraction module and a trained student model.

2. The document-level relation extraction method based on multi-task learning and knowledge distillation according to claim 1, characterized in that, Both the teacher model and the student model are multi-task learning models, which include a feature extraction module, an entity recognition module, an evidence extraction module, and a relation extraction module.

3. The document-level relation extraction method based on multi-task learning and knowledge distillation according to claim 1, characterized in that, The overall loss of the teacher model during training is expressed as: L(T)=L RE +αL ET +βL ER Where L(T) represents the total loss of the teacher model, L ET L represents the cross-entropy loss for multi-class classification. RE L represents the classification cross-entropy loss. ER Let α represent the focal loss of the binary cross-entropy, where α represents the first hyperparameter and β represents the second hyperparameter.

4. The document-level relation extraction method based on multi-task learning and knowledge distillation according to claim 1, characterized in that, In S101, the feature extraction module encodes the manually annotated document data, extracts character representations, and generates mention representations based on these character representations. Specifically: S201: Preprocess the input document data, specifically including: summarizing the beginning and end of each sentence s in the document data d, inserting the classifier token "[CLS]" and the separator "[SEP]" at the beginning and end of each sentence respectively; and marking the entity with an entity tag "*" on each entity mention package m, where "*" indicates the position of the entity mention; S202: Input the preprocessed document data into BERT, and after encoding, output the characters and entity types; S203: Input the characters and entity types output by BERT into the Transformer encoder. After encoding, entity representations and mention representations are obtained. Each entity representation is obtained by embedding the "*" mark before the entity mention and the "[CLS]" mark at the beginning of the sentence. Each mention representation is obtained by embedding the "*" mark in the sentence and the "[CLS]" mark at the beginning of the sentence.

5. The document-level relation extraction method based on multi-task learning and knowledge distillation according to claim 1, characterized in that, The entity type probability is specifically represented as follows: In the formula, W represents the probability of the entity type output by the entity recognition module. c The weight matrix representing entity type classification, Wc∈R |C|×H R represents the real number space, |C| represents the number of entity types, i.e., the number of possible entity types, and H represents the entity representation e. v The dimension, e v Let tanh(·) represent the representation vector of any entity, δ(·) represent the tanh function, and b represent the softmax function. c Let bc represent the bias term vector for entity type classification, and bc ∈ R. |C| b c ∈R |C| b c The dimension is consistent with the number of categories.

6. The document-level relation extraction method based on multi-task learning and knowledge distillation according to claim 1, characterized in that, The probability of the triplet evidence set of the entity pair is specifically expressed as follows: In the formula, Let represent the probability of a triplet evidence set for entity pairs, where h represents the head entity (the subject of the relation in the triplet), t represents the tail entity (the object of the relation in the triplet), s represents the statement or evidence sentence of the triplet (the specific sentence that may contain the relation of the head entity h and the tail entity t), and σ(·) represents the sigmoid function. This represents the transpose of the context embedding of the entity pair at position k, which includes the contextual information of the entity pair at that specific position in the document. Let s represent the weight matrix at position k. k b represents the feature representation of sentence s at position k. p This indicates the bias term.

7. The document-level relation extraction method based on multi-task learning and knowledge distillation according to claim 1, characterized in that, The relational probability of the entity pair is specifically represented as follows: In the formula, This represents the relation probability between entity pairs, specifically the probability that a relation r exists between them. `h` represents the head entity (the subject in the triple relation), `t` represents the tail entity (the object in the triple relation), and `r` represents the relation type, i.e., the relationship between the entity pairs. This represents the transpose of the context-type fusion representation of the head entity h at the k-th layer, obtained by combining the embedded representation of the head entity with the head entity features extracted from the context. Represents the relation weight matrix. Let b represent the context-type fusion representation of the tail entity t at the k-th layer, and let b represent the combination of the tail entity's embedding representation and the tail entity features extracted from the context. r This indicates the bias term.

Images