Document-level relation extraction method based on multi-task learning and knowledge distillation

Abstract

The invention discloses a document-level relation extraction method based on multi-task learning and knowledge distillation, and the method comprises the steps: obtaining a to-be-extracted document text, inputting the to-be-extracted document text into a trained document-level relation extraction model, and outputting a corresponding document-level text relation; the document-level relation extraction model comprises a feature extraction module, a teacher model and a student model, and the training process of the model comprises the following steps: pre-training by utilizing manually labeled document data through multi-task learning to obtain the teacher model; using a pre-trained teacher model to carry out evidence complementation on the document data of which the evidence information is not labeled; training a student model by using the document data after evidence completion; the trained document-level relation extraction model comprises a feature extraction module and a trained student model. According to the method, the context semantic information in the document can be accurately extracted, the problem of data sparsity is effectively relieved through knowledge distillation, efficient extraction of the document-level relationship is realized, and the performance of the model in a complex context is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a document-level relationship extraction method based on multi-task learning and knowledge distillation. Background Art

[0002] With the rapid development of artificial intelligence and natural language processing (NLP), relation extraction (RE) technology has been widely used in downstream applications such as knowledge graph construction, question-answering systems, and digital teaching. The purpose of relation extraction technology is to identify and extract the relationship between entities from unstructured text for the subsequent generation and storage of structured data. At present, most relation extraction technologies have made significant progress in processing sentence-level entity relationship recognition. However, sentence-level relation extraction methods are usually limited to the relationship judgment of a single sentence. There are problems such as information loss and insufficient long-distance dependency in the relationship recognition of multiple sentences or even the entire document, and it is difficult to handle complex relationships in long texts.

[0003] To solve the above problems, document-level relation extraction has become an important research direction in the field of relation extraction. This method uses the full-text semantics and context information to achieve more comprehensive relation identification. Existing document-level relation extraction technologies can be divided into sequence-based, graph-based, and pre-trained language model-based methods.

[0004] Sequence-based methods extract relations by sequential dependencies of text information, which is suitable for shorter texts, but they are prone to losing cross-sentence semantic information when processing long texts with large spans.

[0005] Graph-based methods model entity relationships through graph structures and can mine multi-level relationships, but they introduce high computational complexity in large-scale document construction.

[0006] The method based on pre-trained language models relies on the context understanding ability of pre-trained models to effectively capture long-distance dependencies at the document level, and has become the mainstream research direction in this field in recent years.

[0007] The Chinese invention patent application with publication number CN118095278A discloses a document-level relationship extraction method based on a pre-trained model, which uses a Transformer-based pre-trained model to capture long-distance dependencies and improve the accuracy of relationship recognition. However, this method mainly focuses on single-task training and cannot effectively use the semantic information of different tasks for multi-task joint training.

[0008] The Chinese invention patent application with publication number CN118798369A discloses a document-level relationship extraction method based on a graph structure, which constructs an entity graph for relationship modeling and shows good performance when processing long text relationship extraction. However, due to the high complexity of the graph model when processing long documents, the computational cost increases and it is impossible to quickly process large-scale document data.

[0009] The Chinese invention patent application with publication number CN115688901A discloses a multi-task learning method that applies knowledge distillation technology to text processing. The knowledge distillation technology is used to improve the generalization ability of the model, but the method is not combined with the document-level relationship extraction task and cannot effectively solve the problem of long-distance dependencies in documents. Summary of the invention

[0010] In order to solve the above technical problems, the present invention proposes a document-level relationship extraction method based on multi-task learning and knowledge distillation, the method comprising: obtaining a document text of a document-level relationship to be extracted, inputting the document text into a trained document-level relationship extraction model, and outputting a corresponding document-level text relationship;

[0011] The document-level relationship extraction model includes a feature extraction module, a teacher model and a student model. The training process of the document-level relationship extraction model includes:

[0012] The teacher model is obtained by pre-training with multi-task learning using manually annotated document data;

[0013] Use the pre-trained teacher model to complete the evidence of document data without labeled evidence information;

[0014] Use the document data after evidence completion to train the student model;

[0015] The trained document-level relationship extraction model includes a feature extraction module and a trained student model.

[0016] Beneficial effects of the present invention: First, the present invention can realize semantic information sharing of entity recognition, evidence retrieval and relationship recognition tasks by training the teacher model through multi-task learning, thereby improving the accurate understanding of entity relationships in complex documents and effectively overcoming the shortcomings of existing models in processing long-distance dependency relationships. Secondly, the present invention introduces knowledge distillation and uses the teacher model to complete the evidence of remote supervision data, thereby alleviating the problem of data sparsity. The student model is trained with the data after evidence completion, so that the model can maintain good generalization performance even in the case of data imbalance. The performance of document-level relationship extraction using the trained student model is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is an overall flow chart of an embodiment of the present invention;

[0018] Figure 2 Flow chart of multi-task learning of the teacher model in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] The embodiment of the present invention proposes a document-level relationship extraction method based on multi-task learning and knowledge distillation. Figure 1 As shown, the method includes:

[0021] Obtain the document text of the document-level relationship to be extracted, input it into the trained document-level relationship extraction model, and output the corresponding document-level text relationship;

[0022] The document-level relationship extraction model includes a feature extraction module, a teacher model and a student model. The training process of the document-level relationship extraction model includes:

[0023] S1: Use manually annotated document data to obtain a teacher model through multi-task learning pre-training;

[0024] S2: Use the pre-trained teacher model to complete the document data without labeled evidence information;

[0025] S3: Train the student model using document data after evidence completion;

[0026] The trained document-level relationship extraction model includes a feature extraction module and a trained student model.

[0027] Figure 1In the embodiment of the present invention, the process includes two stages, namely, the training stage and the practical application stage. The document-level relationship extraction model includes a feature extraction module, a teacher model and a student model. The feature extraction module includes a BERT network and a Transformer encoder. Unlike the traditional knowledge distillation model, the embodiment of the present invention only uses the teacher model to transfer knowledge to the student model, and does not compress the model. Both the student model and the teacher model adopt a multi-task learning model, and their structure and size settings are consistent, and both include an evidence extraction module, an entity recognition module and a relationship extraction module. First, manually annotated document data is used to obtain a teacher model through multi-task learning pre-training. The pre-trained teacher model is used to complete the evidence of document data without annotated evidence (i.e., soft label data, without evidence information, such as the remote supervision data set: DocRED) to obtain document data after evidence completion (i.e., soft label data); the student model is trained using the document data after evidence completion. Using the manually annotated document data, after extracting features, the trained student model is input for fine-tuning. In the practical application stage, the trained document-level relationship extraction model includes a feature extraction module and a trained student model. Obtain the document text of the document-level relationship to be extracted, use the feature extraction module to extract features, input them into the trained student model, and output the corresponding document-level text relationship.

[0028] Figure 2 Flow chart of multi-task learning of the teacher model in an embodiment of the present invention. Figure 2 In the paper, the teacher model is obtained by hard-shared multi-task learning training using manually annotated document data (also known as golden data). The specific process includes: inputting the manually annotated document data into BERT for encoding to obtain characters and entity types. These characters and entity types are input into the Transformer encoder to further extract deep character representations, generate mention representations based on the character representations, and then generate entity representations based on the mention representations. Based on the manually annotated entities, an entity pair representation is constructed to provide input for subsequent modules. The entity representation is input into the entity recognition module to obtain the probability distribution of the entity type, and the multi-classification cross entropy loss L is calculated. ET The entity pair representation is input into the evidence extraction module to obtain the probability of the entity pair triple evidence set and calculate the focus loss L of the binary cross entropy ER , which is used to evaluate the effect of evidence extraction. The entity pair representation is input into the relationship recognition module to obtain the relationship probability of the entity pair and calculate the classification cross entropy loss L RE , which measures the accuracy of relationship recognition. The multi-classification cross entropy loss L ET , the focal loss L of binary cross entropy ER and categorical cross entropy loss L REAfter weighted summation, the overall loss L(T) of the teacher model is obtained, and the teacher model is iteratively optimized by minimizing the overall loss L(T).

[0029] In some preferred embodiments, reference Figure 1 , 2 As shown, both the teacher model and the student model are multi-task learning models, which include a feature extraction module, an entity recognition module, an evidence extraction module and a relationship extraction module.

[0030] In some preferred embodiments, reference Figure 1 , 2 As shown, the feature extraction module includes a BERT network and a Transformer encoder. The BERT network is used to perform document representation and entity representation encoding on the input document data, and the Transformer encoder is used to perform in-depth feature processing on the preliminary document and entity representation generated by the BERT network to obtain a more accurate and rich feature representation.

[0031] Specifically, after preprocessing, the document data is input into the BERT network, and the BERT network is used to perform preliminary encoding on the input document data. Through character-level and sentence-level encoding processing, BERT provides basic document representation and entity representation, which can capture the vocabulary and syntactic information in the document, and use the Transformer encoder to perform in-depth feature processing based on the preliminary representation generated by BERT. The Transformer encoder uses the self-attention mechanism to capture the contextual dependencies and long-distance dependency information at different positions in the document, and weights the entity information at different positions through the multi-head self-attention layer (Multi-headAttention), so as to integrate the contextual information and enhance the correlation between entities, so that the model can more comprehensively understand the relationship between entities at the document level, and obtain more accurate and rich feature representations, which are specifically character representations. Mention representations are generated based on character representations, and then entity representations are generated based on mention representations. Finally, entity pair representations are constructed based on entity annotations.

[0032] In some preferred embodiments, reference Figure 1 , 2 As shown in the figure, the teacher model is obtained by pre-training with multi-task learning using manually annotated document data. The specific process includes:

[0033] S101: Encode manually annotated document data using a feature extraction module to extract character representations; generate mention representations based on the character representations, and generate entity representations based on the mention representations; and construct entity pair representations based on manually annotated entities.

[0034] Specifically, first, the manually annotated document data is preprocessed and then input into the BERT network, which is used to encode the text and extract character-level representations. Then, the character representations are input into the Transformer encoder to further generate deep contextual representations. Mention representations are generated based on the character representations, and then entity representations are generated based on the mention representations. Finally, entity pair representations are constructed based on entity annotations to provide input for subsequent modules.

[0035] S102: Input the entity representation into the entity recognition module, classify each entity, and obtain the probability distribution of the entity type; based on this probability distribution, calculate the multi-classification cross entropy loss L ET , which is used to measure the accuracy of entity recognition results and help the model better distinguish different types of entities.

[0036] S103: Input the entity pair representation into the evidence extraction module, perform probability calculation on the entity pair triple evidence set, extract evidence of the existence of the relationship, and calculate the focus loss L of the binary cross entropy based on the predicted evidence probability. ER . Use the binary cross entropy focal loss L ER Optimize the model so that it performs well in the task of evidence extraction. Focus loss L ER It can increase the model's attention to difficult-to-identify samples and minority categories, and improve the accuracy of evidence extraction.

[0037] S104: Input the entity pair representation into the relationship recognition module, classify the relationship of each entity pair, obtain the probability distribution of the relationship type, and calculate the classification cross entropy loss L based on this probability distribution. RE Using the classification cross entropy loss L RE Optimizes the model's accuracy on the relation identification task. This loss enables the model to accurately distinguish multiple relation types between entity pairs.

[0038] S105: Multi-classification cross entropy loss L ET , the focal loss L of binary cross entropy ER and categorical cross entropy loss L RE Through weighted summation, the overall loss L(T) of the teacher model is obtained. The teacher model is iteratively optimized by minimizing the overall loss L(T) so that it can achieve optimal performance in entity recognition, evidence extraction and relationship recognition tasks at the same time.

[0039] In some preferred embodiments, reference Figure 1 , 2 As shown, in S101, the manually annotated document data is encoded using a feature extraction module, character-level representations are extracted, and mention representations are generated based on the character representations, which are specifically:

[0040] S201: Preprocess the input document data, specifically including: aggregating the beginning and end of each sentence s in the document data d, inserting the classifier token "[CLS]" and the separator "[SEP]" at the beginning and end of each sentence respectively; and marking the entity "*" on each entity mention package m, where the "*" indicates the location of the entity mention.

[0041] Specifically: For each document data d, divide it into multiple sentences s by sentence. Insert the classifier token "[CLS]" at the beginning of each sentence and the separator "[SEP]" at the end to clarify the sentence boundaries and provide global information. Doing so allows the model to use the "[CLS]" embedding to represent the characteristics of the entire sentence. For each entity mention bag m in the document, insert the entity marker "*" at the entity position to identify the location of the entity mention. This marker helps the model locate the entity more accurately and focus on the context in which the entity is located.

[0042] S202: Input the preprocessed document data into BERT, and after encoding, output characters and entity types.

[0043] S203: Input the characters and entity types output by BERT into the Transformer encoder, and obtain entity representation and mention representation after encoding processing. Each entity representation is obtained by the embedded combination of the "*" mark before the entity mention and the "[CLS]" mark at the beginning of the sentence, and each mention representation is obtained by the embedded combination of the "*" mark in the sentence and the "[CLS]" mark at the beginning of the sentence.

[0044] Specifically, the characters and entity types output by BERT are input into the Transformer encoder, and the deep contextual representation is extracted through the self-attention mechanism to generate character-level representation, mention representation and entity representation. The representation of each entity is obtained by combining the embeddings of the "*" tag before the entity mention and the "[CLS]" tag at the beginning of the sentence. These embeddings summarize the local information of the entity (provided by the "*" tag) and the global contextual information (provided by the "[CLS]" tag). The representation of each mention includes the embedding of the position mark "*" of the mention in the sentence and the "[CLS]" tag at the beginning of the sentence. The mention representation generated in this way includes the local information of the mention in the sentence and the information of the sentence as a whole.

[0045] Reference Figure 2 As shown, the step S102 is specifically as follows:

[0046] The present invention integrates the mention set M of entity e through the LogSumExp pooling function according to the entity mention representation e , get the embedded representation of entity e: Where m represents the embedding representation of the special character “*” at the beginning of the entity mention. This entity embedding e represents all the mention information of the entity, providing a more comprehensive entity representation for the model. Since entity e can appear as the head or the tail in an entity pair, two separate linear layers are used to distinguish the head entity embedding e h and the tail entity embedding e t , the specific formula is:

[0047]

[0048] Among them, e h represents the head entity embedding, e t represents the tail entity embedding, is the weight matrix of the linear transformation, which is used for the calculation of the head and tail entity embeddings respectively, and are bias parameters corresponding to the linear transformation of the head and tail entities, respectively, and Therefore, input any entity representation e v (where v∈{h,t}, represents head or tail), and calculates the probability of the entity type.

[0049] In some preferred embodiments, the specific calculation formula of the probability of the entity type is:

[0050]

[0051] In the formula, W represents the entity type probability output by the entity recognition module, that is, the model estimates the probability that the entity belongs to different types. c Represents the weight matrix of entity type classification, which is used to map entity representation into entity type space, and Wc∈R |C|×H , R represents the real number space, |C| represents the number of entity types, that is, the number of possible entity types, and H represents the entity representation e v The dimension, e v Represents the representation vector of any entity (the representation of the head entity or the tail entity), e v is the contextual representation of the entity generated by the feature extraction module. tanh(·) represents the tanh activation function, which is used to map the input value to the interval (-1, 1) to increase the nonlinear expression ability of the model. δ(·) represents the softmax function, which is used to map the output to the probability distribution of the entity type. c Represents the bias vector of entity type classification, which is used to adjust different types of biases, and bc∈R |C| , b c ∈R |C| Indicates b c The dimension is consistent with the number of categories.

[0052] In the entity recognition task, due to the imbalance problem of recognition categories, the multi-classification cross entropy loss function is used to calculate the loss value. The specific calculation formula is:

[0053]

[0054] In the formula, Represents the entity recognition loss value of document d. It is obtained by accumulating the recognition losses of all entities in the document. Represents the probability that entity e is identified as category c, which is calculated by the entity recognition module. Indicates that entity e belongs to category c. This is a binary indicator variable. When entity e belongs to category c, otherwise e represents an entity in the current document d, E d represents the set of all entities in document d, c represents a type in the entity category, and C represents the set of all possible entity categories.

[0055] Reference Figure 2 As shown, the step S103 is specifically as follows:

[0056] In this paper, the attention calculation in the encoder of the Transformer is guided by the evidence retrieval task to identify the triple (e h ,e t ,r) A collection of relevant important evidence Where V h,t,r Represents the triple (e h ,e t ,r) A set of sentences related to .

[0057] In order to obtain better contextual semantics, the character-level attention A is calculated by the equation H, A = Encoder (d), and the unique contextual embedding representation c containing the head and tail entities is calculated. h,t , the specific formula is as follows:

[0058]

[0059] Among them, c h,t represents the contextual embedding representation containing the head and tail entities, ⊙ is the element-wise product, A h Indicates all e in the whole document identified by the entity tag "*" h The average of the mention-level attention, A t All e in the full document identified by the entity tag "*" tThe average value of the mentioned attention weights, H represents the document feature representation matrix, each column corresponds to the embedding representation of a character, generated by the Transformer encoder, and T represents the transpose operation of the matrix, which converts the column vector of the matrix H into a row vector.

[0060] The entity pair (e h ,e t )'s contextual embedding c h,t and document sentence set S d Input a set of bilinear layers, composed of c h,t Predict sentence s∈S d is the probability of the entity for the set of triple evidence.

[0061] In some preferred embodiments, the probability of the triplet evidence set of the entity pair is specifically expressed as:

[0062]

[0063] in, The probability of the triple evidence set representing the entity pair, that is, the probability of the model judging whether there is evidence for the entity pair (h, t, s), is used in the evidence extraction module. h represents the head entity, which is the subject entity of the relationship in the triple. t represents the tail entity, which is the object entity of the relationship in the triple. s represents the sentence or evidence sentence of the triple, that is, the specific sentence that may contain the relationship between h and t. σ(·) represents the sigmoid function, which maps the input value to the range of (0, 1) and is used to output the probability value. Represents an entity pair (e h ,e t ) is the transpose of the context embedding at position k, which contains the contextual information of the specific position of the entity pair in the document, represents the weight matrix at position k, which is used for the bilinear transformation between entity pair context features and sentence features, and b p represents the bias term, and b p ∈R,s k Represents the feature representation of sentence s at position k.

[0064] The evidence set also faces the problem of class imbalance. 97.1% of entity pairs in the DocRED dataset have no valid relationship or supporting evidence, so the positive examples (supporting evidence of entity pairs) and negative examples (non-supporting evidence) are seriously unbalanced. DistilRE uses focal loss to alleviate the class imbalance problem by reducing the contribution of easily classified samples to the gradient, so that the model pays more attention to difficult-to-classify samples.

[0065] The focal loss function of the binary cross entropy Specifically:

[0066]

[0067] in, Represents an entity pair (e h ,e t ) is a positive example in sentence s. If sentence s∈v h,t (That is, the sentence s belongs to the entity pair (e h ,e t ), then otherwise The model predicts that the sentence s belongs to the entity pair (e h ,e t ), γ ER is a hyperparameter of the focal loss, controlling the degree of gradient reduction for easy-to-classify samples. ER The value makes the model pay more attention to samples that are difficult to classify. is the category weight, the weight of positive and negative examples is It is inversely proportional to the frequency of . It is used to balance the problem of uneven class ratios and give higher weights to rare samples. h ∈ε d Represents the head entity e h The set of entities ε belonging to document d d , e t ∈ε d Represents the tail entity e t The set of entities ε belonging to document d d , s∈S d Indicates that sentence s belongs to the sentence set S of document d d . , v h,t Represents an entity pair (e h ,e t ) is a collection of supporting evidence.

[0068] Reference Figure 2 As shown, the S104 is specifically:

[0069] For any entity pair (e h ,e t ), c h,t The entity pair related context is encoded in e h and e t The type information of the entity is preserved in c. Therefore, two separate linear layer networks are used to h,t The context features needed to extract the head entity and the tail entity are:

[0070]

[0071] In the formula, c′ h Represents the context feature representation of the head entity, c′ t Represents the context feature representation of the tail entity, The weight matrix representing the context features of the head entity, The weight matrix representing the contextual features of the tail entity, The bias term representing the context feature of the head entity, c′ represents the bias term of the tail entity context feature. h Embed c from the context of the entity pair through linear transformation h,t The head entity features extracted from are used to capture the contextual information of the entity pair as the head entity. t Embed c from the context of the entity pair through linear transformation h,t The tail entity features extracted from the ,is used to capture the contextual information when the entity pair is used as the tail entity. The context feature c of the entity pair h,t Mapped to the head entity feature space, its dimension is H×H. The context feature c of the entity pair h,t Mapped to the tail entity feature space, its dimension is also H×H. It is used to translate the context features of the head entity in the linear transformation, and its dimension is H. It is used to translate the context features of the tail entity in the linear transformation, and its dimension is H.

[0072] Represent the head and tail entities with contextual information h and head and tail entity representations with type information t Combine to generate new head and tail entity representations:

[0073] e″ h =tanh(e h +c′ h )

[0074] e″ t =tanh(e t +c′ t )

[0075] In the formula, e″ h Represents a new header entity representation, which combines the type information of the header entity h and context information c′ h Generated features. Through this combination, the new head entity representation e″ h At the same time, the type information and context feature information of the head entity are retained, so that the representation can more accurately reflect the role of the entity in the specific relationship. e″t Represents a new tail entity representation, which combines the type information of the tail entity e t and context information c′ t Generated features. The new tail entity represents e″ t It also contains the type information and context feature information of the tail entity. h represents the embedding of the head entity, which includes type information representing the inherent attributes and category information of the head entity. This is the embedding generated by the model for the head entity type information during the encoding process. t represents the tail entity embedding, which includes the type information representing the inherent attributes and category information of the tail entity. This is the embedding generated by the model for the tail entity type information during the encoding process. c′ h The context feature representation of the head entity is obtained by linearly transforming the context embedding c h,t It is extracted from the head entity to capture the contextual information of the head entity under a specific relationship. t The context feature representation of the tail entity is obtained by linearly transforming the context embedding c h,t , which is used to capture the contextual information of the tail entity under a specific relationship. tanh(·) represents the tanh activation function, which is used to perform a nonlinear transformation on the result of adding two embeddings and map the result to the interval (-1,1). The introduction of the tanh function increases the nonlinear expression ability of the model, making the newly generated head and tail entity representation e″ h and e″ t More discriminating.

[0076] Let the new head entity be represented by e″ h , the new tail entity represents e″ t Input the bilinear layer network to calculate the new head and tail entity representation e″ h , e″ t The relationship probability r.

[0077] In some preferred embodiments, the relationship probability between the entity pairs is specifically expressed as:

[0078]

[0079] In the formula, Represents the probability of the relationship between entity pairs (h, t). is the probability value of the model predicting that there is a specific relationship r between the head entity h and the tail entity t. h represents the head entity, that is, the subject entity in the entity pair, t represents the tail entity, that is, the object entity in the entity pair, and r represents the relationship type, that is, the possible relationship category between the head entity and the tail entity. represents the transpose of the context-type fused feature representation of the head entity h at the kth position. Includes the entity as the head entity's type information and context features, Represents the relationship weight matrix at the kth layer, which is used to calculate the relationship probability between the head entity and the tail entity. The dimension is R H / K×H / K , H represents the total dimension of hidden layer features, K represents the number of layers of relationship prediction, that is, the number of spatial dimensions into which the features are divided, Represents the context-type fused feature representation of the new tail entity t at the kth position. Include the type information and context features of the entity as the tail entity, b r Represents the bias term in relationship prediction, which is used to linearly adjust the output in relationship prediction and belongs to the real number space R.

[0080] Since only some entity pairs in the dataset have valid relationships, and multiple relationships can exist between a pair of entities at the same time, in order to solve the problem of multi-label imbalanced classification, the threshold relationship class TH is introduced according to the method of Zhou et al.

[24] , and the adaptive threshold loss is used to calculate the loss value. The calculation formula is:

[0081]

[0082] In the formula, represents the relation recognition loss value of document d, which is obtained by accumulating the relation recognition loss of each pair of entities in the document. h Indicates the head entity, e t Indicates the tail entity. e h ∈ε d Represents the head entity e h The set of entities ε belonging to document d d , e t ∈ε d Represents the tail entity e t The set of entities ε belonging to document d d . In each document d, they form an entity pair (e h ,e t ). h,t Represents an entity pair (e h ,e t ), that is, the set of relations with valid relations, r∈P h,t Represents a relation r that belongs to an entity pair (e h ,e t ) of the positive relation set P h,t , that is, the relationship is one of the target relationships that the model needs to correctly identify. h,t Represents an entity pair (e h ,e t), that is, a set of negative relations without valid relations, is used to introduce an auxiliary category in the loss calculation to solve the problem of category imbalance. TH represents the threshold relation class, which is used to distinguish valid relations from invalid relations. By introducing TH, the problem of relation label imbalance can be solved more effectively. Represents an entity pair (e h ,e t ) is a relation score on relation r, which is calculated by a bilinear layer network and is used to evaluate the probability that a specific relation r exists between entity pairs. Represents the relationship score Take the exponential value used to calculate the normalized probability in softmax. Represents the normalized denominator, and the calculated relation r belongs to the positive relation set P h,t The probability of , which includes the threshold relationship TH. Represents the normalized denominator of the negative relationship set, used to calculate whether the relationship TH belongs to the negative relationship set N h,t probability.

[0083] Finally, the total loss function in the multi-task learning pre-training process is:

[0084]

[0085] Where l represents the total loss function in the multi-task learning pre-training process, which is obtained by accumulating the multi-task losses of all documents in the training set. train Represents the training dataset, that is, the collection of all documents used for training. Represents the loss value of document d in the relationship identification task, which is used to measure the performance of the model in the relationship identification task. Represents the loss value of document d in the entity recognition task, which is used to evaluate the classification accuracy of the model in the entity recognition task. Represents the loss value of document d in the evidence extraction task, which measures the performance of the model in the evidence extraction task. η ET , η ER The hyperparameters representing the weights of the balanced tasks are used to adjust the contribution of the entity recognition task and the evidence extraction task to the total loss. These weights allow the model to take into account the importance of different tasks in multi-task learning.

[0086] In some preferred embodiments, reference Figure 2 As shown, the overall loss of the teacher model during training is expressed as:

[0087] L(T)=L RE +αL ET +βL ER

[0088] Among them, L(T) represents the overall loss of the teacher model during the training process, that is, the weighted sum of the losses of different tasks in multi-task learning, which is used to guide the teacher model to optimize on all tasks. ET represents the multi-classification cross entropy loss, which is used to measure the classification accuracy of the teacher model in the entity recognition task. L ET Used to supervise the entity recognition module, by minimizing the loss to optimize the model's ability to recognize different entity types.

[0089] L RE represents the classification cross entropy loss, which is used to measure the classification accuracy of the teacher model in the relation recognition task. RE It is used to supervise the relationship recognition module and optimize the model's prediction of entity-to-relationship types by minimizing the loss, so that the model can better distinguish different relationship categories. ER Represents the focal loss of binary cross entropy, which is used to measure the performance of the teacher model in the evidence extraction task. Focal loss is a weighted binary cross entropy loss, which is used to alleviate the class imbalance problem, especially when there are few positive samples (supporting evidence), by paying more attention to difficult-to-classify samples to improve the model's evidence extraction ability. α represents the first hyperparameter, and β represents the second hyperparameter, which is used to balance the contribution of entity recognition, relation recognition, and evidence extraction tasks to the overall loss, so that the model can properly allocate weights according to the importance of tasks in multi-task learning.

[0090] Reference Figure 1 As shown, the teacher model learns various semantic knowledge through multi-task. In the model training process, the present invention applies the trained teacher model to document data with unlabeled evidence (such as remote supervision dataset: DocRED), and supplements the incomplete data in the document data through teacher model reasoning, thereby obtaining the complete document data.

[0091] The present invention uses complete document data (such as remote supervision dataset: DocRED) to train the student model, denoted as Stu. The objective function of the student model is consistent with that of the teacher model:

[0092] L(Stu)=L RE +γL ET +θL ER

[0093] Where L(Stu) represents the overall loss function of the student model Stu during the training process, which is obtained by weighted summation of the losses of different tasks. L is the loss function of different tasks L task =Loss task (PreStu task ,Labels task), task∈{ET,ER,RE}, corresponding to the entity recognition task (ET), evidence extraction task (ER) and relation recognition task (RE), respectively. task Labels represent the predicted value of the student model on a specific task (such as entity recognition, evidence extraction, or relationship recognition). task Represents manually annotated labels, which are used for supervised training and guide the model to learn the target task. γ and θ represent hyperparameters for balancing task weights, which are used to adjust the contribution of entity recognition and evidence extraction tasks to the overall loss, so that the model can balance the importance of different tasks in multi-task learning. Through remote supervised learning, the student model acquires preliminary relationship extraction capabilities.

[0094] After the remote supervision training is completed, the present invention uses manually annotated data to further fine-tune the student model to improve its performance and generalization ability, so that it can adapt to the complex context in real scenes. The fine-tuning loss function is defined as L finetune , the specific formula is similar to the training stage:

[0095] L finetune (Stu)=L RE +γL ET +θL ER

[0096] Where L finetune (Stu) is the overall loss function in the fine-tuning phase, which further optimizes the student model through manually annotated data to adapt to more complex real-world scenarios. Other parameters L RE , L ET , L ER The meanings of ,γ,θ are consistent with those in the training stage, and are used to balance the contribution of each task loss to the overall loss. After fine-tuning, the final output is the relation extraction model, which is the final student model after remote supervision training and fine-tuning optimization.

[0097] In some preferred embodiments, reference Figure 1 As shown in the figure, the trained teacher model is used to complete the document data without labeled evidence. The process includes:

[0098] S301 Initialization: Obtain document data of unlabeled evidence, which contains entity pairs and corresponding relationship labels, but lacks clear evidence sentence information. The document data of unlabeled evidence is usually automatically generated by an external knowledge base and lacks the accuracy of manual annotation. The document data of unlabeled evidence is initialized.

[0099] S302 Feature extraction and preliminary encoding: Each document in the document data of unlabeled evidence is input into the feature extraction module of the teacher model, which performs preliminary encoding on the document data through the BERT network and Transformer encoder to generate document-level and entity-level context representations.

[0100] S303 Entity pair context feature generation: Generate each entity pair (e h ,e t ), which includes the global information of the entity pair in the document.

[0101] S304: Screening of candidate evidence sentences: Screen the sentences in each document and construct a set of candidate evidence sentences Sd. Each sentence s in the set of candidate evidence Sd includes context information related to the entity pair (eh, et).

[0102] S305 Evidence Probability Calculation: Each sentence s in the evidence candidate sentence set Sd is combined with the entity pair context embedding, and the evidence extraction module of the teacher model is used to calculate the probability of each sentence s becoming evidence. The probability is obtained by the following formula:

[0103]

[0104] Where σ represents the sigmoid function, which is used to convert the evidence score into probability. The sentences with higher probabilities in the candidate sentences are regarded as strong evidence.

[0105] S306 Evidence completion: Each sentence in the evidence candidate sentence set is evaluated according to the probability Sort and select the sentence with the highest probability as the evidence sentence for the entity pair, and complete this information in the document data without labeled evidence.

[0106] S307: Through steps S301-S306, document data containing evidence information is generated, in which each data record includes both entity pairs and their relationship labels, and evidence sentences completed by the teacher model.

[0107] Experimental data and evaluation:

[0108] The present invention conducts experiments on the DocRED dataset. Compared with other datasets, DocRED has richer data in terms of the number of documents, words, sentences, entities, especially relationship types, relationship instances, and relationship facts, and is large in scale. The large-scale DocRED dataset can better reflect the model's ability to extract document-level relationships. The statistical results of the dataset are shown in Table 1:

[0109] Table 1 Statistics of DocRED dataset

[0110] Table1 The statistics of the DocRED datase

[0111]

[0112] DocRED covers a variety of entity types, including persons (18.5%), places (30.9%), institutions (14.4%), times (15.8%), and quantities (5.1%), as well as other miscellaneous entity names (15.2%) such as events, works of art, laws, etc., with each entity annotated 1.34 times on average. DocRED includes 96 frequent relation types from Wikidata, covering a wider range of relations related to science (33.3%), art (11.5%), time (8.3%), personal life (4.2%), etc., which means that relation facts are not restricted to any specific domain. In addition, relation types are organized in a well-defined hierarchy and taxonomy, which can provide rich information for document-level relation extraction systems.

[0113] Following the benchmark setup of Yao et al., two benchmark setups were designed for supervised and remote supervised scenarios. For both setups, the data was randomly split into training, development, and test sets, and the models were evaluated on high-quality manually annotated datasets. Specifically, the development and test sets were split from manually annotated datasets in both scenarios, and the size was 1000 documents. The training set was split from manually annotated datasets in the supervised scenario and the size was 3053 documents, and in the remote supervised scenario it was remote supervised data, the size was 101873 documents.

[0114] The evaluation indicators of DocRED are F1 value and IgnF1 value. The IgnF1 value was proposed by Yao et al., which excludes the relationship facts shared by the training set, validation set and test set, making the performance indicator more objective.

[0115] To verify the recognition performance of DistilRE, we compare it with the following baseline models for document-level relation extraction:

[0116] CNN: Training document data based on convolutional neural network (CNN);

[0117] BiLSTM: Training document data based on Bidirectional Long Short-Term Memory Network (BiLSTM);

[0118] ATLOP: Zhou et al. proposed a document-level relation extraction task that uses adaptive thresholding and localized context pooling to solve multi-label and multi-entity problems;

[0119] SAIS: A document-level relation extraction model proposed by Xiao et al. that solves the problem of unexplainable model prediction by introducing evidence data enhancement methods and integrated reasoning methods;

[0120] EIDER: A document-level relation extraction model proposed by Xie et al. that enhances DocRE by effectively extracting evidence and fusing evidence extracted from reasoning;

[0121] KD-DocRE: Tan et al. adopted adaptive focal loss and knowledge distillation technology to enhance the learning of interdependencies between entity pairs through the axial attention module to solve the problem of document category imbalance;

[0122] SRLR: Proposed by Huang et al., it captures multi-level relation representations at the mention level in documents by separating relation representation and logical reasoning, and uses evidence sentences for complex reasoning;

[0123] PSD: Wang et al. proposed to use online self-knowledge distillation to gradually soften hard labels and improve document-level relationship extraction performance through richer supervision signals. Because PSD is a general framework, according to the description in the original paper, this paper chooses DocuNet+PSD as the benchmark model for comparison. The results are shown in Table 2:

[0124] Table 2 Experimental results on DocRED dataset (unit: %)

[0125] Table2 Experiment results on DocRED(unit:%)

[0126]

[0127]

[0128] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which can include: ROM, RAM, disk or CD, etc.

[0129] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A document-level relationship extraction method based on multi-task learning and knowledge distillation, characterized in that: include: Obtain the document text of the document-level relationship to be extracted, input it into the trained document-level relationship extraction model, and output the corresponding document-level text relationship; The document-level relationship extraction model includes a feature extraction module, a teacher model, and a student model. Its training process includes: The teacher model is obtained by pre-training with multi-task learning using manually annotated document data; Use the pre-trained teacher model to complete the evidence of document data without labeled evidence information; Use the document data after evidence completion to train the student model; The trained document-level relationship extraction model includes a feature extraction module and a trained student model.

2. The document-level relationship extraction method based on multi-task learning and knowledge distillation according to claim 1 is characterized in that: The teacher model and the student model are both multi-task learning models, which include a feature extraction module, an entity recognition module, an evidence extraction module and a relationship extraction module.

3. The document-level relationship extraction method based on multi-task learning and knowledge distillation according to claim 1 is characterized in that: The feature extraction module includes a BERT network and a Transformer encoder. The BERT network is used to perform document representation and entity representation encoding on the input document data, and the Transformer encoder is used to perform in-depth feature processing on the preliminary document and entity representations generated by the BERT network to obtain a more accurate and rich feature representation.

4. The document-level relationship extraction method based on multi-task learning and knowledge distillation according to claim 1 is characterized in that: The teacher model is obtained by pre-training with multi-task learning using manually annotated document data. The specific process includes: S101: Encode manually annotated document data using a feature extraction module to extract character representations; generate mention representations based on the character representations, and generate entity representations based on the mention representations; construct entity pair representations based on manually annotated entities; S102: Input the entity representation into the entity recognition module, classify each entity, obtain the probability distribution of the entity type, and calculate the multi-classification cross entropy loss L based on this probability distribution. ET ; S103: Input the entity pair representation into the evidence extraction module, perform probability calculation on the entity pair triple evidence set, extract evidence of the existence of the relationship, and calculate the focus loss L of the binary cross entropy based on the predicted evidence probability. ER ; S104: Input the entity pair representation into the relationship recognition module, classify the relationship of each entity pair, obtain the probability distribution of the relationship type, and calculate the classification cross entropy loss L based on this probability distribution. RE ; S105: Multi-classification cross entropy loss L ET , the focal loss L of binary cross entropy ER and categorical cross entropy loss L RE The overall loss L(T) of the teacher model is combined through weighted summation, and the teacher model is iteratively optimized by minimizing the overall loss L(T).

5. The document-level relationship extraction method based on multi-task learning and knowledge distillation according to claim 4 is characterized in that: The overall loss of the teacher model during training is expressed as: L(T)=L RE +αL ET +βL ER Among them, L(T) represents the overall loss of the teacher model, L ET represents the multi-classification cross entropy loss, L RE represents the classification cross entropy loss, L ER represents the focal loss of binary cross entropy, α represents the first hyperparameter, and β represents the second hyperparameter.

6. The document-level relationship extraction method based on multi-task learning and knowledge distillation according to claim 4 is characterized in that: In S101, the manually annotated document data is encoded using a feature extraction module to extract character representations, and a mention representation is generated based on the character representations, which is specifically: S201: Preprocessing the input document data, specifically including: summarizing the beginning and end of each sentence s in the document data d, inserting a classifier token "[CLS]" and a separator "[SEP]" at the beginning and end of each sentence respectively; marking an entity "*" on each entity mention package m, where the "*" indicates the location of the entity mention; S202: Input the preprocessed document data into BERT, and after encoding, output characters and entity types; S203: Input the characters and entity types output by BERT into the Transformer encoder, and obtain entity representation and mention representation after encoding processing. Each entity representation is obtained by the embedded combination of the "*" mark before the entity mention and the "[CLS]" mark at the beginning of the sentence, and each mention representation is obtained by the embedded combination of the "*" mark in the sentence and the "[CLS]" mark at the beginning of the sentence.

7. The document-level relationship extraction method based on multi-task learning and knowledge distillation according to claim 4 is characterized in that: The entity type probability is specifically expressed as: In the formula, represents the entity type probability output by the entity recognition module, W c Represents the weight matrix for entity type classification, Wc∈R |C|×H , R represents the real number space, |C| represents the number of entity types, that is, the number of possible entity types, and H represents the entity representation e v The dimension, e v represents the representation vector of any entity, tanh(·) represents the tanh function, δ(·) represents the softmax function, b c Represents the bias vector for entity type classification, and bc∈R |C| , b c ∈R |C| Indicates b c The dimension is consistent with the number of categories.

8. The document-level relationship extraction method based on multi-task learning and knowledge distillation according to claim 4 is characterized in that: The probability of the triple evidence set of the entity pair is specifically expressed as: In the formula, represents the probability of the triple evidence set of the entity pair, that is, the probability that the model judges whether there is evidence for the entity pair (h, t, s). h represents the head entity, that is, the subject entity of the relationship in the triple. t represents the tail entity, that is, the object entity of the relationship in the triple. s represents the sentence or evidence sentence of the triple, that is, the specific sentence that may contain the relationship between the head entity h and the tail entity t. σ(·) represents the sigmoid function. Represents an entity pair (e h ,e t ) is the transpose of the context embedding at position k, which contains the contextual information of the specific position of the entity pair in the document, represents the weight matrix at position k, s k represents the feature representation of sentence s at position k, b p Represents the bias term.

9. The document-level relationship extraction method based on multi-task learning and knowledge distillation according to claim 4 is characterized in that: The relationship probability of the entity pair is specifically expressed as: In the formula, Represents the relationship probability of entity pairs, that is, entity pairs (e h ,e t ) between them, e h Indicates the head entity, e t represents the tail entity, h represents the head entity, i.e. the subject entity in the triple relationship, t represents the tail entity, i.e. the object entity in the triple relationship, and r represents the relationship type, i.e. the entity pair (e h ,e t ), represents the transpose of the context-type fusion representation of the head entity h at layer k, represented by the embedding representation e of the head entity h Combined with the head entity features extracted from the context, represents the relationship weight matrix, represents the context-type fusion representation of the tail entity t at the kth layer, and represents the embedded representation e of the tail entity t Combined with the tail entity features extracted from the context, b r Denotes the bias term.

10. The document-level relationship extraction method based on multi-task learning and knowledge distillation according to claim 1, characterized in that: The trained teacher model is used to complete the document data without labeled evidence. The process includes: S301: Obtain document data of unlabeled evidence, which contains entity pairs and corresponding relationship labels, but lacks clear evidence sentence information; S302: Input each document in the document data of unlabeled evidence into the feature extraction module of the teacher model, which preliminarily encodes the document data through the BERT network and the Transformer encoder to generate document-level and entity-level context representations; S303: Generate each entity pair (e according to the context representation at the document level and entity level) h ,e t ), which includes the global information of the entity pair in the document; S304: Screen the sentences in each document and construct a candidate sentence set S d , the evidence candidate set S d Each sentence s in includes entity pairs (e h ,e t ) related context information; S305: The evidence candidate sentence set S d Each sentence s in is combined with the entity pair context embedding, and the probability of each sentence s becoming evidence is calculated using the evidence extraction module of the teacher model. S306: For each sentence in the evidence candidate sentence set, the probability Sort and select the sentence with the highest probability as the evidence sentence for the entity pair, and complete this information in the document data without labeled evidence; S307: Through steps S301-S306, document data containing evidence information is generated, in which each data record includes both entity pairs and their relationship labels, and evidence sentences completed by the teacher model.

Images