Incremental relation extraction method based on knowledge distillation

Abstract

The application discloses an incremental relation extraction method based on knowledge distillation, and comprises the following steps: constructing an incremental relation extraction model comprising a teacher model and a student model, wherein the teacher model is used for identifying old relation categories; the student model is a clone of the teacher model and is extended with a classification layer, and is used for identifying old relation categories and new relation categories; training the incremental relation extraction model based on a data set and a loss function; in the training process, the teacher model takes old relation category data as input; the student model takes new relation category data as input and fuses the old relation categories output by the teacher model in the training process; and the trained student model is used for incremental relation extraction on input sentences. The application can well retain the knowledge of old relation categories previously learned by the model while learning new relation categories.

Description

Technical Field

[0001] This invention relates to the field of relation extraction technology, and more particularly to an incremental relation extraction method based on knowledge distillation. Background Technology

[0002] In the era of big data, online information is exploding, with most of it stored in unstructured text form. Information extraction is a crucial task in natural language processing, widely applied in knowledge graph construction, recommender systems, and more. Relation extraction is a particularly important subtask within information extraction, aiming to identify the relationships between paired entities in text and automatically extract structured factual knowledge from text data. Compared to traditional methods that rely on manually designed features, deep learning-based relation extraction methods have achieved significant results.

[0003] In many real-world scenarios, new relation categories are frequently introduced, requiring retraining of the relation extraction model to support these new categories. Directly learning new relation categories on a pre-trained model leads to the loss of information about older relation categories, resulting in a decline in performance for recognizing them. The simplest solution is to update the annotation specification so that the predefined relation set covers the new relation categories, re-annotate the training data, then expand the relation extraction model's output layer to support learning the new categories, and finally retrain the model on the modified model using the re-annotated data. However, re-annotating training data for new relation categories is prohibitively costly, and annotating a new dataset for all relation categories becomes impractical and error-prone due to the increasing number of relation categories.

[0004] Therefore, more flexible strategies are needed to handle the ever-increasing number of new relation categories in real-world scenarios. Currently, memory-based methods, such as storing some old relation category data and replaying it when learning new relation categories, have proven to be effective in solving incremental relation extraction. However, memory limitations result in a limited amount of data that can be stored. As the number of relation categories increases, the imbalance between old and new relation category data becomes increasingly prominent. In this situation, the model may tend to predict sample data as new relation categories during training; at the same time, the model is prone to overfitting the small amount of stored old relation category data, which may gradually lead to unavoidable confusion between existing relations. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes an incremental relation extraction method based on knowledge distillation. This incremental relation extraction method can effectively retain the knowledge of previously learned old relation categories while learning new relation categories.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] The incremental relation extraction method based on knowledge distillation includes the following steps:

[0008] An incremental relation extraction model is constructed, which includes a teacher model and a student model. The teacher model is used to identify old relation categories, and the student model is a clone of the teacher model with an extended classification layer, used to identify old relation categories and new relation categories.

[0009] The incremental relation extraction model is trained based on the dataset and loss function. During the training process, the teacher model takes the old relation category data as input, and the student model takes the new relation category data as input and integrates the old relation categories output by the teacher model during the training process.

[0010] An incremental relation extraction model trained on the input sentence is used to extract incremental relations.

[0011] Preferably, the method further includes the following steps:

[0012] Create an empty list to store a predetermined number of old relation category data;

[0013] Extract a portion of data from the new relation category data, wherein the amount of extracted data is the same as the amount of old relation category data stored in advance;

[0014] The incremental relation extraction model is fine-tuned using equal amounts of new relation category data and old relation category data at a low learning rate.

[0015] Preferably, the dataset is divided into several non-overlapping relation clusters using a relation-level fully random sampling strategy, with each relation cluster containing only labeled sample instances of relation categories within the cluster.

[0016] Preferably, the number of relation clusters is consistent with the number of incremental learning steps.

[0017] Preferably, the loss function Loss of the incremental relation extraction model is:

[0018] Loss=αL KD +βL CE ,

[0019] Where α and β are the loss weights, L KD L represents the entity information distillation loss between the teacher and student models. CE Cross-entropy loss during student model training.

[0020] Preferably, the distillation loss L KD Defined as:

[0021]

[0022] Where, q i It is the predicted vector of the old relation category output by the teacher model output layer after smoothing with temperature T and then applying softmax. i The output layer of the student model is a predicted vector of the old relation categories after being smoothed by temperature T and then passed through softmax, where u is the number of old relation categories.

[0023] Preferably, the cross-entropy loss L CE Defined as:

[0024]

[0025] Where, p i y is the predicted vector of the new relation category output by the softmax layer of the student model output layer. i The one-hot representation of the new relationship category label, where v is the number of new relationship categories.

[0026] Preferably, the incremental relation extraction model includes an embedding layer, a Bi-LSTM layer, and a classification layer.

[0027] Based on the above technical solution, the beneficial effects of the present invention are:

[0028] 1. This invention constructs an incremental relation extraction model based on knowledge distillation. The relation extraction model trained using old relation categories is considered the teacher model, while the model incrementally learning new relation categories is considered the student model. The student model is encouraged to simulate the output of the teacher model, enabling it to retain some knowledge of old relation categories while learning new ones. Each time a new relation category is learned, only the labeled data for the new relation and a small portion of the stored old relation category data are used. The soft labels output by the teacher model guide the student model to retain knowledge of old relation categories, while the hard labels guide the student model to learn the new relation category. This reduces the "catastrophic forgetting" problem that occurs when the model incrementally learns new relation category data.

[0029] 2. This invention extracts a portion of data from the new relation category data, with the data size matching the amount of stored old relation category data. This equal amount of new and old relation category data is then used to fine-tune the model at a low learning rate. Fine-tuning the model with equal amounts of new and old relation category data effectively avoids the problem of the model tending to predict sample data as the new relation category during training due to an imbalance in the new and old relation category data. Attached Figure Description

[0030] Figure 1 This is a flowchart of an incremental relation extraction method based on knowledge distillation in one embodiment;

[0031] Figure 2 This is an architecture diagram of an incremental relation extraction model in one embodiment. Detailed Implementation

[0032] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0033] like Figure 1 As shown, this embodiment provides an incremental relation extraction method based on knowledge distillation, including the following steps:

[0034] Step S1: Construct an incremental relation extraction model that includes a teacher model and a student model. The teacher model is used to identify old relation categories, and the student model is a clone of the teacher model with an extended classification layer, used to identify old relation categories and new relation categories.

[0035] Step S2: The incremental relation extraction model is trained based on the dataset and loss function. During the training process, the teacher model takes the old relation category data as input; the student model takes the data labeled with the new relation categories as input and integrates the old relation categories output by the teacher model during the training process.

[0036] Step S3: Use the trained incremental relation extraction model to perform incremental relation extraction on the input sentence.

[0037] This invention constructs an incremental relation extraction model based on knowledge distillation. The relation extraction model trained using old relation categories is considered the teacher model, while the model incrementally learning new relation categories is considered the student model. The student model is encouraged to mimic the output of the teacher model, enabling it to retain some knowledge of old relation categories while learning new ones. This reduces the "catastrophic forgetting" problem that occurs when the model incrementally learns new relation category data.

[0038] The specific explanation is as follows:

[0039] 1. Construct an incremental relation extraction model

[0040] The core idea of ​​knowledge distillation is to use a model with a complex network structure but high learning and generalization abilities as the teacher model, and a model with a simple network structure but lower learning and generalization abilities as the student model. In the incremental learning setting of this patent, the relation extraction model trained using old relation category data is regarded as the teacher model, and the relation extraction model that incrementally learns new relation categories is regarded as the student model. For example... Figure 2 As shown, the student model is a clone of the teacher model, with an extended new output layer for recognizing new relation categories.

[0041] When the student model learns a new relation category, the training data contains sentences with the new relation category and some stored old relation categories. The sentences are passed to the teacher model and the student model respectively. The teacher model outputs soft targets containing semantic information of the old relation categories. The soft targets are used to guide the student model to retain the knowledge of the old relation categories. The hard targets are the labels of the new relation categories, which are used to guide the student model to learn the knowledge of the new relation categories.

[0042] The incremental relation extraction model consists of three parts: an embedding layer, a Bi-LSTM layer, and a classification layer.

[0043] 1) Embedding layer

[0044] For a given sentence W = {w1, w2, ..., w n The embedding layer, consisting of the labeled entity pairs e = [e1, e2], is used to implement vector transformation of natural language text. Within each sentence, additional special markers are inserted to highlight the position of entity pairs and their entity types, and each character w in the input sentence is used to represent these entities. i The mapping is done to word vectors. The sentence output after the embedding layer mapping is a word embedding matrix, which serves as the model's input. The model's input is defined as matrix X: X = (x1, x2, ..., x...). n ) T ∈R n×m In the formula, n is the sentence length; m is the dimension of the mapped vector.

[0045] 2) Bi-LSTM layer

[0046] LSTM is a variant of recurrent neural network that preserves contextual information during data serialization. Through gating structures and memory cells, it can selectively save and discard contextual information. Bi-LSTM combines forward and backward LSTM, making fuller use of sentence context and enriching semantic features. Bi-LSTM performs forward and backward computation on the input sequence simultaneously, concatenating the outputs to obtain the final hidden state as the output of the Bi-LSTM layer.

[0047] The Bi-LSTM encoding layer receives the word embedding matrix X as input and generates the forward hidden layer representation of each token segment. and backward hidden layer representation The forward and backward hidden layer representations are concatenated to obtain the feature representation of each word, including temporal features, syntactic structure, and semantic information.

[0048] 3) Classification layer

[0049] In the incremental relation extraction model, the classification layer passes the output features of the Bi-LSTM layer through a fully connected network. For the teacher model, the features are first smoothed by temperature T and then passed through a softmax function to output a u-dimensional vector, where u is the number of old relation categories. For the student model, the old relation category output layer is first smoothed by temperature T and then passed through a softmax function to output a u-dimensional vector. The new relation category output layer is passed through a softmax function to output a v-dimensional vector, where v is the number of new relation categories.

[0050] 2. Definition of loss function

[0051] During incremental training, the student model observes sentences labeled only for the new relation categories and partially for the stored old relation categories. To obtain the student model's soft target labels, each sentence is also fed into the teacher model. To reduce the student model's forgetting of old relation categories, it is trained by minimizing the distillation loss between the teacher and student model output distributions; for learning new relation categories, it is trained by minimizing the cross-entropy loss between the student model's output distribution and the true labels.

[0052] Distillation loss is defined as Where q i It is the predicted vector of the old relation category output by the teacher model output layer after smoothing with temperature T and then applying softmax. i The student model output layer is smoothed using temperature T and then softmaxed to output the predicted vector of the old relation categories, where u is the number of old relation categories. The smoothing probability formula is: If T = 1, then the formula is the softmax function; if T is close to 0, then q i The maximum value in T will be close to 1, and other values ​​will be close to 0, similar to one-hot encoding; if T is larger, the output will be smoother, and the teacher model will retain more similar information of the old relation categories.

[0053] Cross-entropy loss is defined as Where, p i y is the predicted vector of the new relation category output by the softmax layer of the student model output layer. i The one-hot representation of the new relationship category label, where v is the number of new relationship categories.

[0054] By training with a loss function that minimizes the sum of two weighted sums, the student model can learn new relation categories while simultaneously learning knowledge of old relation categories from the teacher model. The total loss is: Loss = αL KD +βL CE, where α and β are loss weights used to adjust the weight of the two loss functions.

[0055] 3. Dataset partitioning and model initialization

[0056] 1) Dataset partitioning:

[0057] To enable the model to incrementally learn the relation categories in the dataset, the dataset must first be partitioned. A completely random sampling strategy at the relation level is used to partition the data. If the number of incremental learning steps is s, then the dataset R is divided into s disjoint relation clusters R1, R2, ..., R... s Each relation cluster contains only labeled sample instances of relation categories within the cluster.

[0058] 2) Model initialization:

[0059] Initialize the teacher model parameters θ0. Use pre-trained GloVe word vectors to initialize the weights of the embedding layer. To avoid consuming computational resources by updating word vectors during training, the embedding layer parameters are set to be non-trainable. Initialize an empty list Memory_Samples to store old relation category data, with the number of samples equal to the set memory size. For the Kth relation cluster, the test dataset used is... The old relationship category is The new relationship category is The student model is initialized to None.

[0060] 4. Train the incremental relation extraction model

[0061] For the first relation cluster R1, sentences from each training batch are iteratively extracted, and an initial model M1 is trained on R1. The trained M1 is cloned, and a new classification layer is extended to serve as the student model M2. M1 then acts as the teacher model, guiding M2 to learn new relation categories on relation cluster R2 while retaining knowledge of the old relation categories.

[0062] For not the first relation cluster R i The sentence is retrieved from each training batch in a loop, and then processed by the teacher model M. i-1 And student model M i M i-1 The output prediction vector serves as the guide M i Softtarget tags for learning old relational category knowledge, M i The output old relation category prediction vector is used to calculate the distillation loss, M. i The output new relation category prediction vector and R i The cross-entropy loss is calculated using the labeled ground truth labels. M is trained with the goal of minimizing both the defined distillation loss function and the cross-entropy loss function.i .

[0063] In incremental step t (training relation cluster R) t The old relationship category is The new relationship category is R. t , will The trained model M serves as the teacher. t-1 Student Model M t For M t-1 Based on the clone, a new classification layer is extended with dimensions h*m, where h is the dimension of the sentence semantic feature vector output by the Bi-LSTM layer, and m is the R... t The number of new relation categories. The specific training process of incremental step t is as follows:

[0064] Step 1: Text Vectorization

[0065] For each sentence in the training batch, W = {w1, w2, ..., w...} n First, an embedding layer is used to embed each w in the sentence. i This is mapped to a 300-dimensional vector representation, where w i A word is the smallest unit after a sentence has been segmented; in Chinese, it is a character, and in English, it is a word.

[0066] Step 2: Extraction of sentence semantic features

[0067] The encoded sentence input is then entered into M. t-1 and M t The Bi-LSTM layer in the process is used for semantic feature extraction. The bidirectional LSTM network calculates the forward and backward hidden layer representations respectively, and then concatenates them as the final output of the Bi-LSTM layer.

[0068] Step 3: Relationship Classification

[0069] M t The output features of the Bi-LSTM layer are passed through a fully connected network (FC layer), smoothed by temperature T, and then subjected to a softmax operation to obtain the prediction vector; M t-1 After the output features of the Bi-LSTM layer pass through the fully connected network (FCLayer), the old relation category output layer is first smoothed by temperature T and then a softmax operation is performed to obtain the prediction vector, while the extended new relation category output layer directly performs a softmax operation to obtain the prediction vector.

[0070] Step 4: Train the student model

[0071] M t-1 The output prediction vector serves as the guide M tSoft target tags for learning old relational category knowledge, M t The output old relation category prediction vector is used to calculate the distillation loss, M. t The output new relation category prediction vector and R t The cross-entropy loss is calculated using the labeled hard target tags, and M is trained with the goal of minimizing the defined distillation loss function and cross-entropy loss function. t .

[0072] Step 5: Fine-tune the student model

[0073] From relation cluster R t Extract a portion of the new relation category data, with the data size matching the number of stored old relation category data. Use an equal amount of new and old relation category data to fine-tune the model at a low learning rate.

[0074] Step 6: Model Cloning and Expansion

[0075] After training is complete, M t Cloning and expanding the new classification layer serves as a student model M for learning new relational categories. t+1 M t Then, as a teacher model, guide M t+1 In relation cluster R t+1 Training.

[0076] Step 7: Test the model

[0077] The current accuracy of the test model on the test data of the current relation cluster and the incremental accuracy on the test data of all relation clusters learned incrementally before.

[0078] The model training ends when the relation cluster R has been traversed.

[0079] The above description is merely a preferred embodiment of the incremental relation extraction method based on knowledge distillation disclosed in this invention, and is not intended to limit the scope of protection of the embodiments in this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments in this specification should be included within the scope of protection of the embodiments in this specification.

Claims

1. An incremental relation extraction method based on knowledge distillation, characterized in that, The method comprises the following steps: An incremental relation extraction model comprising a teacher model and a student model is constructed, wherein the teacher model is used to identify old relation categories; the student model is a clone of the teacher model and extends a classification layer to identify old relation categories and new relation categories; The incremental relation extraction model is trained based on a data set and a loss function, and in the training process, the teacher model takes old relation category data as input; the student model takes new relation category data as input and fuses the old relation categories output by the teacher model in the training process; Incremental relation extraction is performed on an input sentence by using the trained incremental relation extraction model, The method further comprises the following steps: An empty list is established to store a predetermined amount of old relation category data; Part of the new relation category data is taken out, and the amount of the taken-out part of the data is consistent with the amount of the pre-stored old relation category data; The same amount of new relation category data and old relation category data is used to fine-tune the incremental relation extraction model at a low learning rate, The loss function Loss of the incremental relation extraction model is: Loss = aL + βL KD + βL CE , wherein a and b are loss weights, L KD is the entity information distillation loss between the teacher model and the student model, L CE is the cross-entropy loss during the student model training process, The distillation loss L KD is defined as: where q i is the prediction vector of old relationship classes output by the teacher model output layer after smoothing using temperature T and passing through softmax, s i is the prediction vector of old relationship classes output by the student model output layer after smoothing using temperature T and passing through softmax, and u is the number of old relationship classes. The cross-entropy loss L CE is defined as: where p i is the prediction vector of new relation class output by the student model output layer after softmax, y i is the one-hot representation of the new relation class label, and v is the number of new relation classes. 2.The knowledge-distillation-based incremental relation extraction method of claim 1, wherein, The data set adopts a complete random sampling strategy at a relation level to divide the data to form a plurality of disjoint relation clusters, and each relation cluster only contains labeled sample instances of intra-cluster relation categories. 3.The knowledge-distillation-based incremental relation extraction method of claim 2, wherein, The number of the relation clusters is consistent with the number of steps of incremental learning. 4.The knowledge-distillation-based incremental relation extraction method of claim 1, wherein, The incremental relation extraction model comprises an embedding layer, a Bi-LSTM layer and a classification layer.

Images