Relationship extraction model construction method and system based on semantic enhancement and knowledge distillation

Abstract

The invention provides a relation extraction model construction method and system based on semantic enhancement and knowledge distillation, relates to the technical field of relation extraction, and aims to solve the problem that in the prior art, when continuous learning is processed, a model is often difficult to keep memorizing an old relation when a new relation is learned, and forgetting is caused. Model overfitting is easily caused by excessive dependence on a small amount of data, so that a new relationship cannot be effectively generalized. The method comprises the steps of obtaining a relationship prototype of a training sample and a typical sample, performing semantic enhancement on the relationship prototype to obtain an expanded enhanced data set and an enhanced memory set, and performing feature extraction, relationship classification prediction and relationship extraction based on a student model and a multi-teacher model. According to the method, the overfitting problem caused by scarcity of small sample data is effectively relieved through a semantic enhancement technology, old relation forgetting caused by new relation learning in the continuous learning process is reduced through a multi-teacher knowledge distillation method, and the classification accuracy and generalization ability of the model facing different relations are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of relationship extraction, and in particular to a method and system for constructing a relationship extraction model based on semantic enhancement and knowledge distillation. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In the field of natural language processing (NLP), relation extraction (RE) is a key task to identify semantic relations between entities from text, and is widely used in downstream tasks such as language understanding and knowledge graph construction. However, traditional relation extraction methods usually rely on a large amount of labeled data and assume that all relation types are known during the training phase, which has limitations in practical applications.

[0004] In many practical application scenarios, such as finance, law, and medicine, new entity relationships continue to emerge, while the relevant annotated data is often extremely limited. Taking the medical field as an example, the information in medical literature and electronic medical records is usually unstructured, including patients' medical records, symptom descriptions, diagnosis results, etc. The potential disease relationships and medical knowledge in this information are of great significance to improving the accuracy of disease diagnosis and the formulation of treatment plans. For example, when identifying emerging diseases and treatment plans, doctors and researchers need to extract relevant relationships from a large amount of literature. However, due to the continuous emergence of new diseases and relationships, traditional relationship extraction methods often cannot adapt to this change, and in the absence of sufficient annotated data, the performance of the model is not satisfactory.

[0005] At present, the mainstream methods in the field of relationship extraction mainly include rule-based methods, machine learning-based methods, and deep learning-based methods. Although rule-based methods have certain interpretability, they rely on manually designed rules, making it difficult to cover all language phenomena and have poor adaptability. Machine learning-based methods automatically identify the relationship between entities by learning features. Although they are suitable for processing large-scale data sets, they are not flexible enough for new relationship types, feature engineering is complex, and they often perform poorly when processing newly emerging relationships. Deep learning-based methods can automatically learn relationship features from data, but usually require a large number of labeled samples to train the model. Their performance is limited under small sample conditions, especially when faced with ever-changing types of relationships, and they are prone to catastrophic forgetting.

[0006] Therefore, as the data environment continues to change and the types of relationships gradually expand, real-world application scenarios require methods that can continuously learn new relationships. This means that the method not only needs to learn newly emerging relationships with a small number of samples, but also needs to retain the knowledge of learned relationships when introducing new relationships. To this end, continuous small-sample relationship extraction came into being, aiming to continuously train the model to identify new relationships with a small amount of labeled data, while avoiding catastrophic forgetting and overfitting problems caused by data sparsity.

[0007] However, in the field of continuous small-sample relation extraction (CFRE), although existing technical methods have made certain progress, there are still many key problems and shortcomings. When dealing with continuous learning, the existing technology often finds it difficult to keep the memory of old relations when learning new relations, resulting in catastrophic forgetting. In addition, excessive reliance on a small amount of data can easily lead to overfitting of the model, making it unable to effectively generalize to new relations. This is because the existing technology lacks effective data enhancement methods and cannot make full use of limited data resources. Existing studies have proven the effectiveness of knowledge distillation in transferring knowledge, but many existing methods only rely on a single teacher model to guide the learning of the student model. This approach can easily lead to insufficient knowledge transfer, and the limitations of the teacher model itself may be transferred to the student model, affecting the overall learning effect. Summary of the invention

[0008] In order to overcome the deficiencies of the above-mentioned prior art, the present invention provides a method and system for constructing relation extraction based on semantic enhancement and knowledge distillation. Combining semantic enhancement and multi-teacher knowledge distillation methods, the overfitting problem under small sample conditions is alleviated by enriching training data through semantic enhancement; the phenomenon of forgetting old relations caused by learning new relations in the continuous learning process is reduced through the multi-teacher knowledge distillation method; the gradient optimization temperature adjustment mechanism is adopted to improve the knowledge fusion effect, so that the classification accuracy and generalization ability of the model when facing different relation types are significantly improved.

[0009] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0010] In a first aspect, a method for constructing a relationship extraction model based on semantic enhancement and knowledge distillation is disclosed, comprising:

[0011] Acquire training data, process the training data to obtain typical samples, obtain relationship prototypes of typical samples based on the typical samples, and obtain a memory set based on the typical samples and the relationship prototypes;

[0012] Perform data augmentation on the training data and memory set to obtain the expanded augmented data set and augmented memory set, which are mixed and input into the teacher model and the student model.

[0013] The student model extracts features from the mixed data to obtain entity feature representation of the student model, and obtains the hidden representation of the student model based on the entity feature representation;

[0014] The teacher model extracts features from the mixed data to obtain the output feature representation of each teacher model, and obtains the hidden representation of each teacher model based on the output feature representation;

[0015] Relation classification is performed on the hidden representations of the student model and each teacher model, and the prediction probabilities of the student model and each teacher model are obtained respectively.

[0016] As a further technical solution, the specific process of processing the training data to obtain typical samples is as follows:

[0017] Encode the training data through the encoder to obtain entity feature representation;

[0018] According to the entity feature representation, a hidden representation is obtained;

[0019] Cluster the hidden representations to obtain typical samples;

[0020] As a further technical solution, the data enhancement of the training data and the memory set to obtain the expanded enhanced data set and enhanced memory set is specifically:

[0021] Both the training data and the memory set are augmented by replacing similar semantic entities, expanding vocabulary variants and making random noises, thus obtaining the expanded enhanced data set and enhanced memory set respectively.

[0022] As a further technical solution, based on the entity feature representation of the student model and the output feature representation of the teacher model, a feature distillation loss is constructed, specifically:

[0023] Based on the entity feature representation of the student model and the output feature representation of the teacher model, the cosine embedding loss between each teacher model and the student model is constructed respectively:

[0024] The feature distillation loss is constructed based on the cosine embedding loss between each teacher model and the student model.

[0025] As a further technical solution, based on the hidden representation of the student model and the hidden representation of each teacher model, a hidden representation distillation loss is constructed, and the formula is:

[0026]

[0027] in, is a hyperparameter, represents the hidden representation obtained by the student model, and Represent the hidden representations obtained by the two teacher models, x represents the sample in the enhanced memory set, It is represented as an enhanced memory set, which is a set of typical samples of the current task.

[0028] As a further technical solution, a prediction distillation loss is constructed based on the prediction probabilities of the student model and each teacher model, and the formula is:

[0029]

[0030] Among them, α and β are hyperparameters used to balance the losses of the two teacher models. and are the predicted probabilities of the two teacher models for the i-th category, is the predicted probability of the student model for the i-th category, x represents the sample in the enhanced memory set, It is represented as an enhanced memory set, which is a set of typical samples of the current task.

[0031] As a further technical solution, based on the feature representation distillation loss, the hidden representation distillation loss, and the prediction distillation loss, a total loss is constructed, and the formula is:

[0032] L=λ1L cls +λ2L m_fdl +λ3L hrdl +λ4L tri +λ5L pred ;

[0033] Among them, λ i is the adjustment coefficient of each process; L m_fdl represents the multi-teacher feature distillation loss, L hrdl denotes the multi-teacher hidden representation distillation loss, L pred represents the multi-teacher prediction distillation loss, L cls represents the student model relation classification loss, represents the training data set for the kth task, R k represents the relationship set of the kth task; y r,x ∈{0, 1} indicates whether the true label of x is r. If so, it is 1, otherwise it is 0. P(r|x) represents the probability that the sample x belongs to relation r calculated by the classifier.

[0034] In the second aspect, a relation extraction model construction system based on semantic enhancement and knowledge distillation is disclosed, including:

[0035] A data acquisition module is used to acquire training data, process the training data to obtain typical samples, obtain relationship prototypes of typical samples based on the typical samples, and obtain a memory set based on the typical samples and the relationship prototypes;

[0036] The data enhancement module is used to perform data enhancement on the training data and memory set, and obtain the expanded enhanced data set and enhanced memory set respectively, which are mixed and input into the teacher model and the student model;

[0037] The feature extraction module is used for the student model to extract features from the mixed data set to obtain the entity feature representation of the student model, and the hidden representation of the student model is obtained based on the entity feature representation. The teacher model extracts features from the mixed data to obtain the output feature representation of each teacher model, and the hidden representation of each teacher model is obtained based on the output feature representation.

[0038] The relation classification module is used to perform relation classification on the hidden representations of the student model and each teacher model, and obtain the prediction probabilities of the student model and each teacher model respectively.

[0039] The third aspect of the present invention is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0040] A fourth aspect of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, which executes the steps of the above method when executed by a processor.

[0041] One or more of the above technical solutions have the following beneficial effects:

[0042] In this embodiment, by introducing semantic enhancement technology, the overfitting problem caused by data scarcity in small sample relationship extraction tasks is effectively alleviated; in addition, through the multi-teacher knowledge distillation method, the phenomenon of forgetting old relationships caused by learning new relationships in the continuous learning process is significantly reduced; the introduced gradient optimization temperature adjustment mechanism further enhances the effect of knowledge fusion, so that the classification accuracy and generalization ability of the model when facing different relationship types are significantly improved.

[0043] In this embodiment, not only the overall accuracy of the relationship extraction task is improved, but also the impact of uneven data distribution and insufficient samples on model performance is effectively alleviated through multi-teacher knowledge distillation and gradient optimization temperature adjustment mechanism, thereby enhancing the model's classification ability on a small number of categories. These improvements provide a more robust and efficient solution for handling complex and changeable relationship extraction tasks.

[0044] The performance of this paper on common datasets such as FewRel and TACRED has been significantly improved, providing a more reliable and efficient solution for handling complex relation extraction tasks. We use the classification accuracy of the relations learned by the model in all previous tasks as the evaluation criterion.

[0045] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0047] Figure 1 This is an overall architecture diagram of a method for building a relationship extraction model based on semantic enhancement and knowledge distillation in the first embodiment of the present invention;

[0048] Figure 2 This is a feature extraction flow chart in the first embodiment;

[0049] Figure 3 This is a flow chart of a method for building a relationship extraction model based on semantic enhancement and knowledge distillation in the first embodiment of the present invention. DETAILED DESCRIPTION

[0050] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0051] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0052] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0053] The overall idea proposed in this invention is: through semantic enhancement technology, the number of samples is increased to effectively alleviate the overfitting problem caused by data scarcity in small sample relationship extraction tasks; through the multi-teacher knowledge distillation method, the forgetting of old relationships caused by new relationship learning in the continuous learning process is reduced; through student models and multi-teacher models, the prediction of relationship classification is realized; through the gradient optimization temperature adjustment mechanism, knowledge fusion is improved, and the classification accuracy and generalization ability of the model when facing different relationships are improved.

[0054] Embodiment 1

[0055] This embodiment discloses a method for constructing a relationship extraction model based on semantic enhancement and knowledge distillation.

[0056] In order to more clearly illustrate this embodiment, the process of implementing relation extraction based on semantic enhancement and knowledge distillation can be specifically described as follows:

[0057] A method for constructing a relation extraction model based on semantic enhancement and knowledge distillation, comprising:

[0058] S1. Obtain training data, process the training data to obtain typical samples, obtain relationship prototypes of typical samples based on the typical samples, and obtain a memory set based on the typical samples and the relationship prototypes;

[0059] S2, perform data augmentation on the training data and memory set to obtain the expanded augmented data set and augmented memory set, which are mixed and input into the teacher model and the student model;

[0060] S3, the student model extracts features from the mixed data to obtain entity feature representation of the student model, and obtains the hidden representation of the student model based on the entity feature representation;

[0061] S4, the teacher model extracts features from the mixed data to obtain output feature representations of each teacher model, and obtains hidden representations of each teacher model based on the output feature representations;

[0062] S5. Perform relationship classification on the hidden representations of the student model and each teacher model, and obtain the prediction probabilities of the student model and each teacher model respectively.

[0063] In this embodiment, a large general English domain corpus is taken as an example.

[0064] like Figure 1 , Figure 3 As shown, in step S1, training data is acquired, the training data is processed to obtain typical samples, relationship prototypes of the typical samples are obtained based on the typical samples, and a memory set is obtained based on the typical samples and the relationship prototypes.

[0065] S101. Obtain training data.

[0066] The training samples y come from the commonly used datasets of FewRel and TACRED libraries.

[0067] S102: Obtain typical samples and generate relationship prototypes.

[0068] In order to reduce the degree to which the model forgets the relations in the previous tasks, when performing each new task, we select L typical samples for each relation r in the current task and store them in the memory Mem r middle.

[0069] S1021. Encode the training data through an encoder to obtain entity feature representation.

[0070] For task T j In the input training data y, we use the special tag [E 11 ,E 12 ] and [E 21 ,E 22 ] to mark the position of the head entity and the tail entity respectively. If the training data x contains two entities e1 and e2, we insert these tags in the sentence to highlight the position of the entity. For example, for the sentence "The capital of France is Paris.", after processing, it becomes "The capital of [E 11 ]France[E 12 ]is[E 21 ]Paris[E 22 Then, the processed text is input into the BERT model encoder to obtain the entity feature representation f y .

[0071] S1022. Obtain a hidden representation based on the entity feature representation.

[0072] According to the obtained entity feature representation f y , get the hidden representation, the formula is:

[0073] h y =LN(W·Dropout(f y )+b)(1)

[0074] in and are all trainable parameters, d represents the dimension of the hidden layer, h is the dimension of the BERT hidden representation, and LN(·) represents the layer normalization operation.

[0075] S1023. Cluster the hidden representations to obtain typical samples.

[0076] The K-means clustering algorithm is used on the sample representation processed by the pre-trained encoder to cluster the sample representation into L classes. For each cluster, a representative typical sample is selected as the representative of the cluster. Usually, the sample closest to the cluster center is selected as the typical sample, because these samples can best represent the characteristics of the cluster. Finally, the selected typical samples are stored in the memory Mem r .

[0077] S1024. Obtain a relationship prototype based on the selected typical samples.

[0078] Relation prototypes are used for training and optimization of relation extraction models. By providing a standard representation of each relation, the prototype can help the model more accurately identify and distinguish the relation types of new samples and reduce the forgetting of old relations in the CFRE task. In addition, relation prototypes also support memory enhancement mechanisms and the distinction of similar relations, thereby enhancing the overall performance and adaptability of the model. r The hidden representation of L typical samples is averaged and the formula is:

[0079]

[0080] After the above steps, data support can be provided for reducing the degree to which the model forgets the relationships in previous tasks, so that the relationship extraction model can be better trained and optimized.

[0081] like Figure 1 , Figure 3 As shown, in step S2, data enhancement is performed on the training data and the memory set to obtain an expanded enhanced data set and an enhanced memory set, which are mixed and input into the teacher model and the student model.

[0082] S201. For both training data and memory set, similar semantic entities are replaced to increase the number of samples for expanding the data set.

[0083] In order to effectively reduce the errors caused by similar classes, maintain the consistency of relationships and reduce overfitting, the richness of the dataset is increased by replacing similar input entity pairs based on the method of adaptive semantic entity replacement.

[0084] First, the BERT model is used to encode the text and generate entity representations for each training data.

[0085] In this embodiment, in step S1021, the entity feature representation f of the sample has been obtained. y .

[0086] Secondly, calculate the cosine similarity between any two different entities and construct a similarity matrix.

[0087] In this embodiment, the feature representation f of any two entities is obtained i and f j , calculate the cosine similarity S between the two entities ij , construct the similarity matrix, the formula is:

[0088]

[0089] Among them, f i represents the feature representation of the i-th entity, f j Represents the feature representation of the j-th entity.

[0090] Secondly, replaceable entity pairs are selected according to the threshold and the original samples are replaced.

[0091] According to the actual situation, given a threshold γ, γ is a specific value, we only need to compare the cosine similarity with the threshold. ij When it is higher than the γ threshold, that is, S ij >γ, the two entities f i and f j They have a high degree of similarity and can be replaced. For example:

[0092] "Google acquired YouTube in 2006."

[0093] "Facebook bought Instagramin 2012."

[0094] ===》"Google acquired Instagramin 2006."

[0095] The entities "YouTube" and "Instagram" are very similar, so "YouTube" can be replaced with "Instagram" to enrich the data set.

[0096] After the above steps, it is ensured that the replaced entity is semantically similar to the original entity so as not to significantly change the overall meaning of the sentence. By replacing entities, the model can handle different entity types and instances, improving the generalization ability of entities.

[0097] At the same time, the newly generated data is added to the original training data and the memory set as an enhanced data set and enhanced memory sets

[0098] Finally, the threshold is optimized and adjusted.

[0099] Different from the traditional threshold setting, the threshold is updated and adjusted based on the performance of the model on the validation set.

[0100] The threshold is not fixed, but changes with the performance of model training. When the model performance improves, the threshold can be increased accordingly to ensure that only the most similar entity pairs are retained; when the model performance decreases, the threshold can be appropriately lowered, that is, the threshold size is set according to the change of model performance, and the threshold allows more replacements to provide more diverse training samples.

[0101] Through the above steps, the threshold can be adaptively adjusted to ensure that the threshold is always optimal under different training stages and data distributions, and the data quality can be effectively controlled.

[0102] S202. Increase data through vocabulary variant expansion and random noise.

[0103] Lexical variant expansion is a data augmentation technique widely used in NLP tasks. It uses synonyms to replace certain words in the text to generate new text that is similar to the original text but slightly different. WordNet, the Princeton University synonym dictionary, is used as the source of synonyms. Unlike entity replacement, this method aims to enable the model to better understand and process different lexical expressions and improve the model's fine-grained semantic understanding capabilities.

[0104] Random noise is a method to increase the variability of training data by randomly inserting or deleting certain words in the text. This method does not rely on the semantic relationship of words, but introduces certain disturbances through random operations, generating more diverse training data and enhancing the model's ability to handle imperfect input data.

[0105] S203, filter the added data.

[0106] First, based on the cosine similarity matrix, only sample pairs with similarity exceeding the threshold are retained as enhancement candidates; second, redundant samples are avoided to ensure that the enhanced entity is different from the original entity; then, the length of the enhanced sample is ensured to meet the model requirements to avoid samples that are too short or too long; finally, the amount of enhanced data is limited to prevent excessive data expansion. These screening steps ensure the effectiveness and rationality of the enhanced data.

[0107] The generated data is organized and filtered, and any possible error instances are deleted, such as those that are the same as the original entity or whose length does not meet the model requirements, to obtain an effective expanded enhanced dataset D~ j and enhanced memory set M~ j .

[0108] After the above steps, we ensure that only valid and meaningful augmented data are retained, which can effectively expand the existing data set and memory set, introduce diversity, and ensure the accuracy of subsequent models in predicting input data, thereby reducing overfitting and improving the generalization ability of the model.

[0109] like Figure 1 , Figure 2 , Figure 3 As shown, in step S3, the student model extracts features from the mixed data to obtain an entity feature representation of the student model, and based on the entity feature representation, a hidden representation of the student model is obtained.

[0110] In this embodiment, the BERT model is selected as the student model.

[0111] S301, obtaining entity feature representation of mixed data.

[0112] Specifically, the BERT model is used as the student model. The BERT model mainly consists of an encoding layer, a hidden layer, and a task-specific output layer. BERT is an encoder language model based on a bidirectional transformer and is pre-trained on a large general English domain corpus.

[0113] When faced with a new relationship extraction task T j When the extended augmented dataset and enhanced memory sets Mixed as training data and input into the BERT model for a multi-classification task training.

[0114] For task T j The mixed training samples x in the , using special tags [E 11 ,E 12 ] and [E 21 ,E 22 ] to mark the position of the head entity and the tail entity respectively. If the sample x contains two entities e1 and e2, a special marker is inserted in the sentence to highlight the position of the entity.

[0115] For example, for the sentence "The capital of France is Paris.", after processing it becomes "The capital of [E 11 ]France[E 12 ]is[E 21 ]Paris[E 22 Then, the text is input into the BERT model encoder to obtain the entity feature representation f x S .

[0116] S302: Obtain hidden representation according to entity feature representation.

[0117] Acquisition-based entity feature representation , get the hidden representation of the mixed data sample x, the formula is:

[0118] h x S =LN(W·Dropout(f x S )+b)(4)

[0119] in and are all trainable parameters, d represents the dimension of the hidden layer, h is the dimension of the BERT hidden representation, LN(·) represents the layer normalization operation, Represents the entity feature representation of the mixed data sample x.

[0120] like Figure 1 , Figure 2 ,In step S4, the teacher model extracts features from the mixed data to obtain the output feature representation of each teacher model, and obtains the hidden representation of each teacher model based on the output feature representation.

[0121] In this embodiment, there are two teacher models, teacher model 1 adopts the BERT model, and teacher model 2 adopts the RoBERTa model. BERT is an encoder language model based on a bidirectional transformer, which is pre-trained on a large general English domain corpus. The RoBERTa model has the same architecture as the BERT model, but unlike BERT's static masked language modeling (MLM), RoBERTa uses dynamic masking and uses a different masking mode each time during the training process to enhance the generalization ability of the model. RoBERTa removes the next sentence prediction (NSP) task and is pre-trained on a training corpus that is ten times larger than BERT, further improving the performance of the model.

[0122] The selected BERT model and RoBERTa model have been trained with different data and tasks respectively, and have unique feature representations and knowledge. By integrating these different knowledge into the student model, its generalization ability and robustness are improved.

[0123] S401. Perform feature extraction on the mixed data based on the multi-teacher model to obtain output feature representations of the two teacher models.

[0124] The mixed augmented dataset and enhanced memory sets are input into teacher model 1 and teacher model 2 at the same time, that is, after the encoding layer of the BERT model and the RoBERTa model, the output feature representation f of the two models is obtained respectively. x T1 、f x T2 .

[0125] S402. Construct a feature distillation loss based on the entity feature representation of the student model and the output features of the multi-teacher model.

[0126] Since the teacher model usually has richer knowledge and higher representation ability, by minimizing the distance between the feature representation of the student model and the feature representation of the teacher model, the student model can be as close to the performance of the teacher model as possible, thereby improving its own feature extraction ability.

[0127] The cosine embedding loss is used to measure the similarity between the feature representations of the student model and the teacher model. The formula is:

[0128]

[0129] in, and Represent the output features of the student model and the teacher model of sample x, respectively.

[0130] x represents the sample in the enhanced memory set, It is represented as an enhanced memory set, which is a set of typical samples of the current task.

[0131] like Figure 1 , Figure 3 As shown, in this embodiment, teacher model 1 and teacher model 2 are used, and teacher model 1 outputs feature f x T1 With the student model entity feature representation f x S The similarity between them is calculated by using the cosine embedding loss to construct the feature distillation loss L between the teacher model 1 and the student model. cos1 , the formula is:

[0132]

[0133] Among them, f x T1 and f x S They represent the output features of the teacher model 1 and the entity feature representation of the student model of sample x respectively. x represents the sample in the enhanced memory set, It is represented as an enhanced memory set, which is a set of typical samples of the current task.

[0134] When the feature distillation loss L of teacher model 1 cos1 When minimizing, the entity feature representation of the student model is closest to the output feature representation of the teacher model 1. The student model can improve its own feature extraction ability based on the performance of the teacher model 1.

[0135] Teacher model 2 outputs feature f x T2 With the student model entity feature representation f x S The similarity between them is calculated by using the cosine embedding loss to construct the feature distillation loss L between the teacher model 2 and the student model. cos2 , the formula is:

[0136]

[0137] Among them, f x T2 and f x S They represent the output features of the teacher model 2 and the entity feature representation of the student model of sample x respectively. x represents the sample in the enhanced memory set, It is represented as an enhanced memory set, which is a set of typical samples of the current task.

[0138] When the feature distillation loss L of teacher model 2 cos2 When minimizing, the entity feature representation of the student model is closest to the output feature representation of the teacher model 2. The student model can improve its own feature extraction capability based on the performance of the teacher model 2.

[0139] When multiple teacher models are used at the same time, in order to make the entity feature representation of the student model more consistent with the output feature representation of the teacher model, it is necessary to integrate the feature distillation loss between the two teacher models and the student model to construct the feature distillation loss. The formula is:

[0140] L m_fdl =λL cos1 +(1-λ)L cos2 (8)

[0141] Among them, λ is a hyperparameter, L cos1 and L cos2 They are the feature distillation losses between teacher model 1, teacher model 2, and the student model.

[0142] When L cos1 and L cos2 When minimizing, L m_fdl At the minimum, the student model is closer to the teacher model and can learn richer knowledge and higher expression ability of the teacher model, which can optimize the entity feature representation of the student model, that is, the entity feature representation of the student model is optimal.

[0143] S403. Obtain a hidden representation of the teacher model based on the output features of the teacher model.

[0144] Input the output features of teacher model 1 into the dropout layer of teacher model 1 to obtain the hidden representation of teacher model 1 The formula is:

[0145]

[0146] Input the output features of teacher model 2 into the dropout layer of teacher model 2 to obtain the hidden representation of teacher model 2 The formula is:

[0147]

[0148] in, and are all trainable parameters, d represents the dimension of the hidden layer, h is the dimension of the BERT hidden representation, LN(·) represents the layer normalization operation, and Represent the output features of teacher model 1 and teacher model 2 respectively, and They represent the hidden representations obtained by teacher model 1 and teacher model 2 respectively.

[0149] After the above steps, the hidden representation of the multi-teacher model can be accurately obtained, providing effective data support for constructing operations such as hidden representation distillation loss.

[0150] S404. Construct a hidden representation distillation loss based on the hidden representation of the student model and the hidden representation of the teacher model.

[0151] In order to enable the student model to learn the hidden representation of the teacher model and make the hidden representation of the student model closer to the hidden representation of the teacher model, the cosine similarity is used to measure the student model so that it can be as close as possible to the performance of the teacher model, thereby improving its own hidden representation extraction ability.

[0152] Based on the hidden representation of the student model and the hidden representation of the teacher model, the cosine similarity between the student model and the teacher model is calculated to measure the similarity between the hidden representation of the student model and the teacher model. The formula is:

[0153]

[0154] in, and denote the hidden representations of the student model and the teacher model, respectively.

[0155] In this embodiment, teacher model 1 and teacher model 2 are used, and the hidden representation h of the student model is x S , the hidden representation of teacher model 1 Calculate the cosine similarity between the student model and the teacher model 1, the formula is:

[0156]

[0157] in, represents the hidden representation of the student model, Hidden representation of teacher model 1.

[0158] The larger the cosine similarity of teacher model 1, the closer the student model is to the teacher model, which enables the student model to better learn the hidden representation of teacher model 1.

[0159] Hidden representation h based on the student model x S , the hidden representation of teacher model 2 Calculate the cosine similarity between the student model and the teacher model 2, the formula is:

[0160]

[0161] in, represents the hidden representation of the student model, Hidden representation of teacher model 2.

[0162] The larger the cosine similarity of teacher model 2, the closer the student model is to the teacher model, which enables the student model to better learn the hidden representation of teacher model 2.

[0163] In order to make the hidden representation of the student model more consistent with the hidden representation of the teacher model, it is necessary to comprehensively consider the cosine similarity of teacher model 1 and teacher model 2.

[0164] Based on the cosine similarity, the hidden representation distillation loss L is obtained hrdl , to minimize the difference in hidden representation between the student model and the teacher model, the formula is:

[0165]

[0166] in, is a hyperparameter, represents the hidden representation obtained by the student model, and Represent the hidden representations obtained by the two teacher models, x represents the sample in the enhanced memory set, It is represented as an enhanced memory set, which is a set of typical samples of the current task.

[0167] When L hrdl When is the smallest, the difference in hidden representation between the student model and the teacher is the smallest, the student model is closer to the teacher model, and can learn richer knowledge and higher expression ability of the teacher model, which can optimize the hidden representation of the student model, that is, the hidden representation of the student model is optimal.

[0168] S405, construct triplet loss

[0169] In feature embedding learning, we hope that the model can map similar samples to nearby areas in the feature space, and map different samples to distant areas. If the model does not have a good distance constraint, it may generate low-quality feature embeddings, making similar samples unable to be correctly distinguished. Therefore, in this paper we introduce the triplet loss L tri , which is expressed as follows:

[0170]

[0171] Where N is the total number of triplets, d(a i ,p i ) and d(a i ,n i ) represent the distance between the benchmark sample and the positive sample and the negative sample respectively. δ is a hyperparameter used to control the distance between the benchmark sample and the negative sample. i represents the benchmark sample, p i ,n i represent positive samples and negative samples respectively.

[0172] The reference sample, positive sample and negative sample are determined according to the relation prototype of formula (2) and are determined by calculating the similarity between the relation prototype and the sample. The formula is:

[0173]

[0174] in, represents the hidden representation of the student model, P r Relation prototypes representing typical samples.

[0175] The anchor sample can be selected as a sample similar to a certain relation prototype. The positive samples are those samples under the same relation prototype as the anchor sample. The negative samples are samples that are dissimilar to the anchor sample and may correspond to different relation prototypes.

[0176] If the similarity value is greater than the set high threshold, it is a positive sample, and if the similarity value is less than the set low threshold, it is a negative sample.

[0177] By introducing triplet loss, the model can better learn the semantic distance between samples, so that sentences of the same relation category are embedded in adjacent areas in the feature space, while sentences of different relation categories are separated. This can enhance the model's ability to distinguish different relation categories, especially when some relation categories are very similar in the surface form of sentences, triplet loss can effectively increase their distance in the feature space.

[0178] In this embodiment, in the relation extraction task, the goal of the model is to predict the relationship between entity pairs in a sentence. For example, given a sentence "John works at Google", the model needs to predict that the relationship between the entities "John" and "Google" is "employee-company".

[0179] However, many sentences may look similar in surface structure but actually express different relationships, such as:

[0180] “John works at Google” → employee-company

[0181] “John visited Google” → Visitor-Company

[0182] The entities in both sentences are "John" and "Google", and the sentence structures are similar, but they express different relationships.

[0183] The triplet loss can solve the problem that many sentences are similar in structure but have different actual relationships by using distance constraints in the feature space. The purpose is to improve the model's ability to distinguish different relationship categories and improve the accuracy of prediction.

[0184] like Figure 1 , Figure 3 As shown, in step S5, the relationship classification is performed on the hidden representations of the student model and each teacher model, and the prediction probabilities of the student model and each teacher model are obtained respectively.

[0185] S501. Based on the student model, perform relationship classification on the hidden representation in the student model to obtain the prediction probability of the student model.

[0186] Use a linear softmax classifier for h x Classify and predict the relationship label. The relationship distribution probability of each sample x in the student model is as follows:

[0187]

[0188] in, and are all trainable parameters, W cls represents the weight matrix, b cls represents the bias term, R j represents the number of relationship categories, represents the hidden representation of the student model sample x.

[0189] According to the output probability distribution, the model will select the relationship with the highest probability. Finally, the cross entropy loss is used to calculate the relationship classification loss L of the current task.cls , the formula is:

[0190]

[0191] in, represents the training dataset of the kth task, R k represents the relationship set of the kth task; y r,x ∈{0, 1} indicates whether the true label of x is r. If so, it is 1, otherwise it is 0. P(r|x) represents the probability that the sample x belongs to relation r calculated by the classifier.

[0192] When L cls When the minimum, the true label y r,x ∈{0, 1} is closer to the predicted relationship distribution probability, the obtained relationship classification is more accurate, that is, the predicted classification label is more accurate and effective.

[0193] This cross entropy loss function is used to measure the difference between the probability distribution predicted by the model and the true label distribution. By minimizing the cross entropy loss, the model can better identify and classify different relationships, thereby improving the overall task performance.

[0194] S502: Based on the multi-teacher model, perform relationship classification on the hidden representation in each teacher model to obtain the prediction probability of each teacher model.

[0195] In this embodiment, two teacher models are used, and a linear softmax classifier is used to classify and Classify and predict the relationship labels. The relationship distribution probabilities of teacher model 1 and teacher model 2 are

[0196] P T1 =softmax(W cls h x T1 +b cls ) (19)

[0197] P T2 =softmax(W cls h x T2 +b cls ) (20)

[0198] Among them, P T1 , P T2 Represent the relationship distribution probabilities of teacher model 1 and teacher model 2 respectively, and are all trainable parameters, W cls represents the weight matrix, b clsrepresents the bias term, R j represents the number of relationship categories, h x T1 、h x T2 denote the hidden representations of teacher model 1 and teacher model 2 respectively.

[0199] According to the above formula, the relationship distribution probabilities of teacher model 1 and teacher model 2 can be obtained respectively.

[0200] S503. Adopting the gradient optimization temperature adjustment mechanism, the distribution probability of the student model and the multi-teacher model is optimized.

[0201] In order to ensure that the student model can not only learn the knowledge of the teacher model in the feature space and hidden layer, but also get guidance from the final prediction of the teacher model, we pay attention to the probability distribution of the student model and the teacher model in the output layer, hoping to guide the learning of the student model through the probability distribution of the teacher model, so that the student model can better understand the relative relationship between categories.

[0202] The softmax operation is performed on the original score logits output by the student model and the teacher model in the last layer to obtain the output probability distribution of the model. However, since the use of an exponential function will amplify the logits, resulting in a larger probability difference, we use temperature to scale the logits so that the probability distribution after softmax does not differ too much. The probability distribution formulas of the student model and the teacher model are:

[0203]

[0204] Where n is the number of categories, and Represent the logits values ​​of the i-th category of the student model and the teacher model respectively, and They represent the predicted probability distribution of the i-th category of the student model and the teacher model respectively, and T is the temperature parameter used to adjust the smoothness of softmax.

[0205] Formula (21) is the probability expression of formula (17) under the gradient optimization temperature adjustment mechanism. The optimized student model predicts the distribution probability. For example, if the prediction sentence "John works at Google" is input, the model will calculate the probability of each category according to the respective probability formula.

[0206] When there are three relationship categories:

[0207] A.Employee-Company

[0208] B. Customer-Company

[0209] C. Visitor-Company

[0210] The probability calculated using softmax:

[0211] P(y=employee-company)≈0.65

[0212] P(y=customer-company)≈0.24

[0213] P(y=visitor-company)≈0.11

[0214] According to the output probability distribution, the model will select the relationship with the highest probability, which is 0.65, employee-company, and predict the classification label as "employee-company".

[0215] S504. The multi-teacher model obtained based on the probability distribution of the student model and the multi-teacher model predicts the distillation loss.

[0216] For knowledge distillation of multiple teacher models, in order to make the prediction probability of the student model closer to the prediction probability of the teacher model, the prediction results of multiple teacher models need to be comprehensively considered.

[0217] Align the predictions of the student model with the two teacher models to construct the prediction distillation loss of the multi-teacher model, as follows:

[0218]

[0219] Among them, α and β are hyperparameters used to balance the losses of the two teacher models. and are the predicted probabilities of the two teacher models for the i-th category, x represents the sample in the enhanced memory set, It is represented as an enhanced memory set, which is a set of typical samples of the current task.

[0220] When L pred When it is minimized, the prediction probability of the student model is more consistent with the prediction of the teacher model, which means that the student model can learn richer knowledge and higher expression ability from the teacher model, and the prediction ability of the student model can be optimized, that is, the prediction probability of the student model is more accurate and the relationship classification is optimal.

[0221] When L pred When minimizing, the maximum prediction probability is obtained, that is, the classification label corresponding to the maximum prediction probability is the relationship classification result.

[0222] S505. Obtain sharpness difference based on probability distribution of student model and multi-teacher model

[0223] In the field of knowledge distillation, temperature plays a key role in regulating label softness methods. However, traditional methods usually adopt a static temperature throughout the KD process, which fails to address the subtle complexities of samples with different difficulty levels and ignores the unique capabilities of different teacher-student pairings, which leads to less than ideal knowledge transfer. In order to improve the knowledge propagation process, the GOTA method is used, which adjusts the temperature parameter so that the model can flexibly adjust the smoothness of the probability distribution at different training stages. In particular, we use "sharpness" as a measure to quantify the smoothness of the model output distribution. Assume that the output vector of the model is z = [z1, z2, ..., z n ], where n is the number of categories, and the sharpness difference formula is:

[0224]

[0225] Among them, z represents the model output vector, n is the number of categories, and z i Represents the output vector for the i-th category.

[0226] S506. Obtain a specific temperature by minimizing the sharpness difference between the teacher model and the student model.

[0227] First, an initial temperature temp is given, and the parameter δ is continuously updated using the gradient descent method according to the sharpness difference. Each update is based on

[0228] teacher_temp=temp+δ

[0229] Student_temp=temp-δ

[0230] Finally, the specific temperatures of the teacher model and the student model are obtained.

[0231] After the above steps, the specific temperatures of the teacher model and the student model are obtained, and the specific temperatures are substituted into formulas (21) and (22), which can better optimize the distribution probabilities of the teacher model and the student model so that the probability distribution after using softmax does not have too much difference.

[0232] S507. Construct a total loss based on the student model relationship classification loss, feature distillation loss, hidden representation distillation loss, and prediction distillation loss.

[0233] According to the student model relation classification loss L cls , characteristic distillation loss L m_fdl , hidden representation distillation loss L hrdl and predict distillation loss L pred , get the total loss L of the whole process, update the model parameters through back propagation to guide model learning, the formula is:

[0234] L=λ1L cls +λ2L m_fdl +λ3L hrdl +λ4L tri +λ5L pred (25)

[0235] Among them, L represents the total loss, λ i is the adjustment coefficient of each process, L cls represents the student model relation classification loss, L m_fdl represents the feature distillation loss, L hrdl represents the hidden representation distillation loss, L pred represents the predicted distillation loss.

[0236] When L cls , L m_fdl , L hrdl and L pred When minimized, the total loss is minimized and the classification performance is better.

[0237] The total loss combines multiple loss functions together, calculates the gradient of the total loss relative to the model parameters through back propagation, and uses the gradient descent optimization algorithm to update the model parameters to better adapt to the current learning task and improve the classification performance.

[0238] The model parameters are optimized through the total loss, so that the student model can predict better probability and better learn the richer knowledge and higher expression ability of the teacher model.

[0239] According to the predicted probability, the relationship prediction is performed on the data to be predicted, and the relationship extraction is performed.

[0240] In this embodiment, prediction is performed based on the trained prediction model according to the prediction probability. For example, the data to be predicted is "Google announced the acquisition of YouTube to further expand its influence in the video market."

[0241] First, the input text is preprocessed to annotate entities (such as “Google” and “YouTube”).

[0242] Entity annotation: "[E11] Google[E12] announced the acquisition of [E21] YouTube[E22], further expanding its influence in the video market."

[0243] Secondly, the text of the labeled entity is input into the encoder to obtain the entity feature representation. According to the entity feature representation, it is substituted into formula (1) to obtain the sample hidden representation. The hidden representation is input into the linear Softmax classifier, and the model will generate logits for each known relationship category (for example, "acquisition", "cooperation", "competition", etc.).

[0244] Then, the logits are converted into probability distributions through the Softmax function. For example, the relationship prediction probability output by the model may be:

[0245] - Acquisition: 0.70

[0246] - Cooperation: 0.20

[0247] -Competition: 0.05

[0248] -Irrelevant: 0.05

[0249] Finally, based on the predicted probabilities, relationship prediction is performed and the relationship category with the highest probability is selected. For example, in the above output, the model predicts that the relationship between "Google" and "YouTube" is "acquisition" with a probability of 0.70.

[0250] And based on the predicted relationship type and entity information, construct the extracted structured data. For example, the following structure can be generated:

[0251] "entity1":"Google",

[0252] "entity2":"YouTube",

[0253] "relation":"acquisition".

[0254] After the above steps, relationship classification and relationship extraction can be performed accurately based on the trained prediction model.

[0255] Through the relationship extraction model construction method in this implementation, not only the overall accuracy of the relationship extraction task is improved, but also the impact of data imbalance and insufficient samples on model performance is effectively alleviated through multi-teacher knowledge distillation and gradient optimization temperature adjustment mechanism, thereby enhancing the model's classification ability on a small number of categories. These improvements provide a more robust and efficient solution for handling complex and changeable relationship extraction tasks.

[0256] Embodiment 2

[0257] The purpose of this embodiment is to provide a relationship extraction model construction system based on semantic enhancement and knowledge distillation, including:

[0258] A data acquisition module is used to acquire training data, process the training data to obtain typical samples, obtain relationship prototypes of typical samples based on the typical samples, and obtain a memory set based on the typical samples and the relationship prototypes;

[0259] The data enhancement module is used to perform data enhancement on the training data and memory set, and obtain the expanded enhanced data set and enhanced memory set respectively, which are mixed and input into the teacher model and the student model;

[0260] The feature extraction module is used for the student model to extract features from the mixed data set to obtain the entity feature representation of the student model, and the hidden representation of the student model is obtained based on the entity feature representation. The teacher model extracts features from the mixed data to obtain the output feature representation of each teacher model, and the hidden representation of each teacher model is obtained based on the output feature representation.

[0261] The relation classification module is used to perform relation classification on the hidden representations of the student model and each teacher model, and obtain the prediction probabilities of the student model and each teacher model respectively.

[0262] Based on providing a relationship extraction model construction system based on semantic enhancement and knowledge distillation, the method steps in Example 1 are implemented.

[0263] Embodiment 3

[0264] The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0265] Embodiment 4

[0266] The purpose of this embodiment is to provide a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the above method are performed.

[0267] The steps involved in the apparatus of the above embodiment correspond to the method embodiment 1, and the specific implementation method can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0268] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0269] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A method for constructing a relationship extraction model based on semantic enhancement and knowledge distillation, characterized in that: include: Acquire training data, process the training data to obtain typical samples, obtain relationship prototypes of typical samples based on the typical samples, and obtain a memory set based on the typical samples and the relationship prototypes; Perform data augmentation on the training data and memory set to obtain the expanded augmented data set and augmented memory set, which are mixed and input into the teacher model and the student model. The student model extracts features from the mixed data to obtain entity feature representation of the student model, and obtains the hidden representation of the student model based on the entity feature representation; The teacher model extracts features from the mixed data to obtain the output feature representation of each teacher model, and obtains the hidden representation of each teacher model based on the output feature representation; Relation classification is performed on the hidden representations of the student model and each teacher model, and the prediction probabilities of the student model and each teacher model are obtained respectively.

2. The method for constructing a relationship extraction model based on semantic enhancement and knowledge distillation according to claim 1, characterized in that: The specific process of processing the training data to obtain typical samples is as follows: Encode the training data through the encoder to obtain entity feature representation; According to the entity feature representation, a hidden representation is obtained; Cluster the hidden representations to obtain typical samples.

3. The method for constructing a relationship extraction model based on semantic enhancement and knowledge distillation according to claim 1, characterized in that: The training data and the memory set are enhanced to obtain an expanded enhanced data set and an enhanced memory set, specifically: Both the training data and the memory set are augmented by replacing similar semantic entities, expanding vocabulary variants and making random noises, thus obtaining the expanded enhanced data set and enhanced memory set respectively.

4. The method for constructing a relationship extraction model based on semantic enhancement and knowledge distillation according to claim 1, characterized in that: Based on the entity feature representation of the student model and the output feature representation of the teacher model, the feature distillation loss is constructed as follows: Based on the entity feature representation of the student model and the output feature representation of the teacher model, the cosine embedding loss between each teacher model and the student model is constructed respectively; The feature distillation loss is constructed based on the cosine embedding loss between each teacher model and the student model.

5. The method for constructing a relationship extraction model based on semantic enhancement and knowledge distillation according to claim 4, characterized in that: Based on the hidden representation of the student model and the hidden representation of each teacher model, the hidden representation distillation loss is constructed as follows: in, is a hyperparameter, represents the hidden representation obtained by the student model, and Represent the hidden representations obtained by the two teacher models, x represents the sample in the enhanced memory set, It is represented as an enhanced memory set, which is a set of typical samples of the current task.

6. The method for constructing a relationship extraction model based on semantic enhancement and knowledge distillation according to claim 5, characterized in that: According to the prediction probability of the student model and each teacher model, the prediction distillation loss is constructed as follows: Among them, α and β are hyperparameters used to balance the losses of the two teacher models. and are the predicted probabilities of the two teacher models for the i-th category, is the predicted probability of the student model for the i-th category, x represents the sample in the enhanced memory set, It is represented as an enhanced memory set, which is a set of typical samples of the current task.

7. The method for constructing a relationship extraction model based on semantic enhancement and knowledge distillation according to claim 6, characterized in that: Based on the feature representation distillation loss, hidden representation distillation loss, and prediction distillation loss, the total loss is constructed as follows: L=λ1L cls +λ2L m_fdl +λ3L hrdl +λ4L tri +λ5L pred ; Among them, λ i is the adjustment coefficient of each process; L m_fdl represents the multi-teacher feature distillation loss, L hrdl denotes the multi-teacher hidden representation distillation loss, L pred represents the multi-teacher prediction distillation loss, L cls represents the student model relation classification loss; represents the training dataset of the kth task, R k represents the relationship set of the kth task; y r,x ∈{0, 1} indicates whether the true label of x is r. If so, it is 1, otherwise it is 0. P(r|x) represents the probability that the sample x belongs to relation r calculated by the classifier.

8. A relation extraction model construction system based on semantic enhancement and knowledge distillation, characterized in that: include: A data acquisition module is used to acquire training data, process the training data to obtain typical samples, obtain relationship prototypes of typical samples based on the typical samples, and obtain a memory set based on the typical samples and the relationship prototypes; The data enhancement module is used to perform data enhancement on the training data and memory set, and obtain the expanded enhanced data set and enhanced memory set respectively, which are mixed and input into the teacher model and the student model; The feature extraction module is used for the student model to extract features from the mixed data set to obtain the entity feature representation of the student model, and the hidden representation of the student model is obtained based on the entity feature representation. The teacher model extracts features from the mixed data to obtain the output feature representation of each teacher model, and the hidden representation of each teacher model is obtained based on the output feature representation. The relation classification module is used to perform relation classification on the hidden representations of the student model and each teacher model, and obtain the prediction probabilities of the student model and each teacher model respectively.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are performed.

Images