Few-Shot Relation Classification Method Combining Knowledge Distillation and Parameter-Efficient Fine-Tuning

Abstract

This application relates to a few-shot relation classification method that combines knowledge distillation and parameter-efficient fine-tuning. The method constructs a few-shot relation classification model including a hybrid prompt input module, an encoding module, a relation classifier, and a consistency discriminator. This model adopts a parameter-efficient fine-tuning strategy, only needing to optimize a small number of trainable virtual tokens in the hybrid prompt input while freezing all other model parameters, significantly improving the training efficiency. Moreover, this model constructs a teacher-student framework with consistency constraints through knowledge distillation technology, which can effectively extract factual knowledge from the pre-trained model and obtain a more stable relation representation from the perspective of memory enhancement. In addition, this model can further refine and extract a more accurate prototype representation by taking the weighted average of the relation representations of the support set instances, reducing noise interference, and finally improving the classification prediction accuracy of the query set instances in the few-shot relation classification task.

Description

Technical Field

[0001] This application relates to the technical field of natural language processing, and particularly to a few-shot relation classification method that combines knowledge distillation and parameter-efficient fine-tuning. Background Art

[0002] Few-shot relation classification is a core task in natural language processing, aiming to identify the semantic relationships between entity pairs in text with extremely few labeled instances. Existing studies usually rely on prototype networks to learn dense representations of relation categories, mainly by fusing the text descriptions of entities and relations or pre-trained language models to enhance prototype representations. However, these methods usually fine-tune all parameters, which may lead to overfitting problems in few-shot learning. At the same time, they fail to fully utilize the factual knowledge in pre-trained language models, resulting in inaccurate prototype representations and affecting the accuracy of few-shot relation classification. Summary of the Invention

[0003] Based on this, in view of the above technical problems, it is necessary to provide a few-shot relation classification method that combines knowledge distillation and parameter-efficient fine-tuning. The parameter fine-tuning based on hybrid prompts can efficiently capture the factual knowledge in pre-trained language models, and based on knowledge distillation, it can optimize the prototype representations of relation categories through memory enhancement, thereby obtaining more stable and effective relation representations and improving the accuracy of few-shot relation classification.

[0004] A few-shot relation classification method that combines knowledge distillation and parameter-efficient fine-tuning, the method comprising:

[0005] Dividing the text dataset into a training set, a validation set, and a test set, and randomly extracting multiple meta-tasks composed of a support set and a query set from each of the divided datasets; the support set and the query set contain multiple relation categories and multiple instances corresponding to each relation category, and each instance is composed of a sentence and the semantic relationship of the entity pair in the sentence;

[0006] Constructing a few-shot relation classification model including a hybrid prompt input module, an encoding module, a relation classifier, and a consistency discriminator; wherein, the hybrid prompt input module is used to splice the virtual tokens in the constructed teacher model and student model with the sentences and entity pairs in the input instances respectively to obtain a hybrid prompt input; the encoding module is used to encode the hybrid prompt input according to the pre-trained language model to obtain the relation representations of the support set instances and the query set instances; the relation classifier is used to perform weighted averaging on the relation representations of the support set instances to obtain the prototype representation of each relation category, and predict the relation category to which the query set instance belongs by calculating the similarity between the prototype representation and the relation representation of the query set instance; the consistency discriminator is used to construct a consistency constraint between the teacher model and the student model;

[0007] Construct a training objective that includes relation prediction loss and consistency constraint loss, and perform model training and evaluation on the meta-tasks extracted from the training set and the validation set until a few-shot relation classification model that meets the training objective is obtained and the few-shot relation classification task in the test set is executed; among them, only the virtual labels are optimized during model training, and all other model parameters are kept frozen.

[0008] In one embodiment, the text dataset is divided into a training set, a validation set, and a test set, and multiple meta-tasks composed of a support set and a query set are randomly extracted from each of the divided datasets, including:

[0009] The text dataset containing multiple relation categories and corresponding instances is divided into a training set, a validation set, and a test set by category; among them, the relation categories included in the training set, the validation set, and the test set are non-overlapping;

[0010] Randomly extract multiple meta-training tasks, meta-validation tasks, and meta-test tasks from the training set, the validation set, and the test set respectively; each meta-task consists of a support set and a query set ; among them, is the number of relation categories in the support set or the query set, K is the number of support set instances included in each relation category in the support set, is the number of query set instances included in each relation category in the query set, and the support set instances and the query set instances are non-overlapping; and respectively represent the sentence and the semantic relation of the entity pair in the sentence in the th support set instance; and respectively represent the sentence and the semantic relation of the entity pair in the sentence in the jth query set instance;

[0011] For each instance in the support set and the query set , represents a sentence containing a pair of entities, represents the semantic relation between the entity pair in the sentence; the objective of the meta-task is to use the small number of labeled instances in the support set to predict the relation category of the entity pair in the sentence in any query set instance; among them represents a sentence containing words, is the i-th word in the sentence and , and respectively represent the head entity and the tail entity in the sentence, represents and The semantic relationship between is a predefined set of relationships.

[0012] In one embodiment, the virtual tokens in the constructed teacher model and student model are respectively concatenated with the sentence and entity pair in the input instance to obtain a mixed prompt input, including:[[]]

[0013] The teacher model and student model are constructed using the mean teacher algorithm, and the continuous dense vectors of the learnable virtual tokens in the teacher model and student model are respectively obtained ; where represents the i-th vector, and ;

[0014] The continuous dense vectors in the teacher model and student model are respectively concatenated with the sentence in the input instance and the triple composed of the head entity , the mask token and the tail entity to obtain the mixed prompt input of the teacher model and the mixed prompt input of the student model with consistent structures, both represented as:

[0015] ;

[0016] where is the mixed prompt input; the input instance includes the support set instance and the query set instance.

[0017] In one embodiment, the mixed prompt input is encoded according to the pre-trained language model to obtain the relationship representation of the support set instance and the query set instance, including:[[]]

[0018] The pre-trained language model is represented as a function that maps the mixed prompt input to the feature representation of the mask token , represented as:

[0019] ;

[0020] where is the hidden representation of the mask token , and is used as the relationship representation of the support set instance and the query set instance obtained from the teacher model or the student model; are the fixed parameters in the backbone module of the pre-trained language model, are the trainable parameters, specifically representing the parameters to be trained in the teacher model or the student model, and .

[0021] In one embodiment, the training step the parameters of the teacher model are the exponential moving average weights of the parameters of the student model, expressed as:

[0022] ;

[0023] wherein, is the smoothing coefficient hyperparameter, and the initial parameters of the student model are randomly initialized, and the initial parameters of the teacher model are equal to .

[0024] In one embodiment, a weighted average is performed on the relational representations of the support set instances to obtain a prototype representation for each relation category, including:

[0025] A weighted average is performed on the relational representations of the support set instances obtained from the teacher model and the student model to obtain a prototype representation for each relation category, and the expression is:

[0026] ;

[0027] wherein, represents the prototype representation of the i-th relation category; is the teacher model t or the student model s; represents the relational representation of the j-th support set instance of the i-th relation category in the support set S obtained from the teacher model t or the student model s, is the support set the number of support set instances included in each relation category; is an adjustable parameter, defined as:

[0028] .

[0029] In one embodiment, by calculating the similarity between the prototype representation and the relational representation of the query set instance, the relation category to which the query set instance belongs is predicted, including:

[0030] According to the Euclidean distance function calculate the similarity between the prototype representation of the i-th relation category and the relational representation of the query set instance to predict the sentence of the query set instance the probability that the entity pair in belongs to the i-th relation category , expressed as:

[0031] ; ​

[0032] Among them, represents the i-th relationship category, is the semantic relationship between entity pairs in the sentence, and N is the number of relationship categories in the support set or query set.

[0033] In one embodiment, a training objective including a relationship prediction loss and a consistency constraint loss is constructed, including:

[0034] According to the relationship category prediction probability of entity pairs in the sentence in the query set instance construct a relationship prediction loss , which is expressed as:

[0035] ;

[0036] Construct a consistency constraint loss between the teacher model and the student model according to the normalized relationship representations of each instance in the support set and the query set , which is expressed as:

[0037] ;

[0038] Among them, K is the number of support set instances included in each relationship category in the support set , is the query set the number of query set instances included in each relationship category in; and respectively represent the normalized relationship representations of the i-th instance obtained from the student model s and the teacher model t , and , is the relationship representation of the i-th instance;

[0039] Integrate the relationship prediction loss and the consistency constraint loss , and the final training objective is:

[0040] ;

[0041] Among them, are the fixed parameters in the backbone module of the pre-trained language model, and respectively represent the parameters of the teacher model and the student model; is a dynamic balance coefficient.

[0042] The above few-shot relation classification method that combines knowledge distillation and parameter-efficient fine-tuning constructs a few-shot relation classification model including a hybrid prompt input module, an encoding module, a relation classifier, and a consistency discriminator. This model adopts a parameter-efficient fine-tuning strategy, only needs to optimize a small number of trainable virtual tokens in the hybrid prompt input, and freezes all other model parameters at the same time, significantly improving the training efficiency. Moreover, this model constructs a teacher-student framework with consistency constraints through knowledge distillation technology, can effectively extract factual knowledge from the pre-trained model, and obtain a more stable relation representation from the perspective of memory enhancement. In addition, this model can further refine and extract a more accurate prototype representation by weighted averaging the relation representations of the support set instances, reduce noise interference, and finally improve the classification prediction accuracy of the query set instances in the few-shot relation classification task. Description of the Drawings

[0043] Figure 1 It is a schematic flowchart of the few-shot relation classification method that combines knowledge distillation and parameter-efficient fine-tuning in an embodiment;

[0044] Figure 2 It is a schematic diagram of the overall architecture of the few-shot relation classification model in an embodiment. Detailed Embodiments

[0045] In order to make the objectives, technical solutions, and advantages of this application clearer, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit this application.

[0046] In one embodiment, as Figure 1 shown, a few-shot relation classification method that combines knowledge distillation and parameter-efficient fine-tuning is provided, including the following steps:

[0047] Step S1, divide the text data set into a training set, a validation set, and a test set, and randomly extract multiple meta-tasks composed of a support set and a query set from each of the divided data sets.

[0048] Specifically, step S1 includes:

[0049] First, divide the text data set containing multiple relation categories and corresponding instances into a training set, a validation set, and a test set according to categories; among them, the relation categories included in the training set, the validation set, and the test set are non-overlapping. Such a setting can ensure that the model makes accurate predictions and evaluations when facing unseen relation categories.

[0050] Then, multiple meta-training tasks, meta-validation tasks, and meta-test tasks are randomly sampled from the training set, validation set, and test set respectively; in a typical N-way-K-shot (N classes and K samples) setting, each meta-task consists of a support set and a query set ; where is the number of relationship classes in the support set or query set, K is the number of support set instances included in each relationship class in the support set, is the number of query set instances included in each relationship class in the query set, and the support set instances and query set instances are non-overlapping; and respectively represent the sentence and the semantic relationship of the entity pair in the sentence in the th support set instance; and respectively represent the sentence and the semantic relationship of the entity pair in the sentence in the jth query set instance.

[0051] Support set and query set In each instance in, represents a sentence containing a pair of entities, represents the semantic relationship between the entity pair in the sentence; the goal of the meta-task is to use a small number of labeled instances in the support set to predict the relationship class of the entity pair in the sentence of any query set instance; where represents a sentence containing words, is the ith word in the sentence and , and respectively represent the head entity and the tail entity in the sentence, represents and The semantic relationship between them, is a predefined set of relationships.

[0052] Step S2, construct a few-shot relationship classification model including a hybrid prompt input module, an encoding module, a relationship classifier, and a consistency discriminator.

[0053] The overall architecture of the few-shot relationship classification model is as shown in Figure 2 . Figure 2 In it, the yellow color block and the orange color block respectively represent the learnable virtual tokens from the student model and the teacher model, the blue color block represents the frozen parameters of the model, and the green color block represents the relationship representation of the instances in the model. The specific structure and data processing logic of this few-shot relationship classification model are as follows:

[0054] (1)Hybrid Prompt Input Module: For the relation classification task, this application proposes a new hybrid prompt input method, which consists of some learnable task-specific virtual tokens and the structural patterns of triples. Specifically, the process of constructing the hybrid prompt input includes:

[0055] Construct a teacher model and a student model using the mean teacher algorithm, and respectively obtain the continuous dense vectors of the learnable virtual tokens in the teacher model and the student model ; where represents the i-th vector, and ;

[0056] Respectively concatenate the continuous dense vectors in the teacher model and the student model with the sentences in the input instance and the triples composed of the head entity , the mask token and the tail entity to obtain the hybrid prompt input of the teacher model and the hybrid prompt input of the student model with consistent structures, both of which are represented as:

[0057] ;

[0058] where is the hybrid prompt input; the input instance includes a support set instance and a query set instance.

[0059] (2)Encoding Module: Use the pre-trained language model as the encoder, and represent the pre-trained language model as a function that maps the hybrid prompt input to the feature representation of the mask token , which is represented as:

[0060] ;

[0061] where is the hidden representation of the mask token , and use as the relation representation of the support set instance and the query set instance obtained from the teacher model or the student model; are the fixed parameters in the backbone module of the pre-trained language model, are the trainable parameters, specifically representing the parameters to be trained in the teacher model or the student model, and . Compared with the traditional method of optimizing all parameters The model training method only needs to optimize the continuous embedding vectors of some virtual tokens in the training hybrid prompt input during the training of this application, while keeping all other model parameters frozen, so as to achieve efficient and concise parameter fine-tuning, avoid the overfitting problem, and improve the model training efficiency.

[0062] Constructing the teacher model and the student model based on the mean teacher algorithm above is a knowledge distillation method. It forms new semantic memories by using the exponential moving average of the student model parameters as the parameters of the teacher model during the training process. Different from directly sharing the parameters of the student model, the parameters of the teacher model use the exponential moving average weights of the student model parameters, which can be regarded as enhancing the model through memory without directly optimizing. For few-shot relation classification, the mean teacher is a self-ensemble intermediate model state that can obtain better relation representations. Specifically, the training steps of the parameters of the teacher model are the exponential moving average weights of the parameters of the student model and are expressed as:

[0063] ;

[0064] where is the smoothing coefficient hyperparameter, the initial parameters of the student model are randomly initialized, and the initial parameters of the teacher model

[0065] (3) Relation classifier: The main idea of the traditional prototype network is to use prototype representations to represent each relation. The traditional method for calculating the prototype representation is to average the relation representations of all support set instances in the support set. Thus, the traditional prototype representation of the th relation is:

[0066] ;

[0067] where is the relation representation of the th support set instance of the th relation category from the support set . Compared with the traditional prototype representation calculation method, this application can obtain the relation representation These representations may be subject to noise interference because the input instances are not customized for the current relationship. Therefore, these representations must be further refined to improve the accuracy of the prototype representation by retaining high-quality representations and weakening low-correlation representations that are irrelevant to the current relationship. To this end, in the relationship classifier of this application, the relationship representations of the support set instances obtained from the teacher model and the student model are first weighted and averaged to obtain the prototype representation of each relationship category, and the expression is:

[0068] ;

[0069] Among them, represents the prototype representation of the i-th relationship category; is the teacher model t or the student model s; represents the relationship representation of the j-th support set instance of the i-th relationship category in the support set S obtained from the teacher model t or the student model s, is the support set is the number of support set instances included in each relationship category; is an adjustable parameter, defined as:

[0070] .

[0071] Then, according to the Euclidean distance function calculate the similarity between the prototype representation of the i-th relationship category and the relationship representation of the query set instance, and predict the probability that the entity pair in the sentence of the query set instance belongs to the i-th relationship category, which is expressed as:

[0072] ;

[0073] Among them, represents the i-th relationship category, is the semantic relationship between the entity pairs in the sentence, and N is the number of relationship categories in the support set or the query set. Figure 2 In and respectively represent the relationship representations of each support set instance in the support set S obtained from the student model s and the teacher model t.

[0074] Further, according to the predicted probability of the relationship category of the entity pair in the sentence of the query set instance, construct a relationship prediction loss in the form of a cross-entropy objective function , which is expressed as:

[0075] .

[0076] (4) Consistency discriminator: used to construct the consistency constraint between the teacher model and the student model. Specifically, a consistency constraint loss is constructed between the teacher model and the student model according to the normalized relational representations of each instance in the support set and the query set , expressed as:

[0077] ;

[0078] where K is the number of support set instances included in each relation category in the support set , is the query set and the number of query set instances included in each relation category in it; and respectively represent the normalized relational representations of the i-th instance obtained from the student model s and the teacher model t , and , is the relational representation of the i-th instance. Figure 2 in and respectively represent the relational representations of each query set instance in the query set Q obtained from the student model s and the teacher model t.

[0079] Step S3, construct a training objective that includes the relation prediction loss and the consistency constraint loss, and perform model training and evaluation on the meta-tasks extracted from the training set and the validation set until a few-shot relation classification model that meets the training objective is obtained and the few-shot relation classification task in the test set is executed; among them, only the virtual labels are optimized during model training, and all other model parameters are kept frozen.

[0080] Specifically, by integrating the relation prediction loss and the consistency constraint loss , the final training objective is obtained as:

[0081] ;

[0082] where are the fixed parameters in the backbone module of the pre-trained language model, and respectively represent the parameters of the teacher model and the student model; is the dynamic balance coefficient.

[0083] Further, experiments are conducted on two widely used public datasets, FewRel 1.0 and FewRel 2.0, to evaluate the performance of the few-shot relation classification model constructed in this application.

[0084] The FewRel 1.0 dataset contains 100 relation categories, with 700 instances for each relation category. These instances are all extracted from Wikipedia articles. According to the official evaluation setting, Fewrel 1.0 is divided into a training set, a validation set, and a test set based on 100 relation categories. Among them, the training set contains 64 relation categories, the validation set contains 16 relation categories, and the test set contains 20 relation categories, with 700 instances for each relation category.

[0085] The FewRel 2.0 dataset mainly focuses on the problem of domain adaptability. Its training set is the same as that of FewRel 1.0; the validation set is the SemEval-2010 task 8 dataset, which contains 17 relation categories, with 520 instances for each relation category; the test set is the PubMed dataset, which comes from biomedical literature and contains 25 relation categories, with 100 instances for each relation category.

[0086] It should be noted that all the data of FewRel 1.0 belongs to the Wikipedia domain, while the validation set and test set of FewRel 2.0 come from the biomedical domain. The domain difference brings more challenges to the model's fast learning and generalization, so FewRel 2.0 is more challenging than FewRel 1.0.

[0087] The experimental settings include four N-way-K-shot few-shot learning settings: 5-way-1-shot (hereinafter referred to as 5-w-1-s, and the same for others), 5-way-5-shot, 10-way-1-shot, and 10-way-5-shot to evaluate the performance of the few-shot relation classification model constructed in this application, and the average accuracy is used as the evaluation metric. During training, 30,000 tasks are randomly sampled from the training data for training, 10,000 tasks are sampled from the validation data for evaluation, and 20,000 tasks are sampled from the test data for testing. Specifically, a 12-layer Transformer is used and initialized with BERT_BASE (the BERT base model) as the pre-trained language model. During the training process, a batch size of 4 is used. The optimizer is AdamW, and the learning rate is 2e-3, which is used to optimize the learning of continuous embeddings.

[0088] Table 1 Comparison results of different model performances on the FewRel 1.0 test set

[0089]

[0090] Table 2 Comparison results of different model performances on the FewRel 2.0 test set

[0091]

[0092] Tables 1 and 2 respectively list the comparison results of different model performances on the FewRel 1.0 and FewRel 2.0 test sets. The few-shot relation classification model constructed in this application is marked as ETKD in the table. Other comparison models include: MAML-BERT (BERT model adapted by meta-learning), REGRAB (relation-enhanced graph attention BERT), BERT-PAIR (BERT sentence pair model), Proto-BERT (BERT model enhanced by prototype network), MTB (BERT pre-trained for matching tasks), Big ProtoBERT (large-scale prototype BERT), Proto-HP (hybrid prompt prototype BERT), Proto-SP (soft prompt prototype BERT), Proto-HbP (hybrid block prompt prototype BERT), and Proto-HbP (Efficient) (efficient hybrid block prompt prototype BERT).

[0093] As can be seen from Tables 1 and 2, the few-shot classification model constructed in this application not only efficiently trains model parameters but also achieves the best classification performance in all N-way-K-shot few-shot learning settings, confirming the effectiveness of the few-shot relation classification method that combines knowledge distillation and parameter-efficient fine-tuning in capturing key information in few-shot learning.

[0094] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as within the scope described in this specification.

[0095] The above-described embodiments merely represent several implementation manners of this application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of this application. It should be noted that for those of ordinary skill in the art, without departing from the concept of this application, several modifications and improvements can still be made, and these all belong to the protection scope of this application. Therefore, the protection scope of this application should be subject to the appended claims.

Claims

1. A few-shot relation classification method integrating knowledge distillation and parameter-efficient fine-tuning, characterized in that The method includes: Dividing a text dataset into a training set, a validation set, and a test set, and randomly extracting multiple meta-tasks composed of a support set and a query set from each of the divided datasets; the support set and the query set contain multiple relation categories and multiple instances corresponding to each relation category, and each instance consists of a sentence and the semantic relation of the entity pair in the sentence; Constructing a few-shot relation classification model including a hybrid prompt input module, an encoding module, a relation classifier, and a consistency discriminator; wherein, the hybrid prompt input module is used to splice the virtual tokens in the constructed teacher model and student model with the sentence and entity pair in the input instance respectively to obtain a hybrid prompt input; the encoding module is used to encode the hybrid prompt input according to a pre-trained language model to obtain the relation representations of the support set instances and the query set instances; the relation classifier is used to perform weighted averaging on the relation representations of the support set instances to obtain the prototype representation of each relation category, and predict the relation category to which the query set instance belongs by calculating the similarity between the prototype representation and the relation representation of the query set instance; the consistency discriminator is used to construct a consistency constraint between the teacher model and the student model; Constructing a training objective including a relation prediction loss and a consistency constraint loss, training and evaluating the model on the meta-tasks extracted from the training set and the validation set until a few-shot relation classification model that meets the training objective is obtained and performing the few-shot relation classification task in the test set; wherein, only the virtual tokens are optimized during model training, and all other model parameters are kept frozen.

2. The method according to claim 1, characterized in that, Dividing a text dataset into a training set, a validation set, and a test set, and randomly extracting multiple meta-tasks composed of a support set and a query set from each of the divided datasets, including: Dividing the text dataset containing multiple relation categories and corresponding instances into a training set, a validation set, and a test set according to categories; wherein, the relation categories contained in the training set, the validation set, and the test set are non-overlapping; Randomly extract a plurality of meta-training tasks, meta-validation tasks, and meta-test tasks from the training set, validation set, and test set respectively; each meta-task consists of a support set and a query set ; where is the number of relationship categories in the support set or query set, K is the number of support set instances included in each relationship category in the support set, is the number of query set instances included in each relationship category in the query set, and the support set instances and query set instances are non-overlapping; and respectively represent the sentence and the semantic relationship of the entity pair in the sentence in the th support set instance; and respectively represent the sentence and the semantic relationship of the entity pair in the sentence in the jth query set instance; Support set and the query set each instance in in, represents a sentence containing a pair of entities, represents the semantic relationship between entity pairs in the sentence; the goal of the meta-task is to use the support set a small number of labeled instances in to predict the sentence of any query set instance the relationship category of the entity pair in ; where represents a sentence containing is the i-th word in the sentence and , and respectively represent the head entity and the tail entity in the sentence, represents and the semantic relationship between, is a predefined set of relationships.

3. The method according to claim 2, wherein Splicing the virtual tokens in the constructed teacher model and student model with the sentence and entity pair in the input instance respectively to obtain a hybrid prompt input, including: Construct a teacher model and a student model using the mean teacher algorithm, and respectively obtain the continuous dense vectors of the learnable virtual tokens in the teacher model and the student model ; where represents the i-th vector, and ; Separate the continuous dense vectors in the teacher model and the student model from the sentences in the input instance and the triple composed of the head entity , the mask token and the tail entity are concatenated to obtain the hybrid prompt inputs of the teacher model and the student model with consistent structures, both represented as: ； Among them, is the mixed prompt input; the input examples include the support set examples and the query set examples.

4. The method according to claim 3, wherein Encoding the hybrid prompt input according to a pre-trained language model to obtain the relation representations of the support set instances and the query set instances, including: A pre-trained language model is represented as a function that maps a mixed prompt input to a feature representation of masked tokens and is expressed as: ； Among them, is the masked token 's hidden representation, and is used as the relationship representation between the support set instance and the query set instance obtained from the teacher model or the student model; is the fixed parameter in the backbone module of the pre-trained language model, is the trainable parameter, including the parameters that need to be trained in the teacher model or the student model, and .

5. The method according to claim 4, wherein Training steps The parameters of the teacher model Are the parameters of the student model The exponential moving average weights of, expressed as: ； Among them, is the smoothing coefficient hyperparameter, and the initial parameters of the student model are randomly initialized, and the initial parameters of the teacher model are equal to .

6. The method according to claim 4, wherein Performing weighted averaging on the relation representations of the support set instances to obtain the prototype representation of each relation category, including: Performing weighted averaging on the relation representations of the support set instances obtained from the teacher model and the student model to obtain the prototype representation of each relation category, and the expression is: ； Among them, represents the prototype representation of the i-th relationship category; is the teacher model t or the student model s; represents the relationship representation of the j-th support set instance of the i-th relationship category in the support set S obtained from the teacher model t or the student model s, is the support set is the number of support set instances included in each relationship category in; is an adjustable parameter, defined as: 。 7. The method according to claim 6, wherein Predicting the relation category to which the query set instance belongs by calculating the similarity between the prototype representation and the relation representation of the query set instance, including: According to the Euclidean distance function Calculate the prototype representation of the i-th relation category And the relation representation of the query set instance The similarity between them is used to predict the sentence of the query set instance The probability that the entity pair in belongs to the i-th relation category, which is expressed as: ； Among them, represents the i-th relationship category, is the semantic relationship between entity pairs in the sentence, and N is the number of relationship categories in the support set or query set.

8. The method according to claim 7, wherein Constructing a training objective including a relation prediction loss and a consistency constraint loss, including: Sentence according to the query set instance Prediction probability of the relationship category of the entity pair in Construct the relationship prediction loss , expressed as: ； Construct the consistency constraint loss between the teacher model and the student model according to the normalized relational representation of each instance in the support set and the query set , which is expressed as: ； Among them, K is the support set The number of support set instances included in each relationship category in is the query set The number of query set instances included in each relationship category in; and respectively represent the normalized relationship representations of the i-th instance obtained from the student model s and the teacher model t , and , is the relationship representation of the i-th instance; Comprehensive relationship prediction loss and the consistency constraint loss , the final training objective is obtained as follows: ； Among them, are the fixed parameters in the backbone module of the pre-trained language model, and represent the parameters of the teacher model and the student model respectively; is the dynamic balance coefficient.

Images