Consistency regularization and knowledge distillation entropy constraint pseudo-label text abstract method

Abstract

The invention relates to a consistency regularization and knowledge distillation entropy constraint pseudo-label text summarization method, and belongs to the technical field of computer natural language processing. The method comprises the following steps: 1, performing knowledge distillation on a pre-trained language model; 2, carrying out two-stage consistency regular training on the student model itself and between the models; compared with the prior art, the method has the advantages that an entropy constraint pseudo-label technology based on two-stage consistency regularization and knowledge distillation is adopted, the high-performance extraction type text abstract model can be trained under the condition that resources are limited, and high practicability is achieved.

Description

Technical Field

[0001] The present invention relates to a consistency regularization and knowledge distillation entropy constraint pseudo-label text summarization method, belonging to the technical field of computer natural language processing. Background Art

[0002] In the current field of text summarization research, although large deep pre-trained language models (PLMs) perform well when fine-tuned on downstream tasks, their large number of parameters poses a significant challenge for resource-constrained devices. In addition, high-performance text summarization models rely on large amounts of high-quality annotated data, which is often expensive to obtain.

[0003] Therefore, how to improve the performance of automatic text summarization models under the constraints of limited labeled samples and lightweight models has become an urgent problem to be solved. Summary of the Invention

[0004] The purpose of this invention is to solve the technical problem of improving the performance of automatic text summarization models under the constraints of limited labeled samples and lightweight models, and propose a pseudo-label text summarization method with consistency regularization and knowledge distillation entropy constraints.

[0005] The workflow of the present invention is as follows: First, using task-independent knowledge distillation technology, a fully parameterized pre-trained language model is used as a teacher model to distill two lighter student models on a Wikipedia corpus. Then, a two-stage consistency regularization method is used to co-train the two student models on a dataset containing a large amount of unsupervised data and a small amount of supervised data. Finally, during the training process, an entropy-constrained pseudo-labeling technique is used to dynamically select high-quality pseudo-labels and incorporate them into the supervised training process.

[0006] The purpose of the present invention is achieved through the following technical solutions:

[0007] The present invention provides a pseudo-label text summarization method based on consistency regularization and knowledge distillation entropy constraint, comprising the following steps:

[0008] Step 1: Perform knowledge distillation on the pre-trained language model;

[0009] Step 1.1: Initialize the teacher model and student model of the pre-trained language model;

[0010] Step 1.2: Use the large-scale corpus wiki to perform knowledge distillation based on the Transformer layer on the student model and the teacher model using the method shown in formula (1);

[0011]

[0012] Among them L layerThe loss function of a given model layer is calculated using the attention matrix distillation of step 1.2.1 and the hidden state of step 1.2.2. and denote the behavior functions of the mth layer in the student and teacher networks, respectively, and λ m is a hyperparameter indicating the importance of the m-th layer of distillation.

[0013] Step 1.2.1: Distill the student model and the teacher model using the attention matrix distillation as shown in formula (2);

[0014]

[0015] where h is the number of attention heads, is the attention matrix corresponding to the i-th head in the teacher or student network, l is the input text length, and MSE() represents the mean squared error loss function.

[0016] Step 1.2.2: Distill the student model and the teacher model using the hidden state as shown in formula (3);

[0017]

[0018] Among them, the matrix and denote the hidden states of the student network and the teacher network respectively.

[0019] Step 1.3: Construct the mapping relationship between the student model and the teacher model using the interval method and the continuous method, as shown in Equation (4) and Equation (5) respectively;

[0020] i=n+k-1 (4)

[0021] i=2n-1 (5)

[0022] Among them, i is the teacher layer of the mapping, n is the layer of the student model, and k is a custom constant.

[0023] Step 2: Perform two-stage consistency regularization training on the student model itself and between models;

[0024] Step 2.1: The student model itself undergoes two-stage consistency regularization training;

[0025] Step 2.1.1: Define the input of the labeled part as x and the input of the unlabeled part as x';

[0026] Step 2.1.2: Data augmentation φ using adversarial attack and dropout on x' i (·) method to obtain the enhanced unsupervised part input

[0027] Step 2.1.3: and x' are input into the student model and output the data enhanced p i (φ i (x ′ )) and p without data augmentation i (x ′ );

[0028] Step 2.1.4: Place p i (φ i (x ′ )) and p i (x ′ ) are calculated by Softmax respectively, and p i (φ i (x ′ )) Take the logarithm and get log(Softmax(p i (φ i (x ′ )))) and Softmax(p i (x ′ ));

[0029] Step 2.2: Use entropy-constrained pseudo-labels to filter the unlabeled part x' in training;

[0030] Step 2.2.1: Define the batch size of the supervised part and the unsupervised part as B and b respectively; set x ′ and x are input to the student model and after Softmax calculation, the output y of the unsupervised part is obtained respectively. ′ and the output y of the supervised part;

[0031] Step 2.2.2: For the supervised part and the unsupervised part, the entropy is calculated as shown in formula (6) and formula (7) to obtain the entropy of the supervised part. and the entropy of the unsupervised part

[0032]

[0033] Among them, i represents a sample in the supervised batch size B, i ′ represents the i-th in the unsupervised batch size b ′ samples, j represents the value of the jth element in y, and k represents the total number of sentences in the sample;

[0034]

[0035] Step 2.2.3: In the student model and Use formula (8) to calculate the confidence level;

[0036]

[0037] in, n is the dimension of the pseudo label vector of the unlabeled sample, and N is the average dimension of the pseudo label vector of the labeled sample.

[0038] Step 2.2.4: Compare the confidence with the pre-set hyperparameter accuracy threshold ρ, and add the pseudo-labels with conf>ρ as the preferred ones to the pseudo-label queue.

[0039] Step 2.3: Use step 2.1 to get log(Softmax(p i (φ i (x ′ )))) and Softmax(p i (x ′ )) After back propagation of parameters, the weight parameters of the student model are updated and the student model is optimized through equations (9) and (10);

[0040]

[0041] Among them, i represents the i-th student model, log represents the logarithm, and the overall goal is expressed as:

[0042]

[0043] Among them, λ is a hyperparameter, which represents the weight of the consistency loss part; i represents the i-th student model.

[0044] Step 2.4: Perform two-stage consistency regularization training between student models to share complementary information; take the consistency loss between student models as the training target and use MSE to obtain the objective function as shown in Equations (11) and (12); further obtain the overall target between student models as shown in Equation (13);

[0045]

[0046] Among them, MSE represents the root mean square error, p i represents the calculation output of the model numbered i, φ i (·) represents the data augmentation method of the input of model number i;

[0047]

[0048] Where T is the total number of training steps, t is the current number of training steps, and omega(t,T) is defined as It indicates that the weight value of the training between student models increases with the number of training steps, and η is a hyperparameter used to adjust the weight of the second stage training.

[0049] Beneficial effects:

[0050] Compared with the existing technology, the method of the present invention adopts entropy-constrained pseudo-labeling technology based on two-stage consistency regularization and knowledge distillation, which can train a higher-performance extractive text summarization model under limited resources and has strong practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flow chart of the present invention;

[0052] Figure 2 This is a diagram of the knowledge distillation process of the present invention;

[0053] Figure 3 This is the entropy constrained pseudo-labeling flow chart of the present invention;

[0054] Figure 4 This is the first stage flow chart of the two-stage consistency regularization of the present invention. DETAILED DESCRIPTION

[0055] In order to better illustrate the purpose and advantages of the present invention, the following is a further description of the invention in conjunction with the accompanying drawings and examples. It should be noted that the implementation of the present invention is not limited to the following embodiments, and any form of modification or change made to the present invention will fall within the scope of protection of the present invention.

[0056] Example

[0057] like Figure 1 As shown in FIG, a pseudo-label text summarization method based on consistency regularization and knowledge distillation entropy constraint of the present invention is specifically implemented as follows:

[0058] Step 1: Perform knowledge distillation on the pre-trained language model;

[0059] Step 1.1: Initialize the teacher model and student model of the pre-trained language model;

[0060] Step 1.2: Use the large-scale corpus wiki to perform knowledge distillation based on the Transformer layer on the student model and the teacher model using the method shown in formula (1);

[0061]

[0062] Among them L layer The loss function of a given model layer is calculated using the attention matrix distillation of step 1.2.1 and the hidden state of step 1.2.2. and denote the behavior functions of the mth layer in the student and teacher networks, respectively, and λ m is a hyperparameter indicating the importance of the m-th layer of distillation.

[0063] Step 1.2.1: Distill the student model and the teacher model using the attention matrix distillation as shown in formula (2);

[0064]

[0065] where h is the number of attention heads, is the attention matrix corresponding to the i-th head in the teacher or student network, l is the input text length, and MSE() represents the mean squared error loss function.

[0066] Step 1.2.2: Distill the student model and the teacher model using the hidden state as shown in formula (3);

[0067]

[0068] Among them, the matrix and denote the hidden states of the student network and the teacher network respectively.

[0069] Step 1.3: Construct the mapping relationship between the student model and the teacher model using the interval method and the continuous method, as shown in Equation (4) and Equation (5) respectively;

[0070] i=n+k-1 (4)

[0071] i=2n-1 (5)

[0072] Among them, i is the teacher layer of the mapping, n is the layer of the student model, and k is a custom constant.

[0073] In the embodiment, a 12-layer pre-trained language model BERT is selected as the teacher model, and a 4-layer Transformer structure model is initialized as the student model. Then, task-independent knowledge distillation is performed, and the dataset Wiki-103 is used as the distillation dataset. Figure 2 As shown in the figure, the distillation process involves calculating the root mean squared error (RMSE) between the student model's multi-head self-attention matrix and the hidden layer matrix and the corresponding matrix of the teacher model. During training, the distillation goal is to continuously reduce the RMS error. During the distillation process, two strategies, gap layer distillation and continuous layer distillation, are used to obtain two different student models for subsequent collaborative training.

[0074] Step 2: Perform two-stage consistency regularization training on the student model itself and between models;

[0075] Step 2.1: The student model itself undergoes two-stage consistency regularization training;

[0076] Step 2.1.1: Define the input of the labeled part as x and the input of the unlabeled part as x';

[0077] Step 2.1.2: Data augmentation φ using adversarial attack and dropout on x' i (·) method to obtain the enhanced unsupervised part input

[0078] Step 2.1.3: and x' are input into the student model and output the data enhanced p i (φ i (x ′ )) and p without data augmentation i (x ′ );

[0079] Step 2.1.4: Place p i (φ i (x ′ )) and p i (x ′ ) are calculated by Softmax respectively, and p i (φ i (x ′ )) Take the logarithm and get log(Softmax(p i (φ i (x ′ )))) and Softmax(p i (x ′ ));

[0080] Step 2.2: Use entropy-constrained pseudo-labels to filter the unlabeled part x' in training;

[0081] Step 2.2.1: Define the batch size of the supervised part and the unsupervised part as B and b respectively; set x ′ and x are input to the student model and after Softmax calculation, the output y of the unsupervised part is obtained respectively. ′ and the output y of the supervised part;

[0082] Step 2.2.2: For the supervised part and the unsupervised part, the entropy is calculated as shown in formula (6) and formula (7) to obtain the entropy of the supervised part. and the entropy of the unsupervised part

[0083]

[0084] Among them, i represents a sample in the supervised batch size B, i ′ represents the i-th in the unsupervised batch size b ′ samples, j represents the value of the jth element in y, and k represents the total number of sentences in the sample;

[0085]

[0086] Step 2.2.3: In the student model and Use formula (8) to calculate the confidence level;

[0087]

[0088] in, n is the dimension of the pseudo label vector of the unlabeled sample, and N is the average dimension of the pseudo label vector of the labeled sample.

[0089] Step 2.2.4: Compare the confidence with the pre-set hyperparameter accuracy threshold ρ, and add the pseudo-labels with conf>ρ as the preferred ones to the pseudo-label queue.

[0090] Step 2.3: Use step 2.1 to get log(Softmax(p i (φ i (x ′ )))) and Softmax(p i (x ′ )) After back propagation of parameters, the weight parameters of the student model are updated and the student model is optimized through equations (9) and (10);

[0091]

[0092] Among them, i represents the i-th student model, log represents the logarithm, and the overall goal is expressed as:

[0093]

[0094] Among them, λ is a hyperparameter, which represents the weight of the consistency loss part; i represents the i-th student model.

[0095] Step 2.4: Perform two-stage consistency regularization training between student models to share complementary information; take the consistency loss between student models as the training target and use MSE to obtain the objective function as shown in Equations (11) and (12); further obtain the overall target between student models as shown in Equation (13);

[0096]

[0097] Among them, MSE represents the root mean square error, p i represents the calculation output of the model numbered i, φ i (·) represents the data augmentation method of the input of model number i;

[0098]

[0099] Where T is the total number of training steps, t is the current number of training steps, and omega(t,T) is defined as It indicates that the weight value of the training between student models increases with the number of training steps, and η is a hyperparameter used to adjust the weight of the second stage training.

[0100] In the embodiment, the labeled part and the unlabeled part of the training set are converted into vector representations respectively, and they are input into the two student models obtained in step 1. First, the consistency regularization calculation is performed in the first stage, and the parameters of each student model are updated after the calculation is completed. Figure 3 As shown, the second stage of consistency regularization is performed, that is, the consistency regularization between student models. After the calculation is completed, the model parameters are optimized and updated again. In each round of training, the entropy is calculated based on the output results of the unlabeled part model and the output results of the labeled part model, as shown in Figure 4 As shown in the figure, the output of the unlabeled part with relatively high quality is selected according to entropy as the pseudo label selected by the model, which will be added to the training of the supervised part later.

[0101] In order to further illustrate the advantages of the present invention, an experiment is conducted to compare the present invention with the existing method.

[0102] Table 1 Experimental comparison table

[0103]

[0104] As shown in Table 1, our proposed method, CRPLDSum, achieves the best performance on ROUGE-1, ROUGE-2, and ROUGE-L, three metrics for measuring text summarization accuracy, compared to previously advanced extractive text summarization methods, Disco, UDA, and TinyBERT, using the same number of parameters and labeled data. Our proposed method also outperforms BERT, BERTSUM, and CPSUM, which have larger model parameters, on all three metrics.

[0105] The results show that the method proposed in this paper can effectively improve the performance of the extractive text summarization model in low-resource scenarios.

Claims

1. A consistency regularization and knowledge distillation entropy constraint pseudo-label text summarization method, characterized by: The following steps are included: Step 1: Distill the knowledge of the pre-trained language model; Step 2: Perform two-stage consistency regularization training on the student model itself and between models; Step 2.1: The student model itself undergoes two-stage consistency regularization training; Step 2.2: Use entropy-constrained pseudo-labels to filter the unlabeled part x' in training; Step 2.3: Use step 2.1 to get log(Softmax(p i (φ i (x ′ )))) and Softmax(p i (x ′ )) After back propagation of parameters, the weight parameters of the student model are updated and the student model is optimized through equations (9) and (10); Among them, i represents the i-th student model, log represents the logarithm, and the overall goal is expressed as: Among them, λ is a hyperparameter, which is expressed as the weight of the consistency loss part; i represents the i-th student model; Step 2.4: Perform two-stage consistency regularization training between student models to share complementary information; take the consistency loss between student models as the training target and use MSE to obtain the objective function as shown in equations (11) and (12); further obtain the overall target between student models as shown in equation (13); Among them, MSE represents the root mean square error, p i represents the calculation output of the model numbered i, φ i (·) represents the data augmentation method of the input of the model numbered i; Where T is the total number of training steps, t is the current number of training steps, and omega(t,T) is defined as It indicates that the weight value of the training between student models increases with the number of training steps, and η is a hyperparameter used to adjust the weight of the second stage training.

2. The method for pseudo-label text summarization with consistency regularization and knowledge distillation entropy constraints as claimed in claim 1, characterized in that: Step 1 is implemented as follows: Step 1.1: Initialize the teacher model and student model of the pre-trained language model; Step 1.2: Use the large-scale corpus wiki to perform knowledge distillation based on the Transformer layer on the student model and the teacher model in the manner shown in formula (1); Where L layer is the loss function of a given model layer, calculated using the attention matrix distillation of step 1.2.1 and the hidden state of step 1.2.2, and denote the behavior functions of the mth layer in the student and teacher networks, respectively, and λ m is a hyperparameter indicating the importance of the m-th layer of distillation; Step 1.2.1: Distill the student model and the teacher model using the attention matrix distillation as shown in formula (2); where h is the number of attention heads, is the attention matrix corresponding to the i-th head in the teacher or student network, l is the input text length, and MSE() represents the mean square error loss function; Step 1.2.2: Distill the student model and the teacher model using the hidden state shown in formula (3); Among them, the matrix and Represent the hidden states of the student network and the teacher network respectively; Step 1.3: Use the interval method and the continuous method to construct the mapping relationship between the student model and the teacher model, as shown in equation (4) and equation (5) respectively; i=n+k-1 (4) i=2n-1 (5) Among them, i is the teacher layer of the mapping, n is the layer of the student model, and k is a custom constant.

3. The method for pseudo-label text summarization with consistency regularization and knowledge distillation entropy constraints as claimed in claim 1, characterized in that: Step 2.1 is implemented as follows: Step 2.1.1: Define the input of the labeled part as x and the input of the unlabeled part as x'; Step 2.1.2: Data augmentation φ using adversarial attack and dropout on x' i (·) method to obtain the enhanced unsupervised part input Step 2.1.3: and x' are input into the student model and output the data-enhanced p i (φ i (x ′ )) and p without data augmentation i (x ′ ); Step 2.1.4: Place p i (φ i (x ′ )) and p i (x ′ ) are calculated by Softmax respectively, and p i (φ i (x ′ )) Take the logarithm and get log(Softmax(p i (φ i (x ′ )))) and Softmax(p i (x ′ )).

4. The method for pseudo-label text summarization with consistency regularization and knowledge distillation entropy constraints as claimed in claim 1, characterized in that: Step 2.2 is implemented as follows: Step 2.2.1: Define the batch size of the supervised part and the unsupervised part as B and b respectively; set x ′ and x are input into the student model and after Softmax calculation, the output y of the unsupervised part is obtained respectively. ′ and the output y of the supervised part; Step 2.2.2: For the supervised part and the unsupervised part, the entropy is calculated as shown in equation (6) and equation (7) to obtain the entropy of the supervised part: and the entropy of the unsupervised part Among them, i represents a sample in the supervised batch size B, i ′ represents the i-th ′ samples, j represents the value of the jth element in y, and k represents the total number of sentences in the sample; Step 2.2.3: In the student model and Use formula (8) to calculate the confidence level; in, n is the dimension of the pseudo label vector of the unlabeled sample, and N is the average dimension of the pseudo label vector of the labeled sample; Step 2.2.4: Compare the confidence with the preset hyperparameter accuracy threshold ρ, and add the pseudo-labels with conf>ρ to the pseudo-label queue as the preferred ones.

Images