A consistent regular and knowledge distillation entropy constraint pseudo-label text summarization method

Abstract

This invention relates to a text summarization method based on consistency regularization and knowledge distillation entropy constraint pseudo-labels, belonging to the field of computer natural language processing technology. The invention includes the following steps: 1. Performing knowledge distillation on a pre-trained language model; 2. Performing two-stage consistency regularization training on the student model, both within the model itself and among models. Compared with existing technologies, this invention employs entropy constraint pseudo-labeling technology based on two-stage consistency regularization and knowledge distillation, enabling the training of a high-performance extractive text summarization model under limited resources, thus demonstrating strong practicality.

Description

Technical Field

[0001] This invention relates to a pseudo-label text summarization method constrained by consistency regularization and knowledge distillation entropy, belonging to the field of computer natural language processing technology. Background Technology

[0002] In the current field of text summarization research, although large deep pre-trained language models (PLMs) perform excellently in fine-tuning downstream tasks, their massive number of parameters poses a significant challenge for resource-constrained devices. Furthermore, high-performance text summarization models rely on large amounts of high-quality labeled data, which are often expensive to obtain.

[0003] Therefore, how to improve the performance of automatic text summarization models under the constraints of limited labeled samples and lightweight models has become an urgent problem to be solved. Summary of the Invention

[0004] The purpose of this invention is to address the technical problem of improving the performance of automatic text summarization models under the constraints of limited labeled samples and lightweight models, and to propose a pseudo-label text summarization method constrained by consistency regularization and knowledge distillation entropy.

[0005] The workflow of this invention is as follows: First, using task-agnostic knowledge distillation, a pre-trained language model with two full parameters is used as the teacher model to distill two lighter student models from a wiki corpus. Then, a two-stage consistency regularization method is used to co-train the two student models on a dataset containing a large amount of unsupervised data and a small amount of supervised data. Finally, during the training process, an entropy-constrained pseudo-labeling technique is used to dynamically select high-quality pseudo-labels to be added to the supervised training process.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] This invention discloses a pseudo-label text summarization method constrained by consistency regularization and knowledge distillation entropy, comprising the following steps:

[0008] Step 1: Perform knowledge distillation on the pre-trained language model;

[0009] Step 1.1: Initialize the teacher and student models of the pre-trained language model;

[0010] Step 1.2: Use the large-scale corpus wiki to perform knowledge distillation on the student model and teacher model in the manner shown in Equation (1) based on the Transformer layer;

[0011]

[0012] Where L layerThe loss function for a given model layer is calculated using the attention matrix distillation in step 1.2.1 and the hidden state in step 1.2.2, respectively. and Let λ represent the behavioral functions of the m-th layer in the student and teacher networks, respectively. m It is a hyperparameter representing the importance of the m-th distillation layer.

[0013] Step 1.2.1: Distill the student model and the teacher model using the attention matrix distillation shown in Equation (2);

[0014]

[0015] Where h is the number of attention heads. is the attention matrix corresponding to the i-th head in the teacher or student network, l is the length of the input text, and MSE() represents the mean squared error loss function.

[0016] Step 1.2.2: Distill the student model and teacher model using the hidden states shown in equation (3);

[0017]

[0018] Among them, matrix and These represent the hidden states of the student network and the teacher network, respectively.

[0019] Step 1.3: Construct the mapping relationship between the student model and the teacher model using the interval method and the continuous method respectively, as shown in Equation (4) and Equation (5);

[0020] i = n + k - 1 (4)

[0021] i = 2n-1 (5)

[0022] Where i is the teacher layer of the mapping, n is the student model layer, and k is a user-defined constant.

[0023] Step 2: Perform two-stage consistency regularization training on the student model, both within the model itself and among different models;

[0024] Step 2.1: The student model itself undergoes two-stage consistency regularization training;

[0025] Step 2.1.1: Define the labeled part of the input as x and the unlabeled part of the input as x';

[0026] Step 2.1.2: Apply adversarial attacks and Dropout to data augmentation φ on x'. i (·) method to obtain the input of the enhanced unsupervised part.

[0027] Step 2.1.3: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation x' and x' are respectively input into the student model and p is output after data augmentation. i (φ i (x ′ )) and p without data augmentation i (x ′ );

[0028] Step 2.1.4: Put p i (φ i (x ′ )) and p i (x ′ ) were calculated using Softmax respectively, and p i (φ i (x ′ Taking the logarithm, we get log(Softmax(p)). i (φ i (x ′ )))) and Softmax(p i (x ′ ));

[0029] Step 2.2: Use entropy-constrained pseudo-labels to filter the unlabeled portion x' in the training;

[0030] Step 2.2.1: Define the batch size for the supervised and unsupervised parts as B and b, respectively; set x... ′ The unsupervised output y is obtained by inputting x into the student model and performing Softmax computation. ′ And the output y of the supervised part;

[0031] Step 2.2.2: Calculate the entropy of the supervised part and the unsupervised part using equations (6) and (7) to obtain the entropy of the supervised part. and the entropy of the unsupervised part

[0032]

[0033] Where i represents a sample in the supervised batch size B, i ′ Represents the i-th element in the unsupervised batch size b. ′ There are 1 sample, where j represents the value of the j-th element in y, and k represents the total number of sentences in the sample;

[0034]

[0035] Step 2.2.3: In the student model and The confidence level is calculated using equation (8);

[0036]

[0037] in, n is the dimension of the pseudo-label vector of the unlabeled sample, and N is the average dimension of the pseudo-label vector of the labeled sample.

[0038] Step 2.2.4: Compare the confidence level with the pre-set hyperparameter accuracy threshold ρ, and add pseudo-labels with conf>ρ as preferred to the pseudo-label queue.

[0039] Step 2.3: Use step 2.1 to obtain log(Softmax(p) i (φ i (x ′ )))) and Softmax(p i (x ′ After backpropagation of the parameters, the weight parameters of the student model are updated and the student model is optimized by equations (9) and (10).

[0040]

[0041] Where i represents the i-th student model, log represents taking the logarithm, and the overall objective is expressed as:

[0042]

[0043] Where λ is a hyperparameter representing the weight of the consistency loss component; i represents the i-th student model.

[0044] Step 2.4: Two-stage consistency regularization training is performed between student models to share complementary information; the consistency loss between student models is used as the training objective and the objective function is obtained by MSE as shown in equations (11) and (12); the overall objective between student models is further obtained as shown in equation (13);

[0045]

[0046] Where MSE represents the root mean square error, p i φ represents the computational output of model number i. i (·) indicates the data augmentation method for the input of model number i;

[0047]

[0048] Where T is the total number of training steps, t is the current number of training steps, and omega(t,T) is defined as follows: This indicates that the weights assigned to student models during training increase with the number of training steps. η is a hyperparameter used to adjust the weights assigned to the second-stage training.

[0049] Beneficial effects:

[0050] Compared with existing technologies, the method of this invention adopts an entropy-constrained pseudo-labeling technique based on two-stage consistency regularization and knowledge distillation, which can train a high-performance extractive text summarization model under limited resources and has strong practicality. Attached Figure Description

[0051] Figure 1 This is a flowchart of the present invention;

[0052] Figure 2 This is a diagram illustrating the knowledge distillation process of the present invention;

[0053] Figure 3 This is a flowchart of the entropy-constrained pseudo-label of the present invention;

[0054] Figure 4 This is a flowchart of the first stage of the two-stage consistency regularization of the present invention. Detailed Implementation

[0055] To better illustrate the purpose and advantages of this invention, the invention will be further described below with reference to the accompanying drawings and examples. It should be noted that the implementation of this invention is not limited to the following embodiments, and any modifications or alterations made to this invention will fall within the scope of protection of this invention.

[0056] Example

[0057] like Figure 1 As shown, the specific implementation steps of the consistency regularization and knowledge distillation entropy constraint pseudo-label text summarization method of the present invention are as follows:

[0058] Step 1: Perform knowledge distillation on the pre-trained language model;

[0059] Step 1.1: Initialize the teacher and student models of the pre-trained language model;

[0060] Step 1.2: Use the large-scale corpus wiki to perform knowledge distillation on the student model and teacher model in the manner shown in Equation (1) based on the Transformer layer;

[0061]

[0062] Where L layer The loss function for a given model layer is calculated using the attention matrix distillation in step 1.2.1 and the hidden state in step 1.2.2, respectively. and Let λ represent the behavioral functions of the m-th layer in the student and teacher networks, respectively. m It is a hyperparameter representing the importance of the m-th distillation layer.

[0063] Step 1.2.1: Distill the student model and the teacher model using the attention matrix distillation shown in Equation (2);

[0064]

[0065] Where h is the number of attention heads. is the attention matrix corresponding to the i-th head in the teacher or student network, l is the length of the input text, and MSE() represents the mean squared error loss function.

[0066] Step 1.2.2: Distill the student model and teacher model using the hidden states shown in equation (3);

[0067]

[0068] Among them, matrix and These represent the hidden states of the student network and the teacher network, respectively.

[0069] Step 1.3: Construct the mapping relationship between the student model and the teacher model using the interval method and the continuous method respectively, as shown in Equation (4) and Equation (5);

[0070] i = n + k - 1 (4)

[0071] i = 2n-1 (5)

[0072] Where i is the teacher layer of the mapping, n is the student model layer, and k is a user-defined constant.

[0073] In this embodiment, a 12-layer pre-trained language model BERT is selected as the teacher model, and a 4-layer Transformer model is initialized as the student model. Then, task-independent knowledge distillation is performed, using the wiki-103 dataset as the distillation dataset. Figure 2 As shown, the distillation process requires calculating the root mean square error (RMSE) between the multi-head self-attention matrix and the hidden layer matrix of the student model and the corresponding matrix of the teacher model. During training, the goal of distillation is to continuously reduce the value of the RMS error. Two strategies, interval distillation and continuous layer distillation, are used to obtain two different student models for subsequent co-training.

[0074] Step 2: Perform two-stage consistency regularization training on the student model, both within the model itself and among different models;

[0075] Step 2.1: The student model itself undergoes two-stage consistency regularization training;

[0076] Step 2.1.1: Define the labeled part of the input as x and the unlabeled part of the input as x';

[0077] Step 2.1.2: Apply adversarial attacks and Dropout to data augmentation φ on x'. i (·) method to obtain the input of the enhanced unsupervised part.

[0078] Step 2.1.3: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation x' and x' are respectively input into the student model and p is output after data augmentation. i (φ i (x ′ )) and p without data augmentation i (x ′ );

[0079] Step 2.1.4: Put p i (φ i (x ′ )) and p i (x ′ ) were calculated using Softmax respectively, and p i (φ i (x ′ Taking the logarithm, we get log(Softmax(p)). i (φ i (x ′ )))) and Softmax(p i (x ′ ));

[0080] Step 2.2: Use entropy-constrained pseudo-labels to filter the unlabeled portion x' in the training;

[0081] Step 2.2.1: Define the batch size for the supervised and unsupervised parts as B and b, respectively; set x... ′ The unsupervised output y is obtained by inputting x into the student model and performing Softmax computation. ′ And the output y of the supervised part;

[0082] Step 2.2.2: Calculate the entropy of the supervised part and the unsupervised part using equations (6) and (7) to obtain the entropy of the supervised part. and the entropy of the unsupervised part

[0083]

[0084] Where i represents a sample in the supervised batch size B, i ′ Represents the i-th element in the unsupervised batch size b. ′ There are 1 sample, where j represents the value of the j-th element in y, and k represents the total number of sentences in the sample;

[0085]

[0086] Step 2.2.3: In the student model and The confidence level is calculated using equation (8);

[0087]

[0088] in, n is the dimension of the pseudo-label vector of the unlabeled sample, and N is the average dimension of the pseudo-label vector of the labeled sample.

[0089] Step 2.2.4: Compare the confidence level with the pre-set hyperparameter accuracy threshold ρ, and add pseudo-labels with conf>ρ as preferred to the pseudo-label queue.

[0090] Step 2.3: Use step 2.1 to obtain log(Softmax(p) i (φ i (x ′ )))) and Softmax(p i (x ′ After backpropagation of the parameters, the weight parameters of the student model are updated and the student model is optimized by equations (9) and (10).

[0091]

[0092] Where i represents the i-th student model, log represents taking the logarithm, and the overall objective is expressed as:

[0093]

[0094] Where λ is a hyperparameter representing the weight of the consistency loss component; i represents the i-th student model.

[0095] Step 2.4: Two-stage consistency regularization training is performed between student models to share complementary information; the consistency loss between student models is used as the training objective and the objective function is obtained by MSE as shown in equations (11) and (12); the overall objective between student models is further obtained as shown in equation (13);

[0096]

[0097] Where MSE represents the root mean square error, p i φ represents the computational output of model number i. i (·) indicates the data augmentation method for the input of model number i;

[0098]

[0099] Where T is the total number of training steps, t is the current number of training steps, and omega(t,T) is defined as follows: This indicates that the weights assigned to student models during training increase with the number of training steps. η is a hyperparameter used to adjust the weights assigned to the second-stage training.

[0100] In this embodiment, the labeled and unlabeled portions of the training set are converted into vector representations, which are then input into the two student models obtained in step one. First, a consistency regularization calculation is performed in the first stage, after which the parameters of each student model are updated. For example... Figure 3 As shown, the second stage of consistency regularization, namely consistency regularization among student models, is performed. After calculation, the model parameters are optimized and updated again. During each training round, entropy is calculated based on the output results of the unlabeled part of the model and the output results of the labeled part of the model, such as... Figure 4 As shown, the output of the unlabeled part with relatively high quality is selected based on entropy and used as the pseudo-label for model selection, which will be added to the training of the supervised part later.

[0101] To further illustrate the superiority of the present invention, an experimental method is used to compare it with existing methods.

[0102] Table 1. Experimental Comparison Table

[0103]

[0104] As shown in Table 1, the proposed method CRPLDSum achieves the best performance on the three metrics measuring text summarization accuracy—ROUGE-1, ROUGE-2, and ROUGE-L—compared to previous advanced extractive text summarization methods Disco, UDA, and TinyBERT, under the same parameter count and labeled data volume. The proposed method also outperforms BERT, BERTSUM, and CPSUM, which have larger model parameter counts, in all three metrics.

[0105] The results show that the method proposed in this invention can effectively improve the performance of extractive text summarization models in low-resource scenarios.

Claims

1. A pseudo-label text summarization method constrained by consistency regularization and knowledge distillation entropy, characterized in that: Includes the following steps, Step 1: Perform knowledge distillation on the pre-trained language model; Step 1.1: Initialize the teacher and student models of the pre-trained language model; Step 1.2: Use the large-scale corpus wiki to perform knowledge distillation on the student model and teacher model in the manner shown in Equation (1) based on the Transformer layer; (1) in The loss function for a given model layer is calculated using the attention matrix distillation in step 1.2.1 and the hidden state in step 1.2.2, respectively. and These represent the first and second students in the student and teacher networks, respectively. The behavior function of the layer, It means the first Hyperparameters of importance for layer distillation; Step 1.2.1: Distill the student model and the teacher model using the attention matrix distillation shown in Equation (2); (2) in It's about the number of heads to focus on. It corresponds to the first in the teacher or student network. Attention matrix of size, This is the length of the input text. This represents the mean squared error loss function; Step 1.2.2: Distill the student model and teacher model using the hidden states shown in equation (3); (3) Among them, matrix and These represent the hidden states of the student network and the teacher network, respectively. Step 1.3: Construct the mapping relationship between the student model and the teacher model using the interval method and the continuous method respectively, as shown in Equation (4) and Equation (5); （4） （5） in, For the mapped teacher level, For the student model layers, For user-defined constants; Step 2: Perform two-stage consistency regularization training on the student model, both within the model itself and among different models; Step 2.1: The student model itself undergoes two-stage consistency regularization training; Step 2.1.1: Define the labeled input as... The unlabeled part of the input is defined as ; Step 2.1.2: For Data augmentation using adversarial attacks and Dropout This method obtains the input of the enhanced unsupervised part. ; Step 2.1.3: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation and Input them into the student model respectively and output the data augmented version respectively. and unenhanced data ; Step 2.1.4: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation and Through respectively Perform calculations and... Taking the logarithm, we get and ; Step 2.2: Use entropy-constrained pseudo-labels to process the unlabeled parts during training. Perform the screening; Step 2.3: Obtain using step 2.1 and After backpropagation of the parameters, the weight parameters of the student model are updated and the student model is optimized by equations (9) and (10); （9） in, Indicates the first For each student model, log represents the logarithm, and the overall objective is expressed as: （10） in, is a hyperparameter, representing the weight of the consistency loss component; Indicates the first One student model; Step 2.4: Two-stage consistency regularization training is performed between student models to share complementary information; the consistency loss between student models is used as the training objective and the objective function is obtained by MSE as shown in equations (11) and (12); the overall objective between student models is further obtained as shown in equation (13); (11) (12) Where MSE represents the root mean square error. Indicates the number is The calculation output of the model, Indicates the number is The data augmentation method for the model's input; (13) in, That is the total number of training steps. This is the current number of training steps. The definition of This indicates that the weights assigned to student models during training increase with the number of training steps. Hyperparameters used to adjust the weights used in the second phase of training.

2. The pseudo-label text summarization method with consistency regularization and knowledge distillation entropy constraints as described in claim 1, characterized in that: Step 2.2 is implemented as follows: Step 2.2.1: Define the batch size for the supervised and unsupervised parts as follows: and ;Will and Input into the student model The outputs of the unsupervised parts are obtained after calculation. and the output of the supervised part ; Step 2.2.2: Calculate the entropy of the supervised part and the unsupervised part using the methods shown in equations (6) and (7). and the entropy of the unsupervised part ; （6） in, Indicates supervised batch size One of the samples, Indicates unsupervised batch size The first in One sample, express The first in The value of each element, This represents the total number of sentences in the sample; （7） Step 2.2.3: In the student model and The confidence level is calculated using equation (8); （8） in, , Let be the dimension of the pseudo-label vector of the unlabeled sample. This represents the average dimension of the pseudo-label vectors of the labeled samples; Step 2.2.4: Compare the confidence level with the pre-set hyperparameter accuracy threshold. Comparison, will The pseudo-labels are selected as the preferred ones and added to the pseudo-label queue.

Images