A BERT model fusion method based on attention feature information

Abstract

The present invention discloses a BERT model fusion method based on attention feature information, which uses the hidden representation information of each original model on the input text to calculate the weight of each original model, and then uses the weight information of each original model to perform weighted fusion on the attention feature information of the original model, and finally uses the fused attention feature information to adjust the new model to improve the performance of the new model. This method can make full use of the attention feature information of the BERT model and improve the performance of the fused BERT model. Compared with other fusion methods, this method can effectively improve the attention feature information of the fused model, so that the attention feature of the fused model to the text is more complete, thereby improving the performance of the fused model.

Description

Technical Field

[0001] The present invention relates to the field of BERT model reuse and BERT model fusion, and in particular to a BERT model fusion method based on attention feature information. Background Art

[0002] BERT model fusion is an important model fusion task, which aims to fuse and reuse BERT models that have been fine-tuned on multiple single tasks to generate a new BERT model that can perform well on multiple tasks.

[0003] Current model fusion methods are mostly focused on the field of image recognition and convolutional networks, while there are relatively few methods for fusing natural language processing and the BERT model. These methods often cannot fully utilize the feature information of the original BERT model, so the fused BERT model cannot achieve good performance.

[0004] Currently, BERT model reuse methods mostly focus on the knowledge distillation and reuse of a single BERT model, focusing on the compression of the original model. They do not consider the feature fusion between multiple BERT models, nor do they consider the improvement of the multi-task processing capabilities of the new BERT model. Summary of the invention

[0005] The purpose of the present invention is to address the shortcomings of the existing BERT model fusion technology and propose a BERT model fusion method based on attention feature information. The present invention can improve the attention feature information of the fused BERT model and improve the performance of the fused model.

[0006] The objective of the present invention is to achieve the following technical solution: a BERT model fusion method based on attention feature information, the specific steps of the method are as follows:

[0007] (1) First, for a given set of original models, construct the hidden representation information of each model for the input text to represent the encoding information of the corresponding model for the input text.

[0008] (2) Then, the weight information of each model is calculated by using the difference between the hidden representation information of each original model and the feature information of the input text.

[0009] (3) Afterwards, the weight information of each original model is used to perform weighted fusion on the attention feature information within the model to generate fused attention feature information.

[0010] (4) Finally, the fused attention feature information is used to complete the training of the new model and generate a new model with better performance and stronger generalization ability.

[0011] The present invention has the following beneficial effects: the BERT model fusion method based on attention feature information improves the attention feature information of the fused BERT model, and generates a new BERT model with better performance and stronger generalization ability. The new model can replace the corresponding original model in the corresponding task, has stronger generalization ability, and is more suitable for relatively complex application scenarios than the original model. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a flowchart of specific implementation steps of the present invention;

[0013] Figure 2 This is a module diagram of the BERT model fusion method proposed in the present invention. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiment of the present invention, other embodiments obtained by ordinary technicians in this field without making creative work are all within the scope of protection of the present invention.

[0015] Aiming at the BERT model fusion task, the present invention utilizes unlabeled data to fuse the attention feature information of multiple original BERT models to generate a new BERT model with better performance and generalization ability.

[0016] like Figure 1 As shown, the specific implementation process and working principle of the present invention are as follows:

[0017] 1. Prepare the original model and fine-tune the BERT model using the labeled data on different tasks to obtain the fine-tuned BERT model on different tasks as the original model.

[0018] 2. Prepare the unlabeled data required for the fusion process. These unlabeled data are used to obtain the feature information of the original model in the subsequent fusion process.

[0019] 3. Use Figure 2 The method shown in the figure fuses the original model to generate a new model with stronger performance and generalization ability. i represents the teacher model, S represents the student model, and H x Represents the hidden layer feature information of the teacher model, A x Represents the attention feature information of the teacher model, P xRepresents the response knowledge information of the teacher model. The BERT model fusion method proposed in this invention is mainly divided into three parts, namely dynamic weight calculation, teacher attention fusion, and student model training.

[0020] (1) Dynamic weight calculation, mainly using the implicit representation factor F of the teacher model T And the hidden layer feature information H output by the student model S , to calculate the weight information of each teacher model. Its input is the implicit representation factor F of each teacher model T And the hidden layer feature H output by the last layer Transformer of the student model S The output is the weight information W of each teacher model. T The weight calculation process is as follows: First, the implicit representation factor F of the teacher model is calculated T The hidden layer of the student model represents the feature H S The importance of each teacher model is obtained by using the mean square error function value (MSE) between them. The larger the mean square error function value, the smaller the corresponding teacher model weight. Then the softmax function is used to normalize the importance of each teacher model to obtain the weight information W corresponding to each teacher model. T .

[0021] (2) Teacher attention fusion mainly uses the teacher model weight to fuse the teacher model’s attention features, and also fuses the hidden layer feature information and response knowledge information. Its input is the attention feature A output by the last layer Transformer of the teacher model. T and hidden layer features H T , the response knowledge P output by the teacher model T , and the weight information W of the teacher model obtained by dynamic weight calculation T ; Its output is the fused attention feature A M , the fused hidden layer feature H M , and the fused response knowledge P M The calculation process of attention fusion mainly uses the weight information of each teacher model to perform weighted average of the attention features of different teachers. The calculation process of hidden layer feature fusion and response knowledge fusion also uses the weighted average method for fusion.

[0022] (3) Student model training mainly uses the fused teacher model knowledge to train and adjust the student model. Its input is the fused teacher model knowledge (A M ,H M ,P M ) and the output of the student model (AS ,H S ,P S ), whose output is the student model parameter θ and the teacher model implicit representation factor F T The student model training mainly calculates the loss function for three different types of knowledge, and adjusts the student model parameters and the implicit representation factor of the teacher model according to the final loss function. The three types of loss functions are attention feature loss function, loss function, and teacher model implicit representation factor. A , hidden layer feature loss function Loss H And the response knowledge loss function Loss P , where the loss function of the attention feature is calculated using KL divergence, the loss function of the hidden layer feature is calculated using mean square error function, and the loss function of the response knowledge is calculated using cross entropy (CE). Then, these three types of loss functions are used to calculate the final loss function Loss, which is mainly calculated using weighted sum. Finally, the loss function results are used to update the student model parameters θ and implicit representation factor F T .

[0023] 4. Evaluate and test the new model. We conducted experiments on the GLUE dataset, and the results showed that our attention feature fusion method improved the accuracy by an average of 0.70 percentage points compared to the direct fusion and feature fusion methods, proving the effectiveness of our method. In order to intuitively explain the effect of the attention fusion of the present invention, the present invention selected a piece of data from the RTE dataset, and combined the model fusion experiments on the three tasks of MNLI, QNLI and RTE to visualize the attention features. The experimental results show that the attention distribution information in the three original models is different, and the attention distribution map of the fused model contains the attention distribution information of the corresponding three models. Compared with the three original models, the attention information of the fused model is more complete, which illustrates the effectiveness of the attention fusion method in this chapter.

[0024] Personnel in the technical field can easily make various changes and modifications based on the text description, drawings and claims provided by the present invention without departing from the concept and scope of the present invention defined by the claims. Any modification or equivalent change made to the above embodiments based on the technical concept and essence of the present invention shall fall within the scope of protection defined by the claims of the present invention.

Claims

1. A BERT model fusion method based on attention feature information, characterized in that: The specific steps of this method are as follows: (1) First, for a given set of multiple original models, construct the hidden representation information of each model for the input text to represent the encoding information of the corresponding model for the input text; (2) Then, the weight information of each model is calculated by using the difference between the hidden representation information of each original model and the feature information of the input text; (3) Afterwards, the weight information of each original model is used to perform weighted fusion on the attention feature information within the model to generate fused attention feature information; (4) Finally, the fused attention feature information is used to complete the training of the new model, generating a new model with better performance and stronger generalization ability; The original model given in step (1) is the teacher model, and T i The new model generated in step (4) is the student model, denoted by S; H x The hidden layer of the teacher model represents information, A x Represents the attention feature information of the teacher model, P x Represents the response knowledge information of the teacher model; In step (2), the teacher model is used to implicitly represent the factor F T And the hidden layer feature information H output by the student model S , to calculate the weight information of each teacher model; Its input is the implicit representation factor F of each teacher model T And the hidden layer feature H output by the last layer Transformer of the student model S ; Its output is the weight information W of each teacher model T ; The weight calculation process is as follows: First, calculate the implicit representation factor F of the teacher model T The hidden layer of the student model represents the feature H S The importance of each teacher model is obtained by using the mean square error function value between them. The larger the mean square error function value, the smaller the corresponding teacher model weight. Then, the importance of each teacher model is normalized using the softmax function to obtain the weight information W corresponding to each teacher model. T ; In step (3), the weight information W corresponding to each teacher model obtained in step (2) is used T To fuse the attention feature information of the teacher model, and also fuse the hidden layer feature information and response knowledge information; Its input is the attention feature A output by the last layer Transformer of the teacher model T and hidden layer features H T , the response knowledge P output by the teacher model T , and the weight information W of the teacher model obtained by dynamic weight calculation T ; Its output is the fused attention feature A M , the fused hidden layer feature H M , and the fused response knowledge P M ; The calculation process of attention fusion mainly uses the weight information of each teacher model to perform weighted average of the attention features of different teachers; the calculation process of hidden layer feature fusion and response knowledge fusion also uses the weighted average method for fusion; In the step (4), the fused teacher model knowledge is used to train and adjust the student model; Its input is the fused teacher model knowledge (A M ,H M ,P M ) and the output of the student model (A S ,H S ,P S ), whose output is the student model parameter θ and the teacher model implicit representation factor F T ; Student model training mainly calculates loss functions for three different types of knowledge, and adjusts student model parameters and teacher model implicit representation factors according to the final loss function; the three types of loss functions are attention feature loss function Loss A , hidden layer feature loss function Loss H And the response knowledge loss function Loss P , where the loss function of the attention feature is calculated using KL divergence, the loss function of the hidden layer feature is calculated using the mean square error function, and the loss function of the response knowledge is calculated using the cross entropy; then these three types of loss functions are used to calculate the final loss function Loss, and the final loss function is mainly calculated using the weighted sum method; finally, the loss function results are used to update the student model parameters θ and the implicit representation factor F T .

Images