Compression method and platform of pre-trained language model based on knowledge distillation

Abstract

The invention discloses a compression method and platform of a pre-trained language model based on knowledge distillation. The method first designs a universal knowledge distillation strategy for feature transfer, and in the process of distilling the knowledge of the teacher model to the student model, the The feature map of each layer of the student model is close to the characteristics of the teacher, focusing on the expressive ability of small samples in the middle layer of the teacher model, and using these features to guide the student model; then using the self-attention distribution of the teacher model to detect the semantic and The syntactic ability builds a cross-knowledge distillation method based on self-attention; finally, in order to improve the learning quality of the learning model in the early stage of training and the generalization ability in the later stage of training, a linear migration strategy based on the Bernoulli probability distribution is designed and gradually completed from Knowledge transfer with teacher-to-student feature maps and self-attention distributions. Through the present invention, the multi-task-oriented pre-training language model is automatically compressed, and the compression efficiency of the language model is improved.

Description

technical field [0001] The invention belongs to the field of automatic compression of multi-task-oriented pre-training language models, and in particular relates to a compression method and platform of a pre-training language model based on knowledge distillation. Background technique [0002] With the popularization of smart devices, the application of large-scale language models on embedded devices such as smart phones and wearable devices is becoming more and more common. In addition to its application on smart devices such as mobile phones, the current response method is still one-way from the knowledge distillation of the teacher model to the compression method of the student model, but the problem of difficult generalization of small samples in the process of large-scale language model compression still exists . Contents of the invention [0003] The object of the present invention is to provide a compression method and platform of a pre-trained language model based...

AI Technical Summary

Problems solved by technology

[0002] With the popularization of smart devices, the application of large-scale language models on embedded devices such as smart phones and wearable devices is becoming more and more common. In addition to its application on smart devices such as mobile phones, the current response method is still one-way from the knowledge distillation of the teacher model to the compression method of the student model, but the problem of difficult generalization of small samples in the process of large-scale language model compression still exists

Images