Pre-trained language model compression method and platform based on Knowledge distillation

Abstract

The invention discloses a pre-trained language model compression method and platform based on knowledge distillation. The method comprises the steps of firstly designing a universal knowledge distillation strategy of feature migration, mapping and approaching features of each layer of a student model to features of a teacher in a process of distilling knowledge of a teacher model to the student model, focusing on the feature expression ability of a small sample in the middle layer of the teacher model, and guiding the student model by utilizing the features; constructing a distillation methodbased on self-attention crossover knowledge by utilizing the ability of self-attention distribution of a teacher model to detect semantics and syntax among words; and in order to improve the learningquality of the learning model at the early stage of training and the generalization ability of the learning model at the later stage of training, designing a linear migration strategy based on Bernoulli probability distribution to gradually complete feature mapping from teachers to students and knowledge migration of self-attention distribution. According to the method and the device, the multi-task-oriented pre-trained language model is automatically compressed, so that the compression efficiency of the language model is improved.

Description

technical field [0001] The invention belongs to the field of automatic compression of multi-task-oriented pre-training language models, and in particular relates to a compression method and platform of a pre-training language model based on knowledge distillation. Background technique [0002] With the popularization of smart devices, the application of large-scale language models on embedded devices such as smart phones and wearable devices is becoming more and more common. In addition to its application on smart devices such as mobile phones, the current response method is still one-way from the knowledge distillation of the teacher model to the compression method of the student model, but the problem of difficult generalization of small samples in the process of large-scale language model compression still exists . Contents of the invention [0003] The object of the present invention is to provide a compression method and platform of a pre-trained language model based...

AI Technical Summary

Problems solved by technology

[0002] With the popularization of smart devices, the application of large-scale language models on embedded devices such as smart phones and wearable devices is becoming more and more common. In addition to its application on smart devices such as mobile phones, the current response method is still one-way from the knowledge distillation of the teacher model to the compression method of the student model, but the problem of difficult generalization of small samples in the process of large-scale language model compression still exists

Images