Multi-task-oriented pre-trained language model automatic compression method and platform

Abstract

The invention discloses a multi-task-oriented pre-training language model automatic compression method and platform. This method designs a meta-network of a structure generator, constructs a knowledge distillation encoding vector based on the knowledge distillation method of Transformer layer sampling, and uses the structure generator to generate a distillation structure model corresponding to the current input encoding vector; at the same time, a Bernoulli distribution sampling is proposed The method of training the structure generator; in each round of iteration, use the Bernoulli distribution sampling method to migrate each encoder unit to form the corresponding encoding vector; by changing the input encoding vector of the structure generator and the training data of the small batch, the joint Training the structure generator and the corresponding distillation structure can learn the structure generator that can generate weights for different distillation structures; at the same time, on the basis of the trained meta-learning network, the evolutionary algorithm is used to search for the optimal compression structure, thus obtaining the same as An optimal general compression architecture for task-independent pretrained language models.

Description

technical field [0001] The invention belongs to the field of language model compression, and in particular relates to a multi-task-oriented pre-training language model automatic compression method and platform. Background technique [0002] Large-scale pre-trained language models have achieved excellent performance in both natural language understanding and generation tasks. However, deploying pre-trained language models with massive parameters to devices with limited memory still faces great challenges. In the field of model compression, the existing language model compression methods are all task-specific language model compression. Although knowledge distillation for specific tasks is effective, fine-tuning and inference on large models is still time-consuming and computationally expensive. When facing other downstream tasks, the pre-trained model generated by distillation of specific task knowledge still needs to re-fine-tune the large model and generate related large-s...

AI Technical Summary

Problems solved by technology

However, under small sample data, training is easily affected by overfitting, and the estimation error increases significantly and propagates layer by layer.

Therefore, the core challenge of neural network compression in the case of small samples is that the compressed model is easy to overfit on the few-shot training instances, resulting in a large estimation error between the original network and the inference process.

Estimation errors may accumulate and propagate layer by layer, eventually corrupting the network output

[0004] In addition, the existing knowledge distillation methods are mainly data-driven sparse constraints or artificially designed distillation strategies; considering that a BERT network usually has 12 layers of Transformer units, each unit contains 8 heads of self-attention units; self-attention units There are hundreds of millions of possible connection methods. Due to the limitation of computing resources, it is almost impossible to manually design all possible distillation structures and find the optimal structure.

Images