Language model training method and device based on knowledge distillation and text classification method and device

Abstract

The invention discloses a language model training method and device based on knowledge distillation and a text classification method and device. The language model training method comprises the steps of inputting a training corpus into a first model and a second model for processing to obtain corresponding intermediate layer data and an output result; calculating and obtaining a first hidden layer sentence content and a second hidden layer sentence content by utilizing the corresponding intermediate layer data, constructing a comparison learning positive and negative example based on the first hidden layer sentence content and the second hidden layer sentence content, and training a second model by utilizing the comparison learning positive and negative example, the corresponding intermediate layer data and an output result, and determining the trained second model as a language model. Through the classification model, rich sentence grammars and semantic representations of the first model can be migrated to the second model, so that the second model obtained through distillation has better migration capability, and cross-domain application requirements are met.

Description

technical field [0001] The present application relates to the field of natural language processing and model compression, in particular to a language model training method based on knowledge distillation, a text classification method and a device. Background technique [0002] Knowledge distillation is a teacher-student-based model compression method proposed by Hinton et al. in 2015. By introducing a large-scale teacher model to induce the training of a small-scale student model, knowledge transfer is realized. The method is to train a teacher model first, and then use the output of the teacher model and the annotation labels of the data to train the student model, so that the student model can not only learn how to judge the correct sample category from the labeled data, but also learn from the teacher model. relation. [0003] The existing pre-trained language model distillation methods usually use the distillation method of aligning the output scores and the middle laye...

AI Technical Summary

Problems solved by technology

[0003] The existing pre-trained language model distillation methods usually use the distillation method of aligning the output scores and the middle layer. This method can make the output scores of the student model close to the output scores of the teacher model on the data of a specific task. However, , if the data in a new field is tested, the transfer ability of the student model obtained by distillation cannot be guaranteed, and the rich sentence grammar and semantic representation of the teacher model cannot be accurately transferred, resulting in the accuracy of the student model being lower than that of the teacher model. Many, thus unable to meet cross-field application requirements

Images