Natural language processing method, device and equipment for large language model knowledge distillation

Abstract

The application provides a natural language processing method, device and equipment for large language model knowledge distillation, and relates to the technical field of knowledge distillation. The application can aggregate and reduce the dimensions of the intermediate layer features of the teacher model in the supervised fine-tuning stage, align the intermediate layer features of the student model, and dynamically map the intermediate layer features of the teacher model to the intermediate layers of the student model. After the alignment of the intermediate layer features, a loss function is constructed based on the differences between the intermediate layer features of the teacher model and the student model to train the student model. As a result, the student model obtained by training can not only imitate the output results of the teacher model, but also learn the deep features such as reasoning logic and decision path of the teacher model. The student model can learn the feature information of different levels of the teacher model, thereby better understanding and imitating the reasoning process of the teacher model, and improving the accuracy of natural language processing of the student model.

Images