Knowledge distillation method and device based on self-attention and computer device

Abstract

The invention relates to the field of artificial intelligence, and discloses a knowledge distillation method based on self-attention, which comprises the following steps: inputting input data into a first model to obtain a first feature matrix output by a middle layer of the first model, and inputting the input data into a second model to obtain a second feature matrix output by a middle layer ofthe second model, wherein the first model is a trained teacher model, and the second model is a to-be-trained student model; calculating first self-attention weight distribution corresponding to the teacher model according to the first feature matrix, and calculating second self-attention weight distribution corresponding to the student model according to the second feature matrix; calculating a distribution difference between the first self-attention weight distribution and the second self-attention weight distribution; taking the distribution difference as a knowledge distillation loss function between the teacher model and the student model; according to the knowledge distillation loss function, migrating the data mapping relation of the middle layer of the teacher model to the middle layer of the student model, and knowledge distillation training of models of different task types can be met.

Description

technical field [0001] This application relates to the field of artificial intelligence, in particular to a self-attention-based knowledge distillation method, device and computer equipment. Background technique [0002] Knowledge Distillation is a special transfer learning method whose purpose is to compress the volume of the training model while ensuring the training effect. Using the trained teacher model to guide the learning of the small-volume student model to be trained, and learning the knowledge of the large model through training the small model, compared with directly training the small model, the effect is better and the speed is faster. [0003] At present, the loss function used for knowledge distillation is more for the classification model, which requires the number of categories of the large model and the small model or the dimension of the network output features to be consistent, which limits the application range of knowledge distillation and cannot meet ...

Images