Model distillation method and device, electronic equipment and storage medium

Abstract

The invention discloses a model distillation method and device, electronic equipment and a storage medium, and relates to the technical field of deep learning. According to the specific implementationscheme, the method comprises steps: firstly, obtaining a teacher model and a student model; then, according to the first data processing amount of the first middle full-connection layer of the teacher model and the second data processing amount of the second middle full-connection layer of the student model, converting the second middle full-connection layer into an amplification full-connectionlayer and a reduction full-connection layer, and replacing the second middle full-connection layer with the amplification full-connection layer and the reduction full-connection layer to generate a training student model; and then carrying out distillation training on the trained student model according to the teacher model. According to the method, the second middle full-connection layer is replaced with the enlargement full-connection layer and the reduction full-connection layer, and distillation training is carried out on the training student model according to the teacher model, so thatdistillation is carried out on the middle layer of the training student model, an extra full-connection layer does not need to be introduced, parameter redundancy is avoided, and the distillation efficiency and effect are greatly improved.

Description

technical field [0001] The present application relates to the technical field of artificial intelligence, specifically to the technical field of deep learning, and in particular to a model distillation method, device, electronic equipment, and computer-readable storage medium. Background technique [0002] Currently, deep neural network models are widely used in the field of artificial intelligence. Among them, most effective models require complex calculations, and it is difficult to achieve real-time calculations in Internet scenarios. [0003] In related technologies, there are two main ways to solve the above problems by obtaining a small model with a small amount of calculation through distillation calculation of a complex large model. The first way is to distill the large model in the last layer of the neural network. The predicted results of the model are used as soft labels to assist the training of the small model; the second way is to distill in the middle layer o...

AI Technical Summary

Problems solved by technology

However, in the above first method, since distillation is only performed on the last layer of the neural network, the efficiency of distillation is low, and the overall effect is poor; in the above second method, an additional fully connected layer is used for distillation, and some parameters are wasted , the distillation effect is not ideal

Images