Model distillation method and device based on network depth compression and medium

Abstract

The invention relates to the field of model compression, in particular to a model distillation method and device based on network depth compression and a medium. According to the method, a cosine distance is used for calculating the characteristic relation of different data passing through a model, and a loss function is constructed based on a Spearman correlation formula; matching the feature relationship of different layers of the simple model with the feature relationship of the last layer of the complex model, constructing a loss function, and guiding the data feature relationship of different layers of the simple model to be close to the data feature relationship of the deepest layer of the complex model, so that the shallow layer of the simple model learns the feature information of the deeper layer; therefore, deep compression of the network is realized. Compared with most existing knowledge distillation methods which are mainly applied to network layers with similar depths of a teacher network and a student network, the method considers the influence of the depth on network improvement, directly carries out distillation learning from a shallow layer to a deep layer, is simple and convenient to implement, is remarkable in effect improvement, and is suitable for popularization and application. And the method can be used together with the existing distillation method to improve the effect.

Description

technical field [0001] The present invention relates to the field of model compression, in particular to a model distillation method, device and medium based on deep network compression. Background technique [0002] So far, deep learning has been widely used in various industries as a mainstream branch of machine learning. However, the high complexity and high computational cost of most deep models make it difficult not only to run on mobile or embedded devices, but also difficult to apply to real-time tasks. How to effectively compress the model to reduce the time and calculation cost while minimizing the loss of model accuracy has received extensive attention in recent years. [0003] As one of the important techniques of model compression, knowledge distillation has been extensively researched and applied in industry since it was first proposed by Hinton in 2015. The main idea of ​​knowledge distillation is to train a complex network model first, and use the complex ne...

AI Technical Summary

Problems solved by technology

[0004] In order to solve the problems existing in the prior art that the deep model has a large time calculation cost, high network complexity, and is difficult to run on mobile terminals or embedded devices, and most of the commonly used knowledge distillation methods do not consider the improvement of model accuracy by network depth, the main Knowledge distillation is carried out on the features or logits of the similar depth of the teacher model and the student model. For this, the present invention proposes a model distillation method, device and medium based on network depth compression. The specific technical scheme is as follows:

Images