Neural network acceleration method based on cross-resolution knowledge distillation

Abstract

The invention discloses a neural network acceleration method based on cross-resolution knowledge distillation. The method comprises the steps of acquiring high-resolution and low-resolution training samples; constructing high-resolution and low-resolution student networks; pre-training a teacher network through the high-resolution sample data; fixing teacher network parameters, and extracting teacher network output from the high-resolution image; extracting low-resolution image features by using a student network, and constraining the output features of the high-resolution teacher network andthe low-resolution student network to be consistent through cross-resolution distillation loss; in the test stage, extracting robust features from a low-resolution input image by using a student network. According to the invention, knowledge propagation between high-resolution and low-resolution fields is realized by using cross-resolution distillation loss; by extracting the feature accelerationnetwork from the low-resolution image, the calculation complexity is reduced, the discrimination capability and generalization capability of the depth rule are improved by using the prior knowledge ofthe high-resolution image, and the excellent recognition performance is maintained while the calculation complexity of the depth network is greatly reduced.

Description

technical field [0001] The invention relates to the field of deep learning, in particular to a neural network acceleration method based on cross-resolution knowledge distillation. Background technique [0002] With the popularity of big data and the advancement of deep learning technology, deep networks have made great progress, and have made major breakthroughs in many research tasks such as face recognition, pedestrian re-identification, and object classification. However, the current technology faces the problems of high computational complexity and slow operation speed in the application scenarios, which makes many deep networks unable to meet the application requirements of real-time and resource-constrained scenarios. [0003] Aiming at the high computational complexity of neural networks, Hinton et al. proposed a knowledge distillation framework: learning to distinguish robust features through a deep network, and using the network as a teacher network. For the same i...

AI Technical Summary

Problems solved by technology

The current knowledge distillation method only considers the structure of the compressed student network and the number of network parameters, and does not consider the impact of the resolution of the input image on the computational complexity

Obviously, reducing the resolution of the input image can significantly reduce the computational complexity of neural network extraction features, but at the same time it will also reduce the robustness and generalization ability of deep features, which is not conducive to the application of neural networks.

Images