Multi-network combined auxiliary generative knowledge distillation method

Abstract

The invention discloses a multi-network joint auxiliary generation type knowledge distillation method. The method comprises the following steps: firstly, preprocessing an image classification data set; selecting a teacher network model according to the determined image classification data set and training the teacher network model; selecting a difficult sample generator G1 and a student network according to the determined image classification data set to form an adversarial knowledge distillation framework; establishing a target function for generating confrontation knowledge distillation; carrying out iterative training on the built confrontation knowledge distillation framework; and finally, introducing a simple sample generator G2, and alternately adjusting the student network by using the difficult sample generator G1 and the simple sample generator G2 to obtain a final result. According to the method, the simple sample generator is additionally introduced, the simple sample generator directly copies the trained difficult sample generator, the calculation amount is not increased, and the operation is simple. Under the condition that the simple sample generator helps the students to review the simple samples through the network, a better effect is finally achieved on the target task.

Description

technical field [0001] The invention belongs to the field of knowledge distillation in the field of computer vision, and specifically proposes a multi-network joint auxiliary generative knowledge distillation method for image classification tasks. Background technique [0002] Convolutional Neural Network (CNN) has achieved remarkable achievements in image classification, segmentation, detection and other fields with its powerful feature extraction and expression capabilities. However, the structures of neural networks with high expressive ability are often complex and have a huge amount of parameters. In this case, deploying a complete CNN often requires huge memory overhead and high-performance computing units, and on embedded devices with limited computing resources and mobile terminals with high real-time requirements, the application of CNNs has limitations. Therefore, CNN urgently needs to be lightweight. [0003] Knowledge distillation is widely used as a model comp...

AI Technical Summary

Problems solved by technology

[0008] There is a problem with the above method of resisting knowledge distillation, that is, the generator randomly generates difficult samples and sends them to the student network for training. In the end, the student network may forget the simple samples, resulting in prediction errors and overall performance degradation.

Images