Heterogeneous neural network knowledge recombination method based on common feature learning

Abstract

The heterogeneous neural network knowledge recombination method based on common feature learning comprises the steps of obtaining a plurality of pre-trained neural network models, and naming the neural network models as teacher models; guiding the training of the student model through a common feature learning and soft target distillation method by utilizing the features output by the teacher model and the output prediction result; in the common feature learning process, projecting features of a plurality of heterogeneous networks to a common feature interval, so that a student model integrates knowledge of a plurality of teacher models; the soft target distillation method enables the prediction result of the student model to be consistent with the prediction result of the teacher model, thereby obtaining a stronger student model with the task processing capability of all teacher models. Since only the prediction result of the teacher model needs to be simulated, the student model canbe obtained by training without any manual annotation. The method is suitable for knowledge reorganization of a neural network model, especially knowledge reorganization of a heterogeneous image classification task model.

Description

technical field [0001] The present invention relates to the field of machine learning, in particular to a knowledge reorganization method of heterogeneous neural networks based on common feature learning Background technique [0002] In recent years, deep neural networks (DNNs) have achieved impressive success in numerous artificial intelligence tasks, such as computer vision and natural language processing. However, despite the remarkable results, the training of DNN models is extremely dependent on large-scale human-annotated datasets, and its training takes a long time. In order to ease the reproduction work, more and more researchers have begun to publish the trained models on the Internet so that users can download and use them immediately. It will be very meaningful to reuse these published models to obtain customized models with multi-tasking capabilities without manual labeling of data. However, due to the rapid development of deep learning and the consequent emerg...

AI Technical Summary

Problems solved by technology

The traditional knowledge distillation method is only for a single teacher model, and the goal is to compress the model, that is, use a small network model to imitate and learn the prediction results of a well-trained large network model. For details, see "Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.” Also, direct hierarchical feature fusion learning is infeasible since the teachers’ architectures may be different

Images