Multi-module knowledge distillation method based on MoCo model

Abstract

The invention discloses a multi-module knowledge distillation method based on a MoCo model, and the method comprises the steps: dividing a teacher network and a student network into a plurality of corresponding modules through the characteristic that features generated in an intermediate process have similarity, extracting the features generated by each module of the teacher network and the student network through the MoCo model, calculating the similarity, and carrying out the knowledge distillation of the teacher network and the student network. And the purpose that the teacher network guides the student network is achieved by using the similarity. According to the method, the sample features can be automatically and dynamically updated on the basis of only a small number of tags, the memory efficiency of the method is higher, the problem of training a large-scale data set under the condition of a limited memory is solved, and a student network under the guidance of a teacher network has robustness and generalization at the same time.

Description

technical field [0001] The invention belongs to a model lightweight technology, and in particular relates to a multi-module knowledge distillation method based on a MoCo model. Background technique [0002] In recent years, machine learning and deep learning have made remarkable progress in computer vision, natural language processing, prediction, and audio processing. Deploying it on devices is difficult. In knowledge distillation, a larger cumbersome network (teacher model) trained on a large dataset can transfer the learned knowledge well to a smaller and lighter network as a student model. [0003] In the research based on cues from slender networks, a two-stage strategy is introduced to train deep networks, but there is no significant speed improvement; deep mutual learning proposes teacher-student networks to learn from each other and update at the same time, but it is difficult to extract learning More detailed information brings greater errors; in the regeneration ...

AI Technical Summary

Problems solved by technology

[0004] The purpose of the present invention is to provide a multi-module knowledge distillation method based on the MoCo model, which solves the problem of training large-scale data sets in the case of limited memory, and achieves In order to reduce the amount of calculation and improve the effect of memory efficiency

Images