Relational self-distillation method, device and system, and storage medium

Abstract

The embodiment of the invention provides a relational self-distillation method, device and system, and a storage medium. The method comprises the following steps: acquiring N sample images and corresponding annotation data; inputting the images into a student model to obtain image classification information; under the condition that the i is greater than or equal to a preset threshold value, for each category in the K categories, determining current classification information based on classification information corresponding to all included sample images; for each category pair in at least one category pair, calculating the distance between the current classification information corresponding to two categories to obtain current intra-category / inter-category distance information; obtaining historical intra-class / inter-class distance information of each category pair in the at least one category pair, wherein the historical intra-class / inter-class distance information is obtained based on image classification information corresponding to previous iterations respectively. The student model is distilled by matching the current intra-class / inter-class distance information with the historical intra-class / inter-class distance information for each same class pair, so that relation modeling of a classification level related to high-level semantics can be realized.

Description

technical field [0001] The present invention relates to the technical field of machine learning, and more specifically relates to a relational self-distillation method, device, system and storage medium. Background technique [0002] In recent years, neural network models have been widely used in more and more fields. As the difficulty of the problems handled by the neural network model increases, the network structure of the neural network model becomes more and more complex. Some complex models are huge, with millions (or even billions) of parameters, making them difficult to deploy on edge devices. In order to solve this problem, a method of knowledge distillation is developed, through which a more simplified network model can be trained. Knowledge distillation mainly adopts the idea of ​​model compression, using a larger trained neural network model to teach a smaller neural network model, so that the small model can learn the fitting ability of the large model. A lar...

AI Technical Summary

Problems solved by technology

[0004] However, the above-mentioned existing methods are relational modeling at sample granularity, and do not involve relational modeling at the classification level related to high-level semantics, which has great limitations.

Images