Output regularization method based on teacher model classification layer weight

Abstract

The invention relates to a teacher model classification layer weight-based output regularization method, which comprises the following steps of: converting the weight of a classification layer of a teacher model subjected to supervised training into a correlation matrix among categories, and taking each row in the matrix as a soft label of a corresponding category to provide additional information for a student model and participate in training of the student model; and selecting the student model with the highest accuracy as a final target model. According to the method, the information provided by the teacher model is fully utilized, the problems that the teacher model occupies too large training resources and the overall training time is too long in the training process are reduced, even if some neural network models can only provide the weight of a teacher model classifier layer, the student model can be trained through the method, and the training efficiency is improved. Compared with the prior art, the method has higher classification accuracy and wider model applicability, has higher training speed, only needs fewer training resources, and can further regularize the network model under the condition of fewer resources.

Description

technical field [0001] The present invention relates to the technical field of calculation; calculation or counting, in particular to an output regularization method based on the weight of the classification layer of the teacher model in the field of image classification in deep learning. Background technique [0002] In deep neural networks, neural network models with a large number of parameters have achieved excellent performance in supervised learning tasks for image classification. However, such neural network models often overfit labeled training samples, resulting in poor generalization. ability. This overfitting phenomenon is the main and common problem after training modern deep neural network models with millions of parameters with labeled datasets. In order to solve this overfitting problem, researchers at home and abroad have proposed different Solutions, including regularization methods. [0003] The regularization method includes regularization at the input a...

AI Technical Summary

Problems solved by technology

[0005] The present invention solves the problems existing in the prior art, provides an optimized output regularization method based on the weight of the classification layer of the teacher model, and overcomes the excessive resources required for training the student model by using the teacher model in the prior art , the problem of too long training time

Images