Model training method, device and electronic device based on knowledge distillation

Abstract

The present disclosure provides a model training method, device, electronic device and storage medium based on knowledge distillation, and relates to the field of computers, in particular to the fields of artificial intelligence technologies such as computer vision and NLP. The specific implementation scheme is as follows: input the feature vectors obtained based on the training samples into the first coding layer and the second coding layer respectively, wherein the first coding layer belongs to the first model and the second coding layer belongs to the second model; The result output from a coding layer is subjected to aggregation processing to obtain the first feature vector; the second feature vector is determined according to the output of the second coding layer; the first feature vector and the second feature vector are distilled to obtain the updated The first eigenvector. This scheme is used for model compression distillation training, which can be flexibly used in any layer of the model, and the compression effect is good. The compressed model can be used for image recognition and can be deployed on a variety of devices with limited computing power.

Description

technical field [0001] The present disclosure relates to the field of computer technology, in particular to the field of artificial intelligence technologies such as computer vision and NLP (Natural Language Processing), and in particular to a model training method, device, electronic device and storage medium based on knowledge distillation. Background technique [0002] With the development of information technology, neural network models are widely used to inject into machine learning tasks such as computer vision, information retrieval, and information recognition. However, for better learning effect, neural network models often have a large number of parameters, which generally require huge calculation examples for inference and deployment. Such large neural network models cannot be deployed accordingly on devices of . That is to say, while ensuring excellent performance, due to the large scale of the model and the large amount of data, large neural network models ofte...

AI Technical Summary

Problems solved by technology

However, for a better learning effect, the neural network model often has a large number of parameters, and generally requires a huge calculation example for inference and deployment, that is, it will occupy a large amount of computing resources during the training and inference phase, so some resources are limited. Such a large neural network model cannot be deployed on the device

That is, while ensuring excellent performance, due to the large scale of the model and the large amount of data, large-scale neural network models often have high requirements for the deployment environment, which greatly limits the use of such models

Images