2-exponential power deep neural network quantification method based on knowledge distillation training

Abstract

The invention relates to the technical field of neural networks. The invention further discloses a 2-exponential power deep neural network quantification method based on knowledge distillation training. The method comprises a teacher model and a student model with exponential power quantification of 2, and is characterized in that the teacher network model selects a network model with more parameters and higher precision, and the student model generally selects a network model with fewer parameters and lower precision than the teacher model. According to the invention, an exponential power quantification deep neural network method in which a neural network weight value is quantified into 2 is adopted; an error with a full-precision weight value can be reduced; the precision of the trainednetwork and the precision loss of the unquantified network are effectively reduced; moreover, the exponential power weight multiplication operation of 2 can be completed by displacement, the method has obvious calculation advantages in hardware equipment deployment, the calculation efficiency of neural network hardware can be improved, and the neural network model trained based on the knowledge distillation algorithm can effectively improve the accuracy of the quantitative network.

Description

technical field [0001] The invention relates to the technical field of neural networks, in particular to a method for quantifying a power-of-two deep neural network based on knowledge distillation training. Background technique [0002] Artificial neural network, also referred to as neural network or connection model for short, is an algorithmic mathematical model that imitates the behavior characteristics of animal neural networks and performs distributed parallel information processing. This kind of network depends on the complexity of the system, and achieves the purpose of processing information by adjusting the interconnection relationship between a large number of internal nodes. [0003] The existing methods for training quantized neural networks are mainly to quantify the weight value of each layer to a low-precision value (such as +1, -1, 0, etc. integers) in the forward propagation stage of the neural network, and then calculate its The output of the layer network...

AI Technical Summary

Problems solved by technology

[0005] Aiming at the deficiencies of the prior art, the present invention provides a 2-exponential power deep neural network quantization method based on knowledge distillation training, which solves the problem that the gradient of the quantized weight value adopts full precision when the network performs backpropagation in the existing method of training the quantized neural network. The gradient accumulated by the weight value is approximated, and the quantized weight value of the neural network will have an error with the actual full-precision weight value in the network. Although the neural network has certain robustness, this will also lead to large errors, resulting in its The problem that the accuracy of the quantized neural network is lost relative to the unquantized neural network

Images