A Method for Layer-by-Layer Optimization of Three-Valued Neural Networks

Abstract

The present invention relates to a method for layer-by-layer optimization of a ternary neural network, belonging to the technical field of deep learning training algorithms. This method uses a training set for layer-by-layer quantization. Each layer's quantization is based on the quantization results of the previous layer. Starting from the first layer, the weights of the deep neural network model are quantized layer by layer. Each time of training only quantizes the weights of one layer in the neural network model. A neural network training method that trains the quantizer and the weights simultaneously is adopted until the quantization training of the last layer is completed, completing the quantization of the neural network. The quantized network model is saved. The quantization accuracy is higher, the operation and storage costs are lower, and the application scope of the deep neural network is greatly expanded.

Description

Technical Field

[0001] The present invention relates to a method for optimizing a ternary neural network layer by layer, belonging to the technical field of deep learning training algorithms. Background Art

[0002] In recent years, deep neural networks have been widely applied in the fields of computer vision, semantic recognition, etc. However, their excellent performance comes at the cost of a huge number of parameters and huge computational costs. Therefore, deploying deep neural networks with excellent performance to devices such as computer software and mobile devices that can only provide limited resources (computing power, storage capacity, etc.) will face many challenges, which has also triggered research on deep learning model compression. When applying deep learning tasks related to images such as image classification, object detection, and natural language processing to mobile devices, performing such tasks usually requires using large deep learning networks such as AlexNet to achieve good results, and the neural network needs to be quantized before being applied to mobile devices.

[0003] Currently, deep learning model compression mainly includes channel pruning, weight quantization, weight sparsity, etc. Among them, weight quantization mainly uses integers with a lower number of bits than floating-point type to replace the parameters of the model, so as to achieve the acceleration effect at the storage and calculation levels. Weight ternarization is a special case of weight quantization, which quantizes floating-point weights into a ternary case composed of -1, 0, and 1, greatly reducing the size of the model, and the quantized neural network using this quantization method is called a ternary neural network.

[0004] The defects of existing ternary quantization techniques are as follows: training the quantized weights with a given quantizer causes inevitable accuracy loss; quantizing and training all layers of the deep neural network simultaneously results in huge training computational costs.

[0005] Therefore, there is an urgent need for a layer-by-layer optimization method that simultaneously optimizes the ternary quantizer and the neural network to solve the above technical problems. Summary of the Invention

[0006] In view of the deficiencies of the prior art, the present invention provides a method for optimizing a ternary neural network layer by layer.

[0007] The technical solution of the present invention is as follows:

[0008] A method for optimizing a ternary neural network layer by layer includes:

[0009] Step 1, obtaining a trained deep neural network model as the neural network model to be quantized; or loading a neural network model and pre-training the neural network model using a traditional neural network training method to obtain a pre-trained deep neural network model;

[0010] Step 2: Quantize and train the weights of the deep neural network model layer by layer starting from the first layer. Each time, only quantize the weights of one layer in the neural network model until the quantization training of the last layer is completed, thus completing the quantization of the neural network, and save the quantized network model.

[0011] Preferably, in Step 2, assuming that the weights of the first j layers have been quantized, now perform quantization training on the weights of the (j + 1)-th layer. The specific steps include:

[0012] Forward propagation to calculate the loss function of the output and the true label;

[0013] Backward calculate the gradient of the loss with respect to the weights from the (j + 1)-th layer to the last layer and update the weights;

[0014] Determine the optimal scaling factor and the optimal ternary quantizer for the (j + 1)-th layer of the neural network by the truncated Gaussian distribution, and the quantization of the (j + 1)-th layer of the network model is completed.

[0015] More preferably, in Step 2, the steps for quantizing and training the weights of the (j + 1)-th layer include the following:

[0016] Step 21: Obtain the L-layer deep neural network model in which the first j layers have been quantized. The output of the j-th layer is To quantize the (j + 1)-th layer.

[0017] Step 22: First is the weight update phase. Forward propagate to calculate the loss between the final output and the true label and the gradient of the loss with respect to the weights from the (j + 1)-th layer to the L-th layer, and update the weights from the (j + 1)-th layer to the L-th layer. Update them all at the same time once;

[0018] Forward propagate to calculate the loss:

[0019]

[0020] Where is the output of the j-th layer; is the output of the L-th layer, that is, the predicted label; Y true is the true label; CrossEntrop(·) is the cross-entropy loss function.

[0021] Update the weights from the (j + 1)-th layer to the L-th layer:

[0022]

[0023] Where, W i is the weight of the i-th layer, W i new is the updated weight of the i-th layer, i = j + 1, …, L, is the gradient of the loss with respect to the weight of the i-th layer, and η is the learning rate.

[0024] Step 23. Next, enter the quantizer training phase. First, determine the initialization scaling factor of the weights of the (j + 1)-th layer after the network model is updated and the initialization ternary quantizer, i.e., the quantization threshold.

[0025] In a specific implementation, optionally, the initialization quantization threshold can be arbitrarily selected from the following options: δ l = 0.05max(|W l |), δ l = 0.1max(|W l |), δ l = 0.15max(|W l |), where W l is the weight of the l-th layer; the scaling factor is determined according to the quantization threshold by formula (3).

[0026] Step 24. Quantize the weights of the (j + 1)-th layer by the scaling factor and the ternary quantizer, and calculate the loss between the output and the true label and the gradient of the loss with respect to the quantization threshold of the (j + 1)-th layer again through forward propagation, and update the quantization threshold of the (j + 1)-th layer to obtain a new scaling factor and a new ternary quantizer.

[0027] In a specific implementation, preferably, the determination method of the scaling factor is:

[0028]

[0029] where μ j+1 is the mean of all elements of the weight matrix of the (j + 1)-th layer; σ j+1 is the standard deviation of all elements of the weight matrix of the (j + 1)-th layer; δ j+1 is the quantization threshold of the (j + 1)-th layer, and the initialization quantization threshold in step 23 is used when performing quantizer optimization for the first time; is the probability density function of the standard normal distribution; Φ(·) is the distribution function of the standard normal distribution;

[0030] In a specific implementation, preferably, the determination method of the quantizer is:

[0031]

[0032] where w j+1,i is the element of the weight matrix of the (j + 1)-th layer; μ j+1 is the mean of all elements of the weight matrix of the (j + 1)-th layer; δ j+1 is the quantization threshold of the (j + 1)-th layer, and the initialization quantization threshold in step 23 is used when performing quantizer optimization for the first time;

[0033] Calculate the loss between the output and the true label again through forward propagation:

[0034]

[0035] Among them, is the output of the j-th layer; W new is the weight from the (j + 1)-th layer to the L-th layer, W j+1 = S(μ j+1 , σ j+1 , δ j+1 ) * Ter(w j+1,i , μ j+1 , δ j+1 ) is the quantized weight after the above quantization operation, W j+2 , …, W L are the updated weights in step 22; is the output of the L-th layer, that is, the predicted label; Y true is the true label; CrossEntrop(·) is the cross-entropy loss function;

[0036] Update the quantization threshold of the (j + 1)-th layer:

[0037]

[0038] Among them, δ j+1 is the quantization threshold of the (j + 1)-th layer, i = j + 1, …, L, is the gradient of the loss with respect to the quantization threshold of the (j + 1)-th layer, and η is the learning rate.

[0039] Substitute the new quantization threshold of the (j + 1)-th layer to obtain a new scaling factor:

[0040]

[0041] New ternary quantizer:

[0042]

[0043] Among them, among them, w j+1,i is the element of the weight matrix of the (j + 1)-th layer; μ j+1 is the mean of all elements of the weight matrix of the (j + 1)-th layer; σ j+1 is the standard deviation of all elements of the weight matrix of the (j + 1)-th layer; is the new quantization threshold of the (j + 1)-th layer; is the probability density function of the standard normal distribution; Φ(·) is the distribution function of the standard normal distribution.

[0044] Step 25, Quantize the weights of the (j + 1)-th layer using the new scaling factor and the new ternary quantizer, calculate the loss through forward propagation, and perform forward propagation calculation of the loss in step 24 again. If the loss no longer decreases, it converges; if the loss decreases, repeat step 24;

[0045] Forward propagation to calculate the loss:

[0046]

[0047] Step 26: At this time, the scaling factor and the ternary quantizer are the optimal scaling factor and ternary quantizer for the weights of the (j + 1)-th layer, and the quantization of the (j + 1)-th layer of the network model is completed.

[0048] The beneficial effects of the present invention are as follows:

[0049] 1. The present invention adopts a neural network training method that trains the quantizer and the weights simultaneously, which has the advantage of reducing precision loss and solves the problem of inevitable precision loss caused by the existing method of giving a fixed quantizer.

[0050] 2. The present invention adopts a layer-by-layer training framework, which has the advantages of low computational cost and low storage requirements. Compared with the traditional overall training framework, the computational and storage costs are greatly reduced.

[0051] 3. The present invention uses the training set for layer-by-layer quantization. Each layer of quantization is based on the quantization results of the previous layer, rather than being independent for each layer, so the quantization accuracy is higher.

[0052] 4. The present invention trinary quantizes the quantized values of the weights of the network model (+1, 0, -1). The trinary operation is a trainable quantizer, and the quantizer threshold can be adaptively adjusted. Moreover, the quantization result of this method is {-1, 0, 1}, with low arithmetic storage cost. The network model after quantization can be deployed to devices such as computer software and mobile devices that can only provide limited resources (low computing power and storage capacity). The arithmetic storage cost of the model is greatly reduced. This method can be applied to the training of neural networks with various structures for deep learning tasks such as image processing and language processing, greatly expanding the application scope of deep neural networks. Description of the Drawings

[0053] Figure 1 is a flowchart of a method for layer-by-layer quantization optimization of a deep neural network model in the present invention;

[0054] Figure 2 is a flowchart of simultaneous training of the weights and quantizer of the (j + 1)-th layer of a deep neural network model in the present invention. Detailed Embodiments

[0055] The present invention will be further described below through examples in conjunction with the drawings, but is not limited thereto.

[0056] Example 1:

[0057] Refer to Figure 1, a flowchart of a layer-by-layer quantization optimization method for a deep neural network model in the present invention is given. The following is an explanation in combination with specific steps.

[0058] Step 1: Load the neural network model, and pre-train the network model using a traditional neural network training method such as the backpropagation training method to obtain a pre-trained deep neural network model as the neural network model to be quantized.

[0059] In a specific implementation, a trained deep neural network model can also be obtained as the neural network model to be quantized. For example, the model module in PyTorch contains pre-trained networks such as ResNet and AlexNet, which can be directly loaded and used by code.

[0060] Step 2: Starting from the first layer, quantize the weights of the deep neural network model layer by layer. Each time of training, only quantize the weights of one layer in the neural network model until the quantization training of the last layer is completed, completing the quantization of the neural network, and saving the quantized network model.

[0061] In a specific implementation, the weights that have been quantized in the network model will no longer change. The output of the quantized layer is used as the input of the current quantized layer, and quantization is performed until the last layer in the network model. At this time, all layers have been quantized, and the neural network quantization is completed.

[0062] Among them, when quantizing a certain layer of the neural network model to be quantized, the following method is used for quantization processing. Taking the L-layer deep neural network model with the weights of the previous j layers already quantized as an example, in combination with Figure 2 The process of jointly training the weights of the (j + 1)-th layer and the quantizer is described. The process of jointly training the weights of the (j + 1)-th layer and the quantizer can include the following steps:

[0063] Step 21: Obtain the L-layer deep neural network model with the weights of the previous j layers already quantized. The output of the j-th layer is To quantize the (j + 1)-th layer.

[0064] Step 22: First is the weight update stage. Forward propagate to calculate the loss between the final output and the true label and the gradient of the loss with respect to the layers from the (j + 1)-th layer to the L-th layer, and update the weights of the layers from the (j + 1)-th layer to the L-th layer, and update them all at the same time once from the (j + 1)-th layer to the L-th layer;

[0065] Forward propagate to calculate the loss:

[0066]

[0067] Among them is the output of the j-th layer; is the output of the L-th layer, that is, the predicted label; Y trueis the true label; CrossEntrop(·) is the cross-entropy loss function.

[0068] Update the weights of the (j + 1)-th layer to the L-th layer:

[0069]

[0070] where, W i is the weight of the i-th layer, and W i new is the updated weight of the i-th layer, i = j + 1, …, L, is the gradient of the loss with respect to the weight of the i-th layer, η is the learning rate, usually set to 0.001, and can be tuned for optimization.

[0071] Step 23: Next, enter the quantizer training stage. First, determine the initial scaling factor and the initial ternary quantizer, i.e., the quantization threshold, of the updated weight of the (j + 1)-th layer of the network model;

[0072] In a specific implementation, optionally, the initial quantization threshold can be arbitrarily selected from the following options: δ l = 0.05max(|W l |), δ l = 0.1max(|W l |), δ l = 0.15max(|W l |), where W l is the weight of the l-th layer; the scaling factor is determined according to the quantization threshold by formula (3).

[0073] Step 24: Quantize the weight of the (j + 1)-th layer by the scaling factor and the ternary quantizer, and then perform forward propagation again to calculate the loss between the output and the true label and the gradient of the loss with respect to the quantization threshold of the (j + 1)-th layer, and update the quantization threshold of the (j + 1)-th layer to obtain a new scaling factor and a new ternary quantizer.

[0074] In a specific implementation, preferably, the determination method of the scaling factor is:

[0075]

[0076] where, μ j+1 is the mean of all elements of the weight matrix of the (j + 1)-th layer; σ j+1 is the standard deviation of all elements of the weight matrix of the (j + 1)-th layer; δ j+1 is the quantization threshold of the (j + 1)-th layer, and the initial quantization threshold in step 23 is used when performing the quantizer optimization for the first time; is the probability density function of the standard normal distribution; Φ(·) is the distribution function of the standard normal distribution;

[0077] In a specific implementation, preferably, the determination method of the quantizer is as follows:

[0078]

[0079] where w j + 1,i is an element of the weight matrix of the (j + 1)-th layer; μ j+1 is the mean of all elements of the weight matrix of the (j + 1)-th layer; δ j+1 is the quantization threshold of the (j + 1)-th layer, and the initialization quantization threshold in step 23 is used for the first quantizer optimization;

[0080] Calculate the loss between the output and the true label again through forward propagation:

[0081]

[0082] where is the output of the j-th layer; W new is the weight from the (j + 1)-th layer to the L-th layer, and W j+1 = S(μ j+1 , σ j+1 , δ j+1 ) * Ter(w j+1,i , μ j+1 , δ j+1 ) is the quantized weight after the above quantization operation, and W j+2 , …, W L are the updated weights in step 22; is the output of the L-th layer, that is, the predicted label; Y true is the true label; CrossEntrop(·) is the cross-entropy loss function;

[0083] Update the quantization threshold of the (j + 1)-th layer:

[0084]

[0085] where δ j+1 is the quantization threshold of the (j + 1)-th layer, i = j + 1, …, L, is the gradient of the loss with respect to the quantization threshold of the (j + 1)-th layer, and η is the learning rate.

[0086] Substitute the new quantization threshold of the (j + 1)-th layer to obtain a new scaling factor:

[0087]

[0088] New ternary quantizer:

[0089]

[0090] where, w j+1,i is an element of the weight matrix of the (j + 1)-th layer; μ j+1 is the mean of all elements of the weight matrix of the (j + 1)-th layer; σ j+1 is the standard deviation of all elements of the weight matrix of the (j + 1)-th layer; is the new quantization threshold of the (j + 1)-th layer; is the probability density function of the standard normal distribution; Φ(·) is the distribution function of the standard normal distribution.

[0091] Step 25: Quantize the weights of the (j + 1)-th layer using the new scaling factor and the new ternary quantizer, calculate the loss through forward propagation, and perform the forward propagation calculation of the loss in Step 24 again. If the loss no longer decreases, it converges; if the loss decreases, repeat Step 24;

[0092] Forward propagation calculation of the loss:

[0093]

[0094] Step 26: At this time, the scaling factor and the ternary quantizer are the optimal scaling factor and ternary quantizer for the weights of the (j + 1)-th layer, and the quantization of the (j + 1)-th layer of the network model is completed.

[0095] Experimental Example 1

[0096] An image classification processing method and system based on a ternary neural network model after a layer-by-layer optimization method. When performing tasks such as image classification, large deep learning neural networks such as Alexnet are usually used to achieve good results. When applied to mobile devices, the neural network needs to be quantized before application. The training method of the present invention completes this quantization process. First, load the Alexnet neural network model as the neural network model to be quantized, perform the layer-by-layer quantization method described in Embodiment 1, layer-by-layer quantize the weights of the Alexnet deep neural network model, and only quantize the weights of one layer in the neural network model each time during training until the quantization training of the last layer is completed, completing the quantization of the Alexnet neural network, saving the quantized network model, and applying the quantized network model to a mobile device for image classification processing.

[0097] A computer system, on which a computer program is stored, and when the computer program is executed by a processor, it implements the image classification processing steps based on the ternary neural network model after the layer-by-layer optimization method.

Claims

1. A method for layer-by-layer quantization of a deep neural network model when an image classification application is applied to a mobile device, characterized in that, The steps are as follows: Step 1, when performing an image classification task, obtain a trained Alexnet deep neural network model as the neural network model to be quantized; or load a neural network model and pre-train it using a traditional neural network training method to obtain a pre-trained deep neural network model; Step 2, layer by layer, starting from the first layer, quantize and train the weights of the deep neural network model. Each time, only quantize the weights of one layer in the neural network model until the quantization training of the last layer is completed, completing the quantization of the neural network. Save the quantized network model and apply the quantized network model to a mobile device; Assume that the weights of the first j layers have been quantized. Now, perform quantization training on the weights of the (j + 1)-th layer. The specific steps for the quantization training of the weights of the (j + 1)-th layer include: Forward propagation, calculate the loss function between the output and the true label; Backward calculate the gradient of the loss with respect to the weights from the (j + 1)-th layer to the last layer and update the weights; Determine the optimal scaling factor and the optimal ternary quantizer for the (j + 1)-th layer of the neural network from a truncated Gaussian distribution. The quantization of the (j + 1)-th layer of the network model is completed; The steps for the quantization training of the weights of the (j + 1)-th layer include the following: Step 21, obtain the L-layer deep neural network model whose first j layers have been quantized, and the output of the j-th layer is to quantize the (j + 1)-th layer; Step 22, first is the weight update stage. Forward propagation calculates the loss between the final output and the true label and the gradient of the loss with respect to the weights from the (j + 1)-th layer to the L-th layer, and updates the weights from the (j + 1)-th layer to the L-th layer. All layers from the (j + 1)-th layer to the L-th layer are updated simultaneously once; Forward propagation calculates the loss: wherein is the output of the j-th layer; is the output of the L-th layer, i.e., the predicted label; Y true is the true label; CrossEntrop(·) is the cross-entropy loss function; Update the weights from the (j + 1)-th layer to the L-th layer: where, W i is the weight of the i-th layer, and W i new is the updated weight of the i-th layer, where i = j + 1, …, L, is the gradient of the loss with respect to the weight of the i-th layer, and η is the learning rate; Step 23, then enter the quantizer training stage. First, determine the initial scaling factor and the initial ternary quantizer, i.e., the quantization threshold, for the updated weights of the (j + 1)-th layer of the network model; Step 24, quantize the weights of the (j + 1)-th layer using the scaling factor and the ternary quantizer. Forward propagation again calculates the loss between the output and the true label and the gradient of the loss with respect to the quantization threshold of the (j + 1)-th layer, and updates the quantization threshold of the (j + 1)-th layer to obtain a new scaling factor and a new ternary quantizer; The method for determining the scaling factor is: Among them, μ j+1 is the mean value of all elements of the weight matrix of the (j + 1)-th layer; σ j+1 is the standard deviation of all elements of the weight matrix of the (j + 1)-th layer; δ j+1 is the quantization threshold of the (j + 1)-th layer, and the initial quantization threshold in step 23 is used when the quantizer is optimized for the first time; is the density function of the standard normal distribution; Φ(·) is the distribution function of the standard normal distribution; The method for determining the quantizer is: where w j+1,i is an element of the weight matrix of the (j + 1)-th layer; μ j+1 is the mean of all elements of the weight matrix of the (j + 1)-th layer; δ j+1 is the quantization threshold of the (j + 1)-th layer, and the initialization quantization threshold in step 23 is used when the quantizer is optimized for the first time; Forward propagation again calculates the loss between the output and the true label: Among them, is the output of the j-th layer; W new is the weight from the (j + 1)-th layer to the L-th layer, and W j+1 = S(μ j+1 , σ j+1 , δ j+1 ) * Ter(w j+1,i , μ j+1 , δ j+1 ) is the quantized weight after the above quantization operation, and W j+2 , …, W L are the updated weights in step 22; is the output of the L-th layer, that is, the predicted label; Y true is the true label; CrossEntrop(·) is the cross-entropy loss function; Update the quantization threshold of the (j + 1)-th layer: where, δ j+1 is the quantization threshold of the (j + 1)-th layer, i = j + 1, …, L, is the gradient of the loss with respect to the quantization threshold of the (j + 1)-th layer, and η is the learning rate; Bring in the new quantization threshold of layer j + 1 to obtain a new scaling factor: New ternary quantizer: where, where, w j+1,i is an element of the weight matrix of the (j + 1)-th layer; μ j+1 is the mean of all elements of the weight matrix of the (j + 1)-th layer; σ j+1 is the standard deviation of all elements of the weight matrix of the (j + 1)-th layer; is the new quantization threshold of the (j + 1)-th layer; is the probability density function of the standard normal distribution; Φ(·) is the distribution function of the standard normal distribution; Step 25, quantize the weights of the (j + 1)-th layer using the new scaling factor and the new ternary quantizer. Forward propagation calculates the loss. Perform the forward propagation calculation of the loss in Step 24 again. If the loss no longer decreases, it converges; if the loss decreases, repeat Step 24; Forward propagation calculates the loss: Step 26, at this time, the scaling factor and the ternary quantizer are the optimal scaling factor and ternary quantizer for the weights of the (j + 1)-th layer. The quantization of the (j + 1)-th layer of the network model is completed.

2. The method for layer-by-layer quantization of a deep neural network model when the image classification according to claim 1 is applied to a mobile device, characterized in that, In step 23, the quantization threshold can be initialized with any of the following options: δ l = 0.05max(|W l |), δ l = 0.1max(|W l |), δ l = 0.15max(|W l |), where W l is the weight of the l-th layer; The scaling factor is determined according to the quantization threshold by formula (3).

Images