Gradient quantization-based binarization network training method and device

Abstract

The application provides a binarization network training method and device based on gradient quantization, which comprises the following steps: in a full-precision model full-quantization training process taking a sample image as a training sample and taking a class label of the sample image as a training label, quantizing weights and input activation values of the full-precision model to obtain first quantization data with a target bit width; in a case of outputting activation value gradients by performing back propagation on the first quantization data with the target bit width, quantizing the activation value gradients to obtain second quantization data with the target bit width; and updating the activation value gradients and weight gradients according to the first quantization data and the second quantization data to obtain a trained binarization network. The method realizes binarization gradient training, effectively reduces the training power consumption of the model and the requirements for memory and computing power, and further improves the training efficiency and inference performance of the classification model in a complex scene.

Description

Technical Field

[0001] This invention relates to the field of neural network technology, and in particular to a method and apparatus for training binary networks based on gradient quantization. Background Technology

[0002] With the popularization of smart devices and the rapid development of technologies such as computer vision and speech recognition, classification tasks on terminal devices (such as smartphones, tablets, smart home devices, and embedded systems) have become an important research area.

[0003] Classification models are typically trained in the cloud and then deployed to end devices. While models can be efficiently trained in the cloud and then deployed to end devices, further training on end devices remains necessary and advantageous for reasons including privacy protection, reduced data transmission and latency, and maintaining model personalization. However, end devices have relatively limited computing power, making it difficult to support the training and inference of complex deep learning models.

[0004] In related technologies, terminal devices use traditional deep learning models to deploy classification models. However, due to the relatively weak computing power of these models, complex model training and inference scenarios typically require a large amount of computing resources to process high-dimensional data and complex network structures. This poses a significant challenge for terminal devices with limited computing power. For example, models in image classification tasks usually contain a large number of convolutional and fully connected layers, requiring a large number of floating-point operations. This results in low performance of the classification models on terminal devices with limited computing resources, making it difficult to adapt to the classification needs in complex scenarios. Summary of the Invention

[0005] This invention provides a binary network training method and apparatus based on gradient quantization, which solves the problem that traditional deep learning models in the prior art suffer from limited computing resources when performing classification tasks in complex scenarios, resulting in low model training and inference efficiency, and improves the training efficiency and inference performance of classification models in complex scenarios.

[0006] This invention provides a binary network training method based on gradient quantization, comprising:

[0007] The full-precision model is trained using sample images as training samples and class labels of the sample images as training labels. During the forward propagation of the full-precision model, the weights and input activation values ​​of the full-precision model are quantized to obtain first quantized data with the target bit width. After backpropagation of the first quantized data with the target bit width and outputting the activation value gradient, the activation value gradient is quantized to obtain second quantized data with the target bit width.

[0008] The activation value gradient and weight gradient are updated based on the first quantized data and the second quantized data, and the trained binarized network is obtained under the condition of satisfying the maximum number of training times.

[0009] According to a gradient quantization-based binarization network training method provided by the present invention, before quantizing the activation value gradient, the method further includes:

[0010] The bit widths corresponding to the multiple activation value gradients output by the full-precision model during backpropagation are compared with a bit width threshold. If the bit width corresponding to the target activation value gradient is less than the bit width threshold, the target activation value gradient is removed from the multiple activation value gradients to obtain a new activation value gradient.

[0011] According to a gradient quantization-based binarization network training method provided by the present invention, after obtaining the new activation value gradient, the method further includes:

[0012] The new activation gradient is losslessly decomposed using the following formula to update the calculation strategy for the activation gradient and weight gradient of the full-precision model during backpropagation:

[0013] ;

[0014] in, for The element in the first position i The value of the bit. i The number of bits in the element. i ∈{0,…, b}

[0015] According to a gradient quantization-based binarization network training method provided by the present invention, after obtaining the first quantized data with a target bit width, the method further includes:

[0016] When the full-precision model performs backpropagation based on the first quantized data, the gradients between multiple output activation values ​​of the model are quantized by channel grouping to obtain the gradients of the output activation values; the gradients between multiple weights of the model are quantized by sample grouping to obtain the weight gradients.

[0017] According to a gradient quantization-based binarization network training method provided by the present invention, before performing full quantization training on a full-precision model using sample images as training samples and the class labels of the sample images as training labels, the method further includes:

[0018] The sample image is processed by removing invalid data, handling missing values, and eliminating duplicates to obtain the processed sample image.

[0019] According to the present invention, a binary network training method based on gradient quantization is provided, wherein the activation value gradient calculation strategy of the full-precision model during backpropagation is expressed by the following formula:

[0020] ;

[0021] in, For the first l The quantized activation gradient corresponding to the layer network, For the first l The transpose of the weights corresponding to the layer network, where sign is a function; For the first l The quantized activation gradient corresponding to the layer network, for b Bit-level sample grouping quantizer; For pruning masking;

[0022] The calculation strategy for the weight gradient of the full-precision model during backpropagation is expressed by the following formula:

[0023] ;

[0024] in, This represents the quantized weight gradient corresponding to the l-th layer of the network. For the first l The activation value of layer -1 for b Bit channel grouping quantization operation, For pruning masking.

[0025] The present invention also provides a binary network training device based on gradient quantization, comprising:

[0026] The quantization module is used to perform full-quantization training on the full-precision model using sample images as training samples and the class labels of the sample images as training labels. During the forward propagation of the full-quantization training, the weights and input activation values ​​of the full-precision model are quantized to obtain first quantized data with the target bit width. After backpropagating the first quantized data with the target bit width and outputting the activation value gradient, the activation value gradient is quantized to obtain second quantized data with the target bit width.

[0027] The training module is used to update the input activation value based on the first quantized data and the second quantized data, and to obtain the trained binarized network when the maximum number of training iterations is met.

[0028] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the gradient quantization-based binarization network training method as described above.

[0029] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the gradient quantization-based binarization network training method as described above.

[0030] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the gradient quantization-based binarization network training method as described above.

[0031] The present invention provides a gradient quantization-based binarized network training method and apparatus. During the forward propagation process of fully quantized training of a full-precision model, the weights and input activation values ​​are quantized to the target bit width. After backpropagation of the first quantized data with the target bit width and outputting the activation value gradient, the activation value gradient is quantized to the same bit width, and the gradients of the input activation values ​​and weights are updated accordingly. This achieves binarized gradient training, effectively reduces the training power consumption of the model and the requirements for memory and computing power, thereby improving the training efficiency and inference performance of classification models in complex scenarios. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0033] Figure 1 This is one of the flowcharts for training a binary network based on gradient quantization provided by the present invention.

[0034] Figure 2 This is the second schematic diagram of the training process of a binary network based on gradient quantization provided by the present invention.

[0035] Figure 3 This is a schematic diagram of the structure of the gradient quantization-based binarized network training device provided by the present invention.

[0036] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0038] The following is combined with Figures 1-3 The present invention describes a gradient quantization-based binarization network training method and apparatus.

[0039] Figure 1 This is one of the flowcharts illustrating the training process of a binary network based on gradient quantization provided by this invention, such as... Figure 1 As shown, the method includes the following steps:

[0040] Step 110: Using the sample images as training samples and the class labels of the sample images as training labels, perform full-quantization training on the full-precision model. During the forward propagation of the full-quantization training, quantize the weights and input activation values ​​of the full-precision model to obtain the first quantized data with the target bit width. After backpropagating the first quantized data with the target bit width and outputting the activation value gradient, quantize the activation value gradient to obtain the second quantized data with the target bit width.

[0041] Step 120: Update the activation value gradient and weight gradient according to the first quantization data and the second quantization data, and obtain the trained binarized network under the condition of satisfying the maximum number of training times.

[0042] In the above steps, the sample images and corresponding labels serve as input data for the binarization network to perform the target image processing task; for example, this binarization network can be applied to image classification tasks as well as object recognition tasks. For different downstream tasks, the backbone network of the binarization network (model) is similar, and the computational cost of the backbone network accounts for the majority of the total network cost. Therefore, only quantization of the backbone network is needed to achieve a high speedup.

[0043] In this embodiment, the target bit width can be set according to user needs. The target bit width can be a numerical range, for example, the target bit width is 1 bit to 4 bits; the target bit width can also be a specific value, for example, the target bit width is 1 bit.

[0044] In this embodiment, the maximum number of training iterations can be set according to user needs; for example, the maximum number of training iterations is 2000.

[0045] The following section uses a binarized network to perform an image classification task as an example to illustrate the training method of a gradient quantization-based binarized network.

[0046] In this embodiment, the multi-class image dataset includes sample images from multiple classes. For example, the multi-class image dataset includes: Cifar10: ten classes including airplanes, cars, birds, cats, dogs, deer, frogs, horses, boats, and trucks; Cifar100: 20 major classes, each containing 5 subclasses including animals, food, and plants; Flowers: 102 types of flowers; Cars: 196 classes of cars; Pets: 37 classes including cats and dogs; CUB: 200 classes of birds.

[0047] In this embodiment, the classification evaluation metrics include classification accuracy and speedup ratio; the selected classification model includes ResNet-18 or VGGNet-16, and the corresponding backbone network can be trained by the above-mentioned gradient quantization-based binarization network training method.

[0048] In this embodiment, multiple images and their class labels are used as input samples to the full-precision model, and a binarized model is obtained after multiple full-quantization trainings.

[0049] Specifically, define a parameter as of The full-precision model of the layer is denoted as In each training iteration, a batch of data is sampled from the above-mentioned multiple types of image data. Then input it into the model, as shown in the following formula:

[0050] ;in, It is a feature map. It's batch size. It is the feature dimension; It is an integer; It is the model's first Layers, weights are represented as Loss based on a given batch of data Calculate the gradient of the parameters And update the parameters; for simplicity, here we use and Representing the activation value and parameter gradient, the calculation of the activation value and weight during backpropagation is expressed by the following formula:

[0051] ;

[0052] Among them, the function The gradient calculation formula, for example, for a fully connected layer. The corresponding activation value gradient is calculated as follows:

[0053] ;

[0054] The corresponding weight gradient is calculated as follows:

[0055] .

[0056] To reduce the model's storage requirements and computational costs, the quantization method uses a smaller bit width to represent the model's parameters and activation values, allowing the model to be deployed even with limited device resources. The quantization process is as follows:

[0057] ;

[0058] ;

[0059] in, and It is a quantizer for activation values ​​and weights. and These are the quantized activation values ​​and weights; the forward process is quantized as follows:

[0060] ;

[0061] The above and Since it belongs to low bit-width data, the inference process can be effectively implemented on low bit-width computing kernels, thereby improving the inference speed of the model.

[0062] It should be noted that quantization leads to a loss of accuracy, especially at extremely low bit widths where the degree of accuracy degradation is unacceptable. Therefore, this embodiment uses quantization-aware training (quantizing the model first, then training the model) to reduce the degree of accuracy degradation. Based on this strategy, the activation values ​​and weights of the model can be quantized to a bit width of 1 bit, while ensuring that the performance degradation of the model is within an acceptable range.

[0063] For example, the activation values ​​and weights of the model can be quantized to 1 bit using the sign function shown below:

[0064] ;

[0065] The forward propagation process of the model is represented by the following equation:

[0066] ;

[0067] in, The scaling factor, shared by weights and activation values, is a learnable parameter; the backpropagation process of the model is as follows:

[0068] ;

[0069] It should be noted that for some non-differentiable operations, such as sign functions, this method uses a passthrough to skip these operations. While this method accelerates model inference, the backpropagation process is not accelerated because the gradients are not quantized. In this embodiment, to quickly train the binarized network, a quantizer is used to quantize the gradients of each layer based on quantization-aware training, specifically as follows:

[0070] ;

[0071] in, It is a gradient quantizer; the forward propagation process of full quantization training and quantization-aware training is the same, and the backpropagation process is quantized as follows:

[0072] ;

[0073] In this embodiment, the forward and backward propagation of the model are accelerated by the above formula, and the gradient of the fully quantized training is an unbiased estimate of the gradient in the quantized perceptual training, thereby ensuring that the two can converge to the same point.

[0074] In this embodiment, the Min-max quantization method can be used to quantize the activation gradient and weight gradient of the model; for example, given a gradient tensor quantify it to For each element, first calculate the maximum and minimum values ​​of the tensor, then scale it using the following formula:

[0075] ;

[0076] in, It is the number of quantization intervals. R The range of the gradient can be expressed as: , It is the zero point of the gradient. The input is converted into an integer through random rounding.

[0077] In this embodiment, to reduce the quantization loss, the scaled gradient tensor is dequantized using the following formula to obtain the updated activation gradient and weight gradient:

[0078] .

[0079] Figure 2 This is the second schematic diagram of the training process for a binary network based on gradient quantization provided by this invention. Figure 2In the illustrated embodiment, during the forward propagation of the full-precision model training, the weights and activation values ​​are quantized into 1-bit data (i.e., the first quantized data) using a sign function, and then backpropagated. During the backpropagation process, the gradient of the output activation value of each layer is also quantized into 1 bit (i.e., the second quantized data). The quantized output activation value is calculated together with the quantized input activation value and the weights to obtain the gradient of the input activation value and the parameters. The above operation is repeated until the training rounds are completed, and finally the binary model is returned, which is the trained binary network.

[0080] The gradient quantization-based binarized network training method provided in this invention quantizes the weights and input activation values ​​to the target bit width during the forward propagation process of fully quantized training of a full-precision model. Then, by backpropagating the first quantized data with the target bit width and outputting the activation value gradient, the activation value gradient is quantized to the same bit width, and this is used to update the gradient of the input activation value and the gradient of the weights. This achieves binarized gradient training, effectively reducing the training power consumption, memory, and computing power requirements of the model, thereby improving the training efficiency and inference performance of classification models in complex scenarios.

[0081] It should be noted that some samples have a large gradient range, while others have a small gradient range. This phenomenon is also more pronounced across channels, resulting in a large variance when the model performs quantization calculations, which affects the model's training efficiency and accuracy.

[0082] To address the aforementioned issues, before quantizing the activation value gradient, the method further includes: comparing the bit widths corresponding to multiple activation value gradients output by the full-precision model during backpropagation with a bit width threshold; if the bit width corresponding to the target activation value gradient is less than the bit width threshold, removing the target activation value gradient from the multiple activation value gradients to obtain a new activation value gradient.

[0083] In this embodiment, the quantizer variance is reduced by an activation value gradient pruning strategy. Since the gradients of the relevant parameters of the full-precision network are heterogeneous, the activation value gradient pruning strategy is used to prune smaller groups and allocate the saved computational cost to larger groups, thereby increasing the bit width of the elements within these groups.

[0084] Specifically, to complete 1-bit full quantization training using the above pruning strategy, the following conditions must be guaranteed: (1) If the bit width of the retained group is b Then only (1) Groups can be retained, thus ensuring that the average bit width of the gradient is 1; (2) Random pruning is used instead of deterministic pruning to ensure the unbiasedness of pruning; (3) Groups with larger ranges are more likely to retain rows.

[0085] Based on the above requirements, this embodiment first assigns a possibility to each group, represented as follows:

[0086] .

[0087] In order to satisfy the above conditions (1) and (3), The following conditions must be met:

[0088] and ; .

[0089] Then, the method defines a random mask. To subtract unimportant groups, and then perform group quantification on the remaining groups, in the form of: Furthermore, since the gradient variance mainly comes from the gradient set with a larger range, cutting off the set with a smaller range and increasing the bit width of the set with a larger range can reduce the variance.

[0090] In this embodiment, the bit width threshold can be set according to user needs, and this threshold is used to filter out groups with a smaller range.

[0091] In this embodiment, the forward and backward propagation processes of a single fully connected layer are shown in Table 1 below.

[0092] Table 1: Record of Forward and Backward Propagation Processes in a Single Fully Connected Layer

[0093]

[0094] in, , yes A bit-sized sample group quantizer.

[0095] The gradient quantization-based binarization network training method provided in this invention compares the bit widths corresponding to multiple activation value gradients output by the full-precision model during backpropagation with a bit width threshold. If the bit width corresponding to the target activation value gradient is less than the bit width threshold, the target activation value gradient is removed from the multiple activation value gradients. This gradient pruning strategy reduces the heterogeneity of the gradients corresponding to the full-precision model and reduces the quantization variance, thereby improving the convergence of the model and thus improving the network training efficiency.

[0096] It should be noted that the gradient pruning strategy described above reduces the average bit width of all elements to 1 bit, but since the retained groups are actually... The data is limited to a few bits, thus reducing the actual speedup effect.

[0097] To address this issue, after obtaining the new activation gradient, the method further includes: performing a lossless decomposition of the new activation gradient using the following formula to update the calculation strategy for the activation gradient and weight gradient of the full-precision model during backpropagation:

[0098] ;

[0099] in, for The element in the first position i The value of the bit. i The number of bits in the element. i ∈{0,…, b}

[0100] In this embodiment, by means of The bit data underwent lossless decomposition, and the above decomposition operation was extended to the entire tensor. Backpropagation then becomes:

[0101] ;

[0102] The gradient quantization-based binarization network training method provided in this invention improves the training speed of the binarization network and thus improves the network training efficiency by losslessly decomposing the gradient of the new activation value to update the backpropagation process of the full-precision model.

[0103] Furthermore, after obtaining the first quantized data with the target bit width, the method further includes: performing channel grouping quantization on the gradients between multiple output activation values ​​of the model to obtain the output activation value gradients while the full-precision model performs backpropagation based on the first quantized data; and performing sample grouping quantization on the gradients between multiple weights of the model to obtain the weight gradients.

[0104] It should be noted that, due to the reference to random rounding, the unbiased quantizer... It is an unbiased quantizer, and based on convergence theory, reducing the variance of the quantizer is the key to ensuring convergence. However, outliers in the gradient will increase the range of the gradient, thereby increasing the variance.

[0105] This embodiment employs grouped quantization to calculate gradients during model training, which reduces the impact of outliers on variance. Specifically, in grouped quantization, gradients are first grouped, for example, by sample or channel, and then each group is assigned an independent range instead of sharing the same range. For example, sample-based grouped quantization of gradients takes the following form:

[0106] ;

[0107] in, The first The range and zero point of the activation value gradient for each sample.

[0108] In this embodiment, to ensure the compatibility of the model training method with low-bit-width hardware, a sample-channel joint quantization scheme is adopted. Specifically, this method uses channel grouping quantization when calculating the weight gradient, and sample grouping quantization when calculating the input activation value gradient, thus making the method easily portable to low-bit-width hardware. The backpropagation process is as follows:

[0109] ;

[0110] in, , Representing the The range of channels; the joint quantization scheme of this sample channel accelerates the calculation of weight and activation value gradients.

[0111] In this embodiment, the sample channel joint quantization scheme is integrated with the above-mentioned activation value gradient pruning strategy to obtain the complete form, that is, the activation value gradient calculation strategy of the full-precision model during backpropagation is expressed by the following formula:

[0112] ;

[0113] in, For the first l The quantized activation gradient corresponding to the layer network; For the first l The quantized activation gradient corresponding to the layer network, For the first l The transpose of the weights in the layer network, where sign is a function. for b Bit-level sample grouping quantizer; For pruning masking;

[0114] The strategy for calculating the weight gradient during backpropagation of the full-precision model is expressed by the following formula:

[0115] ;

[0116] in, For the first l The quantized weight gradient corresponding to the layer network, For the first l The activation value of layer -1 for b Bit channel block quantization operation, For pruning masking.

[0117] In this embodiment, the weights, activation values, and gradients during the training process are all quantized into binary data, which can be accelerated by binary matrix multiplication. Specifically, the operation part of binary matrix multiplication includes encoding and binary multiplication. The encoding stores the binary data into integer data, as shown in Table 2.

[0118] Table 2: Coding Flowchart

[0119]

[0120] In this embodiment, binary matrix multiplication can be achieved by using XOR operations and bit counting operations on the encoded integer matrix.

[0121] The gradient quantization-based binarization network training method provided in this invention obtains the output activation value gradient by performing channel group quantization on the gradients between multiple output activation values ​​of the model; and obtains the weight gradient by performing sample group quantization on multiple weights of the model. This allows the model training process to be implemented directly on low-bit hardware, improving the deployment efficiency of the binarization network.

[0122] Furthermore, before performing full-quantization training on the full-precision model using the sample images as training samples and the class labels of the sample images as training labels, the method also includes: removing invalid data, processing missing values, and eliminating duplicates from the sample images to obtain processed sample images.

[0123] In this embodiment, during the data collection and preparation phase, sample images are acquired and processed in the following manner:

[0124] (1) Determine the scope and objectives of data collection: Clearly define the channels for selecting public datasets or collecting images.

[0125] (2) Data crawling: Based on the data collection objectives, use web crawling technology to select keywords to obtain publicly available data.

[0126] (3) Data cleaning and preprocessing: The collected raw data may contain noise, duplicates or incomplete content. In the data cleaning and preprocessing stage, the data is cleaned, including removing invalid data, handling missing values, and eliminating duplicates, to ensure data quality.

[0127] (4) Data display and analysis: Visualize and analyze the acquired data to better understand and utilize it.

[0128] In some embodiments, this implementation acquires and processes sample images in the following manner:

[0129] (1) Hardware adaptation: Adapt and optimize the hardware characteristics and performance of the target terminal device; Since different terminal devices have different hardware architectures and computing capabilities, the quantization model needs to be appropriately hardware optimized to achieve the best performance and effect.

[0130] (2) Performance evaluation: After deployment, the performance of the quantization model on the terminal device is evaluated and tested. Evaluation indicators include the inference speed, accuracy and power consumption of the model, so as to ensure that the model meets the needs and requirements of actual application.

[0131] (3) Deployment and update: As terminal devices and application requirements change, it may be necessary to update and optimize the quantization model regularly. The update and optimization include retraining for new datasets, adjusting quantization parameters, updating hardware adaptation, and optimizing model deployment.

[0132] The gradient quantization-based binarization network training method provided in this invention improves the quality of sample images by removing invalid data, processing missing values, and eliminating duplicates, thereby enhancing the reliability of binarization network training.

[0133] The gradient quantization-based binarization network training device provided by the present invention will be described below. The gradient quantization-based binarization network training device described below can be referred to in correspondence with the gradient quantization-based binarization network training method described above.

[0134] Figure 3 This is a schematic diagram of the structure of the gradient quantization-based binarized network training device provided by the present invention, as shown below. Figure 3 As shown, the gradient quantization-based binarization network training device includes:

[0135] The present invention also provides a binary network training device based on gradient quantization, comprising: a quantization module 310 and a training module 320.

[0136] The quantization module 310 is used to perform full-quantization training on the full-precision model using sample images as training samples and class labels of sample images as training labels. During the forward propagation of the full-quantization training, the weights and input activation values ​​of the full-precision model are quantized to obtain the first quantized data with the target bit width. After backpropagating the first quantized data with the target bit width and outputting the activation value gradient, the activation value gradient is quantized to obtain the second quantized data with the target bit width.

[0137] The training module 320 is used to update the input activation values ​​based on the first quantized data and the second quantized data, and to obtain the trained binarized network when the maximum number of training iterations is met.

[0138] The gradient quantization-based binarized network training device provided in this invention quantizes the weights and input activation values ​​to the target bit width during the forward propagation process of fully quantized training of a full-precision model. Then, by backpropagating the first quantized data with the target bit width and outputting the activation value gradient, the activation value gradient is quantized to the same bit width, and this is used to update the gradient of the input activation value and the gradient of the weights. This achieves binarized gradient training, effectively reducing the training power consumption, memory, and computing power requirements of the model, thereby improving the training efficiency and inference performance of classification models in complex scenarios.

[0139] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 4 As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440. The processor 410, communications interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a gradient quantization-based binarized network training method. This method includes: performing full-quantization training on a full-precision model using sample images as training samples and the class labels of the sample images as training labels; during the forward propagation of the full-precision training, quantizing the weights and input activation values ​​of the full-precision model to obtain first quantized data with a target bit width; after backpropagating the first quantized data with the target bit width and outputting the activation value gradient, quantizing the activation value gradient to obtain second quantized data with the target bit width; updating the activation value gradient and weight gradient based on the first and second quantized data; and obtaining the trained binarized network when the maximum number of training iterations is satisfied.

[0140] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0141] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the gradient quantization-based binarized network training method provided by the above methods. The method includes: performing full-quantization training on a full-precision model using sample images as training samples and class labels of the sample images as training labels; quantizing the weights and input activation values ​​of the full-precision model during the forward propagation of the full-quantization training to obtain first quantized data with a target bit width; quantizing the activation value gradient by backpropagating the first quantized data with the target bit width and outputting the activation value gradient to obtain second quantized data with the target bit width; updating the activation value gradient and weight gradient according to the first quantized data and the second quantized data; and obtaining the trained binarized network under the condition of satisfying the maximum number of training iterations.

[0142] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a gradient quantization-based binarized network training method provided by the above methods. The method includes: performing full-quantization training on a full-precision model using sample images as training samples and class labels of the sample images as training labels; quantizing the weights and input activation values ​​of the full-precision model during the forward propagation of the full-quantization training to obtain first quantized data with a target bit width; quantizing the activation value gradient by backpropagating the first quantized data with the target bit width and outputting the activation value gradient to obtain second quantized data with the target bit width; updating the activation value gradient and weight gradient according to the first quantized data and the second quantized data; and obtaining a trained binarized network while satisfying the maximum number of training iterations.

[0143] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0144] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A gradient quantization-based binarization network training method, wherein the binarization network is applied to image classification tasks, characterized in that, include: The full-precision model is trained using sample images as training samples and class labels of the sample images as training labels. During the forward propagation of the full-precision model, the weights and input activation values ​​of the full-precision model are quantized to obtain the first quantized data with the target bit width. By backpropagating the first quantized data with the target bit width and outputting the activation value gradient, the activation value gradient is quantized to obtain the second quantized data with the target bit width. The activation gradient and weight gradient are updated based on the first quantized data and the second quantized data, and the trained binarized network is obtained under the condition of satisfying the maximum number of training times. The activation gradient calculation strategy of the full-precision model during backpropagation is expressed by the following formula: in, The gradient of the quantized activation value corresponding to the l-th layer of the network. Let be the transpose of the weights corresponding to the l-th layer of the network, and let sign be a function; The gradient of the quantized activation value corresponding to the l-th layer of the network. A b-bit sample group quantizer; M PSQ For pruning masking; The calculation strategy for the weight gradient of the full-precision model during backpropagation is expressed by the following formula: in, This represents the quantized weight gradient corresponding to the l-th layer of the network. The activation value of the (l-1)th layer. For b-bit channel group quantization operations, M PCQ For pruning masking.

2. The binary network training method based on gradient quantization according to claim 1, characterized in that, Before quantizing the activation value gradient, the method further includes: The bit widths corresponding to the multiple activation value gradients output by the full-precision model during backpropagation are compared with a bit width threshold. If the bit width corresponding to the target activation value gradient is less than the bit width threshold, the target activation value gradient is removed from the multiple activation value gradients to obtain a new activation value gradient.

3. The binary network training method based on gradient quantization according to claim 2, characterized in that, After obtaining the new activation value gradient, the method further includes: The new activation gradient is losslessly decomposed using the following formula to update the calculation strategy for the activation gradient and weight gradient of the full-precision model during backpropagation: Where, x i Let i be the value of the i-th bit of the b-bit element, where i is the number of bits in the element and i∈{0,…,b}.

4. The binary network training method based on gradient quantization according to claim 1, characterized in that, After obtaining the first quantized data with the target bit width, the method further includes: When the full-precision model performs backpropagation based on the first quantized data, the gradients between multiple output activation values ​​of the model are quantized by channel grouping to obtain the gradients of the output activation values; the gradients between multiple weights of the model are quantized by sample grouping to obtain the weight gradients.

5. The binary network training method based on gradient quantization according to claim 1, characterized in that, Before performing full-quantization training on the full-precision model using the sample images as training samples and the class labels of the sample images as training labels, the method further includes: The sample image is processed by removing invalid data, handling missing values, and eliminating duplicates to obtain the processed sample image.

6. A gradient quantization-based binarization network training device, employing the gradient quantization-based binarization network training method as described in claim 1, characterized in that, include: The quantization module is used to perform full quantization training on the full-precision model using the sample image as the training sample and the class label of the sample image as the training label. During the forward propagation of the full quantization training, the weights and input activation values ​​of the full-precision model are quantized to obtain the first quantized data with the target bit width. By backpropagating the first quantized data with the target bit width and outputting the activation value gradient, the activation value gradient is quantized to obtain the second quantized data with the target bit width. The training module is used to update the input activation value based on the first quantized data and the second quantized data, and to obtain the trained binarized network when the maximum number of training iterations is met.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the gradient quantization-based binarized network training method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the gradient quantization-based binarization network training method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the gradient quantization-based binarization network training method as described in any one of claims 1 to 5.

Images