Binaryzation network training method and device based on gradient quantization

Abstract

The invention provides a binary network training method and device based on gradient quantization, and the method comprises the steps: carrying out the quantization of the weight and an input activation value of a full-precision model in a full-quantization training process of a full-precision model with a sample image as a training sample and a class label of the sample image as a training label, and obtaining a binary network training result; obtaining first quantized data with a target bit width; quantizing the activation value gradient to obtain second quantized data having the target bit width under the condition that the first quantized data having the target bit width is subjected to back propagation and the activation value gradient is output; and updating the activation value gradient and the weight gradient according to the first quantized data and the second quantized data to obtain a trained binary network. According to the method, binarization gradient training is achieved, the training power consumption of the model and the requirements for memory and computing power are effectively reduced, and then the training efficiency and reasoning performance of the classification model in a complex scene are improved.

Description

Technical Field

[0001] The present invention relates to the field of neural network technology, and in particular to a binary network training method and device based on gradient quantization. Background Art

[0002] With the popularization of smart devices and the rapid development of technologies such as computer vision and speech recognition, classification tasks on terminal devices (such as smartphones, tablets, smart home devices, and embedded systems) have become an important research field.

[0003] Classification models are usually trained in the cloud first and then deployed on terminal devices. Although models can be efficiently trained in the cloud and then deployed on terminal devices, further training on terminal devices is still necessary and advantageous for reasons including privacy protection, reducing data transmission and latency, and maintaining model personalization; however, the computing power of terminal devices is relatively weak, making it difficult to support the training and reasoning of complex deep learning models.

[0004] In related technologies, terminal devices use traditional deep learning models to deploy related classification models. However, due to the relatively weak computing power of the model, complex model training and reasoning scenarios usually require a large amount of computing resources to process high-dimensional data and complex network structures. This is a huge challenge for terminal devices with limited computing power. For example, the model in the image classification task usually contains a large number of convolutional layers and fully connected layers, which requires a large number of floating-point operations. This results in low performance of the classification model on the terminal device due to limited computing resources, and it is difficult to adapt to the classification requirements in complex scenarios. Summary of the invention

[0005] The present invention provides a binary network training method and device based on gradient quantization, which is used to solve the defect of limited computing resources when using traditional deep learning models to perform classification tasks in complex scenarios, resulting in low model training and reasoning efficiency, and improves the training efficiency and reasoning performance of classification models in complex scenarios.

[0006] The present invention provides a binary network training method based on gradient quantization, comprising: Fully quantized training is performed on a full-precision model using a sample image as a training sample and a class label of the sample image as a training label. In the forward propagation process of the full-precision training, the weights and input activation values ​​of the full-precision model are quantized to obtain first quantized data with a target bit width; when the first quantized data with the target bit width is back-propagated and the activation value gradient is output, the activation value gradient is quantized to obtain second quantized data with the target bit width; The activation value gradient and the weight gradient are updated according to the first quantized data and the second quantized data, and a trained binary network is obtained when a maximum number of training times is met.

[0007] According to a binary network training method based on gradient quantization provided by the present invention, before quantizing the activation value gradient, the method further includes: The bit widths corresponding to multiple activation value gradients output by the full-precision model during the back-propagation process are compared with the bit width thresholds respectively; when the bit width corresponding to the target activation value gradient is less than the bit width threshold, the target activation value gradient is removed from the multiple activation value gradients to obtain a new activation value gradient.

[0008] According to a binary network training method based on gradient quantization provided by the present invention, after obtaining the new activation value gradient, the method further includes: The new activation value gradient is losslessly decomposed by the following formula to update the calculation strategy of the activation value gradient and weight gradient of the full-precision model during the back-propagation process: ; in, for Bit element i The value of the bit, i is the number of elements, i ∈{0,…, b}.

[0009] According to a binary network training method based on gradient quantization provided by the present invention, after obtaining the first quantized data with the target bit width, the method further includes: When the full-precision model performs back propagation according to the first quantized data, the gradients between multiple output activation values ​​of the model are quantized by channel groups to obtain the output activation value gradients; and the gradients between multiple weights of the model are quantized by sample groups to obtain weight gradients.

[0010] According to a binary network training method based on gradient quantization provided by the present invention, before performing full quantization training on a full-precision model using a sample image as a training sample and a class label of the sample image as a training label, the method further includes: The sample image is processed to remove invalid data, process missing values ​​and eliminate duplicate items to obtain a processed sample image.

[0011] According to a binary network training method based on gradient quantization provided by the present invention, the activation value gradient calculation strategy of the full-precision model in the back-propagation process is expressed by the following formula: ; in, For the l The quantized activation value gradient corresponding to the layer network, For the l The transpose of the weights corresponding to the layer network, sign is a function; For the l The quantized activation value gradient corresponding to the layer network, for b bit sample grouping quantizer; is the pruning mask; The calculation strategy of the weight gradient of the full-precision model during the back-propagation process is expressed by the following formula: ; in, is the quantized weight gradient corresponding to the l-th layer network, For the l -1 layer activation value, for b Channel grouping quantization operation of bits, is the pruning mask.

[0012] The present invention also provides a binary network training device based on gradient quantization, comprising: A quantization module, configured to perform full quantization training on a full-precision model using a sample image as a training sample and a class label of the sample image as a training label, and quantize the weights and input activation values ​​of the full-precision model in a forward propagation process of the full quantization training to obtain first quantized data having a target bit width; and quantize the activation value gradient to obtain second quantized data having the target bit width by performing back propagation on the first quantized data having the target bit width and outputting an activation value gradient; A training module is used to update the input activation value according to the first quantized data and the second quantized data, and obtain a trained binarized network when a maximum number of training times is met.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a binary network training method based on gradient quantization as described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the binary network training method based on gradient quantization as described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the binary network training method based on gradient quantization as described above is implemented.

[0016] The binary network training method and device based on gradient quantization provided by the present invention quantize weights and input activation values ​​to a target bit width during the forward propagation process of full quantization training of a full-precision model; when backpropagating first quantized data with the target bit width and outputting the activation value gradient, the activation value gradient is quantized to the same bit width, and the gradient of the input activation value and the gradient of the weight are updated accordingly, thereby realizing binary gradient training and effectively reducing the training power consumption of the model as well as the requirements for memory and computing power, thereby improving the training efficiency and reasoning performance of the classification model in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 This is one of the flow charts of binary network training based on gradient quantization provided by the present invention.

[0019] Figure 2 This is the second flowchart of the binary network training based on gradient quantization provided by the present invention.

[0020] Figure 3 It is a structural schematic diagram of a binary network training device based on gradient quantization provided by the present invention.

[0021] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] Combine the following Figure 1-Figure 3 Describe the binary network training method and device based on gradient quantization of the present invention

[0024] Figure 1 This is one of the flow charts of binary network training based on gradient quantization provided by the present invention, such as Figure 1 As shown, the method comprises the following steps: Step 110: Perform full quantization training on the full-precision model using the sample image as the training sample and the class label of the sample image as the training label. In the forward propagation process of the full quantization training, the weights and input activation values ​​of the full-precision model are quantized to obtain first quantized data with a target bit width; when the first quantized data with the target bit width is back-propagated and the activation value gradient is output, the activation value gradient is quantized to obtain second quantized data with the target bit width; Step 120: Update the activation value gradient and the weight gradient according to the first quantized data and the second quantized data, and obtain a trained binary network when the maximum number of training times is met.

[0025] In the above steps, the sample image and the corresponding label are the input data for the binary network to perform the target image processing task; for example, the binary network can be applied to image classification tasks and target recognition tasks. For different downstream tasks, the backbone network of the binary network (model) is similar, and the computational cost of the backbone network accounts for most of the entire network cost, so only the backbone network needs to be quantized to achieve a higher speedup ratio.

[0026] In this embodiment, the target bit width can be set according to user needs. The target bit width can be a numerical range, for example, the target bit width is 1 bit to 4 bits; the target bit width can also be a specific value, for example, the target bit width is 1 bit.

[0027] In this embodiment, the maximum number of training times can be set according to user needs, for example, the maximum number of training times is 2000.

[0028] Below, taking the use of a binary network to perform an image classification task as an example, the binary network training method based on gradient quantization is explained.

[0029] In this embodiment, the multi-category image dataset includes multi-category sample images. For example, the multi-category image dataset includes: Cifar10: ten categories including airplanes, cars, birds, cats, dogs, deer, frogs, horses, ships, and trucks; Cifar100: 20 major categories, each major category contains 5 minor categories including animals, food, and plants; Flowers: 102 kinds of flowers; Cars: 196 categories of cars; Pets: 37 categories of cats and dogs; CUB: 200 categories of birds.

[0030] In this embodiment, the classification evaluation indicators include classification accuracy and acceleration ratio, etc. The selected classification models include ResNet-18 or VGGNet-16, and the corresponding backbone network can be trained by the above-mentioned binary network training method based on gradient quantization.

[0031] In this embodiment, multiple images and class labels of the images are input as input samples to the full-precision model, and a binarized model is obtained after multiple full-quantization trainings.

[0032] Specifically, define a parameter as of The full-precision model of the layer is denoted by , in each training iteration, a batch of data is sampled from the above multi-class image data ( ) and input it into the model, which is shown in the following equation: ;in, is the feature map, is the batch size, is the characteristic dimension; is an integer; The model is layer, the weights are expressed as ; Based on the loss of a given batch of data , calculate the parameter gradient And update the parameters; for simplicity, here we use and Represents the activation value and parameter gradient. The calculation of activation value and weight in the back propagation process is expressed as follows: ; Among them, the function is the gradient calculation formula, for example, for a fully connected layer , the corresponding activation value gradient calculation method is: ; The corresponding weight gradient calculation method is: .

[0033] In order to reduce the storage requirements and computational costs of the model, the quantization method uses a smaller bit width to represent the model's parameters and activation values, so that the model can be deployed when device resources are limited; the quantization process is as follows: ; ; in, and is a quantizer for activations and weights, and are the quantized activation values ​​and weights; the forward process is quantized as: ; Above and It belongs to low-bit-width data, and the reasoning process can be effectively implemented on the low-bit-width computing core, thereby improving the reasoning speed of the model.

[0034] It should be noted that quantization brings about a loss in precision, especially at extremely low bit widths, the degree of precision degradation is unacceptable; therefore, this embodiment adopts quantization-aware training (first quantize the model, then train the model) to reduce the degree of precision degradation; based on this strategy, the activation values ​​and weights of the model can be quantized to a bit width of 1 bit, and ensure that the performance degradation of the model is within an acceptable range.

[0035] For example, the activation value and weight of the model are quantized to 1 bit through the sign function shown in the following formula: ; The forward propagation process of the model is expressed as follows: ; in, is the scaling factor shared by weights and activation values, which is a learnable parameter; the back propagation process of the model is: ; It should be noted that for some non-differentiable operations, such as the sign function, this method uses a pass-through to skip this operation. Although this method accelerates the reasoning of the model, the back propagation process is not accelerated because the gradient is not quantized. In order to quickly train the binary network, this embodiment quantizes the gradient of each layer through a quantizer based on quantization-aware training, which is specifically expressed as follows: ; in, is the quantizer of the gradient; the forward propagation process of fully quantized training is the same as that of quantization-aware training, and the backpropagation process is quantized as: ; In this embodiment, both the forward and backward propagation of the model are accelerated by the above formula, and the gradient of the full quantization training is an unbiased estimate of the gradient in the quantization-aware training, thereby ensuring that the two can converge to the same point.

[0036] In this embodiment, the Min-max quantization method can be used to quantize the activation value gradient and weight gradient of the model; for example, given a gradient tensor , quantify it to For each element, first calculate the maximum and minimum values ​​of the tensor, and then scale it by the following formula: ; in, is the number of quantization intervals, R is the range of the gradient, which can be expressed as: , is the zero point of the gradient, Converts the input to an integer using a random truncation operation.

[0037] In this embodiment, in order to reduce the quantization loss, the scaled gradient tensor is dequantized by the following formula to obtain the updated activation value gradient and weight gradient: .

[0038] Figure 2 This is the second flow chart of the binary network training based on gradient quantization provided by the present invention. Figure 2 In the illustrated embodiment, during the forward propagation of full-precision model training, the weights and activation values ​​are quantized to 1-bit data (i.e., first quantized data) through a sign function, and then back-propagated; during the back-propagation process, the output activation value gradient of each layer is also quantized to 1 bit (i.e., second quantized data), and the quantized output activation value is calculated with the quantized input activation value and the weight to obtain the input activation value and the gradient of the parameter, and the above operation is repeated until the number of training rounds is completed, and finally the binary model is returned, that is, a trained binary network is obtained.

[0039] The binary network training method based on gradient quantization provided by an embodiment of the present invention quantizes the weights and input activation values ​​to the target bit width during the forward propagation process of full-quantization training of the full-precision model; when backpropagating the first quantized data with the target bit width and outputting the activation value gradient, the activation value gradient is quantized to the same bit width, and the gradient of the input activation value and the gradient of the weight are updated accordingly, thereby realizing binary gradient training and effectively reducing the training power consumption of the model as well as the requirements for memory and computing power, thereby improving the training efficiency and reasoning performance of the classification model in complex scenarios.

[0040] It should be noted that the gradient range corresponding to some samples is very large, while the gradient range corresponding to other samples is very small; this phenomenon is also more obvious across channels, resulting in a large variance in the model's quantization calculations, affecting the model's training efficiency and accuracy.

[0041] To address the above problem, before quantizing the activation value gradient, the method also includes: comparing the bit widths corresponding to multiple activation value gradients output by the full-precision model during the back-propagation process with the bit width thresholds respectively; when the bit width corresponding to the target activation value gradient is less than the bit width threshold, removing the target activation value gradient from the multiple activation value gradients to obtain a new activation value gradient.

[0042] In this embodiment, the quantizer variance is reduced by an activation value gradient pruning strategy; since the gradients of relevant parameters of the full-precision network are heterogeneous, the activation value gradient pruning strategy is used to prune smaller groups and allocate the saved computational costs to larger groups, thereby enhancing the bit width of the elements within these groups.

[0043] Specifically, the following conditions must be met to complete 1-bit full quantization training using the above pruning strategy: (1) If the bit width of the retained group is b , then only Groups of can be retained, thus ensuring that the average bit width of the gradient is 1; (2) random pruning is used instead of deterministic pruning to ensure the unbiased pruning; (3) The larger the group, the higher the probability of retaining rows.

[0044] Based on the above requirements, this embodiment first assigns a possibility to each group, which is expressed as: .

[0045] In order to satisfy the above conditions (1) and (3), Need to meet: and ; .

[0046] The method then defines the random mask To subtract unimportant groups, and perform group quantization on the retained groups, the form of Since the gradient variance mainly comes from the gradient group with a larger range, cutting off the group with a smaller range and increasing the bit width of the group with a larger range can reduce the variance.

[0047] In this embodiment, the bit width threshold can be set according to user requirements, and the threshold is used to filter out groups with a smaller range.

[0048] In this embodiment, the forward and backward propagation processes of a single fully connected layer are shown in Table 1 below.

[0049] Table 1: Record of forward and backward propagation process of a single fully connected layer

[0050] in, , yes quantizer for grouping samples of bits.

[0051] The binary network training method based on gradient quantization provided by an embodiment of the present invention compares the bit widths corresponding to multiple activation value gradients output by the full-precision model during the back-propagation process with the bit width thresholds respectively; when the bit width corresponding to the target activation value gradient is less than the bit width threshold, the target activation value gradient is removed from the multiple activation value gradients, and the heterogeneity of the corresponding gradients of the full-precision model is reduced and the quantization variance is reduced through the gradient pruning strategy, thereby improving the convergence of the model and thereby improving the network training efficiency.

[0052] It should be noted that the above gradient pruning strategy reduces the average bit width of all elements to 1 bit, but since the retained group is actually bits of data, so the actual acceleration effect is weakened.

[0053] To address this problem, after obtaining the new activation value gradient, the method further includes: losslessly decomposing the new activation value gradient by the following formula to update the calculation strategy of the activation value gradient and the weight gradient of the full-precision model during the back-propagation process: ; in, for Bit element i The value of the bit, i is the number of elements, i ∈{0,…, b}.

[0054] In this embodiment, by The bit data is losslessly decomposed and the above decomposition operation is extended to the entire tensor. At this time, the back propagation becomes: ; The binary network training method based on gradient quantization provided by the embodiment of the present invention further improves the training speed of the binary network by losslessly decomposing the new activation value gradient to update the back propagation process of the full-precision model, thereby improving the network training efficiency.

[0055] Furthermore, after obtaining the first quantized data with the target bit width, the method also includes: when the full-precision model performs back propagation according to the first quantized data, performing channel grouping quantization on the gradients between multiple output activation values ​​of the model to obtain output activation value gradients; performing sample grouping quantization on the gradients between multiple weights of the model to obtain weight gradients.

[0056] It should be noted that due to the reference to random rounding, the unbiased quantizer It is an unbiased quantizer. Based on the convergence theory, reducing the variance of the quantizer is the key to ensuring convergence. However, outliers in the gradient will increase the range of the gradient, thereby increasing the variance.

[0057] This embodiment uses group quantization to calculate gradients during model training, which can reduce the impact of outliers on variance. Specifically, in group quantization, gradients are first grouped, such as by sample or channel, and then an independent range is assigned to each group instead of sharing the same range. For example, sample group quantization is performed on the gradient, and its form is as follows: ; in, Respectively The range and zero point of the activation value gradient for each sample.

[0058] In this embodiment, in order to ensure the compatibility of the model training method with low-bitwidth hardware, a sample channel joint quantization scheme is adopted; specifically, the method uses channel grouping quantization when calculating weight gradients, and uses sample grouping quantization when calculating input activation value gradients, so that the method can be easily transplanted to low-bitwidth hardware; the back propagation process is as follows: ; in, , Representative The sample channel joint quantization scheme accelerates the calculation of weight and activation value gradients.

[0059] In this embodiment, the sample channel joint quantization scheme is integrated with the above-mentioned activation value gradient pruning strategy to obtain a complete form, that is, the activation value gradient calculation strategy of the full-precision model during the back-propagation process is expressed by the following formula: ; in, For the l The quantized activation value gradient corresponding to the layer network; For the l The quantized activation value gradient corresponding to the layer network, For the l The transpose of the weights of the layer network, sign is a function, for b bit sample grouping quantizer; is the pruning mask; The calculation strategy of the weight gradient of the full-precision model during back-propagation is expressed as follows: ; in, For the lThe quantized weight gradient corresponding to the layer network, For the l -1 layer activation value, for b Channel grouping quantization operation of bits, is the pruning mask.

[0060] In this embodiment, the weights, activation values, and gradients in the training process are quantized into binary data, which can be accelerated by binary matrix multiplication. Specifically, the operation part of the binary matrix multiplication includes encoding and binary multiplication, wherein the encoding stores the binary data into integer data, as shown in Table 2.

[0061] Table 2: Coding flow chart

[0062] In this embodiment, for the encoded integer matrix, binary matrix multiplication can be implemented through XNOR operation and bit counting operation.

[0063] The binary network training method based on gradient quantization provided by the embodiment of the present invention obtains the output activation value gradient by performing channel grouping quantization on the gradients between multiple output activation values ​​of the model; and obtains the weight gradient by performing sample grouping quantization on multiple weights of the model, thereby allowing the model training process to be directly implemented on low-bit hardware, thereby improving the deployment efficiency of the binary network.

[0064] Furthermore, before performing full quantization training on the full-precision model using the sample image as a training sample and the class label of the sample image as a training label, the method also includes: removing invalid data, processing missing values, and eliminating duplicates from the sample image to obtain a processed sample image.

[0065] In this embodiment, during the data collection and preparation stage, the present embodiment collects and processes sample images in the following manner: (1) Determine the scope and objectives of data collection: clearly select public data sets or channels for collecting images.

[0066] (2) Data crawling: Based on the data collection objectives, use web crawler technology to select keywords to obtain publicly visible data.

[0067] (3) Data cleaning and preprocessing: The collected raw data may contain noise, duplication or incomplete content. During the data cleaning and preprocessing stage, the data is cleaned, including removing invalid data, processing missing values, eliminating duplicates, etc., to ensure the quality of the data.

[0068] (4) Data display and analysis: Visualize and analyze the acquired data to better understand and utilize the data.

[0069] In some embodiments, the present implementation uses the following method to collect and process sample images: (1) Hardware adaptation: Adapt and optimize the hardware characteristics and performance of the target terminal device. Since different terminal devices have different hardware architectures and computing capabilities, appropriate hardware optimization of the quantization model is required to achieve the best performance and effect.

[0070] (2) Performance evaluation: After deployment, the performance of the quantization model on the terminal device is evaluated and tested. The evaluation indicators include the model's inference speed, accuracy, and power consumption to ensure that the model meets the needs and requirements of actual applications.

[0071] (3) Deployment updates: As terminal devices and application requirements change, the quantization model may need to be updated and optimized regularly. The updates and optimizations include retraining for new data sets, adjusting quantization parameters, updating hardware adaptation, and optimizing model deployment.

[0072] The binary network training method based on gradient quantization provided in the embodiment of the present invention improves the quality of sample images by removing invalid data, processing missing values ​​and eliminating duplicate items, thereby improving the reliability of binary network training.

[0073] The following is a description of a binary network training device based on gradient quantization provided by the present invention. The binary network training device based on gradient quantization described below and the binary network training method based on gradient quantization described above can be referred to each other.

[0074] Figure 3 : is a schematic diagram of the structure of a binary network training device based on gradient quantization provided by the present invention, such as Figure 3 As shown, the binary network training device based on gradient quantization includes: The present invention also provides a binary network training device based on gradient quantization, including: a quantization module 310 and a training module 320.

[0075] The quantization module 310 is used to perform full quantization training on the full-precision model using the sample image as the training sample and the class label of the sample image as the training label. In the forward propagation process of the full quantization training, the weights and input activation values ​​of the full-precision model are quantized to obtain first quantized data with a target bit width; when the first quantized data with the target bit width is back-propagated and the activation value gradient is output, the activation value gradient is quantized to obtain second quantized data with the target bit width; The training module 320 is used to update the input activation value according to the first quantized data and the second quantized data, and obtain a trained binarized network when the maximum number of training times is met.

[0076] The binary network training device based on gradient quantization provided by an embodiment of the present invention quantizes weights and input activation values ​​to a target bit width during the forward propagation process of fully quantized training of a full-precision model; when backpropagating first quantized data with the target bit width and outputting the activation value gradient, the activation value gradient is quantized to the same bit width, and the gradient of the input activation value and the gradient of the weight are updated accordingly, thereby realizing binary gradient training and effectively reducing the training power consumption of the model as well as the requirements for memory and computing power, thereby improving the training efficiency and reasoning performance of the classification model in complex scenarios.

[0077] Figure 4 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute a binary network training method based on gradient quantization, the method comprising: performing full quantization training on a full precision model with a sample image as a training sample and a class label of the sample image as a training label, and in the forward propagation process of the full quantization training, quantizing the weight and input activation value of the full precision model to obtain first quantized data with a target bit width; quantizing the activation value gradient to obtain second quantized data with a target bit width by back-propagating the first quantized data with the target bit width and outputting the activation value gradient; updating the activation value gradient and the weight gradient according to the first quantized data and the second quantized data, and obtaining a trained binary network when the maximum number of training times is met.

[0078] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0079] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the binary network training method based on gradient quantization provided by the above methods, the method including: performing full quantization training on a full-precision model using a sample image as a training sample and a class label of the sample image as a training label, and in the forward propagation process of the full quantization training, quantizing the weights and input activation values ​​of the full-precision model to obtain first quantized data with a target bit width; quantizing the activation value gradient to obtain second quantized data with a target bit width by back-propagating the first quantized data with the target bit width and outputting the activation value gradient; updating the activation value gradient and the weight gradient according to the first quantized data and the second quantized data, and obtaining a trained binary network when the maximum number of training times is met.

[0080] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the binary network training method based on gradient quantization provided by the above-mentioned methods, the method comprising: performing full quantization training on a full-precision model using a sample image as a training sample and a class label of the sample image as a training label, and in the forward propagation process of the full quantization training, quantizing the weights and input activation values ​​of the full-precision model to obtain first quantized data with a target bit width; quantizing the activation value gradient to obtain second quantized data with a target bit width by back-propagating the first quantized data with the target bit width and outputting the activation value gradient; updating the activation value gradient and the weight gradient according to the first quantized data and the second quantized data, and obtaining a trained binary network when the maximum number of training times is met.

[0081] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0082] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A binary network training method based on gradient quantization, characterized in that: include: Performing full quantization training on a full-precision model using a sample image as a training sample and a class label of the sample image as a training label, and quantizing weights and input activation values ​​of the full-precision model during forward propagation of the full quantization training to obtain first quantized data having a target bit width; In a case where the first quantized data having the target bit width is back-propagated and the activation value gradient is output, the activation value gradient is quantized to obtain second quantized data having the target bit width; The activation value gradient and the weight gradient are updated according to the first quantized data and the second quantized data, and a trained binary network is obtained when a maximum number of training times is met.

2. The binary network training method based on gradient quantization according to claim 1 is characterized in that: Before quantizing the activation value gradient, the method further includes: The bit widths corresponding to multiple activation value gradients output by the full-precision model during the back-propagation process are compared with the bit width thresholds respectively; when the bit width corresponding to the target activation value gradient is less than the bit width threshold, the target activation value gradient is removed from the multiple activation value gradients to obtain a new activation value gradient.

3. The binary network training method based on gradient quantization according to claim 2 is characterized in that: After obtaining the new activation value gradient, the method further includes: The new activation value gradient is losslessly decomposed by the following formula to update the calculation strategy of the activation value gradient and weight gradient of the full-precision model during the back-propagation process: ； in, for Bit element i The value of the bit, i is the number of elements, i ∈{0,…, b }.

4. The binary network training method based on gradient quantization according to claim 1 is characterized in that: After obtaining the first quantized data having the target bit width, the method further includes: When the full-precision model performs back propagation according to the first quantized data, the gradients between multiple output activation values ​​of the model are quantized by channel groups to obtain the output activation value gradients; and the gradients between multiple weights of the model are quantized by sample groups to obtain weight gradients.

5. The binary network training method based on gradient quantization according to claim 1 is characterized in that: Before performing full quantization training on the full-precision model using the sample image as a training sample and the class label of the sample image as a training label, the method further includes: The sample image is processed to remove invalid data, process missing values ​​and eliminate duplicate items to obtain a processed sample image.

6. The binary network training method based on gradient quantization according to any one of claims 1 to 4, characterized in that: The activation value gradient calculation strategy of the full-precision model during the back-propagation process is expressed by the following formula: ； in, For the l The quantized activation value gradient corresponding to the layer network, For the l The transpose of the weights corresponding to the layer network, sign is a function; For the l The quantized activation value gradient corresponding to the layer network, for b bit sample grouping quantizer; is the pruning mask; The calculation strategy of the weight gradient of the full-precision model during the back-propagation process is expressed by the following formula: ； in, is the quantized weight gradient corresponding to the l-th layer network, For the l -1 layer activation value, for b Channel grouping quantization operation of bits, is the pruning mask.

7. A binary network training device based on gradient quantization, characterized in that: include: A quantization module, configured to perform full quantization training on a full-precision model using a sample image as a training sample and a class label of the sample image as a training label, and quantize the weights and input activation values ​​of the full-precision model during forward propagation of the full quantization training to obtain first quantized data having a target bit width; In a case where the first quantized data having the target bit width is back-propagated and the activation value gradient is output, the activation value gradient is quantized to obtain second quantized data having the target bit width; A training module is used to update the input activation value according to the first quantized data and the second quantized data, and obtain a trained binarized network when a maximum number of training times is met.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the binary network training method based on gradient quantization as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the binary network training method based on gradient quantization as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the binary network training method based on gradient quantization as described in any one of claims 1 to 6 is implemented.

Images