Method and System for Gradient Clipping for Stable Quantization-Aware Training

Abstract

A gradient clipping method and system for learning after stable quantization are presented. The gradient clipping method for learning after stable quantization proposed in the present invention includes the step of performing dropout to replace the normalization effect during the forwarding process of a classifier after performing quantization of an input model, and the step of performing gradient clipping when the combination of the norm of the gradient and the norm of the weight for learning after quantization exceeds a predetermined threshold.

Description

Technology Field

[0001] The present invention relates to a gradient clipping method and system for learning after stable quantization. Background Technology

[0002] Recently, as the accuracy of artificial neural networks has increased, the number of parameters has also been rising. If the parameter size becomes too large, it becomes difficult to deploy the network on devices such as mobile phones. To reduce this number of parameters, various lightweighting techniques, such as pruning, knowledge distillation, and quantization, are being researched and developed. Among these, quantization is a method that reduces the memory usage and computational cost of an over-parameterized model during inference by lowering the number of bits for weights and activation terms. However, simply lowering the number of bits reduces the representation range of weights and activation terms, leading to a significant decrease in accuracy. This decrease in accuracy becomes even greater, particularly when aggressive quantization of 4 bits or less is performed.

[0003] The technique applied to address this is Quantization-Aware Training (QAT). Also known as simulated quantization, the QAT technique simulates how quantization proceeds in advance by adding fake quantization nodes during the training process. There exists a technique called HAWQ (Hessian-Aware Quantization) that has achieved state-of-the-art performance by effectively utilizing this QAT technique. However, experiments have confirmed a problem where the accuracy of the HAWQ technique gradually decreases due to overfitting as the number of fine-tuning epochs increases during the training process.

[0004] In the existing QAT technique, additional training is performed on the quantized model. At this time, a fake quantization node is added to simulate quantization by making it behave as if actual quantization is taking place. Through this additional training, the accuracy that was reduced by using the existing quantization is restored as much as possible.

[0005] However, the HAWQ method, which demonstrated the highest performance in this QAT, becomes overfitted as the number of training epochs increases, converging to low accuracy. Additionally, QAT attempts to mitigate the loss of accuracy after quantization by performing folding, which involves convolving the batch normalization layer with the weights; however, combining the batch normalization terms in this process prevents the utilization of various normalization effects. Unless the issues regarding overfitting and batch normalization terms arising from these quantization and additional training processes are resolved, it becomes difficult to recover from the accuracy loss caused by quantization. Prior art literature

[0006] [1] Benoit Jacob et al. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference. In CVPR, 2018[2] Andrew Brock et al. High-Performance Large-Scale Image Recognition Without Normalization, arXiv preprint arXiv:2102.06171[3] Zhewei Yao et.al. HAWQV3: Dyadic Neural Network Quantization. In ICML, 2021 The problem to be solved

[0007] The technical problem that the present invention aims to solve is to provide a gradient clipping method and a system for stable post-quantization learning that applies dropout and a gradient clipping method that replaces the effect of a batch normalization layer and prevents a decrease in accuracy. The present invention aims to achieve stable learning through gradient clipping and to demonstrate superior quantitative performance compared to other algorithms by adding dropout. means of solving the problem

[0008] In one aspect, the gradient clipping method for stable post-quantization learning proposed in the present invention includes the step of performing dropout to replace the normalization effect during the forwarding process of a classifier after performing quantization of an input model, and the step of performing gradient clipping when the combination of the norm of the gradient and the norm of the weight for learning after quantization exceeds a predetermined threshold.

[0009] The step of performing gradient clipping when the combination of the gradient norm and the weight norm for learning after the above quantization exceeds a predetermined threshold value limits the magnitude of the gradient by performing gradient clipping when the value obtained by dividing the gradient norm representing the direction of the gradient by the weight norm exceeds a predetermined threshold value.

[0010] The step of performing gradient clipping when the combination of the gradient norm and the weight norm for learning after the above quantization exceeds a predetermined threshold value occurs when the expressible range is reduced in the case of quantization and the gradient is greater than the weight, and when the gradient is greater than the weight, it is determined that learning is complete and the gradient is limited according to the predetermined threshold value to optimize the parameters of the classifier.

[0011] The step of performing dropout to replace the normalization effect during the forwarding process of the classifier after performing quantization of the above input model replaces the normalization effect caused by folding of the batch normalization layer through dropout that probabilistically removes some neurons to obtain an unbiased output value.

[0012] In another aspect, the gradient clipping system for stable post-quantization learning proposed in the present invention includes a dropout performing unit that performs dropout to replace the normalization effect during the forwarding process of a classifier after performing quantization of an input model, and a gradient clipping unit that performs gradient clipping when the combination of the norm of the gradient and the norm of the weight for learning after quantization exceeds a predetermined threshold. Effects of the invention

[0013] The gradient clipping method and system for stable post-quantization learning that apply dropout and replaces the effect of the batch normalization layer and prevents accuracy reduction according to embodiments of the present invention can achieve stable learning through gradient clipping and exhibit superior quantitative performance compared to other algorithms by adding dropout. Brief explanation of the drawing

[0014] FIG. 1 is a flowchart illustrating a gradient clipping method for learning after stable quantization according to one embodiment of the present invention. FIG. 2 is a diagram showing the configuration of a gradient clipping system for stable post-quantization learning according to one embodiment of the present invention. FIG. 3 is a diagram illustrating a dropout process according to an embodiment of the present invention. FIG. 4 is a diagram illustrating the process of object detector learning according to one embodiment of the present invention. FIG. 5 is a graph for comparing the results of dropout according to one embodiment of the present invention with the prior art. Specific details for implementing the invention

[0015] Existing DNN-based quantization techniques are broadly classified into two types: PTQ (Post Training Quantization) and QAT (Quantization-Aware Training). PTQ quantization results in a decrease in accuracy because the number of bits represented is reduced compared to existing FP32 models. To overcome this, QAT, which involves additional training after quantization, is used to achieve accuracy close to that of FP32 models. However, since this QAT technique involves folding the batch normalization layer, it fails to apply various normalization effects of the batch normalization term. Furthermore, this QAT technique converges to lower accuracy as the number of training epochs increases.

[0016] Therefore, the present invention proposes a configuration that applies a gradient clipping method and dropout to replace the effect of this batch normalization layer and prevent a decrease in accuracy. As a result, the present invention achieves stable learning through gradient clipping and demonstrates superior quantitative performance compared to other algorithms by adding dropout. Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0018] FIG. 1 is a flowchart illustrating a gradient clipping method for learning after stable quantization according to one embodiment of the present invention.

[0019] The proposed gradient clipping method for stable post-quantization learning includes a step (110) of performing dropout to replace the normalization effect during the forwarding process of a classifier after performing quantization of an input model, and a step (120) of performing gradient clipping when the combination of the norm of the gradient and the norm of the weight for learning after quantization exceeds a predetermined threshold.

[0020] In step (110), after performing quantization of the input model, dropout is performed to replace the normalization effect during the forwarding process of the classifier.

[0021] According to an embodiment of the present invention, the normalization effect caused by the folding of the batch normalization layer is replaced by dropout (111), which probabilistically removes some neurons to obtain an unbiased output value. Since the normalization effect is lost due to the folding of the batch normalization layer, the normalization effect can be replaced by dropout (111).

[0022] In step (120), gradient clipping is performed when the combination of the gradient norm and the weight norm for learning after quantization exceeds a predetermined threshold.

[0023] According to an embodiment of the present invention, the value obtained by dividing the norm of the slope representing the direction of the slope by the norm of the weight is compared with a predetermined threshold value (121), and if it exceeds the predetermined threshold value, slope clipping is performed to limit the magnitude of the slope.

[0024] In the case of quantization, the expressible range is reduced, and a case occurs where the gradient is greater than the weight. When the gradient is greater than the weight, it is determined that learning is complete, and the gradient can be limited according to a predetermined threshold value to optimize the parameters of the classifier (122).

[0025] Gradient clipping according to an embodiment of the present invention limits the gradient when a specific condition exceeds a threshold value. The specific condition refers to the case where the value obtained by dividing the norm of the gradient by the norm of the weights exceeds a certain threshold value (e.g., default=0.01). The norm of the gradient indicates the direction of the gradient, and the norm of the weights and the threshold value determine the gradient value that limits it.

[0026] Basically, during the initial stages of the learning process, the gradient tends to be large and the weights tend to be small because a lot needs to be learned. Conversely, once learning is somewhat complete, the gradient becomes small and the weights become large. In the case of quantization, the representable range decreases, so there may be instances where the gradient is greater than the weights. In such cases, it is determined that the model has over-learned, and the gradient is limited.

[0027]

[0028] The proposed algorithm, which has two configurations of gradient clipping and dropout, can expect improved performance by limiting excessive gradients during the quantization process to achieve stable learning while adding a regularization effect.

[0030] FIG. 2 is a diagram showing the configuration of a gradient clipping system for stable post-quantization learning according to one embodiment of the present invention.

[0031] A gradient clipping system (200) according to the present embodiment may include a processor (210), a bus (220), a network interface (230), memory (240), and a database (250). The memory (240) may include an operating system (241) and gradient clipping and dropout (242) for learning after quantization. The processor (210) may include a dropout execution unit (211) and a gradient clipping unit (212). In other embodiments, the gradient clipping system (200) may include more components than those of FIG. 2. However, it is not necessary to clearly illustrate most prior art components. For example, the gradient clipping system (200) may include other components such as a display or a transceiver.

[0032] Memory (240) is a computer-readable recording medium and may include a non-perishable mass storage device such as RAM (random access memory), ROM (read only memory), and a disk drive. Additionally, the memory (240) may store an operating system (241) and program code for gradient clipping and dropout (242) for learning after quantization. These software components may be loaded from a computer-readable recording medium separate from the memory (240) using a drive mechanism (not shown). This separate computer-readable recording medium may include a computer-readable recording medium (not shown) such as a floppy drive, disk, tape, DVD / CD-ROM drive, or memory card. In another embodiment, the software components may be loaded into the memory (240) via a network interface (230) rather than a computer-readable recording medium.

[0033] The bus (220) can enable communication and data transmission between components of the slope clipping system (200). The bus (220) can be configured using a high-speed serial bus, a parallel bus, a Storage Area Network (SAN), and / or other suitable communication technology.

[0034] The network interface (230) may be a computer hardware component for connecting the tilt clipping system (200) to a computer network. The network interface (230) may connect the tilt clipping system (200) to a computer network via a wireless or wired connection.

[0035] The database (250) can serve to store and maintain all information necessary for gradient clipping and dropout for learning after quantization. In FIG. 2, the database (250) is shown built and included inside the gradient clipping system (200), but it is not limited thereto and may be omitted depending on the system implementation method or environment, or it is also possible for all or part of the database to exist as an external database built on a separate system.

[0036] The processor (210) may be configured to process instructions of a computer program by performing input / output operations of basic arithmetic, logic, and gradient clipping systems (200). Instructions may be provided to the processor (210) via memory (240) or a network interface (230) and via a bus (220). The processor (210) may be configured to execute program code for a dropout execution unit (211) and a gradient clipping unit (212). Such program code may be stored in a recording device such as memory (240).

[0037] The dropout performing unit (211) and the slope clipping unit (212) can be configured to perform the steps (110–120) of FIG. 1.

[0038] The slope clipping system (200) may include a dropout performing unit (211) and a slope clipping unit (212).

[0039] A dropout performing unit (211) according to an embodiment of the present invention performs dropout to replace the normalization effect during the forwarding process of a classifier after performing quantization of the input model.

[0040] According to an embodiment of the present invention, the normalization effect caused by the folding of the batch normalization layer is replaced by dropout, which probabilistically removes some neurons to obtain an unbiased output value. Since the normalization effect is lost due to the folding of the batch normalization layer, the normalization effect can be replaced by dropout.

[0041] The gradient clipping unit (212) according to an embodiment of the present invention performs gradient clipping when the combination of the gradient norm and the weight norm for learning after quantization exceeds a predetermined threshold value.

[0042] According to an embodiment of the present invention, the value obtained by dividing the norm of the slope representing the direction of the slope by the norm of the weight is compared with a predetermined threshold value, and if it exceeds the predetermined threshold value, slope clipping is performed to limit the magnitude of the slope.

[0043] In the case of quantization, the expressible range is reduced, resulting in a situation where the gradient is greater than the weight. When the gradient is greater than the weight, it is determined that learning is complete, and the parameters of the classifier can be optimized by limiting the gradient according to a predetermined threshold.

[0045] FIG. 3 is a diagram illustrating a dropout process according to an embodiment of the present invention.

[0046] FIG. 3(a) is a diagram illustrating batch normalization according to the prior art, and FIG. 3(b) is a diagram illustrating a dropout process according to an embodiment of the present invention.

[0047] Referring to Fig. 3(a), in the conventional QAT method, additional training is performed on the quantized model. At this time, a fake quantization node is added to simulate quantization by making it behave as if actual quantization is taking place. Through this additional training, the accuracy that was reduced by using conventional quantization is restored as much as possible.

[0048] However, the HAWQ method, which demonstrated the highest performance in this QAT, becomes overfitted as the number of training epochs increases, converging to low accuracy. Additionally, QAT attempts to mitigate the loss of accuracy after quantization by performing folding, which involves convolving the batch normalization layer with the weights; however, combining the batch normalization terms in this process prevents the utilization of various normalization effects. Unless the issues regarding overfitting and batch normalization terms arising from these quantization and additional training processes are resolved, it becomes difficult to recover from the accuracy loss caused by quantization.

[0049] As shown in FIG. 3(b), the present invention utilizes dropout to replace the normalization effect of the batch normalization layer. Dropout is a technique that probabilistically removes some neurons to obtain unbiased output values ​​and exhibits excellent normalization effects. In the present invention, the normalization effect cannot be properly utilized because the batch normalization layer is folded to combine with weights and function as a single weight in order to operate like inference in QAT. Therefore, the normalization effect is replaced by dropout.

[0051] FIG. 4 is a diagram illustrating the process of object detector learning according to one embodiment of the present invention.

[0052] To apply the algorithm proposed in the present invention, the learning process is as follows. First, a pre-trained model (411) is quantized (410). Then, the forward process (421) of a classifier with added dropout is performed in the forward process (420).

[0053] To further train this quantized model, fake quantization nodes (422) are placed in the weights and activation terms to make it behave like a quantized model during training. After the forward process of the classifier with added dropout, the norms of the gradient (G) and weights (W) are calculated (431) to perform gradient clipping in the reverse process (430). The two calculated norms are compared (432) to determine whether to perform gradient clipping. Then, the gradient is proposed through gradient clipping and updated to optimize the classifier parameters.

[0055] FIG. 5 is a graph for comparing the results of dropout according to one embodiment of the present invention with the prior art.

[0056] To demonstrate the effectiveness of the present invention, performance was evaluated on ImageNet, a representative image dataset. The classifier (ResNet18)[1],[2] to which the present invention was applied achieved higher performance than the existing high-performance HAWQ[3], and after additional training, it increased by 0.02% compared to the existing classifier and by 1.685% compared to HAWQ, and when dropout was added, it increased by 0.118% compared to the existing classifier and by 1.722% compared to HAWQ.

[0058] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include a plurality of processing elements and / or a plurality of types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0059] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be embodied in any type of machine, component, physical device, virtual equipment, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0060] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0061] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0062] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

Claim 1 A gradient clipping method performed by a gradient clipping system configured as a computer, wherein the dropout performing unit of the gradient clipping system performs dropout to replace the normalization effect during the forwarding process of a classifier after performing quantization of an input model; The method comprises a step of performing gradient clipping to limit the magnitude of the gradient when the value obtained by dividing the gradient norm, which indicates the direction of the gradient, by the gradient clipping unit of the gradient clipping system for learning after the quantization is exceeded a predetermined threshold value; the step of performing dropout is performed by probabilistically removing some neurons to obtain an unbiased output value in order to replace the normalization effect lost by folding in which the batch normalization layer of the input model is combined with the weights during an additional learning process for the input model that has undergone quantization by placing fake quantization nodes in the weights and activation terms; and the step of performing gradient clipping is a step of determining that learning is completed when the value obtained by dividing the gradient norm by the weight norm exceeds the predetermined threshold value according to the numerical change of the gradient norm and the weight norm that occurs as the expressible range decreases during the additional learning process. A gradient clipping method comprising the step of optimizing the parameters of the classifier by limiting the magnitude of the gradient through an operation of multiplying the original gradient by a scaling ratio obtained by dividing the value obtained by multiplying the norm of the weight by the norm of the gradient to the value obtained by multiplying the predetermined threshold by the norm of the gradient. Claim 2 delete Claim 3 delete Claim 4 delete Claim 5 A dropout execution unit that performs dropout to replace the normalization effect during the forwarding process of a classifier after performing quantization of the input model; and includes a gradient clipping unit for learning after the quantization, which performs gradient clipping to limit the magnitude of the gradient when the value obtained by dividing the gradient norm, representing the direction of the gradient, by the weight norm exceeds a predetermined threshold; the dropout performing unit performs the dropout to probabilistically remove some neurons to obtain an unbiased output value in order to replace the normalization effect lost by folding in the batch normalization layer of the input model, which is performed by placing fake quantization nodes in the weights and activation terms, and the gradient clipping unit determines that learning is complete when the value obtained by dividing the gradient norm by the weight norm exceeds the predetermined threshold according to numerical changes in the gradient norm and the weight norm that occur as the expressible range decreases during the additional learning process, and a scaling ratio obtained by dividing the value obtained by multiplying the predetermined threshold by the weight norm by the gradient norm A gradient clipping system that optimizes the parameters of the classifier by limiting the magnitude of the gradient through an operation of multiplying the original gradient. Claim 6 delete Claim 7 delete Claim 8 delete

Images