Method, computer device, and computer program for network quantization using element-wise division-based rounding

Abstract

Disclosed are a method, a computer device, and a computer program for network quantization using element-wise division-based rounding. The method for network quantization may comprise a step of performing, by at least one processor, quantization by applying a learnable parameter to a pre-learned weight of a neural network, wherein the step of performing quantization may quantize the pre-learned weight by means of an element-wise division-based rounding function using the parameter.

Description

Method, computer device, and computer program for network quantization using element-wise division-based rounding

[0001] The following description describes a technique for lightweighting deep neural network models through network quantization.

[0002] Deep neural networks have recently achieved unprecedented success in various fields such as computer vision, natural language processing, and automatic speech recognition.

[0003] In particular, with the emergence of zero-shot and few-shot performance of large language models (LLMs) such as ChatGPT (generative pre-trained transformer) and GPT-4, interest in improving the latency and throughput of LLMs is growing significantly.

[0004] Network quantization technology is attracting attention as one of the technologies to improve the delay time and throughput of LLM.

[0005] As an example of network quantization technology, Korean Patent Publication No. 10-2019-0043849 (published on April 29, 2019) discloses a technology for quantizing the parameters of a neural network.

[0006] Network quantization techniques can be broadly categorized into quantization-aware training (QAT) and post-training quantization (PTQ). Quantization-aware training (QAT) applies quantization during the training process, while post-training quantization (PTQ) applies quantization after training.

[0007] Quantized Atomic Training (QAT) integrates network quantization into the training process, requiring end-to-end retraining or fine-tuning of the entire training dataset, which consumes significant time and resources to achieve a quantized neural network with competitive performance.

[0008] As quantization-based learning (QAT) requires a lot of time and resources, making it difficult to implement in practice, interest in post-training quantization (PTQ), which does not require the entire training data set or end-to-end training, is increasing.

[0009] In quantizing the pre-learned weights of a network, we provide a quantization technique that can learn not only the parameters for the quantization grid size but also the scale for each weight according to the size of the weight through rounding based on element-wise division of the weights.

[0010] A network quantization method executed on a computer device, wherein the computer device includes at least one processor configured to execute computer-readable instructions contained in a memory, and the network quantization method includes a step of performing quantization by applying learnable parameters to pre-learned weights of a neural network by the at least one processor, and the step of performing quantization is characterized in that the pre-learned weights are quantized through a rounding function based on element-wise division using the parameters.

[0011] According to one aspect, the step of performing the quantization may perform post-training quantization for each layer or block of the network.

[0012] According to another aspect, the step of performing the quantization may include a step of updating quantization parameters for the pre-learned weights so that a difference between the output of the network using the pre-learned weights before quantization and the output of the network using the quantized weights after quantization is minimized.

[0013] According to another aspect, the step of performing the quantization may include a step of jointly learning a common quantization grid size for the entire layer and a division factor corresponding to a scale for the pre-learned weights according to the size of the weights.

[0014] According to another aspect, the step of performing the above quantization may quantize the pre-learned weights through a division-based rounding scheme of the following mathematical formula.

[0015] Mathematical formula: (Here, W is the pre-learned weight, is the quantized weight, s1 is the common quantization grid size across the layers, S is the division coefficient for the pre-trained weights, (means rounding function)

[0016] According to another aspect, the step of performing the quantization is to compare the output of the network using the pre-learned weights (W) before quantization and the quantized weights ( ) may include a step of updating the common quantization grid size (s1) and the division coefficient (S) so as to minimize the difference between the outputs of the network.

[0017] According to another aspect, the division coefficient (S) can be defined as an element-wise product of the common quantization grid size (s1) and a matrix scaling (S2) having the same shape as the pre-learned weights (W).

[0018] According to another aspect, the above matrix scaling (S2) can be supplemented by at least one additional vector that takes into account channel statistics changes depending on the layer type.

[0019] According to another aspect, the above matrix scaling (S2) can be supplemented by an additional vector (s3) for the output channel in the case of a linear layer, and by an additional vector (s3) for the output channel and an additional vector (s4) for the input channel in the case of a 2D convolution.

[0020] According to another aspect, the step of performing the quantization may perform weight quantization of an LLM (large language model) using a rounding function based on the element-wise division.

[0021] A computer program stored in a computer-readable recording medium is provided for executing the above network quantization method on a computer.

[0022] In a computer device, at least one processor configured to execute computer-readable instructions contained in a memory, wherein the at least one processor performs quantization by applying learnable parameters to pre-learned weights of a neural network, and processes a process of quantizing the pre-learned weights through a rounding function based on element-wise division using the parameters.

[0023] According to embodiments of the present invention, when quantizing pre-learned weights of a network, not only parameters for the quantization grid size but also scales for each weight are learned according to the size of the weight through rounding based on element-wise division of the weights, thereby enabling more flexible quantization of weights with large values ​​(i.e., important weights), thereby significantly improving performance after quantization, and in particular, restoring the model to a performance close to that before quantization.

[0024] FIG. 1 is a block diagram illustrating an example of the internal configuration of a computer device according to one embodiment of the present invention.

[0025] FIG. 2 is a flowchart illustrating an example of a network quantization method that can be performed by a computer device according to an embodiment of the present invention.

[0026] FIG. 3 is an exemplary drawing for explaining a rounding process based on element-by-element division for weights in one embodiment of the present invention.

[0027] FIG. 4 is an exemplary drawing for explaining a learnable rounding function in one embodiment of the present invention.

[0028] FIG. 5 is an exemplary drawing for explaining a division coefficient for a weight in one embodiment of the present invention.

[0029] FIG. 6 is an exemplary diagram for explaining a search space according to rounding based on element-wise division of weights in one embodiment of the present invention.

[0030] Figures 7 and 8 illustrate the results of weight update according to quantization using element-by-element division-based rounding in one embodiment of the present invention.

[0031] Figures 9 and 10 illustrate grid shift results according to quantization using element-by-element division-based rounding in one embodiment of the present invention.

[0032] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0033] Embodiments of the present invention relate to a technique for reducing the weight of a deep neural network model through network quantization.

[0034] Embodiments including those specifically disclosed herein can perform network quantization using element-wise division-based learnable rounding.

[0035] A network quantization device according to embodiments of the present invention may be implemented by at least one computer device, and a network quantization method according to embodiments of the present invention may be performed by at least one computer device included in the network quantization device. At this time, a computer program according to an embodiment of the present invention may be installed and run on the computer device, and the computer device may perform the network quantization method according to embodiments of the present invention under the control of the run computer program. The above-described computer program may be stored on a computer-readable recording medium so as to be coupled to the computer device and cause the computer to execute the network quantization method.

[0036] FIG. 1 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention. For example, a network quantization device according to embodiments of the present invention may be implemented by the computer device (100) illustrated in FIG. 1.

[0037] As illustrated in FIG. 1, a computer device (100) may include a memory (110), a processor (120), a communication interface (130), and an input / output interface (140) as components for executing a network quantization method according to embodiments of the present invention.

[0038] The memory (110) is a computer-readable recording medium, and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, the ROM and the permanent mass storage device such as the disk drive may be included in the computer device (100) as a separate permanent storage device distinct from the memory (110). In addition, the memory (110) may store an operating system and at least one program code. These software components may be loaded into the memory (110) from a computer-readable recording medium separate from the memory (110). This separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. In another embodiment, the software components may be loaded into the memory (110) through a communication interface (130) rather than a computer-readable recording medium. For example, software components may be loaded into the memory (110) of a computer device (100) based on a computer program that is installed by files received over a network (160).

[0039] The processor (120) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor (120) via the memory (110) or the communication interface (130). For example, the processor (120) may be configured to execute instructions received according to program code stored in a storage device such as the memory (110).

[0040] The communication interface (130) may provide a function for the computer device (100) to communicate with other devices via a network (160). For example, requests, commands, data, files, etc. generated by the processor (120) of the computer device (100) according to program codes stored in a recording device such as a memory (110) may be transmitted to other devices via the network (160) under the control of the communication interface (130). Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device (100) via the communication interface (130) of the computer device (100) via the network (160). Signals, commands, data, etc. received via the communication interface (130) may be transmitted to the processor (120) or the memory (110), and files, etc. may be stored in a storage medium (the aforementioned permanent storage device) that the computer device (100) may further include.

[0041] The communication method is not limited, and may include not only a communication method that utilizes a communication network (e.g., a mobile communication network, a wired Internet, a wireless Internet, a broadcasting network) that the network (160) may include, but also short-range wired / wireless communication between devices. For example, the network (160) may include any one or more of a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a broadband network (BBN), the Internet, and the like. In addition, the network (160) may include any one or more of a network topology including, but not limited to, a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree, or a hierarchical network.

[0042] The input / output interface (140) may be a means for interfacing with an input / output device (150). For example, the input device may include a device such as a microphone, a keyboard, a camera, or a mouse, and the output device may include a device such as a display or a speaker. As another example, the input / output interface (140) may be a means for interfacing with a device that integrates input and output functions, such as a touchscreen. The input / output device (150) may also be configured as a single device with the computer device (100).

[0043] Furthermore, in other embodiments, the computer device (100) may include fewer or more components than those illustrated in FIG. 1. However, it is not necessary to explicitly illustrate most conventional components. For example, the computer device (100) may be implemented to include at least some of the input / output devices (150) described above, or may further include other components such as a transceiver, a camera, various sensors, a database, and the like.

[0044] Below, a specific embodiment of a network quantization technique using element-wise division-based rounding is described.

[0045] Recently, techniques for quantizing models without retraining have emerged for various tasks such as computer vision, natural language understanding, and natural language generation. Outlier channel splitting replicates the channel associated with an outlier and then splits the outlier in half. While this outlier channel splitting technique explicitly addresses outliers, it significantly reduces accuracy when both weights and activations are quantized to low bit rates. Bit-split, an alternative solution, optimizes by splitting integers into multiple bits. While bit-split techniques perform similarly to full-precision models in low-bit settings, they are ineffective on certain architectures, such as MobileNetV2.

[0046] To overcome the limitations of existing techniques, a weight rounding mechanism called AdaRound can be utilized, and the AdaRound rounding technique minimizes the mean squared error between the output of the full-precision layer and the output of the quantized layer. In addition, there are block-wise reconstruction techniques (called 'BRECQ') to consider cross-layer dependencies along with the possibility of quantizing MobileNetV2 to 4 bits, and techniques ('QDrop' technique) to randomly drop activation quantization during post-training quantization (PTQ) reconstruction so that activation quantization is synchronized with weight quantization. Both BRECQ and QDrop are based on AdaRound, which rounds the maximum weight only up or down with a fixed quantization grid size.

[0047] In early studies, post-training quantization (PTQ) was performed by minimizing the quantization error in parameter space, rounding to the nearest value. This approach resulted in significant performance degradation, and since the loss degradation due to quantization can be approximated by the second-order error of Taylor expansion by treating the quantized weights as perturbation weights, reconstructing the output of a layer or block can be viewed as minimizing the approximate loss degradation due to quantization.

[0048] Recent research has proposed techniques for reconstructing the output of layers or blocks by learning new weight rounding schemes to preserve the performance of fully accurate models even after post-training quantization (PTQ). However, the rounding schemes designed in existing studies merely adaptively round or quantize pre-trained weights through element-wise addition.

[0049] In this embodiment, we adopt a post-training weight quantization method (called "FlexRound") based on an element-wise division approach instead of an element-wise addition approach. FlexRound provides a novel approach to post-training quantization (PTQ) by jointly learning a common quantization grid size and division coefficients for pre-trained weights. Thanks to the reciprocal rules of the derivatives induced by element-wise division, FlexRound can inherently leverage pre-trained weights when updating individual scales for the pre-trained weights. Specifically, FlexRound can explore a relatively wider range of discrete values ​​when quantizing large pre-trained weights. This is because the size of a weight can be interpreted as its relative importance within the network. To maintain the performance of the pre-trained model even after quantization, weights with larger absolute values ​​can be quantized relative to weights with smaller absolute values, considering that larger weights have a greater impact on network performance. That is, important weights can be quantized to one of the two nearest discrete values, as well as to one of the discrete values ​​further apart. Therefore, FlexRound can flexibly quantize the pre-trained weights according to their size, which can improve performance.

[0050] In other words, in this embodiment, we can apply a rounding scheme, FlexRound, based on the principle of element-wise division, to jointly learn a common quantization grid size across groups (e.g., layers, channels, blocks, etc.) as well as separate scales for all pre-learned weights. The rounding scheme, FlexRound, based on element-wise division, can more flexibly quantize large pre-learned weights (i.e., important weights) by considering the sizes of the pre-learned weights when updating their scales.

[0051] FIG. 2 is a flowchart illustrating an example of a network quantization method that can be performed by a computer device according to an embodiment of the present invention.

[0052] The computer device (100) according to the present embodiment can provide a desired service to a client through a dedicated application installed on the client or access to a web / mobile site related to the computer device (100). The computer device (100) may be configured with a network quantization device implemented as a computer. For example, the network quantization device may be implemented in the form of an independently operating program, or may be configured in the form of an in-app of a specific application so as to be operable on the specific application.

[0053] The processor (120) of the computer device (100) may be implemented as a component for performing the following network quantization method. Depending on the embodiment, the components of the processor (120) may be selectively included or excluded from the processor (120). Furthermore, depending on the embodiment, the components of the processor (120) may be separated or merged to express the functions of the processor (120).

[0054] These processors (120) and components of the processor (120) can control the computer device (100) to perform the steps included in the following network quantization method. For example, the processor (120) and components of the processor (120) can be implemented to execute instructions according to the code of the operating system included in the memory (110) and the code of at least one program.

[0055] Here, the components of the processor (120) may be representations of different functions performed by the processor (120) according to instructions provided by the program code stored in the computer device (100).

[0056] The processor (120) can read necessary commands from the memory (110) loaded with commands related to the control of the computer device (100). In this case, the read commands may include commands for controlling the processor (120) to execute steps to be described later.

[0057] The steps included in the network quantization method described below may be performed in a different order than the illustrated order, and some of the steps may be omitted or additional processes may be included.

[0058] Referring to FIG. 2, at step S210, the processor (120) may perform pre-training on a deep neural network utilized in various models such as LLM. In the present embodiment, pre-training of the network may be performed prior to quantization in order to apply post-training quantization (PTQ).

[0059] In step S220, the processor (120) can perform post-training quantization (PTQ) by applying learnable parameters to the pre-trained weights of the network to perform quantization. In particular, the processor (120) can perform quantization through element-wise division-based rounding using separate scales for all pre-trained weights, and at this time, parameters for the weights (common quantization grid size and division coefficient) can be updated to minimize the difference between the network output before quantization and the network output after quantization. In other words, the present invention can perform post-training quantization (PTQ) through element-wise division-based rounding for the weights, and can learn and reflect the common quantization grid size of the entire layer and the scale (division coefficient) for the pre-trained weights together during the quantization process.

[0060] A specific embodiment of a learnable rounding technique based on element-wise division for post-training quantization (PTQ) in the present invention is described as follows.

[0061] Notation

[0062] Scalars, vectors, and matrices (or tensors) are represented by lowercase, bold lowercase, and uppercase letters, respectively, for example, s, s, and S. means the quantized counterpart of W. The input to a 2D convolutional or linear layer is represented by X if all previous layers are intact, and X if all previous layers are quantized. is expressed as . The entry of matrix A is A (i,j) , whereas the entries of a four-dimensional tensor A are represented as A (i,j,k,l) is expressed as . Similar to the broadcasting process of Python Numpy, ⊙ represents element-wise multiplication and / represents element-wise division. And, represents a rounding function, represents the floor function. ||·|| F stands for Frobenius norm.

[0063] Post-Training Quantization (PTQ) Background

[0064] The post-training quantization (PTQ) approach rounds the pre-trained weights W to is quantized so that , where the quantization grid size s1∈R >0 silver is set to minimize. However, minimizing the quantization error in parameter space is not the same as the final task loss. Since the loss reduction due to quantization can be approximated by the quadratic form of the network output and the Hessian matrix, it can be minimized layer-by-layer or block-by-block with respect to a continuous variable V with a small amount of data ( ) here, has a specific function h(·) or It means one of them. However, all the above rounding mechanisms are based on element-wise addition.

[0065] FlexRound

[0066] FIG. 3 is an exemplary diagram illustrating an element-wise division-based rounding process for weights according to one embodiment of the present invention. FIG. 3 illustrates an element-wise division-based rounding process in tensor-wise uniform learning followed by quantization (PTQ) reconstruction.

[0067] Unlike element-wise addition-based approaches, this embodiment applies element-wise division to quantize pre-trained weights. A weight rounding scheme based on element-wise division can be defined as shown in Equation 1.

[0068] [Mathematical Formula 1]

[0069]

[0070] S is a scale with the same shape as the pre-trained weights W, and represents a division coefficient for W. s1 represents the common quantization grid size across the layers. All entries of S and s1 are positive and trainable. S and s1 are the reconstruction error L. It is updated in a direction to minimize .

[0071] Mathematical expression 1 shows that FlexRound's basic formula supports tensor-specific uniform post-training quantization (PTQ). FlexRound simply converts a scalar s1 to a vector While per-channel weight quantization can be adopted as a replacement, tensor-wise uniform post-training quantization (PTQ) via FlexRound can be sufficient to achieve the performance of a full-precision model. Therefore, we focus on reconstructing tensor-wise uniform post-training quantization (PTQ).

[0072] Figure 4 is an exemplary diagram for explaining a learnable rounding function in one embodiment of the present invention. Looking at the rounding function using learnable parameters S and s1, the pre-learned weights are W (2,4) <W (3,2) Ina, each different S (i,j) Using It can be flexibly quantized as follows.

[0073] Here's how to design S: First, we can formalize S as S = s1 ⊙ S2 , where S2 is a matrix or tensor scaling W with the same shape as W , and all elements of S2 are positive and trainable. When S = s1 ⊙ S2 , the performance is comparable to that of a weighted rounding scheme based on element-wise addition in the uniform post-training quantization (PTQ) setting per unit. However, to further improve the performance of the new weighted rounding scheme based on element-wise division, we can complement S2 as follows: Linear layer In this case, S2 is an additional learnable tensor can be supplemented by using . In that the statistics of the output channel can vary greatly, By supplementing S2 with , we can take into account the variation of output channel statistics. In other words, is an additional vector supporting S2 to describe the statistical changes of the output channel in W. In some embodiments, 2D convolution For S2, there are two additional learnable tensors: one for the output channels, and one for the output channels. and additional tensors for the input channels can be complemented. Therefore, S is complemented with at least one additional vector depending on the layer type, and in the case of a linear layer, s1⊙S2⊙ as shown in Fig. 5. It can be formulated as , and in the case of 2D convolution, S is s1⊙S2⊙ ⊙ can be formalized as

[0074] [Equation 2]

[0075]

[0076] Consider Equation 2 as the final formula of FlexRound. Here, S2, , All elements of are rounded using the usual rounding method, i.e. It is initialized to 1 to facilitate learning. , S2, and All parameters ( ) are subject to the constraint that all items are positive. , S2, , )go is updated to minimize .

[0077] FIG. 6 is an exemplary diagram illustrating a search space according to element-wise division-based rounding of weights in one embodiment of the present invention. As illustrated in FIG. 6, Rounding-to-Nearest (RTN) rounds weights to the nearest value, and AdaRound, one of the post-training quantization (PTQ) methods, only rounds weights up or down according to an element-wise addition-based rounding scheme, whereas FlexRound according to the present invention can more flexibly change weights within a wider search space through an element-wise division-based rounding scheme.

[0078] In Equation 2, element-wise division serves a similar purpose to element-wise addition: creating a more efficient rounding scheme than nearest-neighbor rounding. By implementing a new rounding policy through element-wise division, , S2, and can learn all of them. This allows FlexRound to learn the rounding process (e.g., S2⊙ or S2⊙ ⊙ ) with a common quantization grid size (i.e., ) can be learned. In addition, the reciprocal rule of the derivatives induced by element-wise division allows FlexRound to utilize pre-learned weights when learning the corresponding scale.

[0079] Let L be the reconstruction error representing the difference between the network output before quantization and the network output after quantization, and let S' be the matrix (or tensor) that scales the pre-learned weights W in Equation 2, i.e. S'=S2⊙ (or S2⊙ ⊙ ) is used. When using STE (straight-through estimator), S', (or ) the gradient of L for the items of the pre-learned weights W (i,j) (or W (i,j,k,l) ) is proportional to.

[0080] The above proposition holds for linear layers, element S' (i,j) Go W (i,j) Influenced by Also updated and W (i,j) This means that it can be influenced by the pre-learned weights W. (i,j) The larger the size of W during post-training quantization (PTQ) reconstruction, (i,j)The likelihood of receiving larger updates increases. The size of the weight can be considered a scale that measures the importance of the weight for pruning unimportant weights. Consequently, weights with larger values ​​play a more important role than weights with smaller values. To reduce the performance gap between a full-precision pre-trained model and its quantized version, large pre-trained weights (potentially important pre-trained weights) can be quantized by increasing the likelihood of being quantized to one of the two closest quantization grids, as well as to one of the two quantization grids that are further away from the smaller grid. The above implications can be equally applied to 2D convolutions.

[0081] Figures 7 and 8 illustrate the results of weight update according to quantization using element-wise division-based rounding in one embodiment of the present invention. In the first block of MobileNetV2 (Figure 7) and ResNet-18 (Figure 8), the weight update through FlexRound of the first 2D convolution maintains full precision of activation after quantizing the pre-trained weights to 4 bits through FlexRound.

[0082] Figures 7 and 8 show the amount of weight updates through FlexRound for MobileNetV2 and ResNet-18. The histograms on the left and center in Figures 7 and 8 are grouped for small pre-trained weights (|W|<1, left) and large pre-trained weights (|W|>1, center). The scatter plot on the right shows the grid shift of the grid that can be obtained from RTN. We can see that MobileNetV2 and ResNet-18 are distinctly quantized due to FlexRound. For example, in the case of MobileNetV2, W (i,j,k,l) When the absolute value of is greater than 1, it is obtained by minimizing L. The change is more aggressive (i.e., the rounding can deviate from more than one step up or one step down). That is, FlexRound quantizes a large number of pre-trained weights more flexibly, as shown in the dashed box in Fig. 7. The amount of aggressively rounded weights in the first 2D convolution of the first block of MobileNetV2 is about 12.8% of the total. However, for ResNet-18, there is no pre-trained weight with size greater than 1. Therefore, most of the pre-trained weights are rounded up or down, as shown in Fig. 8 (e.g., only about 1.5% of the weights in the first 2D convolution of the first block of ResNet-18 are aggressively rounded).

[0083] FlexRound can take into account the size of the pre-trained weights when updating their scales, and larger pre-trained weights are more likely to be quantized with greater flexibility than smaller weights. Is is directly proportional to W (i,j) No matter how big the size is If is close to 0 becomes 0. In this sense, large-scale pre-trained weights can be quantized into a grid that can be obtained from RTN. When is (significantly) greater than 0, large pre-trained weights can be quantized to grids farther away from the two closest grids, as illustrated in Fig. 9. In short, although the size of the pre-trained weights affects the corresponding scale update of FlexRound, it does not necessarily mean that large weights should be quantized more flexibly than small weights.

[0084] FlexRound can quantize weights more flexibly as the bit width increases. Comparing the right-hand diagrams of Figures 7 and 8 with Figure 10, FlexRound can provide more grid shifting than the grid achievable with RTN as higher bit widths are used. Unlike AdaRound, which only rounds weights up or down regardless of the number of bits used, FlexRound allows for more flexible weight quantization as the bit width increases, making it more suitable for quantizing models that require higher bit widths than AdaRound (e.g., LLM).

[0085] According to embodiments of the present invention, when quantizing the pre-learned weights of a network, not only the parameters for the quantization grid size but also the scale for each weight are learned according to the size of the corresponding weight through rounding based on element-wise division of the weights, thereby enabling more flexible quantization of weights with large values. Consequently, FlexRound, which utilizes the element-wise division-based rounding technique, can perform better than existing techniques in various types of tasks such as image classification, natural language understanding, and natural language generation for various models ranging from small to large models such as ResNet, MobileNet, BERT, GPT-Neo, OPT, and LLaMA. In particular, in the case of LLM, existing techniques suffer from performance degradation after quantization, whereas FlexRound according to the present invention can reduce the performance difference before and after quantization, and can quantize the activations of LLM to 8 bits without any assumption about activation outliers, thereby simultaneously improving the latency and throughput of LLM.

[0086] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0087] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0088] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording media or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0089] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0090] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. In a network quantization method executed on a computer device, The computer device comprises at least one processor configured to execute computer-readable instructions contained in a memory, The above network quantization method is, A step of performing quantization by applying learnable parameters to pre-learned weights of a neural network by at least one processor. Including, The step of performing the above quantization is: Quantizing pre-learned weights through a rounding function based on element-wise division using the above parameters. A network quantization method characterized by .

2. In paragraph 1, The step of performing the above quantization is: Performing post-training quantization for each layer or block of the above network. A network quantization method characterized by .

3. In paragraph 1, The step of performing the above quantization is: A step of updating quantization parameters for pre-learned weights so that the difference between the output of the network using pre-learned weights before quantization and the output of the network using quantized weights after quantization is minimized. A network quantization method including:

4. In paragraph 1, The step of performing the above quantization is: A step of jointly learning the common quantization grid size of the entire layer and the division factor corresponding to the scale of the pre-learned weights according to the size of the weights. A network quantization method including:

5. In paragraph 1, The step of performing the above quantization is: Quantizing pre-trained weights using a division-based rounding scheme of the following mathematical formula: A network quantization method characterized by . Mathematical formula: (Here, W is the pre-learned weight, is the quantized weight, s1 is the common quantization grid size across the layers, S is the division coefficient for the pre-trained weights, (means rounding function) 6. In paragraph 5, The step of performing the above quantization is: The output of the above network using the pre-learned weights (W) before quantization and the quantized weights ( ) to minimize the difference between the outputs of the network. A network quantization method including:

7. In paragraph 5, The above division coefficient (S) is defined as the element-wise product of the common quantization grid size (s1) and the matrix scaling (S2) whose shape is the same as the pre-learned weight (W). A network quantization method characterized by .

8. In paragraph 7, The above matrix scaling (S2) is complemented by at least one additional vector that takes into account channel statistics changes depending on the layer type. A network quantization method characterized by .

9. In paragraph 7, The above matrix scaling (S2) is complemented by an additional vector (s3) for the output channel in the case of a linear layer, and by an additional vector (s3) for the output channel and an additional vector (s4) for the input channel in the case of a 2D convolution. A network quantization method characterized by .

10. In paragraph 1, The step of performing the above quantization is: Performing weight quantization of LLM (large language model) using a rounding function based on the above element-wise division. A network quantization method characterized by .

11. A computer program stored in a computer-readable recording medium for causing a computer to execute the network quantization method of any one of claims 1 to 10.

12. In computer devices, At least one processor configured to execute computer-readable instructions contained in memory Including, At least one processor, A process of performing quantization by applying learnable parameters to pre-learned weights of a neural network, and quantizing the pre-learned weights through a rounding function based on element-wise division using the above parameters. A computer device that processes .

13. In paragraph 12, At least one processor, Performing quantization after learning for each layer or block of the above network A computer device characterized by:

14. In paragraph 12, At least one processor, Updating quantization parameters for pre-learned weights so that the difference between the output of the network using pre-learned weights before quantization and the output of the network using quantized weights after quantization is minimized. A computer device characterized by:

15. In paragraph 12, At least one processor, Jointly learning a common quantization grid size across layers and a division coefficient corresponding to the scale of the pre-learned weights according to the size of the weights. A computer device characterized by:

16. In paragraph 12, At least one processor, Quantizing pre-trained weights using a division-based rounding scheme of the following mathematical formula: A computer device characterized by: Mathematical formula: (Here, W is the pre-learned weight, is the quantized weight, s1 is the common quantization grid size across the layers, S is the division coefficient for the pre-trained weights, (means rounding function) 17. In paragraph 16, At least one processor, The output of the above network using the pre-learned weights (W) before quantization and the quantized weights ( ) to minimize the difference between the outputs of the network using the common quantization grid size (s1) and the division coefficient (S). A computer device characterized by:

18. In paragraph 16, The above division coefficient (S) is defined as the element-wise product of the common quantization grid size (s1) and the matrix scaling (S2) whose shape is identical to the pre-learned weight (W). A computer device characterized by:

19. In paragraph 18, The above matrix scaling (S2) is complemented by at least one additional vector that takes into account channel statistics changes depending on the layer type. A computer device characterized by:

20. In paragraph 18, The above matrix scaling (S2) is complemented by an additional vector (s3) for the output channel in the case of linear layers, and by an additional vector (s3) for the output channel and an additional vector (s4) for the input channel in the case of 2D convolution. A computer device characterized by:

Images