Method, computer device, and computer program for network quantization using rounding based on element-wise division

Abstract

A method, a computer device, and a computer program for network quantization using element-wise division-based rounding are disclosed. The network quantization method may include the step of performing quantization by applying learnable parameters to pre-learned weights of a neural network by the at least one processor, wherein the step of performing quantization may quantize the pre-learned weights through an element-wise division-based rounding function using the parameters.

Description

Technology Field

[0001] The following description concerns a technique for lightweighting deep neural network models through network quantization. Background Technology

[0002] Recently, deep neural networks have been achieving unprecedented success in various fields such as computer vision, natural language processing, and automatic speech recognition.

[0003] In particular, with the emergence of zero-shot and few-shot performance of large language models (LLMs) such as ChatGPT (generative pre-trained transformer) and GPT-4, interest in improving the latency and throughput of LLMs has increased significantly.

[0004] Network quantization technology is gaining attention as one of the technologies to improve the latency and throughput of LLMs.

[0005] As an example of network quantization technology, Korean Published Patent No. 10-2019-0043849 (published on April 29, 2019) discloses a technology for quantizing the parameters of a neural network.

[0006] Network quantization techniques can be broadly classified into quantization-aware training (QAT) and post-training quantization (PTQ). Quantization-aware training (QAT) is a technique that performs quantization during the training process, whereas post-training quantization (PTQ) is a technique that applies quantization after training has been performed.

[0007] Quantized learning (QAT) is a method in which network quantization is integrated into the learning process, requiring end-to-end retraining or fine-tuning of the entire training dataset; consequently, obtaining a quantized neural network with competitive performance requires enormous time and resources.

[0008] As such, quantization learning (QAT) requires a lot of time and resources, making it difficult to implement in reality, so there is growing interest in post-training quantization (PTQ), which does not require the entire training dataset or end-to-end learning. The problem to be solved

[0009] It provides a quantization technique that enables learning not only parameters for the quantization grid size but also the scale for each weight according to its magnitude through rounding based on element-wise division of the weights when quantizing the network's pre-trained weights. means of solving the problem

[0010] A network quantization method executed in a computer device comprises at least one processor configured to execute computer-readable instructions contained in memory, and the network quantization method comprises the step of performing quantization by applying learnable parameters to pre-learned weights of a neural network by the at least one processor, wherein the step of performing quantization is characterized by quantizing the pre-learned weights through an element-wise division-based rounding function using the parameters.

[0011] According to one aspect, the step of performing the quantization may perform post-training quantization for each layer or block of the network.

[0012] According to another aspect, the step of performing the quantization may include updating quantization parameters for the pre-trained weights such that the difference between the output of the network using the pre-trained weights before quantization and the output of the network using the quantized weights after quantization is minimized.

[0013] According to another aspect, the step of performing the quantization may include a step of jointly learning a common quantization grid size of the entire layer and a division factor corresponding to the scale of the pre-learned weights according to the size of the weights.

[0014] According to another aspect, the step of performing the above quantization can quantize the pre-learned weights through a division-based rounding scheme of the following mathematical formula.

[0015] Mathematical formula: (Here, W is the pre-trained weight, is the quantized weight, s1 is the common quantization grid size across the layer, S is the division factor for the pre-trained weight, )

[0016] According to another aspect, the step of performing the quantization comprises the output of the network using pre-learned weights (W) before quantization and the quantized weights ( It may include a step of updating the common quantization grid size (s1) and the division factor (S) so that the difference between the outputs of the network using ) is minimized.

[0017] According to another aspect, the division factor (S) can be defined as the element-wise product of the common quantization grid size (s1) and the matrix scaling (S2) which has the same shape as the pre-learned weight (W).

[0018] According to another aspect, the matrix scaling (S2) may be supplemented by at least one additional vector that takes into account channel statistical changes according to the layer type.

[0019] According to another aspect, the matrix scaling (S2) may be supplemented by an additional vector (s3) for the output channel in the case of a linear layer, and by an additional vector (s3) for the output channel and an additional vector (s4) for the input channel in the case of a 2D convolution.

[0020] According to another aspect, the step of performing the quantization can perform weighted quantization of the LLM (large language model) using the element-by-element division-based rounding function.

[0021] A computer program stored on a computer-readable recording medium is provided to execute the above network quantization method on a computer.

[0022] A computer device comprises at least one processor configured to execute computer-readable instructions contained in memory, wherein the at least one processor performs quantization by applying learnable parameters to pre-learned weights of a neural network, and processes the process of quantizing pre-learned weights through a rounding function based on element-wise division using said parameters. Effects of the invention

[0023] According to embodiments of the present invention, when quantizing the pre-learned weights of a network, by learning not only the parameters for the quantization grid size but also the scale for each weight according to the magnitude of the weight through rounding based on elemental division of the weights, large weights (i.e., important weights) can be quantized more flexibly, thereby significantly improving performance after quantization and, in particular, restoring the model to a performance close to that before quantization. Brief explanation of the drawing

[0024] FIG. 1 is a block diagram illustrating an example of the internal configuration of a computer device in an embodiment of the present invention. FIG. 2 is a flowchart illustrating an example of a network quantization method that can be performed by a computer device according to an embodiment of the present invention. FIG. 3 is an example diagram illustrating a rounding process based on element-by-element division for weights in an embodiment of the present invention. FIG. 4 is an example diagram illustrating a learnable rounding function in an embodiment of the present invention. FIG. 5 is an example diagram illustrating the division factor for weights in an embodiment of the present invention. FIG. 6 is an example drawing for explaining a search space based on element-by-element division of weights in an embodiment of the present invention. FIGS. 7 and 8 illustrate the results of weight updates according to quantization using element-based division rounding in an embodiment of the present invention. FIGS. 9 and 10 illustrate grid shift results according to quantization using element-based division rounding in an embodiment of the present invention. Specific details for implementing the invention

[0025] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0026] Embodiments of the present invention relate to a technique for lightweighting deep neural network models through network quantization.

[0027] Embodiments including those specifically disclosed in this specification can perform network quantization using element-based learnable rounding.

[0028] A network quantization device according to embodiments of the present invention may be implemented by at least one computer device, and a network quantization method according to embodiments of the present invention may be performed through at least one computer device included in the network quantization device. At this time, a computer program according to an embodiment of the present invention may be installed and run on the computer device, and the computer device may perform a network quantization method according to embodiments of the present invention under the control of the run computer program. The above-described computer program may be stored on a computer-readable recording medium to be combined with the computer device to execute the network quantization method on the computer.

[0029] FIG. 1 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention. For example, a network quantization device according to embodiments of the present invention may be implemented by a computer device (100) illustrated in FIG. 1.

[0030] As illustrated in FIG. 1, a computer device (100) may include a memory (110), a processor (120), a communication interface (130), and an input / output interface (140) as components for executing a network quantization method according to embodiments of the present invention.

[0031] Memory (110) is a computer-readable recording medium and may include a non-perishable mass storage device such as RAM (random access memory), ROM (read only memory), and a disk drive. Here, a non-perishable mass storage device such as a ROM and a disk drive may be included in the computer device (100) as a separate permanent storage device distinct from memory (110). Additionally, an operating system and at least one program code may be stored in memory (110). These software components may be loaded into memory (110) from a computer-readable recording medium separate from memory (110). This separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, disk, tape, DVD / CD-ROM drive, or memory card. In another embodiment, software components may be loaded into memory (110) through a communication interface (130) rather than a computer-readable recording medium. For example, software components can be loaded into the memory (110) of the computer device (100) based on a computer program installed by files received through the network (160).

[0032] The processor (120) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor (120) via memory (110) or a communication interface (130). For example, the processor (120) may be configured to execute instructions received according to program code stored in a recording device such as memory (110).

[0033] The communication interface (130) may provide a function for the computer device (100) to communicate with other devices through a network (160). For example, requests, commands, data, files, etc. generated by the processor (120) of the computer device (100) according to program code stored in a recording device such as memory (110) may be transmitted to other devices through the network (160) under the control of the communication interface (130). Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device (100) through the communication interface (130) of the computer device (100) via the network (160). Signals, commands, data, etc. received through the communication interface (130) may be transmitted to the processor (120) or memory (110), and files, etc. may be stored in a storage medium (the permanent storage device described above) that the computer device (100) may further include.

[0034] The communication method is not limited and may include not only communication methods utilizing communication networks (e.g., mobile communication networks, wired internet, wireless internet, broadcasting networks) that the network (160) may include, but also short-range wired / wireless communication between devices. For example, the network (160) may include any one or more networks such as a PAN (personal area network), LAN (local area network), CAN (campus area network), MAN (metropolitan area network), WAN (wide area network), BBN (broadband network), and the Internet. Additionally, the network (160) may include any one or more network topologies such as a bus network, star network, ring network, mesh network, star-bus network, tree or hierarchical network, but is not limited thereto.

[0035] The input / output interface (140) may be a means for interfacing with an input / output device (150). For example, the input device may include a device such as a microphone, keyboard, camera, or mouse, and the output device may include a device such as a display or speaker. As another example, the input / output interface (140) may be a means for interfacing with a device in which the functions for input and output are integrated into one, such as a touchscreen. The input / output device (150) may be composed of a computer device (100) and a single device.

[0036] Additionally, in other embodiments, the computer device (100) may include fewer or more components than the components of FIG. 1. However, it is not necessary to clearly illustrate most of the prior art components. For example, the computer device (100) may be implemented to include at least some of the input / output devices (150) described above, or may include other components such as a transceiver, a camera, various sensors, a database, etc.

[0037] Below, we will describe specific embodiments of network quantization technology using element-based division rounding.

[0038] Recently, techniques have emerged to quantize models for various tasks, such as computer vision, natural language understanding, and natural language generation, without (re)training. Outlier channel splitting duplicates the channels associated with outliers and then splits the outliers in those channels in half. While this technique explicitly resolves outliers, accuracy drops significantly when both weights and activations are quantized to low bits. As an alternative solution, bit-split technology is utilized to optimize integers by splitting them into multiple bits. Although the performance of bit-split technology is comparable to that of full-precision models at low bit settings, it presents a problem in that it is not effective for certain architectures, such as MobileNetV2.

[0039] To overcome the limitations of existing technologies, a weight rounding mechanism called AdaRound can be utilized. The AdaRound rounding technique minimizes the mean squared error between the output of the full precision layer and the output of the quantized layer. In addition, there are techniques such as block-by-block reconstruction ('BRECQ') to account for cross-layer dependencies along with the possibility of quantizing MobileNetV2 to 4 bits, and techniques that induce activation quantization to synchronize with weight quantization by randomly dropping activation quantization during post-training quantization (PTQ) reconstruction ('QDrop' technique). Both BRECQ and QDrop are based on AdaRound, which rounds the maximum weight only up or down with a fixed quantization grid size.

[0040] In early research, post-training quantization (PTQ) is performed to the value closest to rounding by minimizing the quantization error in the parameter space. This approach leads to severe performance degradation, and since the loss degradation due to quantization can be approximated by the second-order error of Taylor expansion by considering the quantized weights as perturbation weights, reconstructing the output of a layer or block can be considered equivalent to minimizing the approximation of the loss degradation due to quantization.

[0041] Recent research proposes techniques to reconstruct the outputs of layers or blocks by training through novel weight rounding schemes in an effort to preserve the performance of full-precision models even after training-based quantization (PTQ). However, the rounding schemes designed in existing studies are merely about adaptively rounding or quantizing pre-trained weights through element-wise addition.

[0042] In this embodiment, instead of the element-wise addition approach, a post-training weight quantization method based on the element-wise division approach (referred to as 'FlexRound') is applied. FlexRound provides a novel approach to post-training quantization (PTQ) by jointly learning a common quantization grid size and a division factor for pre-trained weights. Thanks to the mutual rules of derivatives derived by element-wise division, FlexRound can inherently utilize pre-trained weights when updating individual scales for pre-trained weights. In particular, FlexRound can explore a relatively wider range of discrete values ​​when quantizing large pre-trained weights. This is because the magnitude of a weight can be interpreted as its relative importance within the network. To maintain the performance of the pre-trained model even after quantization, weights with large absolute values ​​can be quantized by comparing them to weights with small absolute values, considering that larger weights have a greater impact on network performance. That is, important weights can be quantized not only into the two closest discrete values ​​but also into one of the more distant discrete values. Therefore, FlexRound can flexibly quantize pre-trained weights according to their own size, so performance can be improved.

[0043] In other words, in this embodiment, a rounding scheme FlexRound based on the elemental division principle can be applied to enable the joint learning of a common quantization grid size across groups (e.g., layers, channels, blocks, etc.) as well as separate scales for all pre-trained weights. The FlexRound rounding scheme based on elemental division can quantize pre-trained weights with large values ​​(i.e., important weights) more flexibly by considering the magnitude of the pre-trained weights when updating the scale.

[0044] FIG. 2 is a flowchart illustrating an example of a network quantization method that can be performed by a computer device according to an embodiment of the present invention.

[0045] The computer device (100) according to the present embodiment can provide a desired service to a client by accessing a dedicated application installed on the client or a web / mobile site related to the computer device (100). The computer device (100) may be configured with a network quantization device implemented in a computer. For example, the network quantization device may be implemented in the form of a program that operates independently, or configured as an in-app of a specific application so that it can operate on said specific application.

[0046] The processor (120) of the computer device (100) may be implemented as a component for performing the following network quantization method. Depending on the embodiment, the components of the processor (120) may be optionally included in or excluded from the processor (120). Additionally, depending on the embodiment, the components of the processor (120) may be separated or merged to represent the function of the processor (120).

[0047] These processors (120) and components of the processor (120) can control the computer device (100) to perform steps included in the following network quantization method. For example, the processor (120) and components of the processor (120) may be implemented to execute instructions according to the code of an operating system included in memory (110) and the code of at least one program.

[0048] Here, the components of the processor (120) may be representations of different functions performed by the processor (120) according to instructions provided by program code stored in the computer device (100).

[0049] The processor (120) can read necessary instructions from memory (110) in which instructions related to the control of the computer device (100) are loaded. In this case, the read instructions may include instructions for controlling the processor (120) to execute the steps to be described later.

[0050] The steps included in the network quantization method described later may be performed in a different order than the one described, and some of the steps may be omitted or additional processes may be included.

[0051] Referring to FIG. 2, in step (S210), the processor (120) can perform pre-training on a deep neural network used in various models such as LLM. In this embodiment, pre-training of the network can be performed first prior to quantization in order to apply quantization (PTQ) after training.

[0052] In step (S220), the processor (120) can perform post-training quantization (PTQ) by applying learnable parameters to the pre-trained weights of the network to perform quantization. In particular, the processor (120) can perform quantization through element-dividend-based rounding using a separate scale for all pre-trained weights, and at this time, parameters for the weights (common quantization grid size and division factor) can be updated to minimize the difference between the network output before quantization and the network output after quantization. In other words, the present invention can perform post-training quantization (PTQ) through element-dividend-based rounding for the weights, and during the quantization process, the common quantization grid size of the entire layer and the scale (division factor) for the pre-trained weights can be learned and reflected together.

[0053] A specific embodiment of the element-based learnable rounding technology for post-learning quantization (PTQ) in the present invention is described as follows.

[0054] notation

[0055] Scalars, vectors, and matrices (or tensors) are represented by lowercase, bold lowercase, and uppercase letters, respectively, for example, s, s , is expressed as S. represents the quantized counterpart of W. The input to a 2D convolution or linear layer is represented as X if all previous layers are intact, and if all previous layers are quantized It is expressed as. The entry of matrix A is A (i,j) While it is expressed as, the items of a 4-dimensional tensor A are A (i,j,k,l) It is expressed as. Similar to the broadcasting process in Python NumPy, ⊙ represents element-wise product and / represents element-wise division. And, represents the rounding function, represents the floor function. means the Frobenius norm.

[0056] Post-training Quantization (PTQ) Background

[0057] The Post-Training Quantization (PTQ) approach uses pre-trained weights W through rounding It is to quantize so that it becomes, where the quantization grid size silver It is set to minimize. However, the minimization of quantization error in parameter space is not equal to the final task loss. Given that the loss degradation due to quantization can be approximated by the second-order form of the network output and the Hessian matrix, it can be minimized layer by layer or block with respect to a continuous variable V with a small amount of data ( Here, is having a specific function h(·) or It means one of them. However, all of the aforementioned rounding mechanisms are based on element-by-element addition.

[0058] FlexRound

[0059] FIG. 3 is an example diagram illustrating a factor-based rounding process for weights in an embodiment of the present invention. FIG. 3 illustrates a factor-based rounding process in quantization (PTQ) reconstruction after uniform learning per tensor.

[0060] Unlike the element-wise addition-based approach, the present embodiment applies element-wise division to quantize pre-learned weights. A weight rounding scheme based on element-wise division can be defined as Equation 1.

[0061] [Mathematical Formula 1]

[0062]

[0063] S represents the division factor for W, scaled to have the same shape as the pre-trained weight W. s1 represents the common quantization grid size across the entire layer. All items in S and s1 are positive and learnable. S and s1 represent the reconstruction error. person It is updated in a direction to minimize.

[0064] Mathematical Equation 1 is the basic formula for FlexRound, indicating that it supports per-tensor uniform learning followed by quantization (PTQ). FlexRound simply treats the scalar s1 as a vector s 1 While it is possible to adopt per-channel weighted quantization by replacing it, tensor-specific uniform learning after quantization (PTQ) via FlexRound may be sufficient to achieve the performance of a full-precision model. Therefore, we focus on the reconstruction of tensor-specific uniform learning after quantization (PTQ).

[0065] FIG. 4 is an exemplary diagram illustrating a learnable rounding function in an embodiment of the present invention. Looking at the rounding function using learnable parameters S and s1, the pre-trained weights or, each different S (i,j) Using It can be flexibly quantized as such.

[0066] The method for designing S is as follows. First, S It can be formulated as follows. Here, S2 is a matrix or tensor-scaled W with the same shape as W, and all elements of S2 are positive and learnable. In this case, performance is good compared to a weight rounding scheme based on element-wise addition in the uniform per-unit learning after quantization (PTQ) setting. However, to further improve the performance of the new weight rounding scheme based on element-wise division, S2 can be supplemented as follows. Linear layer In the case of, S2 is an additional trainable tensor It can be supplemented by using. In terms of the fact that the statistics of the output channel can vary significantly, s 3 By supplementing S2, fluctuations in output channel statistics can be taken into account. In other words, s 3 is an additional vector supporting S2 to describe the statistical change of the output channel in W. Depending on the embodiment, 2D convolution For this, S2 consists of two additional trainable tensors, namely additional tensors for the output channels and additional tensors for input channels It can be supplemented as. Therefore, S is supplemented with at least one additional vector depending on the layer type, and in the case of a linear layer, as shown in FIG. 5 It can be formulated as, and according to the embodiment, in the case of 2D convolution, S is It can be formalized as.

[0067] [Mathematical Formula 2]

[0068]

[0069] Mathematical Equation 2 is considered as the final formula for FlexRound. Here, S2, s 3 , s 4 All elements of the general rounding method, that is It is initialized to 1 to facilitate learning. s 1 , S2, s 3 and s 4 According to the constraint that all items of are positive, all parameters ( s 1 , S2, s 3 , s 4 )go It is updated to minimize.

[0070] FIG. 6 is an example diagram illustrating a search space based on element-by-element division-based rounding of weights in an embodiment of the present invention. As shown in FIG. 6, Rounding-to-Nearest (RTN) rounds weights to the nearest value, and AdaRound, which is one of the learning-post-quantization (PTQ) methods, rounds weights only up or down according to an element-by-element addition-based rounding system, whereas FlexRound according to the present invention can change weights more flexibly within a wider search space through an element-by-element division-based rounding system.

[0071] In mathematical equation 2, elemental division serves a similar purpose to elemental addition, creating a rounding system that is more effective than the round closest to the round. By implementing a new rounding policy through elemental division s 1 , S2, s 3 and s4 It can learn all of this. Through this, FlexRound can learn the rounding process (e.g., or Along with ) common quantization grid size (i.e., s 1 FlexRound can learn ) and, in addition, through the mutual rules of derivatives derived by elemental division, FlexRound can utilize pre-trained weights when learning the scale.

[0072] Let be the reconstruction error representing the difference between the network output before quantization and the network output after quantization, and S' be the matrix (or tensor) scaling the pre-trained weights W in Equation 2, i.e. (or Let ) be. When using STE (straight-through estimator), S', (or Regarding the items of ) The gradient is the pre-trained weight W (i,j) (or W (i,j,k,l) It is proportional to ).

[0073] The above proposition is for the element S' in the case of a linear layer. (i,j) Ga W (i,j) Influenced by It is also updated and W (i,j) This means that it can be influenced by the pre-trained weight W. In other words, the pre-trained weight W (i,j) The larger the magnitude of , the more W during post-training quantization (PTQ) reconstruction (i,j)The likelihood of receiving a larger update increases. The magnitude of the weight can be regarded as a scale measuring the importance of the weight for pruning unimportant weights. Consequently, weights with larger values ​​play a more significant role than those with smaller values. To reduce the performance gap between a fully precise pre-trained model and its quantized version, large pre-trained weights (potentially important pre-trained weights) can be quantized by increasing the likelihood of quantization into one of two quantization grids further away than the two nearest quantization grids. The same applies to 2D convolution.

[0074] FIGS. 7 and 8 illustrate the results of weight updates according to quantization using element-based rounding in an embodiment of the present invention. In the first block of MobileNetV2 (Fig. 7) and ResNet-18 (Fig. 8), the weight update through FlexRound of the first 2D convolution involves quantizing the pre-trained weights into 4 bits through FlexRound, after which the activation is maintained at full precision.

[0075] Figures 7 and 8 show the amount of weight updates via FlexRound for MobileNetV2 and ResNet-18. In Figures 7 and 8, the left and center histograms are grouped for small pre-trained weights (|W|<1, left) and large pre-trained weights (|W|>1, center). It explains the change of. The scatter plot on the right shows the grid shift amount of the grid obtainable in RTN. It can be seen that MobileNetV2 and ResNet-18 are distinctly quantized due to FlexRound. For example, in the case of MobileNetV2, W (i,j,k,l) When the absolute value of is greater than 1, Obtained by minimizing The change is more aggressive (i.e., rounding can deviate from one or more steps of upward or one step of downward). In other words, FlexRound quantizes large pre-trained weights more flexibly, as indicated by the dashed box in Fig. 7. In the first 2D convolution of the first block of MobileNetV2, the amount of aggressively rounded weights is approximately 12.8% of the total. However, in the case of ResNet-18, there are no pre-trained weights with a magnitude greater than 1. Therefore, most pre-trained weights are rounded upward or downward, as shown in Fig. 8 (for example, in the first 2D convolution of the first block of ResNet-18, only about 1.5% of the weights are aggressively rounded).

[0076] When FlexRound updates the scale, it can consider the magnitude of the pre-trained weights, and large pre-trained weights are more likely to be quantized with greater flexibility compared to relatively small weights. Is It is directly proportional to W (i,j) No matter how large the size of If ga is close to 0 It also becomes 0. In this sense, large-scale pre-trained weights can be quantized into a grid obtainable from RTN. When α is (significantly) greater than 0, large-scale pre-trained weights can be quantized from the two nearest grids to the farther grids, as illustrated in Fig. 9. In short, while the magnitude of the pre-trained weights influences the corresponding scale update of FlexRound, this does not necessarily mean that large weights must be quantized more flexibly than small weights.

[0077] FlexRound can quantize weights more flexibly as the bit width increases. Comparing the right-hand diagrams of Figs. 7 and 8 with Fig. 10, FlexRound can provide more grid shifts than can be obtained from RTN as high bit widths are used. Unlike AdaRound, which only rounds weights up or down regardless of the number of bits used, FlexRound enables more flexible weight quantization as the bit width increases, making it more suitable for quantizing models that require higher bit widths than AdaRound (e.g., LLM).

[0078] As such, according to the embodiments of the present invention, when quantizing the pre-trained weights of a network, large weights can be quantized more flexibly by learning not only the parameters for the quantization grid size but also the scale for each weight according to the magnitude of the weight through rounding based on elemental division of the weights. Consequently, FlexRound using the elemental division-based rounding technique can perform better than existing techniques on various types of tasks, such as image classification, natural language understanding, and natural language generation, for various models ranging from small to large models, including ResNet, MobileNet, BERT, GPT-Neo, OPT, and LLaMA. In particular, in the case of LLM, while existing techniques suffer from performance degradation after quantization, FlexRound according to the present invention can reduce the performance difference before and after quantization and can quantize the activation of LLM to 8-bits without any assumptions regarding activation outliers, thereby simultaneously improving the latency and throughput of LLM.

[0079] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0080] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0081] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may continuously store a program executable by a computer, or temporarily store it for execution or download. Additionally, the medium may be various recording or storage means in the form of a single or several hardware combined, and may not be limited to a medium directly connected to a computer system but may exist distributed over a network. Examples of media may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to store program instructions, including ROM, RAM, and flash memory. Additionally, other examples of media may include recording or storage media managed by app stores that distribute applications or sites and servers that supply or distribute various other software.

[0082] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0083] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

Claim 1 A neural network quantization method executed on a computer device, wherein the computer device comprises at least one processor configured to execute computer-readable instructions contained in memory, and the neural network quantization method comprises the step of performing quantization by applying learnable parameters to pre-learned weights of a neural network by the at least one processor, wherein the step of performing quantization comprises quantizing the pre-learned weights through an element-wise division-based rounding function using the parameters, and quantizing the pre-learned weights through a rounding scheme using a common quantization grid size of the entire layer together with a division factor corresponding to the scale of the pre-learned weights. Claim 2 A neural network quantization method according to claim 1, wherein the step of performing the quantization is characterized by performing post-training quantization for each layer or block of the network. Claim 3 A neural network quantization method according to claim 1, wherein the step of performing quantization comprises updating quantization parameters for pre-learned weights such that the difference between the output of the network using pre-learned weights before quantization and the output of the network using quantized weights after quantization is minimized. Claim 4 A neural network quantization method according to claim 1, wherein the step of performing the quantization comprises the step of jointly learning the common quantization grid size and the division coefficient according to the size of the weights. Claim 5 A neural network quantization method according to claim 1, wherein the step of performing the quantization is characterized by quantizing the pre-learned weights through a division-based rounding scheme of the following mathematical formula. (Here, W is the aforementioned pre-trained weight, is the quantized weight, s1 is the common quantized grid size, S is the division factor, ) Claim 6 In paragraph 5, the step of performing the quantization comprises the output of the network using pre-learned weights (W) before quantization and the quantized weights ( A neural network quantization method comprising the step of updating the common quantization grid size (s1) and the division factor (S) so as to minimize the difference between the outputs of the network using ). Claim 7 A neural network quantization method according to claim 5, characterized in that the division factor (S) is defined as the element-wise product of matrix scaling (S2) that is identical in shape to the common quantization grid size (s1) and the pre-learned weight (W). Claim 8 A neural network quantization method according to claim 7, characterized in that the matrix scaling (S2) is supplemented by at least one additional vector that considers channel statistical changes according to layer type. Claim 9 A neural network quantization method according to claim 7, characterized in that the matrix scaling (S2) is supplemented by an additional vector (s3) for the output channel in the case of a linear layer, and by an additional vector (s3) for the output channel and an additional vector (s4) for the input channel in the case of a 2D convolution. Claim 10 A neural network quantization method according to claim 1, wherein the step of performing the quantization is characterized by performing weighted quantization of a large language model (LM) using a rounding function based on the element division. Claim 11 A computer program stored on a computer-readable recording medium for executing the neural network quantization method of any one of claims 1 to 10 on a computer. Claim 12 A computer device comprising at least one processor configured to execute computer-readable instructions contained in memory, wherein the at least one processor performs quantization by applying learnable parameters to pre-learned weights of a neural network, and processes the process of quantizing the pre-learned weights through a rounding function based on element-wise division using said parameters, and wherein the at least one processor quantizes the pre-learned weights through a rounding scheme using a common quantization grid size of the entire layer together with a division factor corresponding to a scale for said pre-learned weights. Claim 13 A computer device according to claim 12, wherein the at least one processor performs quantization after learning for each layer or block of the network. Claim 14 A computer device according to claim 12, wherein at least one processor updates quantization parameters for pre-learned weights such that the difference between the output of the network using pre-learned weights before quantization and the output of the network using quantized weights after quantization is minimized. Claim 15 A computer device according to claim 12, wherein at least one processor is characterized by jointly learning the common quantization grid size and the division factor according to the size of the weights. Claim 16 A computer device according to claim 12, wherein at least one processor quantizes the pre-learned weights through a division-based rounding scheme of the following mathematical formula. (Here, W is the aforementioned pre-trained weight, is the quantized weight, s1 is the common quantized grid size, S is the division factor, ) Claim 17 In paragraph 16, the above at least one processor comprises the output of the network using pre-learned weights (W) before quantization and the quantized weights ( A computer device characterized by updating the common quantization grid size (s1) and the division factor (S) so as to minimize the difference between the outputs of the network using ). Claim 18 A computer device characterized in that, in claim 16, the division factor (S) is defined as the element-wise product of the common quantization grid size (s1) and the matrix scaling (S2) which has the same shape as the pre-learned weight (W). Claim 19 A computer device according to claim 18, wherein the matrix scaling (S2) is supplemented by at least one additional vector that considers channel statistical changes according to layer type. Claim 20 A computer device according to claim 18, wherein the matrix scaling (S2) is supplemented by an additional vector (s3) for the output channel in the case of a linear layer, and by an additional vector (s3) for the output channel and an additional vector (s4) for the input channel in the case of a 2D convolution.

Images