Method for compressing activation value of artificial neural network model, training method using same, recording medium, and computing device

Abstract

An embodiment relates to a method for compressing an activation value. The method comprises: a first step of calculating sensitivity for each layer of an artificial neural network model, wherein the sensitivity is an index indicating an effect of activation values of each layer on training of the artificial neural network model; a second step of allocating a bit of each layer according to the sensitivity calculated in the first step, wherein a layer having high sensitivity is allocated with a higher bit than a layer having low sensitivity; and a third step of compressing the activation values of each layer according to the bit allocated in the second step.

Description

Activation value compression method of artificial neural network model and learning method using the same, and recording medium and computing device

[0001] The present invention relates to a compression method that solves the bottleneck problem of activation memory during learning of an artificial neural network model by combining effective activation value compression using an average value and a computationally efficient sensitivity analysis method.

[0002]

[0003] In recent years, artificial neural network models such as deep learning have achieved remarkable results in language-related tasks, using large language models (LLMs) to achieve performance similar to or even better than humans.

[0004] Behind this success are various efforts to increase model size, as this directly improves performance according to scaling laws. However, the amount of activation memory required for training also increases proportionally, making practical implementation difficult. One cause of the activation memory bottleneck during training is that the backpropagation algorithm (Kelley, 1960) must store all intermediate activations generated during the forward propagation process in activation memory, which is later used to compute parameter gradients during the backpropagation process.

[0005] For example, as shown in Figure 1, GPT-3 with 175 billion parameters and MT-LNG with 1 trillion parameters require 67.3 GB and 132.7 GB of activated memory, respectively, which exceeds the activated memory occupied by parameters and optimizer states. Figure 1 shows the activated memory usage of GPT-3 (22 B / 175 B) and MT-LNG (530 B / 1 T) using data and model parallelism, and the red dotted line represents the 80 GB capacity of the NVIDIA A100 GPU.

[0006] This problem worsens as microbatch sizes and sequence lengths increase, as the memory used to store activation values ​​increases proportionally, while the activation memory occupied by parameters and optimizer states remains constant. Therefore, reducing activation memory is crucial when training large language models (LLMs).

[0007] The prior research related to the present invention is described as follows.

[0008] Activation rematerialization (Chen et al., 2016; Jain et al., 2019; Feng & Huang, 2021) and reversible networks (Gomez et al., 2017; Kitaev et al., 2020; Sander et al., 2021; Cai et al., 2023) store only a portion of the activations and recompute the remainder during backpropagation. These methods require additional computation during backpropagation and slow down the learning rate.

[0009] Reduced-precision training (Micikevicius et al., 2018; Wang et al., 2018; Chen et al., 2020; Sun et al., 2020) aims to reduce computational precision and training memory by representing each variable (e.g., weights, errors, activations, and gradients) using a low-precision data format, such as FP8. However, training accuracy degrades rapidly as precision decreases, and optimized kernels for low-precision operations are required to maximize learning speed.

[0010] On the other hand, Activation Compression Learning (ACT) (Chakrabarti & Moseley, 2019; Chen et al., 2021; Liu et al., 2022b; Pan et al., 2021; Liu et al., 2022a) aims only to reduce activation memory usage by compressing the activation values ​​before storing them during the forward propagation process. However, existing ACT methods suffer from significant performance degradation when the activations are deeply compressed during training when applied to large language models. For example, MESA (Pan et al., 2021) uniformly compresses the activations of all layers with the same compression ratio, but the training performance deteriorates noticeably when the activation precision falls below 8 bits. GACT (Liu et al., 2022a) determines the compression ratio (i.e., the allocated bit precision) based on the sensitivity of each layer and achieves a 4-bit compression ratio in a transformer model. However, the quantized activation values ​​have a significant impact on the learning performance, and the lowest bit precision that can be assigned to a low-sensitivity layer is 1 bit, which limits the flexibility of bit precision allocation and hinders further compression.

[0011]

[0012] The present invention was created against this technical background, and aims to solve the memory bottleneck problem during artificial neural network model learning by combining effective activation value compression using average values ​​and a computationally efficient sensitivity analysis method.

[0013]

[0014] An activation value compression method of one embodiment includes a first step (S10) of calculating a sensitivity for each layer of an artificial neural network model, wherein the sensitivity is an indicator of the influence of activation values ​​of each layer on learning of the artificial neural network model; a second step (S20) of allocating bits to each layer according to the sensitivity calculated in the first step, such that a layer with a high sensitivity has a higher bit than a layer with a low sensitivity; and a third step (S30) of compressing activation values ​​of each layer according to the bits allocated in the second step.

[0015] The above sensitivity is calculated as the difference between the gradient L2 norm when all layers are compressed with the same bits and the gradient L2 norm when only a specific layer has its bits changed.

[0016] The first step compresses the activation values ​​of all layers with the first seed and trains the artificial neural network model, but only saves the L2 norm of the parameter gradient of each layer, changes the seed used only for compressing the activation values ​​of a specific layer among all layers, trains the artificial neural network model again, but only saves the L2 norm of the parameter gradient of each layer, and calculates the sensitivity of the specific layer based on the difference in the L2 norm value of the specific layer obtained in the two trainings.

[0017] In the second step, the bits are allocated to each layer based on a greedy algorithm, i) initializing the bits of each layer, ii) reducing the bits of one layer in a direction that most greatly reduces the objective function of Equation 1 below according to the sensitivity, iii) checking whether the sum of the reduced bits satisfies the boundary condition of Equation 2 below of the memory according to the preset average bits, and iv) if the boundary condition is not satisfied, repeating i) to iii) until the boundary condition is satisfied.

[0018] The above bits are selected from 0.5 bits, 2 bits, 4 bits, and 8 bits, and in the third step, for the layer to which 0.5 bits is allocated, (a) the activation values ​​of each layer are divided into n groups, (b) all the activation values ​​belonging to each group are added to obtain an average value, and (c) the activation values ​​of each group are replaced with the obtained average value to compress them.

[0019] In the third step above, for the layers allocated with bits other than 0.5 bits, the activation values ​​belonging to each layer are compressed according to the number of bits allocated to the layer.

[0020] In addition, another embodiment of the present invention discloses a learning method of an artificial neural network model based on the compression method, the learning method including: (A) calculating a sensitivity for each layer of the artificial neural network model, wherein the sensitivity is an indicator of the influence of the activation values ​​of each layer on the learning of the artificial neural network model; (B) allocating bits to each layer according to the sensitivity calculated in (A), such that a layer with a high sensitivity has a higher bit than a layer with a low sensitivity; (C) compressing activation values ​​according to the bits allocated to each layer of the artificial neural network model in a forward propagation process according to the bits allocated in (B); and (D) restoring the activation values ​​compressed and stored in (C) in a backpropagation process to update weights.

[0021]

[0022] The present invention can achieve a high compression ratio while maintaining model performance by adaptively allocating the number of bits according to sensitivity to minimize gradient variance and using average quantization to minimize gradient differences.

[0023]

[0024] Figure 1 shows the memory usage of GPT-3 (22B / 175B) and MT-LNG (530B / 1T) using data and model parallelism.

[0025] Figure 2 shows the flow of an activation value compression method according to one embodiment.

[0026] Figure 3 shows the overall framework for a learning method of a neural network model according to one embodiment.

[0027] Figure 4 shows a flowchart of the learning method of the neural network model.

[0028] Fig. 5 shows a computational device that implements an activation value compression method according to one embodiment.

[0029]

[0030] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, detailed descriptions of well-known functions or components that may obscure the gist of the present invention will be omitted in the following description and the attached drawings. Additionally, throughout the specification, the term "including" a component does not exclude other components, unless specifically stated otherwise, but rather implies the inclusion of other components.

[0031] Additionally, while terms such as "first" and "second" may be used to describe various components, these components should not be limited by these terms. These terms may be used to distinguish one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

[0032] The terminology used herein is merely used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the terms "comprise" or "have" indicate the presence of a described feature, number, step, operation, component, part, or combination thereof, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0033] Unless specifically defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by those of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning within the context of the relevant technology, and shall not be construed in an idealized or overly formal sense unless explicitly defined herein.

[0034]

[0035] The present invention relates to a novel framework that can significantly improve compression ratio while maintaining learning performance. In this paper, the inventors mathematically prove that compressing activation values ​​to the average value of a group minimizes gradient variance. Leveraging this characteristic, the present invention proposes Average Quantization (AQ), which achieves high-quality, deeply compressed activations with an effective precision of less than 1 bit and improves the flexibility of precision allocation. Furthermore, an efficient and accurate sensitivity calculation algorithm relies solely on the L2 norm variance of parameter gradients, thereby addressing the memory overhead associated with sensitivity calculations. Experimental results demonstrate that the framework significantly reduces activation memory without compromising accuracy, achieving up to a 10x compression ratio in large-scale language models.

[0036]

[0037] An activation value compression method according to one embodiment of the present invention is as disclosed in FIG. 2. The activation value compression method according to one embodiment includes a first step (S10) of calculating a sensitivity for each layer of an artificial neural network model, wherein the sensitivity is an indicator of an influence of activation values ​​of each layer on learning of the artificial neural network model, a second step (S20) of allocating bits to each layer according to the sensitivity calculated in the first step, such that a layer with a high sensitivity has a higher bit than a layer with a low sensitivity, and a third step (S30) of compressing activation values ​​of each layer according to the bits allocated in the second step.

[0038]

[0039] Step 1 (S10)

[0040] Sensitivity is a measure of how much a change in the activation compression method of each layer of an artificial neural network model affects the operation of the model. Here, activation refers to the value output by each neuron in the neural network after receiving an input signal and performing a calculation. During the forward pass of an artificial neural network model, input data is calculated and transformed as it passes through the neurons of each layer, and the values ​​calculated by each neuron are the activation values.

[0041] More specifically, sensitivity indicates how important the activation values ​​of each layer are to model learning. For example, a layer with high sensitivity indicates that its activation values ​​have a significant impact on model learning, while a layer with low sensitivity indicates that its activation values ​​have a relatively small impact on model learning.

[0042] In the present invention, sensitivity serves as a criterion for determining the number of bits to be allocated to each layer when compressing activation values. In the present invention, more bits are allocated to layers with high sensitivity to minimize information loss, and fewer bits are allocated to layers with low sensitivity to maximize the compression ratio.

[0043] This sensitivity allows the present invention to find an optimal balance between model performance and compression ratio, and the present invention adaptively compresses activation values ​​by taking the sensitivity into account.

[0044] In the present invention, sensitivity is calculated through the GradNormVar algorithm, which measures the sensitivity of a layer by observing how much the gradient norm of the entire model changes when the activation compression method of each layer changes slightly (e.g., when a different seed is used).

[0045] Below is the pseudocode for the GradNormVar algorithm. This explains the process of calculating sensitivity in the present invention.

[0046]

[0047] In the present invention, sensitivity is calculated as the difference between the gradient L2 norm when all layers are compressed to the same bit and the gradient L2 norm when only a specific layer has its bits changed.

[0048] To be more specific about this,

[0049] The first seed compresses the activation values ​​of all layers and trains the artificial neural network model, but only saves the L2 norm of the parameter gradient of each layer.

[0050] Retrain the above artificial neural network model by changing the seed used only for compressing the activation values ​​of a specific layer among all layers, but only saving the L2 norm of the parameter gradient of each layer.

[0051] The variance of the difference in the L2 norm values ​​of the specific layer obtained through two learnings becomes the sensitivity of the layer.

[0052] This sensitivity calculation is performed for all layers of the neural network.

[0053] In the present invention, memory usage can be significantly reduced because only the L2 norm of the parameter gradient of each layer is used instead of the parameter gradient.

[0054] For example, if a layer has 1,000 parameters, the existing method (GACT) requires storing all 1,000 gradient values. However, the present invention only requires storing the size (L2 norm) of the 1,000 gradient vectors.

[0055]

[0056] Step 2 (S20)

[0057] In the second step, bits are allocated to each layer according to the sensitivity of each layer calculated in the first step, so that layers with higher sensitivity have higher bits than layers with lower sensitivity.

[0058] In one example, the bits used may be 0.5 bits, 2 bits, 4 bits, or 8 bits. The following description is based on this, but the present invention is not intended to be limited thereto.

[0059] The gradient (or gradient) is a value used when updating the model's weights during learning, and the model can learn well only when this gradient is calculated accurately.

[0060] Compressing activation values ​​naturally affects gradient calculations. The key here is to minimize the "change" in gradient due to compression. Large changes can lead to unstable model learning.

[0061] As announced in the paper (ALAM: AVERAGED LOW-PRECISION ACTIVATION FOR MEMORY-EFFICIENT TRAINING OF TRANSFORMER MODELS), the inventor mathematically analyzed this change amount using the concept of 'slope variance' and proved that this slope variance is smallest when the activation values ​​are replaced with the average value.

[0062] Simply put, unifying all values ​​within a group to the value closest to the average minimizes the amount of variation, and the present invention uses the average value.

[0063] For example, if the values ​​of a group are [1, 2, 3, 4], if all of these values ​​are replaced by the minimum value of 1, the result is [1, 1, 1, 1]. Conversely, if all of these values ​​are replaced by the maximum value of 4, the result is [4, 4, 4, 4]. However, if they are replaced by the average value of 2.5, the result is [2.5, 2.5, 2.5, 2.5].

[0064] If we add up the differences between the original values ​​of the group and the values ​​after compression,

[0065] |1-1| + |2-1| + |3-1| + |4-1| = 6

[0066] |1-4| + |2-4| + |3-4| + |4-4| = 6

[0067] |1-2.5| + |2-2.5| + |3-2.5| + |4-2.5| = 4

[0068] When the average value is used like this, the difference is the smallest, and the smaller this difference is, the smaller the error in calculating the slope can be.

[0069] In the present invention, considering these points, when compressing to a specific bit, the activation value is compressed to an average, which not only reduces memory usage, but also helps maintain the learning performance of the model to the maximum.

[0070] In the present invention, the sensitivity-based bit allocation for each layer is based on a greedy algorithm. This grid algorithm is a problem-solving method that makes the most optimal choice at each step. It makes the best choice in the current situation without considering the future impact of that choice. In other words, this grid algorithm focuses on maximizing immediate benefits.

[0071] The bit allocation algorithm of the present invention based on this grid algorithm is described as follows.

[0072] In the present invention, the second step is,

[0073] i) Initialize the bits of each layer above,

[0074] ii) Lower the bits of one layer in the direction that reduces the objective function of Equation 1 below the most according to the above sensitivity,

[0075]

[0076] In mathematical expression 1, the gradient when using compressed activation values ​​is , L is the total number of layers, cl is the sensitivity of the lth layer, and bl is the number of bits allocated to the lth layer.

[0077] Through mathematical expression 1, we can see that the less the activation value is compressed (the larger the bl), the smaller the gradient variance, the higher the sensitivity of the layer, the greater the influence on the variance, and the upper limit of the overall gradient variance is determined by the sum of the variances of each layer.

[0078] In the present invention, the number of bits (bl) of each layer is determined in a direction that minimizes this gradient variance. That is, more bits are allocated to layers with high sensitivity, and fewer bits are allocated to layers with low sensitivity.

[0079] iii) Check whether the sum of the reduced bits satisfies the boundary condition of the following mathematical expression 2 of the memory according to the preset average bits,

[0080]

[0081] iv) If the above boundary conditions are not satisfied, steps ii) to iii) are repeated until the above boundary conditions are satisfied.

[0082] In other words, bits are allocated differently for each layer to minimize variance from a given average bit (e.g., 1-bit), and the bits are allocated based on Equation 2. In Equation 2, bl is the number of bits in the layer, Bset is the average bit set by the user, and nl is the size of the activation value.

[0083] For example, if all activation values ​​have the same size (nl) and there are 3 layers, and the user sets the average bit to 1 bit, the explanation is as follows.

[0084] i) Initially, all layers' bits are initialized to 8-bit. (8-bit, 8-bit, 8-bit)

[0085] ii) In this case, the bits of the first layer with the greatest sensitivity are lowered in the direction that reduces the target function of Equation 1 the most (8-bit, 8-bit, 8-bit) -> (4-bit, 8-bit, 8-bit)

[0086] iii) Check if this satisfies the boundary conditions of Equation 2. [Check if (4+8+8) <= 3*1 holds.]

[0087] iv) If iii) does not hold, repeat ii) and iii).

[0088] As a result, it becomes (0.5-bit, 0.5-bit, 2-bit) and if the boundary condition (0.5+0.5+2) <= 3*1 is satisfied, compression begins with that bit.

[0089]

[0090] Step 3 (S30)

[0091] The third step is a process of compressing the activation values ​​of each layer according to the bits allocated in the second step.

[0092] In the present invention, activation values ​​are compressed using an average low-precision compression algorithm and a general quantization algorithm.

[0093] General quantization algorithm

[0094] In conventional quantization, each individual activation value is compressed independently according to the bits assigned to the layer.

[0095] For example, 8-bit quantization approximates each activation value to one of 256 (2^8) levels. In this case, each activation value occupies 8 bits of memory.

[0096] In general quantization, 1 bit is the theoretically smallest possible unit, where each value is represented as 0 or 1.

[0097]

[0098] Average low-precision compression algorithm

[0099] In this algorithm, the group average of the activation values ​​is calculated and this average is quantized.

[0100] For example, if four activation values ​​are grouped together and the average of this group is quantized to 8 bits (the number of bits allocated to the layer), each value is effectively represented by 2 bits (8 bits / 4). This means that the individual value level is compressed to a precision lower than 2 bits.

[0101] This algorithm allows for compression to a precision of less than 1 bit, for example, "0.5 bit". This is equivalent to grouping eight values ​​and quantizing their average to 4 bits.

[0102] This average-precision compression algorithm compresses groups of values ​​instead of individual values. This enables ultra-low-precision compression of less than 1 bit, a level of compression not possible with conventional quantization.

[0103] Furthermore, this average low-precision compression algorithm minimizes information loss due to compression by using group averages. The inventors mathematically proved that using group averages minimizes gradient variance. This suggests that this average low-precision compression algorithm is an effective compression method that minimizes the impact on model performance.

[0104] In contrast, conventional quantization compresses individual values ​​independently, making compression less than 1 bit impossible. Furthermore, extreme compression (e.g., 1-bit quantization) can result in significant information loss, significantly degrading model performance.

[0105] Considering these points, the present invention utilizes both methods. First, the possibility of ultra-low-precision compression is opened up through an average low-precision compression algorithm, and then an appropriate precision (number of bits) is allocated according to the sensitivity of each layer through a greedy algorithm. For layers with low sensitivity (e.g., layers allocated 0.5 bits), the average low-precision compression algorithm is used to compress to less than 1 bit, while for layers with high sensitivity (e.g., layers allocated one of 2, 4, or 8 bits), general quantization is used to maintain sufficient precision.

[0106] As described above, the present invention adaptively allocates the number of bits according to sensitivity to minimize gradient variance (S20) and uses average quantization to minimize gradient differences (S30). Through these two methods, the present invention can achieve a high compression ratio while maintaining model performance.

[0107]

[0108] Hereinafter, with reference to FIGS. 3 and 4, a learning method for an artificial neural network model based on the aforementioned compression method will be described. Here, FIG. 3 shows the overall framework for the learning method, and FIG. 4 shows a flowchart of the learning method.

[0109] The learning method of an artificial neural network model according to the present embodiment includes: (A) step (S100) of calculating sensitivity for each layer of an artificial neural network model, wherein the sensitivity is an indicator of the influence of activation values ​​of each layer on the learning of the artificial neural network model; (B) step (S200) of allocating bits to each layer according to the sensitivity calculated in step (A), such that a layer with a high sensitivity has a higher bit than a layer with a low sensitivity; (C) step (S300) of compressing activation values ​​according to the bits allocated to each layer of the artificial neural network model in a forward propagation process according to the bits allocated in step (B); and (D) step (S400) of restoring the activation values ​​compressed and stored in step (C) in a backpropagation process to update weights.

[0110] Here, steps S100 to S300 are substantially the same as steps S10 to S30 described above, and are therefore described in detail here.

[0111] The above sensitivity is calculated as the difference between the gradient L2 norm when all layers are compressed with the same bits and the gradient L2 norm when only a specific layer has its bits changed.

[0112] In the above step (A), the activation values ​​of all layers are compressed with the first seed and the artificial neural network model is trained, but only the L2 norm of the parameter gradient of each layer is saved, and the seed used only for compressing the activation values ​​of a specific layer among all layers is changed to train the artificial neural network model again, but only the L2 norm of the parameter gradient of each layer is saved, and the sensitivity of the specific layer is calculated based on the difference in the L2 norm value of the specific layer obtained in the two trainings. This process is repeated identically for all layers of the neural network.

[0113] In the step (B) above, the bits are allocated to each layer based on a greedy algorithm, and the step (B) is performed by: i) initializing the bits of each layer, ii) reducing the bits of one layer in a direction that most reduces the objective function of Equation 1 according to the sensitivity, iii) checking whether the sum of the reduced bits satisfies the boundary condition of Equation 2 of the memory according to a preset average bit, and iv) repeating i) to iii) above until the boundary condition is satisfied if the boundary condition is not satisfied.

[0114] The above bits can be selected from 0.5 bits, 2 bits, 4 bits, and 8 bits.

[0115] In the above step (C), for the layer to which 0.5 bits are allocated, (a) the activation values ​​of each layer are divided into n groups, (b) the activation values ​​belonging to each group are added to obtain an average value, and (c) the activation values ​​of each group are replaced with the obtained average value to compress them.

[0116] In the above process (a), adjacent activation values ​​can be grouped together.

[0117] In the above step (C), for the layers allocated with bits other than 0.5 bits, the activation values ​​belonging to each layer are compressed according to the number of bits allocated to the layer.

[0118]

[0119] Another embodiment of the present invention relates to a computing device implementing the above-described activation value compression method.

[0120] Figure 5 is a block diagram schematically illustrating the configuration of the computing device. It reconstructs the above-described series of components from a hardware perspective. Therefore, to avoid redundancy, this section provides a brief overview, focusing on the functions and operations of each component.

[0121] The computing device (800) is configured to include a memory (830) that stores a program (820) coded so that the computer can read the activation value compression method described above through FIG. 2, and a processor (810) that executes the program.

[0122] Here, the activation value compression method includes a first step (S10) of calculating a sensitivity for each layer of an artificial neural network model, wherein the sensitivity is an indicator of the influence of the activation values ​​of each layer on the learning of the artificial neural network model, a second step (S20) of allocating bits to each layer according to the sensitivity calculated in the first step, such that a layer with a high sensitivity has a higher bit than a layer with a low sensitivity, and a third step (S30) of compressing the activation values ​​of each layer according to the bits allocated in the second step.

[0123] The above sensitivity is calculated as the difference between the gradient L2 norm when all layers are compressed with the same bits and the gradient L2 norm when only a specific layer has its bits changed.

[0124] The first step compresses the activation values ​​of all layers with the first seed and trains the artificial neural network model, but only saves the L2 norm of the parameter gradient of each layer, changes the seed used only for compressing the activation values ​​of a specific layer among all layers, trains the artificial neural network model again, but only saves the L2 norm of the parameter gradient of each layer, and calculates the sensitivity of the specific layer based on the difference in the L2 norm value of the specific layer obtained in the two trainings.

[0125] In the second step, the bits are allocated to each layer based on a greedy algorithm, i) initializing the bits of each layer, ii) reducing the bits of one layer in a direction that most greatly reduces the objective function of Equation 1 below according to the sensitivity, iii) checking whether the sum of the reduced bits satisfies the boundary condition of Equation 2 below of the memory according to the preset average bits, and iv) if the boundary condition is not satisfied, repeating i) to iii) until the boundary condition is satisfied.

[0126] The above bits are selected from 0.5 bits, 2 bits, 4 bits, and 8 bits, and in the third step, for the layer to which 0.5 bits is allocated, (a) the activation values ​​of each layer are divided into n groups, (b) all the activation values ​​belonging to each group are added to obtain an average value, and (c) the activation values ​​of each group are replaced with the obtained average value to compress them.

[0127] In the third step above, for the layers allocated with bits other than 0.5 bits, the activation values ​​belonging to each layer are compressed according to the number of bits allocated to the layer.

[0128] Meanwhile, the above-described activation value compression method can be implemented as computer-readable code on a computer-readable recording medium. A computer-readable recording medium includes any type of recording device that stores data that can be read by a computer system.

[0129] Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disks, and optical data storage devices. Furthermore, computer-readable recording media can be distributed across network-connected computer systems, allowing computer-readable code to be stored and executed in a distributed manner. Furthermore, functional programs, codes, and code segments for implementing the present invention can be readily inferred by programmers in the technical field to which the present invention pertains.

[0130] The present invention has been described above, focusing on various embodiments thereof. Those skilled in the art will appreciate that the present invention can be implemented in modified forms without departing from its essential characteristics. Therefore, the disclosed embodiments should be considered illustrative rather than limiting. The scope of the present invention is set forth in the claims, not the foregoing description, and all differences within the scope equivalent thereto should be construed as being encompassed by the present invention.

Claims

1. A first step of calculating sensitivity for each layer of an artificial neural network model, wherein the sensitivity is an indicator of the influence of the activation values ​​of each layer on the learning of the artificial neural network model; A second step of allocating bits to each layer according to the sensitivity calculated in the first step, such that a layer with a high sensitivity has a higher bit than a layer with a low sensitivity; and A third step of compressing the activation values ​​of each layer according to the bits allocated in the second step; An activation value compression method including:

2. In paragraph 1, The above sensitivity is an activation value compression method calculated as the difference between the gradient L2 norm when all layers are compressed with the same bit and the gradient L2 norm when only a specific layer has its bits changed.

3. In paragraph 2, The above first step is, The activation values ​​of all layers are compressed as the first seed and the artificial neural network model is trained, but only the L2 norm of the parameter gradient of each layer is saved. Retrain the above artificial neural network model by changing the seed used only for compressing the activation values ​​of a specific layer among all layers, but only saving the L2 norm of the parameter gradient of each layer. Calculating the sensitivity of the specific layer based on the difference in the L2 norm values ​​of the specific layer obtained in the two learnings above. Activation value compression method.

4. In paragraph 1, In the second step, the activation value compression method allocates bits to each layer based on a greedy algorithm.

5. In paragraph 4, The second step above is, i) Initialize the bits of each layer above, ii) Lower the bits of one layer in the direction that reduces the objective function of Equation 1 below the most according to the above sensitivity, iii) Check whether the sum of the reduced bits satisfies the boundary condition of the following formula 2 of the memory according to the preset average bits, iv) If the above boundary conditions are not satisfied, repeat steps ii) to iii) until the above boundary conditions are satisfied. Activation value compression method.

6. In paragraph 1, The above bits are selected from 0.5 bits, 2 bits, 4 bits, and 8 bits, and are an activation value compression method.

7. In paragraph 6, In the third step above, For layers allocated 0.5 bits, (a) Divide the activation values ​​of each layer into n groups, (b) Add up all the activation values ​​belonging to each group and calculate the average value, (c) Compressing the activation values ​​of each group by replacing them with the obtained average value. Activation value compression method.

8. In paragraph 6, In the third step above, An activation value compression method that compresses the activation values ​​belonging to each layer according to the number of bits allocated to the layer, for layers allocated with bits other than 0.5 bits.

9. A recording medium having recorded thereon a computer-readable program coded to perform the activation value compression method described in any one of paragraphs 1 to 8.

10. Step (A) of calculating sensitivity for each layer of an artificial neural network model, wherein the sensitivity is an indicator of the influence of the activation values ​​of each layer on the learning of the artificial neural network model; Step (B) of allocating bits to each layer according to the sensitivity calculated in step (A), such that a layer with high sensitivity has higher bits than a layer with low sensitivity; (C) step of compressing the activation values ​​according to the bits assigned to each layer of the artificial neural network model in the forward propagation process according to the bits assigned in the above (B) step; and Step (D) of restoring the activation values ​​compressed and stored in step (C) in the backpropagation process and updating the weights; A learning method for an artificial neural network model including .

11. In paragraph 10, The above sensitivity is a learning method of an artificial neural network model, calculated as the difference between the gradient L2 norm when all layers are compressed to the same bit and the gradient L2 norm when only a specific layer has its bits changed.

12. In paragraph 11, Step (A) above, The activation values ​​of all layers are compressed as the first seed and the artificial neural network model is trained, but only the L2 norm of the parameter gradient of each layer is saved. Retrain the above artificial neural network model by changing the seed used only for compressing the activation values ​​of a specific layer among all layers, but only saving the L2 norm of the parameter gradient of each layer. Calculating the sensitivity of the specific layer based on the difference in the L2 norm values ​​of the specific layer obtained in the two learnings above. Learning methods for artificial neural network models.

13. In paragraph 10, In the above step (B), a learning method of an artificial neural network model, wherein the bits are allocated to each layer based on a greedy algorithm.

14. In paragraph 13, Step (B) above, i) Initialize the bits of each layer above, ii) Lower the bits of one layer in the direction that reduces the objective function of mathematical expression 1 the most according to the above sensitivity, iii) Check whether the sum of the reduced bits satisfies the boundary condition of mathematical expression 2 of the memory according to the preset average bits, iv) If the above boundary conditions are not satisfied, repeat steps ii) to iii) until the above boundary conditions are satisfied. Learning methods for artificial neural network models.

15. In paragraph 10, The above bits are selected from 0.5 bits, 2 bits, 4 bits, and 8 bits, and are a learning method of an artificial neural network model.

16. In paragraph 15, In the above step (C), For layers allocated 0.5 bits, (a) Divide the activation values ​​of each layer into n groups, (b) Add up all the activation values ​​belonging to each group and calculate the average value, (c) Compressing the activation values ​​of each group by replacing them with the obtained average value. Learning methods for artificial neural network models.

17. In paragraph 16, In the above step (C), A learning method for an artificial neural network model in which the activation values ​​belonging to each layer are compressed according to the number of bits allocated to the layer, for layers allocated with bits other than 0.5 bits.

18. A memory storing a program coded so that a computer can read the activation value compression method described in any one of paragraphs 1 to 8; and A processor executing the above program; A computing device including a .

Images