Low-precision floating-point friendly quantization method and apparatus

Abstract

A low-precision floating-point friendly quantization method and apparatus are provided. A quantization apparatus of the present invention outputs a reduced-precision floating-point deep learning model from an input model. The quantization apparatus according to a preferred embodiment of the present invention comprises: a target layer detection unit for detecting a target layer in an input model; a channel detection unit for detecting a problematic channel for the layer detected by the target layer detection unit; and a quantization unit for selecting weights to which a predetermined quantization method is to be applied for the channel detected by the channel detection unit, and applying the predetermined quantization method to the selected weights.

Description

Low-precision floating-point friendly quantization method and device

[0001] The present invention relates to a low-precision floating-point friendly quantization method and device.

[0002] In a CNN (Convolutional Neural Network), multiple convolutional layers and multiple fully connected layers can be provided between the input layer and the output layer. When processing images using CNN, one convolutional layer can have as many filters as the number of channels of the input image, and the output image of the convolutional layer is generated by applying the filter assigned to each channel. For example, in a convolutional layer that applies a 3x3 filter to an input image in the form of a 4x4x1 tensor, an image in the form of a 2x2x1 tensor is generated through a convolution operation on the image and the filter. FP32 is generally used as a format to express the weights that constitute the filter, but it requires a lot of computational power and memory resources.

[0003] Therefore, quantization is often used to efficiently execute neural networks. The network's weights and activations can be subject to quantization. Quantization can be used to convert data into a lower-bit data type than the most commonly used default format, FP32.

[0004] The types being converted include fixed-point formats such as INT16, INT8, and INT4, and floating-point formats such as FP16 and FP8. Because they use a lower bit width than the default FP32, they are also referred to as reduced precision.

[0005] While FP32 and FP16 are standardized in the IEEE-754 format, other formats, such as INT8 and FP8, are not. Therefore, detailed implementations vary depending on the developer. Furthermore, various formats for neural networks exist, such as BFLOAT16, Flexpoint, and TensorFloat32.

[0006] Running models with reduced precision offers the advantage of reduced computational and memory usage, resulting in shorter execution times. However, the use of a lower bit width to represent numbers results in information loss, which reduces accuracy.

[0007] There are two main approaches to building models using quantization: Quantization-Aware Training (QAT), which incorporates the effects of reduced precision during model training, and Post-Training Quantization (PTQ), which converts a pre-trained model in FP32 to reduced precision.

[0008] The QAT method can create a model suitable for the target type and produce high accuracy results because the effect of reduced precision is reflected during model learning, but training data is required for learning.

[0009] The PTQ method does not require training data, but it suffers from low accuracy due to the loss incurred by applying reduced precision types. Calibration can be performed to address this issue.

[0010] Calibration is the process of running a model in advance using sample data (a calibration set) and verifying the statistical distribution of values ​​within the model. This process allows for appropriately setting the values ​​of some of the variables used in the quantization process.

[0011] Floating-point types are divided into sign, exponent, and mantissa parts. For example, the IEEE-754 single-precision floating-point type has 1 sign bit, 8 exponent bits, and 23 mantissa bits. Types other than FP32 have different bit configurations for the exponent and mantissa. For example, FP16 has 1 sign bit, 5 exponent bits, and 10 mantissa bits.

[0012] The FP8 format, recently used in deep learning, has no established standard. The sign-exponent-mantissa configuration is often attempted in various configurations, such as 1-5-2, 1-4-3, 1-3-4, and 1-2-5, and the details may vary slightly depending on the developer. However, the basic format of the formula is identical to the IEEE-754 single-precision floating-point format.

[0013] That is, the dynamic range is controlled by the exponent, and the precision is controlled by the mantissa. While the INT8 type uses 7 bits for precision if the sign is expressed, and 8 bits if the sign is not expressed, FP8 uses fewer bits for precision. FP quantization increases quantization error as the number increases.

[0014] One of the operations of a neural network is depthwise convolution. This operation is a core element of well-known networks such as MobileNet and EfficientNet, and is also widely used in other networks. Depthwise convolution is a convolution operation that operates independently of the input channel. Unlike general convolution, this operation does not accumulate values ​​for the input channel. In other words, the operation is performed on elements as large as the filter size of the depthwise convolution. For example, a 2-dimensional depthwise convolution operation (Pytorch: DWConv2D(k=3)) with a filter size of 3 performs matrix multiply and accumulation operations on the 3*3 region of the input tensor and the 3*3 region of the filter.

[0015] However, depth-wise convolution operations often suffer from significant accuracy degradation when using reduced-precision floating-point operations. Even in typical convolutions, if the input channels are small and the number of elements being computed is small, using reduced-precision floating-point types (e.g., FP8) can result in significant errors and reduced accuracy.

[0016] The present invention has been made in consideration of these points, and aims to provide a quantization method capable of improving the decrease in accuracy that occurs when general quantization is applied.

[0017] Another object of the present invention is to provide a quantization method that can significantly reduce computational complexity, memory usage, and power consumption when using reduced precision.

[0018] The quantization device of the present invention outputs a reduced-precision floating-point deep learning model from an input model. The quantization device according to a preferred embodiment of the present invention comprises a target layer detection unit that detects a target layer from an input model, a channel detection unit that detects a problematic channel for the layer detected by the target layer detection unit, and a quantization unit that selects weights to which a predetermined quantization method is to be applied for the channel detected by the channel detection unit, and applies the predetermined quantization method to the selected weights.

[0019] In one embodiment, the target layer detection unit detects a depth-based convolution or a convolution having a predetermined number of input channels or less as a target layer. In one embodiment, the channel detection unit may calculate a predetermined metric for each channel of the target layer and determine that the channel is problematic if the metric satisfies a predetermined condition. The predetermined condition may be one of the following: the calculated metric being greater than or equal to a predetermined threshold value; or the calculated metric being greater than or equal to a certain percentage of the top when sorted by size. The metric is defined to indicate how widely the weights are distributed or the size of the weights. In one embodiment, the metric may be one of the mean square error of the weight quantization error, the sum of the weight quantization errors, the variance of the weights per channel, the maximum value of the weights per channel, and the square of the maximum value of the weights per channel.

[0020] In one embodiment, the quantization unit may select weights to which the predetermined quantization method is to be applied using any one of a method of selecting a predetermined number of weights having the largest absolute value of quantization error in the corresponding channel, a method of selecting a weight having the largest exponent value, and a method of selecting all weights of the corresponding channel.

[0021] In one embodiment, the predetermined quantization method may be any one of a first quantization method (Rounding Away From Zero) that selects a value far from 0, a second quantization method (Rounding Towards Zero) that selects a value close to 0, and a method that selects a combination that minimizes the sum of the squares of quantization errors for the entire target weight when applying both the first quantization method and the second quantization method.

[0022] A quantization method according to a preferred embodiment of the present invention is a quantization method for outputting a reduced-precision floating-point deep learning model from an input model, comprising: a target layer detection step for detecting a target layer in an input model; a channel detection step for detecting a problematic channel for the layer detected by the target layer detection unit; and a quantization step for selecting weights to which a predetermined quantization method is to be applied for the channel detected by the channel detection unit, and applying the predetermined quantization method to the selected weights.

[0023] In one embodiment, in the target layer detection step, a depth-based convolution or a convolution with a predetermined number or less of input channels is detected as the target layer.

[0024] In one embodiment, the channel detection step calculates a predetermined metric for each channel of the target layer and determines that the channel is problematic if the metric satisfies a predetermined condition. The predetermined condition may be one of the following: the calculated metric is greater than or equal to a predetermined threshold value, or the calculated metric is greater than or equal to a certain percentage of the upper rank when sorted by size. As the metric, any one of the mean square error of weight quantization error, the sum of weight quantization errors, the variance of weights per channel, the maximum value of weights per channel, and the square of the maximum value of weights per channel may be used.

[0025] In one embodiment, the quantization step may select weights to which the predetermined quantization method is to be applied using any one of a method of selecting a predetermined number of weights having the largest absolute value of quantization error in the corresponding channel, a method of selecting a weight having the largest exponent value, and a method of selecting all weights of the corresponding channel.

[0026] In one embodiment, the predetermined quantization method may be any one of a first quantization method (Rounding Away From Zero) that selects a value far from 0, a second quantization method (Rounding Towards Zero) that selects a value close to 0, and a method that selects a combination that minimizes the sum of the squares of quantization errors for the entire target weight when applying both the first quantization method and the second quantization method.

[0027] According to one embodiment of the present invention, by finding a channel that is problematic for depth-based convolution operation and then using a quantization method according to the present invention, it is possible to significantly recover the accuracy degradation that occurs when applying general quantization.

[0028] According to one embodiment of the present invention, when using reduced precision, the amount of computation, memory usage and power consumption can be reduced.

[0029] According to one embodiment of the present invention, in the case of general convolution, even if the input channel is small and the number of elements to be operated is small, the decrease in accuracy can be recovered.

[0030] FIG. 1 is a block diagram showing the configuration of a low-precision floating-point friendly quantization device according to a preferred embodiment of the present invention.

[0031] FIG. 2 is a flowchart showing the operation flow of a low-precision floating-point friendly quantization method according to a preferred embodiment of the present invention.

[0032] Figure 3 is a diagram illustrating an example of selecting weights according to index values.

[0033] Figure 4 is a diagram illustrating several examples of quantization methods.

[0034] Hereinafter, some embodiments of the present disclosure will be described in detail using exemplary drawings. When designating components in each drawing, it should be noted that, where possible, identical components are given the same reference numerals, even if they appear in different drawings. Furthermore, when describing the present disclosure, detailed descriptions of related known structures or functions will be omitted if they are deemed to obscure the gist of the present disclosure.

[0035] In describing components of embodiments according to the present disclosure, symbols such as first, second, i), ii), a), b) may be used. These symbols are only for distinguishing the components from other components, and the nature, order, or sequence of the components are not limited by the symbols. When a part in the specification is said to 'include' or 'have' a component, this does not mean that other components are excluded, but rather that other components may be further included, unless explicitly stated to the contrary. In addition, terms such as 'part' and 'module' described in the specification mean a unit that processes at least one function or operation, and this may be implemented by hardware, software, or a combination of hardware and software.

[0036] The following description of the invention, together with the accompanying drawings, is intended to illustrate exemplary embodiments of the invention and is not intended to represent the only embodiments in which the invention may be practiced.

[0037] The following description is based on FP8 for convenience, but reduced-precision floating-point can have various bit-widths, including FP8, FP10, FP6, and FP4. FP16 is considered to have a precision close to FP32 and is not considered reduced precision. However, any floating-point data type less than 16 bits can be considered reduced precision.

[0038] Figure 1 is a block diagram illustrating the configuration of a low-precision, floating-point-friendly quantization device according to a preferred embodiment of the present invention. A high-precision, deep learning model developed for floating-point-friendly quantization can be input into the device. For example, an FP16 or FP32 deep learning model can be input.

[0039] A floating-point friendly quantization device according to one embodiment of the present invention comprises a target layer detection unit (110), a channel detection unit (120), and a quantization unit (130).

[0040] The target layer detection unit (110) detects the target operation or layer in the input model. The target layer detection unit (110) can detect depth-based convolution or convolution with small input channels.

[0041] The channel detection unit (120) detects a problematic channel for the operation (layer) detected by the target layer detection unit (110). In one embodiment, the channel detection unit (120) calculates a predetermined metric for each channel of the target layer and finds a case where this metric is large. In order to detect a problematic channel, it is preferable to measure the metric for each channel. As the metric, any one of the mean square error of the weight quantization error, the sum of the weight quantization errors, the variance of the weights for each channel, the maximum value of the weights for each channel, and the square of the maximum value of the weights for each channel can be used.

[0042] The quantization unit (130) applies the proposed quantization method to the channel detected by the channel detection unit (120). In one embodiment, the quantization unit (130) quantizes in a different manner from the conventional RTN (Rounding to nearest) method that quantizes to the nearest value. The quantization unit (130) selects weights to which the different quantization method is to be applied in order to reduce errors in channels that generate many quantization errors, and applies the different quantization method to the selected weights. Various methods may be used as weight selection methods, such as a method of selecting a predetermined number of weights having the largest absolute value of quantization error in the corresponding channel, a method of selecting a weight having the largest exponent value, and a method of selecting all weights of the corresponding channel. As other quantization methods mentioned above, there may be used RAFZ (Rounding Away From Zero) which selects a value far from 0, RTZ (Rounding Towards Zero) which selects a value close to 0, and a method of selecting a combination that minimizes the sum of the squares of quantization errors for all target weights when applying both RAFZ and RTZ.

[0043] This process outputs a reduced-precision floating-point deep learning model. That is, the same model is output using a bit-width smaller than the precision of the input model.

[0044] Figure 2 is a flowchart illustrating the operational flow of a low-precision, floating-point-friendly quantization method according to a preferred embodiment of the present invention. A high-precision, deep learning model developed for floating-point-friendly quantization can be input into the floating-point-friendly quantization device. For example, an FP16 or FP32 deep learning model can be input.

[0045] A floating-point-friendly quantization device detects a target operation or layer in an input model (step S110). The problematic channel is detected for the detected operation (layer) (step S120). The proposed quantization method is applied to the detected channel (step S130). Through this process, a reduced-precision floating-point deep learning model is output. In other words, the same model is output using a bit-width smaller than the precision of the input model.

[0046] Each action step is described in detail below.

[0047] 1. Target motion detection (step S110)

[0048] Detects depth-dependent convolutions or convolutions with small input channels. The criteria for determining whether input channels are small can be determined by the user. Preferably, the number of input channels is 3 or less, but this value is not limited to this. Convolutions with larger input channels can also be targeted, but in this case, additional computation may be required. While performance (accuracy) does not deteriorate after applying the method of the present invention, performance improvement is not guaranteed when the input channels are large.

[0049] 2. Target channel detection method (step S120)

[0050] A specific metric is calculated for each channel of the target layer, and cases where this metric is large are identified. The metric should be able to represent the wide distribution of the input tensor (the operation input) and the weights, or the magnitude of their values. To detect problematic channels, the metric is measured for each channel.

[0051] For example, the following metrics can be used as metrics. In the equation below, W_FP32 represents a weight expressed in the FP32 type, and W_target_type represents a weight expressed in the target type, for example, the FP8 type.

[0052] Below, the metrics are described based on two dimensions, and accordingly, two letters, i and j, are used. In other embodiments, the metrics can be applied to one or three dimensions. In one-dimensional metrics, only the letter i is used, while in three-dimensional metrics, three letters, i, j, and k, are used.

[0053] < Metric 1 >

[0054] The mean squared error of the weight quantization error, expressed as in mathematical expression 1, can be used as a metric.

[0055]

[0056] < Metric 2 >

[0057] The sum of weighted quantization errors expressed as in mathematical expression 2 can be used as a metric.

[0058]

[0059] < Metric 3 >

[0060] The variance of the channel-specific weights, expressed as in mathematical expression 3, can be used as a metric.

[0061]

[0062] < Metric 4 >

[0063] The maximum value of the channel-specific weight expressed as in mathematical expression 4 can be used as a metric.

[0064]

[0065] < Metric 5 >

[0066] The square of the maximum value of the weights per channel, expressed as in mathematical expression 5, can be used as a metric.

[0067]

[0068] A metric being "large" can be determined by exceeding an arbitrary, experimentally determined threshold, or by whether it ranks among the top values ​​when sorted by size. "Top value" can be defined as a percentage, such as the top 10%.

[0069] The lower the threshold or ratio, the greater the number of targeted channels and the greater the likelihood of recovering accuracy. However, the execution overhead (running time of the resulting model, complexity of applying the new method, etc.) varies depending on the solution applied to the targeted channel, so increasing the threshold value unconditionally is not feasible. Furthermore, increasing the threshold value unconditionally does not guarantee infinite recovery from accuracy degradation. Therefore, selecting an appropriate value is advisable.

[0070] 3. Quantization method applied to the problem channel (step S130)

[0071] To reduce errors in channels that generate a lot of quantization errors, the following method can be applied. The target channel is k 2 There are weights, and among these weights, the weights to which a new quantization method is to be applied are selected, and the new quantization method is applied to the selected weights. Meanwhile, in n-dimensional convolution and deconvolution, the number of weights is k. n It could be a dog.

[0072] k existing in the channel 2 Among the dog's weights, the target weight can be specified in various ways as follows.

[0073] < Weight Selection Method 1 >

[0074] This method selects a predetermined number of weights with the largest absolute value of quantization error in the corresponding channel. The quantization error can be defined as the value obtained by subtracting the value when converted to the target type from the value of W_FP32. In one embodiment, selecting the two weights with the largest absolute values ​​of quantization error can result in selecting one positive weight and one negative weight.

[0075] < Weight Selection Method 2 >

[0076] The weight with the largest exponent is selected. In other words, all weights with the largest exponent among sign-exponent-decimal when converted to the desired type are selected. A larger exponent indicates a larger weight. In other embodiments, the weights with the largest and second-largest exponents can be selected.

[0077] Figure 3 illustrates an example of selecting weights based on exponent values. Method 2-1 selects the one with the largest exponent value, and Method 2-2 selects the one with the largest and second largest exponent values. In the example of Figure 3, in the case of Method 2-1, weights 4 and 6, which have the largest exponent value of "1100", are selected, and in the case of Method 2-2, weights 4 and 6, which have the largest exponent value of "1100", and weights 7 and 9, which have the second largest exponent value of "1011", are selected.

[0078] < Weight Selection Method 3 >

[0079] You can select all weights for a given channel. This means that you can select all weights, not just those that satisfy a specific condition.

[0080] Typically, when quantizing, real numbers are rounded to the nearest integer. For example, as shown in (a) of Fig. 4, -3.36, which is between -3.5 and -3.25, is quantized to -3.25 because the difference between -3.5 and -3.25 is 0.14 and 0.12, respectively, which is smaller than the difference between -3.5 and -3.25. 3.19, which is between 3 and 3.25, is quantized to 3.25 because the difference between 3.00 and 3.36 is 0.06, which is smaller than the difference between 3.00 and 3.25, which is 0.19.

[0081] In the present invention, when quantizing selected weights, one of the following quantization methods can be applied instead of the general rounding method.

[0082] < Quantization Method 1 > Rounding away from zero

[0083] Rounding to a value farther from 0. For example, as shown in (b) of Fig. 4, the value -3.36, which is between -3.5 and -3.25, is quantized to -3.5, which is further from 0 than -3.25. The value 3.19, which is between 3 and 3.25, is quantized to 3.25, which is further from 0 than 3.00.

[0084] < Quantization Method 2 > Rounding towards zero

[0085] Rounding to a value closer to 0. For example, as shown in (c) of Fig. 4, the value -3.36, which is between -3.5 and -3.25, is quantized to -3.25, which is closer to 0 than -3.5. The value 3.19, which is between 3 and 3.25, is quantized to 3.00, which is closer to 0 than 3.25.

[0086] < Quantization Method 3 >

[0087] When applying both quantization method 1 (Rounding away from zero) and quantization method 2 (Rounding towards zero) to the target weight, the combination that minimizes the sum of the squares of quantization errors for the entire target weight is selected.

[0088] The accuracy was compared when the existing FP32 format was quantized to NF8 format and when the method of the present invention was applied to the entire depth-wise convolution. When applied to MobileNetV1, when quantized to NF8 format using the existing method, an accuracy of 81% was recorded compared to the FP32 format, whereas when quantized to NF8 format using the method of the present invention, an accuracy of 94% was recorded compared to the FP32 format. When applied to MobileNetV2, when quantized to NF8 format using the existing method, an accuracy of 91% was recorded compared to the FP32 format, whereas when quantized to NF8 format using the method of the present invention, an accuracy of 97% was recorded compared to the FP32 format.

[0089] Each component of the device or method according to the present invention may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.

[0090] Various implementations of the systems and techniques described herein can be implemented as digital electronic circuits, integrated circuits, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations of one or more computer programs executable on a programmable system. The programmable system includes at least one programmable processor (which may be a special purpose processor or a general purpose processor) coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device. Computer programs (also known as programs, software, software applications, or code) include instructions for the programmable processor and are stored on a "computer-readable recording medium."

[0091] A computer-readable recording medium includes any type of recording device that stores data that can be read by a computer system. Such a computer-readable recording medium may be a non-volatile or non-transitory medium such as a ROM, CD-ROM, magnetic tape, floppy disk, memory card, hard disk, magneto-optical disk, storage device, and may further include a transitory medium such as a data transmission medium. Furthermore, the computer-readable recording medium may be distributed across network-connected computer systems, so that computer-readable code can be stored and executed in a distributed manner.

[0092] Although the flowchart of this specification describes each process as being executed sequentially, this is merely an illustrative description of the technical idea of ​​one embodiment of the present disclosure. In other words, a person of ordinary skill in the art to which one embodiment of the present disclosure belongs may modify and apply various modifications and variations by changing the order described in the flowchart of this specification without departing from the essential characteristics of one embodiment of the present disclosure, or by executing one or more of the processes in parallel. Therefore, the flowchart of this specification is not limited to a chronological order.

[0093] The above description is merely an example of the technical idea of ​​the present embodiment, and those skilled in the art to which the present embodiment pertains may make various modifications and variations without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of ​​the present embodiment, but to explain it, and the scope of the technical idea of ​​the present embodiment is not limited by these embodiments. The protection scope of the present embodiment should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of the rights of the present embodiment.

[0094] CROSS-REFERENCE TO RELATED APPLICATION

[0095] This patent application claims priority to Korean Patent Application No. 10-2023-0145070, filed in Korea on October 26, 2023, Korean Patent Application No. 10-2023-0146559, filed in Korea on October 30, 2023, and Korean Patent Application No. 10-2023-0182301, filed in Korea on December 14, 2023, which are incorporated herein by reference in their entirety.

Claims

1. A quantization device that outputs a reduced-precision floating-point deep learning model from an input model, A target layer detection unit that detects the target layer in the input model; A channel detection unit that detects problematic channels for the layers detected by the target layer detection unit, and A quantization unit that selects weights to which a predetermined quantization method is applied for the channels detected by the above channel detection unit, and applies the predetermined quantization method to the selected weights. A quantization device having a .

2. In paragraph 1, The above target layer detection unit detects a depth-based convolution or a convolution with a predetermined number of input channels or less as a target layer. Quantization device.

3. In paragraph 2, The above channel detection unit calculates a predetermined metric for each channel of the target layer and determines that the channel is problematic if the metric satisfies a predetermined condition. Quantization device.

4. In the third paragraph, the predetermined condition is: One of the above calculated metrics being greater than or equal to a predetermined threshold value or one of the above calculated metrics being greater than or equal to a certain percentage of the top when sorted by size. Quantization device.

5. In paragraph 3 or 4, The above metric is one of the mean square error of weight quantization error, sum of weight quantization error, variance of weight per channel, maximum value of weight per channel, and square of maximum value of weight per channel. Quantization device.

6. In any one of paragraphs 1 to 3, the quantization unit, Selecting weights to which the predetermined quantization method is to be applied using any one of the following methods: a method of selecting a predetermined number of weights having the largest absolute value of quantization error in the corresponding channel, a method of selecting a weight having the largest exponent value, and a method of selecting all weights of the corresponding channel. Quantization device.

7. In any one of the first to third paragraphs, the predetermined quantization method, One of the first quantization method (Rounding Away From Zero) that selects a value far from 0, the second quantization method (Rounding Towards Zero) that selects a value close to 0, and the method of selecting a combination that minimizes the sum of the squares of quantization errors for the entire target weight when applying both the first quantization method and the second quantization method. Quantization device.

8. A quantization method for outputting a reduced-precision floating-point deep learning model from an input model, Target layer detection step that detects the target layer in the input model; A channel detection step for detecting problematic channels for the layers detected by the target layer detection unit, and A quantization step of selecting weights to which a predetermined quantization method is applied for the channel detected by the above channel detection unit, and applying the predetermined quantization method to the selected weights. A quantization method having .

9. In paragraph 8, The above target layer detection step detects a depth-based convolution or a convolution with a predetermined number of input channels or less as a target layer. Quantization method.

10. In paragraph 9, The above channel detection step calculates a predetermined metric for each channel of the target layer and determines that the channel is problematic if the metric satisfies a predetermined condition. Quantization method.

11. In paragraph 10, the predetermined condition is: One of the above calculated metrics being greater than or equal to a predetermined threshold value or one of the above calculated metrics being greater than or equal to a certain percentage of the top when sorted by size. Quantization method.

12. In paragraph 10 or 11, The above metric is one of the mean square error of weight quantization error, sum of weight quantization error, variance of weight per channel, maximum value of weight per channel, and square of maximum value of weight per channel. Quantization method.

13. In any one of paragraphs 8 to 10, the quantization step, Selecting weights to which the predetermined quantization method is to be applied using any one of the following methods: a method of selecting a predetermined number of weights having the largest absolute value of quantization error in the corresponding channel, a method of selecting a weight having the largest exponent value, and a method of selecting all weights of the corresponding channel. Quantization method.

14. In any one of the 8th to 10th clauses, the predetermined quantization method, One of the first quantization method (Rounding Away From Zero) that selects a value far from 0, the second quantization method (Rounding Towards Zero) that selects a value close to 0, and the method of selecting a combination that minimizes the sum of the squares of quantization errors for the entire target weight when applying both the first quantization method and the second quantization method. Quantization method.

Images