Calibration method and apparatus using input / output distribution of adjacent layer

Abstract

The present invention relates to a calibration method and apparatus using an input / output distribution of an adjacent layer. According to one aspect of the present invention, provided is a calibration method comprising: an operation process of generating a first operation result in an operation layer of an artificial neural network; a calibration process of acquiring a first effective input range by using information about another layer adjacent to the operation layer; a first quantization process of generating a first quantization result by quantizing the first operation result on the basis of the first effective input range; and an activation process of generating a first activation output by inputting the first quantization result to the first activation function.

Description

Calibration method and device utilizing input / output distribution of adjacent layers

[0001] The present disclosure relates to a calibration method and device utilizing the input / output distribution of adjacent layers.

[0002] The content described below merely provides background information related to the present embodiment and does not constitute prior art.

[0003] Deep learning models consist of a combination of blocks, including layers (convolutional layers, linear layers) that perform linear operations on learnable weights, and activation functions. These blocks can be combined in dozens or even hundreds of combinations.

[0004] Because the parameters and operations of deep learning models require precise representation, floating-point data types such as FP32 and FP16 are generally used.

[0005] When using FP32 or FP16 as the data type, the size of FP32 is 32 bits and the size of FP16 is 16 bits, so a lot of memory is required for the operation.

[0006] Typically, in deep learning models, to accelerate computation and reduce memory usage, computational results are quantized and expressed in data types with fewer bits (e.g., 8-bit INT8 and FP8 data types).

[0007] Quantization is primarily applied to weights, computational results, and activations. When quantizing computational results and quantizing activations (the output of an activation function), thresholds are required to approximate the actual input distribution of each quantization. This process is called calibration.

[0008] The calibration process refers to the step of analyzing the output distribution of a specific layer to determine an appropriate threshold. This threshold is calculated and used when quantizing the operation result and quantizing the activation.

[0009] Conventional calibration uses the most efficient calibration method among the representative calibration methods, namely maximum, percentile, and entropy (max / percentile / entropy) calibration.

[0010] Figure 1a is a diagram illustrating the absolute value frequency distribution of the operation output value of a specific layer and the percentile position that serves as a calibration reference point, and Figure 1b is a diagram illustrating the quantization target area and the quantization exclusion area as a result of the calibration.

[0011] As shown in Fig. 1a, in order to perform symmetric quantization on the operation output value, when the value at the 99.99% percentile point of the absolute value frequency distribution of the operation output value is set as the calibration reference point, as shown in Fig. 1b, symmetric quantization is performed on the operation output between the + calibration reference point and the - calibration reference point corresponding to the value within the 99.99% percentile point to generate the first quantized output, while values ​​whose absolute values ​​of the operation output are larger than the calibration reference point (i.e., the area indicated by Clip in Fig. 1b) are excluded from the quantization target, thereby achieving relatively efficient quantization using a quantization code of a constant number of bits.

[0012] Figure 2 is a diagram showing the input region of the activation function along with the frequency distribution of the entire operation output value.

[0013] As illustrated in Fig. 2, when the valid input region of the activation function is defined as 0≤x≤6, when the first quantized output is input to the activation function, the first quantized output outside the valid input region of the activation function becomes an unnecessary input to the activation function, and an inefficiency occurs in which a quantization code is assigned to the first quantized output that is not unnecessary to the activation function.

[0014] Therefore, when performing calibration using a conventional method, considering the quantization and activation process after calibration, information loss occurs due to the allocation of unnecessary quantization codes when quantizing the calibration result. Therefore, in order to minimize this loss, it is necessary to perform calibration more precisely.

[0015] The main purpose of the present disclosure is to provide a calibration method and device utilizing the input / output distribution of adjacent layers.

[0016] The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0017] According to one aspect of the present disclosure, a calibration method is provided, including: a computational process for generating a first computational result in a computational layer of an artificial neural network; a calibration process for obtaining a first valid input range by using information of another layer adjacent to the computational layer; a first quantization process for generating a first quantized result by quantizing the first computational result based on the first valid input range; and an activation process for inputting the first quantized result to the first activation function to generate a first activation output.

[0018] A computer program stored on a computer-readable recording medium is provided to execute each process included in the above calibration method.

[0019] According to another aspect of the present disclosure, a calibration device is provided, including: a calculation unit that generates a first calculation result in a calculation layer of an artificial neural network; a calibration unit that obtains a first valid input range by using information of another layer adjacent to the calculation layer; a first quantization unit that quantizes the first calculation result based on the first valid input range to generate a first quantization result; and an activation unit that inputs the first quantization result to the first activation function to generate a first activation output.

[0020] According to an embodiment of the present disclosure, there is an effect of making the quantization process prior to activation in an artificial neural network more sophisticated.

[0021] When using an activation function other than the type of standard activation function, it has the effect of effectively finding the valid input range.

[0022] The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0023] Figure 1a is a diagram illustrating the absolute value frequency distribution of the operation output value of a specific layer and the percentile position that serves as a calibration reference point, and Figure 1b is a diagram illustrating the quantization target area and the quantization exclusion area as a result of the calibration.

[0024] Figure 2 is a diagram showing the input region of the activation function along with the frequency distribution of the entire operation output value.

[0025] FIG. 3 is a block diagram illustrating the configuration of a calibration device (300) according to one embodiment of the present disclosure.

[0026] Fig. 4a is a diagram illustrating the input and output of the activation function ReLU6(), Fig. 4b is a diagram illustrating the input and output of the activation function sigmoid(), and Fig. 4c is a diagram illustrating the input and output of the activation function Swish().

[0027] Figure 5 is a diagram illustrating a graph of the activation function Swish() and its derivative, the Swish'() function.

[0028] FIG. 6 is a flowchart illustrating a calibration method according to one embodiment of the present disclosure.

[0029] Hereinafter, some embodiments of the present disclosure will be described in detail using exemplary drawings. When designating components in each drawing, it should be noted that, where possible, identical components are given the same reference numerals, even if they appear in different drawings. Furthermore, when describing the present disclosure, detailed descriptions of related known structures or functions will be omitted if they are deemed to obscure the gist of the present disclosure.

[0030] In describing components of embodiments according to the present disclosure, symbols such as first, second, i), ii), a), b) may be used. These symbols are only for distinguishing the components from other components, and the nature, order, or sequence of the components are not limited by the symbols. When a part in the specification is said to "include" or "have" a component, this does not mean that other components are excluded, but rather that other components may be included, unless explicitly stated otherwise.

[0031] The detailed description set forth below, together with the accompanying drawings, is intended to explain exemplary embodiments of the present disclosure and is not intended to represent the only embodiments in which the present disclosure may be practiced.

[0032] FIG. 3 is a block diagram illustrating the configuration of a calibration device (300) according to one embodiment of the present disclosure.

[0033] As illustrated in FIG. 3, the calibration device (300) according to the present embodiment may be implemented by including a calculating unit (310), a calibrating unit (320), a first quantizing unit (330), an activating unit (340), a second quantizing unit (350), and a table storing unit (360). The calibration device (300) according to the present embodiment may be implemented by omitting some of the components of FIG. 3 or by adding other components not illustrated in FIG. 3.

[0034] The calculating unit (310) generates a first calculation result in a calculation layer during deep learning. For example, the calculating unit (310) performs a calculation of one layer in an artificial neural network, such as a convolutional neural network (CNN) or recurrent neural network (RNN), related to deep learning.

[0035] For example, in the case of performing matrix multiplication using a kernel and a part of the data set of the input image in a convolutional neural network, as illustrated in FIG. 3, the operation unit (310) performs matrix multiplication ∑w using the weights (w1, w2, w3, w4) of the corresponding kernels for each of the four data sets (x1, x2, x3, x4) of the input image. i x i Perform the first operation and output the result.

[0036] The operation unit (310) applies the kernel weight to each of the multiple different data sets, performs matrix multiplication for each, and outputs each first operation result.

[0037] The operation of the calibration unit (320) is performed each time a matrix multiplication is performed by the operation unit (310) and the first operation result is output.

[0038] If the output distribution of the activation function can be predicted in a block consisting of a layer (Convolution Layer, Linear layer) that performs linear operations using weights and an activation function, this output distribution can be utilized in the calibration stage of the adjacent previous layer.

[0039] If the characteristic information of the activation function as well as the output distribution of the activation function for the actual input is utilized in the previous layer, the quantization process of the previous layer can be performed more precisely.

[0040] The calibration unit (320) obtains a first valid input range related to the relationship between the input value of the first activation function and the output value of the first activation function related to the first operation result of the operation unit (110).

[0041] Fig. 4a is a diagram illustrating the input and output of the activation function ReLU6(), Fig. 4b is a diagram illustrating the input and output of the activation function sigmoid(), and Fig. 4c is a diagram illustrating the input and output of the activation function Swish().

[0042] It is assumed that the calibration unit (320) knows what the first activation function related to the first operation result is.

[0043] As shown in Fig. 4a, if the first activation function related to the first operation result is ReLU6(x), then y=ReLU6(x) is the following formula:

[0044] If x<0, then y=0

[0045] If 0≤x≤6, then y=x

[0046] If x>60, then y=6

[0047] is defined as

[0048] Accordingly, the calibration unit (320) obtains the valid input range [0:6] of ReLU6(x), and [0:6] is defined as a value greater than or equal to 0 and less than or equal to 6.

[0049] Also, as shown in Fig. 4b, if the first activation function related to the first operation result is sigmoid(z), y=sigmoid(z)=1 / (1+e -z ), if z<-6 or z>6, y≒0. Therefore, the calibration unit (320) obtains the valid input range [-6:6] of the first activation function sigmoid(z), and the valid input range [-6:6] is defined as a value greater than or equal to -6 and less than or equal to 6.

[0050] In addition, as illustrated in Fig. 4c, when the first activation function related to the first operation result of the activation function is Swish(x), in y=Swish(x)=x*sigmoid(x), if x<-6, y≒0. Accordingly, the calibration unit (320) obtains the valid input range [-6:INF] of the first activation function Swish(x), and the valid input range [-6:INF] is defined as a value greater than or equal to -6.

[0051] The table storage unit (360) stores the valid input range of each activation function for at least one activation function in the valid range table (370).

[0052] In the present embodiment, when one of ReLU6(), sigmoid(), and Swish() is used as an activation function, the table storage unit (360) stores the valid input ranges for each of the multiple candidate activation functions ReLU6(), sigmoid(), and Swish() in the valid range table (370).

[0053] Since the calibration unit (320) knows which function is the first activation function related to the first operation result, the calibration unit (320) searches for the first activation function among the plurality of candidate activation functions within the valid range table (370).

[0054] If the first activation function related to the first operation result exists in the valid range table (370), the calibration unit (320) obtains the valid input range related to the found first activation function from the valid range table (370) and sets the obtained valid input range as the first valid input range.

[0055] The first quantization unit (330) quantizes the first operation result based on the first valid input range to generate the first quantization result. The first quantization unit (330) does not perform quantization on the first operation result that is outside the first valid input range.

[0056] Since the first operation result is quantized within the first valid input range, a more precise quantization is possible compared to cases where the result is calibrated and quantized using methods such as maximum, percentile, and entropy, so there is a relatively high possibility that data loss due to quantization of the first operation result will be minimized.

[0057] There may also be cases where the first activation function related to the first operation result does not exist within the valid range table (370).

[0058] When a user is given a first activation function that does not exist within the valid range table (370), the calibration unit (320) obtains the first derivative of the output value with respect to the input value on the given first activation function, and obtains the first valid input range from the obtained first derivative.

[0059] The first valid input range of the first activation function can utilize the first derivative of the first activation function.

[0060] When the first derivative of the first activation function has a value of 0 continuously for a certain range of input values ​​or the difference from 0 is less than or equal to a preset size continuously for a certain range of input values, the first effective input range of the first activation function is obtained based on the certain range.

[0061] Figure 5 is a diagram illustrating a graph of the activation function Swish() and its derivative, the Swish'() function.

[0062] If information on the valid input range of the activation function Swish() does not exist in the valid range table (370) and Swish() is given as the first activation function, the calibration unit (320) obtains the first valid input range of the first activation function based on the first derivative value of Swish() as the given first activation function.

[0063] As shown in Fig. 5, in the first activation function Swish(), in the input interval -6 or less, cases in which the difference between the first derivative value of Swish() and 0 is less than or equal to a preset size appear continuously for a certain period of time, and in the input interval -6 or more, cases in which the difference between the first derivative value of Swish() and 0 is less than or equal to a preset size do not appear continuously for a certain period of time.

[0064] Accordingly, in the case of Fig. 5, the calibration unit (320) sets the section [-6:INF], excluding the section in which the difference between the first differential value of Swish() and 0 is less than or equal to a preset size, as the first valid input range of the first activation function Swish().

[0065] If information on the valid input range of the activation function ReLU6() does not exist in the valid range table (370) and ReLU6() is given as the first activation function, the calibration unit (320) obtains the first valid input range of the first activation function based on the first derivative value of ReLU6() as the given first activation function.

[0066] In the case of ReLU6(x), when the input value x is less than or equal to 0 and when the input value x is greater than or equal to 6, the output value is constant, so we can see that the first derivative of ReLU6(x) continuously appears as 0.

[0067] Accordingly, the calibration unit (320) sets the section [0:6] excluding the section x<0, x>6 where the first derivative of ReLU6() continuously appears as 0 as the first valid input range of the first activation function ReLU6().

[0068] When a first activation function that does not exist in the valid range table (370) is used, the calibration unit (320) obtains an expected lower limit value and an expected upper limit value related to the first operation result.

[0069] The calibration unit (320) obtains a certain section in which the first derivative value continuously has a value of 0 for input values ​​between the expected lower limit value and the expected upper limit value or in which the first derivative value of the first activation function continuously has a difference from 0 less than or equal to a preset size, and determines the remaining section, excluding the certain section, among the sections between the expected lower limit value and the expected upper limit value, as the first valid input range of the first activation function.

[0070] The calibration unit (320) obtains a first predetermined interval in which the first derivative value continuously has a value of 0 for the input value of the first activation function in the interval from 0 to the expected lower limit value or the difference between the first derivative value and 0 is less than or equal to a preset size, and obtains a second predetermined interval in which the first derivative value of the first activation function continuously has a value of 0 for the input value of the first activation function in the interval from 0 to the expected upper limit value or the difference between the first derivative value and 0 is less than or equal to a preset size, and determines a range greater than or equal to the first predetermined interval and less than or equal to the second predetermined interval as a first valid input range.

[0071] In this way, when performing the process of obtaining the first and second fixed intervals, there is no need to check the first derivative for the first fixed interval or other input values ​​that are smaller than the first fixed interval, and there is no need to check the first derivative for the second fixed interval or other input values ​​that are larger than the second fixed interval, so that the first valid input range can be efficiently determined.

[0072] As described above, the first quantization unit (330) quantizes the first operation result based on the first valid input range to generate the first quantization result. For the first operation result that falls outside the first valid input range, the first quantization unit (330) excludes it from the quantization target, thereby enabling the quantization result for the first operation result to be expressed more accurately.

[0073] The activation unit (150) inputs the first quantization result into the first activation function to generate a first activation output.

[0074] It can be seen that the quantization loss due to the first quantization result is minimized because the effective input range of the first activation function used by the activation unit (150) is identical to the output range of the first quantization result.

[0075] The second quantization unit (350) quantizes the first activation output to generate a second quantization result. During the second quantization, calibration is performed on the first activation output so that an effective quantization input range for the first activation output can be obtained.

[0076] When calibrating the first activation output, the most efficient method among the maximum, percentile and entropy methods can be selected.

[0077] FIG. 6 is a flowchart illustrating a calibration method according to one embodiment of the present disclosure.

[0078] A calibration method according to one embodiment of the present disclosure is performed by a calibration device (300).

[0079] The operation unit (310) performs an operation process to generate a first operation result in the operation layer of the artificial neural network (S610).

[0080] The calibration unit (320) performs a calibration process to obtain a first valid input range by using information from another layer adjacent to the operation layer (S620).

[0081] The first quantization unit (330) performs a first quantization process to generate a first quantization result by quantizing the first operation result based on the first valid input range (S630).

[0082] The activation unit (340) performs an activation process that inputs the first quantization result into the first activation function to generate a first activation output (S640).

[0083] Each component of the device or method according to the present invention may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.

[0084] Various implementations of the systems and techniques described herein may be implemented as digital electronic circuits, integrated circuits, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations of one or more computer programs executable on a programmable system. The programmable system includes at least one programmable processor (which may be a special purpose processor or a general purpose processor) coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device. Computer programs (also known as programs, software, software applications, or code) include instructions for the programmable processor and are stored on a "computer-readable recording medium."

[0085] A computer-readable recording medium includes any type of recording device that stores data that can be read by a computer system. Such a computer-readable recording medium may be a non-volatile or non-transitory medium such as a ROM, CD-ROM, magnetic tape, floppy disk, memory card, hard disk, magneto-optical disk, storage device, and may further include a transitory medium such as a data transmission medium. Furthermore, the computer-readable recording medium may be distributed across network-connected computer systems, so that computer-readable code can be stored and executed in a distributed manner.

[0086] Although the flowchart / timing diagram of this specification describes each process as being executed sequentially, this is merely an illustrative description of the technical idea of ​​one embodiment of the present disclosure. In other words, a person of ordinary skill in the art to which one embodiment of the present disclosure belongs may modify and apply various modifications and variations by changing the order described in the flowchart / timing diagram without departing from the essential characteristics of one embodiment of the present disclosure, or by executing one or more of the processes in parallel. Therefore, the flowchart / timing diagram is not limited to a chronological order.

[0087] The above description is merely an example of the technical idea of ​​the present embodiment, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of ​​the present embodiment, but rather to explain it, and the scope of the technical idea of ​​the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present embodiment.

[0088] [Explanation of symbols]

[0089] 300: Calibration device 310: Operation unit

[0090] 320: Calibration section 330: First quantization section

[0091] 340: Activation section 350: Second quantization section

[0092] 360: Table storage 370: Valid range table

[0093]

[0094] Statement regarding sponsored research or development

[0095] The present invention is a result of a research project (Project Unique Number: 1711117060, Subproject Number: 2020-0-01305-001, Ministry: Ministry of Science and ICT, Project Management (Specialized) Agency: National IT Industry Promotion Agency, Research Project Name: Next-Generation Intelligent Semiconductor Technology Development (Design) R&D, Research Project Name: Development of 2,000 TFLOPS-Class Server Artificial Intelligence Deep Learning Processor and Module, Contribution Ratio: 1 / 1, Project Performing Agency: SK Telecom Co., Ltd., Sapion Korea Co., Ltd., Research Period: 2020.04.01. ~ 2027.12.31.).

[0096]

[0097] CROSS-REFERENCE TO RELATED APPLICATION

[0098] This patent application claims priority to Korean Patent Application No. 10-2023-0162768, filed in Korea on November 21, 2023, and Korean Patent Application No. 10-2023-0181018, filed in Korea on December 13, 2023, the entire contents of which are incorporated herein by reference.

Claims

1. A computational process that generates the first computational result in the computational layer of an artificial neural network; A calibration process for obtaining a first valid input range by using information from another layer adjacent to the above operation layer; A first quantization process that quantizes the first operation result based on the first valid input range to generate a first quantization result; and An activation process that inputs the first quantization result to the first activation function to generate a first activation output. A calibration method comprising:

2. In paragraph 1, The above calibration process is, A calibration method characterized by obtaining the first valid input range related to the relationship between the input value of the first activation function and the output value of the first activation function for the first activation function of the activation layer related to the first operation result.

3. In paragraph 2, The above activation layer is, A calibration method characterized in that the activation layer appears immediately after quantization of the first operation result.

4. In paragraph 2, Further comprising a table storage process for storing the valid input range of each activation function in a valid range table for at least one activation function, The above calibration process is, A calibration method characterized by obtaining the first activation function among the at least one activation function within the above valid range table, and obtaining the valid input range related to the obtained first activation function as the first valid input range.

5. In paragraph 2, The above calibration process is, A calibration method characterized by obtaining a first derivative of the output value with respect to the input value and obtaining the first valid input range from the first derivative.

6. In paragraph 5, The above calibration process is, A calibration method characterized in that the first valid input range is obtained based on the predetermined interval when the first differential value has a value of 0 continuously for a certain interval of the input value or the difference from 0 continuously for a certain interval of the input value is less than or equal to a preset size.

7. In paragraph 6, The above calibration process is, Obtain the expected lower bound value and the expected upper bound value related to the above first operation result, A calibration method characterized in that the first differential value continuously has a value of 0 for the input value between the expected lower limit value and the expected upper limit value, or the first differential value continuously has a difference from 0 less than or equal to a preset size, is obtained in a predetermined interval, and an interval remaining between the expected lower limit value and the expected upper limit value, excluding the predetermined interval, is determined as the first valid input range.

8. In paragraph 6, The above calibration process is, Obtain the expected lower bound value and the expected upper bound value related to the above first operation result, A calibration method characterized in that a first predetermined interval is obtained in which the first derivative continuously has a value of 0 for the input values ​​in the interval from 0 to the expected lower limit value or the difference between the first derivative and 0 is less than or equal to a preset size, a second predetermined interval is obtained in which the first derivative continuously has a value of 0 for the input values ​​in the interval from 0 to the expected upper limit value or the difference between the first derivative and 0 is less than or equal to a preset size, and a range of values ​​greater than or equal to the first predetermined interval and less than the second predetermined interval is determined as the first valid input range.

9. A computer program stored on a computer-readable recording medium for executing each process included in the calibration method according to any one of clauses 1 to 8.

10. A computational unit that generates the first computational result in the computational layer of the artificial neural network; A calibration unit that obtains a first valid input range by using information of another layer adjacent to the above operation layer; A first quantization unit that quantizes the first operation result based on the first valid input range to generate a first quantization result; and An activation unit that inputs the first quantization result to the first activation function to generate a first activation output. A calibration device comprising:

11. In paragraph 10, The above calibration unit, A calibration device characterized in that it obtains the first valid input range related to the relationship between the input value of the first activation function and the output value of the first activation function for the first activation function of the activation layer related to the first operation result.

12. In paragraph 11, Further comprising a table storage unit storing the valid input range of each activation function in a valid range table for at least one activation function, The above calibration unit, A calibration device characterized in that it obtains the first activation function among the at least one activation function within the above valid range table, and obtains the valid input range related to the obtained first activation function as the first valid input range.

13. In paragraph 11, The above calibration unit, A calibration device characterized by obtaining a first derivative of the output value with respect to the input value and obtaining the first valid input range from the first derivative.

Images