Method for improving quantization loss due to statistical characteristics between channels of neural network layer and apparatus therefor

Abstract

By using statistical characteristics of data for each output channel of a neural network convolution layer, quantization scales for the different output channels of the convolution layer are independently calculated. When the convolution layer generates each output channel, quantization is performed using a quantization scale set for each output channel.

Description

Method for improving quantization loss due to statistical characteristics between channels of neural network layers and device therefor

[0001] The present invention relates to a technology for independently setting a quantization scale used in the process of generating output activation of a neural network layer for each output channel in order to control quantization errors occurring during neural network operations.

[0002] Neural networks developed to run on general computing devices or dedicated computing hardware can be trained and operated in data representation formats such as FP32 (Floating Point 32-bit), FP64, and FP128. Based on the training results, the specific values ​​of the parameters used in each layer of the neural network, such as the weights used in the convolutional layer, can be determined.

[0003] When a completed neural network must be executed by computing hardware that represents data with a limited number of bits, the parameters constituting the neural network are quantized into a data representation format using a smaller number of bits, such as FP16 or INT8, and provided to the computing hardware.

[0004] In the present invention, the process of converting a specific value having a high-precision data representation format (e.g., FP32) into a value having a low-precision data representation format (e.g., INT8) may be referred to as quantization, and the conversion constant S used in the conversion process may be referred to as a quantization scale. Setting the quantization scale to an appropriate value increases the likelihood that the quantization error due to the quantization will be reduced.

[0005] The resulting quantization error can be amplified as it passes through the layers of a neural network. Therefore, when converting input activations into output activations in each layer, including the convolutional layer, it is necessary to control the quantization error that occurs during the conversion process.

[0006] The present invention aims to provide a technology for independently quantizing each output channel by independently setting a quantization scale for each output channel in order to minimize the influence of quantization errors that may occur in each output channel of a convolutional layer.

[0007] A convolutional layer of a neural network can perform, for example, normal convolution operations or depth-wise convolution operations.

[0008] The normal convolution operation uses all input channels of the input activations fed to the convolution layer to produce a single value among the values ​​that constitute the output activations. Normal convolution offers better performance than depth-wise convolution, but requires more computational effort.

[0009] In contrast, depth-wise convolution uses only one input channel of the input activations fed to the convolutional layer to produce a single value among the values ​​that constitute the output activations. While depth-wise convolution may perform lower than normal convolution, its lower computational complexity makes it a viable option for streamlining neural networks.

[0010] If a real value input to or output from a specific convolutional layer of a neural network is denoted as r, a value representing a quantization scale is denoted as S, a value representing a zero point is denoted as Z, and a value of the real value quantized according to the quantization scale is denoted as q, then it can be defined as in Equation 1.

[0011] [Formula 1]

[0012] q=round(r*S)+Z

[0013] If the above Z is 0, it corresponds to symmetric quantization, and if it is not 0, it corresponds to asymmetric quantization. Here, the real value r is a value given from outside the layer or a value given by the operation result of the layer, so the quantized value q is determined depending on which value is selected for the quantization scale S. Therefore, it is necessary to select the quantization scale S effectively.

[0014] At this time, depending on the limitations of computing hardware, the number of bits allowed as weights and activations of the convolutional layer is q. bit can be given. The above q bit can be referred to as quantization bits. And the number of bits q bit The maximum value that can be expressed by r max It can be expressed as follows. At this time, the quantization scale S used to convert the given real number r into a quantized value q for use in the computing hardware can be defined as in Equation 2.

[0015] [Formula 2]

[0016] S=2*(q bit -1)-1 / r max

[0017] As in Equation 2, the quantization scale S is the quantization bit q bit and maximum value r max is determined by . Here, the quantization bit q bit Since is a constraint given by the computing hardware, the quantization bit q bit is an uncontrollable value without changing the computing hardware. Therefore, to optimize the quantization scale S, the maximum real number r max should be determined as the optimal value. According to this, among the values ​​related to a specific layer of the neural network, the maximum real value r max If there exists a larger first real value, the first real value cannot exist as it is, and the maximum real value r max It must be provided converted to .

[0018] In a convolutional layer, input activations and weights are convolved, and then the resulting convolution is quantized at a predetermined quantization scale to generate output activations. In one embodiment, only one quantization scale may be used to generate multiple output channels included in the output activations.

[0019] However, the inventors of the present invention have confirmed that the distribution of data values ​​contained in each of the multiple output channels constituting the output activation may differ from one another. The characteristics of this data value distribution can be observed differently for each layer.

[0020] For example, among the plurality of output channels, the maximum value of data included in the first output channel may have a very small first value, while the maximum value of data included in the second output channel may have a very large second value (second value >> first value).

[0021] At this time, it can be assumed that a second quantization scale is selected as a single quantization scale used to generate the output activation, such that the second value can be expressed in the second output channel without loss. At this time, a second original output channel can be generated as a result of convolving the second output channel weight tensor used to generate the second output channel and the input activation. Then, when the second original output channel is quantized with the second quantization scale to generate the second output channel, the second value can be expressed without loss. Similarly, a first original output channel can be generated as a result of convolving the first output channel weight tensor used to generate the first output channel and the input activation. Next, when the first original output channel is quantized to the second quantization scale to generate the first output channel, there is a problem that a quantization error that is too large to be acceptable occurs compared to the size of the values ​​included in the first output channel.

[0022] Conversely, it can be assumed that a first quantization scale is selected as a single quantization scale used to generate the output activation so that the maximum value that the first output channel can express is the first value (the first quantization scale < the second quantization scale). At this time, a first original output channel can be generated as a result of convolving the first output channel weight tensor used to generate the first output channel and the input activation. Then, when the first original output channel is quantized with the first quantization scale to generate the first output channel, the first value can be expressed without loss. Similarly, a second original output channel can be generated as a result of convolving the second output channel weight tensor used to generate the second output channel and the input activation. There is a problem in that when the second original output channel is quantized to the first quantization scale to generate the second output channel, all values ​​included in the second output channel that are greater than the first value are clipped to the first value, resulting in a quantization error.

[0023] The problems described above arise because the quantization scale used to generate the output activations is fixed to one value for the entire output activations.

[0024] To address this issue, one aspect of the present invention independently sets the quantization scale used to generate a plurality of output channels constituting output activation for each output channel. This minimizes the ratio of quantization error in each output channel to the size of the data value contained in each output channel.

[0025] When utilizing the configuration of the present invention described above, a phenomenon occurs in which the different output channels constituting the output activation may be quantized with different quantization scales. Comparing this with the prior art, there is a difference in that according to the prior art, all output channels constituting the output activation are quantized with the same single quantization scale.

[0026] In a situation where different output channels constituting the output activation as in the present invention are quantized with different quantization scales, a downstream second convolutional layer that receives the output activation can be considered. At this time, the input activation of the second convolutional layer is the output activation. Therefore, the input activation of the second convolutional layer has an uneven state in which different input channels are quantized with different quantization scales.

[0027] At this time, if the second convolution layer is to perform the normal convolution operation, a problem may arise in that the operation method must be readjusted in a very complex manner to handle the non-uniform state.

[0028] In contrast, if the second convolution layer is to perform a depth-wise convolution operation, each of the input channels undergoes an independent convolution operation process, so it is relatively easy to modify the operation method in consideration of the non-uniform state.

[0029] Therefore, in a case where different output channels constituting the output activation as in the present invention are quantized with different quantization scales, it is advantageous for the downstream second convolution layer that receives the output activation to be a layer that performs a depth-wise convolution operation.

[0030] That is, the neural network operation method and the neural network quantization error management method provided according to one aspect of the present invention can be applied to the upstream layer among two adjacent layers when the downstream layer performs a depth-wise convolution operation. In this case, preferably, the upstream layer may be a layer that performs a normal convolution operation.

[0031] According to one aspect of the present invention, a method for managing quantization errors in a neural network may be provided, including: a step in which a computing device inputs an input activation to a specific layer of a predetermined neural network to obtain an output activation including a plurality of output channels from the layer; a step in which the computing device calculates a data distribution, which is a distribution of data values ​​included in the output activation, for each output channel; a step in which the computing device divides the plurality of output channels into a plurality of groups based on the data distributions calculated for each output channel; a step in which the computing device independently determines a quantization scale for each of the groups and assigns the determined quantization scale to each of the groups; and a step in which the computing device generates a command file that causes dedicated computing hardware for processing neural network operations to execute a function of the layer. In this case, the computing device is configured to include information regarding a quantization scale assigned to each output channel of the output activation of the layer in the command file.

[0032] At this time, the information about the quantization scale may have a table including a configuration channel ID field that identifies output channels included in each of the plurality of groups, and a group quantization scale field that indicates the quantization scale assigned to each of the plurality of groups.

[0033] At this time, the maximum value among the variances of the data distributions of each of the output channels belonging to the first group among the plurality of groups may be smaller than the minimum value among the variances of the data distributions of each of the output channels belonging to the second group among the plurality of groups.

[0034] At this time, the first quantization scale assigned to the first group may be smaller than the second quantization scale assigned to the second group.

[0035] At this time, the obtaining step may include a step of independently inputting a plurality of different input activations into the specific layer to independently obtain a plurality of output activations. In addition, the calculating step may be configured to calculate the data distribution included in the plurality of output activations for each output channel.

[0036] At this time, the first quantization scale is assigned to all output channels belonging to the first group among the plurality of groups, and the second quantization scale is assigned to all output channels belonging to the second group among the plurality of groups, and the first quantization scale and the second quantization scale may have different values.

[0037] At this time, the layer may be a layer that performs a normal convolution operation.

[0038] At this time, the second layer that receives the output activation of the specific layer among the layers constituting the neural network may be a depth-wise convolution layer.

[0039] At this time, no other convolution operation layer may be included between the specific layer and the second layer.

[0040] At this time, the computing hardware may be configured to execute a convolution operation between the first input activation input to the layer and a predetermined first output channel weight tensor unit to produce an intermediate value, and the first output channel weight tensor unit may be a weight tensor used to obtain the first output channel among the output activations of the layer, and the computing hardware may be configured to produce the first output channel by quantizing the intermediate value with a quantization scale assigned to a group to which the first output channel belongs among the plurality of groups.

[0041] According to another aspect of the present invention, a method for managing quantization errors in a neural network may be provided, including: a step of inputting an input activation to a layer of a neural network to obtain an output activation including a plurality of output channels from the layer; a step of independently calculating, by the computing device, a first quantization scale for a first output channel and a second quantization scale for a second output channel among the plurality of output channels; and a step of generating, by the computing device, a command file that causes dedicated computing hardware for processing neural network operations to execute a function of the layer. In this case, the computing device is configured to include in the command file information regarding the first quantization scale assigned to the first output channel of the layer and information regarding the second quantization scale assigned to the second output channel. And the computing hardware is configured to execute a convolution operation between a first input activation input to the layer and a predetermined first output channel weight tensor unit to produce a first intermediate value, and to execute a convolution operation between the first input activation and a predetermined second output channel weight tensor unit to produce a second intermediate value. And the first output channel weight tensor unit is a weight tensor used to obtain the first output channel, and the second output channel weight tensor unit is a weight tensor used to obtain the second output channel. And the computing hardware is configured to quantize the first intermediate value with the first quantization scale to produce the first output channel, and to quantize the second intermediate value with the second quantization scale to produce the second output channel.

[0042] At this time, the first quantization scale may be determined based on the distribution of data of the first group including data included in the first output channel, and the second quantization scale may be determined based on the distribution of data of the second group including data included in the second output channel.

[0043] At this time, the calculating step may include: a step in which the computing device calculates a data distribution, which is a distribution of data values ​​included in the output activation, for each output channel; a step in which the computing device divides the plurality of output channels into a plurality of groups based on the data distributions calculated for each output channel; and a step in which the computing device independently determines a quantization scale for each group and assigns the quantization scale to each group. In addition, the first quantization scale may be a quantization scale assigned to a group to which the first output channel belongs among the plurality of groups, and the second quantization scale may be a quantization scale assigned to a group to which the second output channel belongs among the plurality of groups.

[0044] According to another aspect of the present invention, a neural network operation method may be provided in which dedicated computing hardware for processing neural network operations executes a neural network operation. The method comprises: a step in which the computing hardware obtains a first input activation input to a specific layer of a predetermined neural network; a calculation step in which the computing hardware executes a convolution operation of the first input activation and a predetermined first output channel weight tensor unit assigned to the layer to calculate a first intermediate value, and a convolution operation of the first input activation and a predetermined second output channel weight tensor unit assigned to the layer to calculate a second intermediate value, wherein the first output channel weight tensor unit and the second output channel weight tensor unit are weight tensors used to obtain a first output channel and a second output channel among output activations of the layer, respectively; And the computing hardware includes an output activation calculation step of quantizing the first intermediate value with a first quantization scale stored in a first location of a memory of the computing hardware to produce the first output channel, and quantizing the second intermediate value with a second quantization scale stored in a second location of the memory to produce the second output channel.

[0045] At this time, the output activation calculation step may include: a step of checking a first record corresponding to the specific layer and the first output channel and a second record corresponding to the specific layer and the second output channel among tables stored in a memory included in the computing hardware; and a step of setting a value of a group quantization scale field included in the checked first record to the first quantization scale, and a step of setting a value of a group quantization scale field included in the checked second record to the second quantization scale.

[0046] At this time, in the step of acquiring the first input activation, the computing hardware may be configured to acquire a layer ID that identifies the specific layer.

[0047] At this time, the first quantization scale and the second quantization scale may be included in a command file downloaded by the computing hardware from another device, and the command file may be generated by a computing device configured to execute a quantization error management method of a predetermined neural network.

[0048] At this time, the method for managing quantization errors of the neural network may include: a step in which the computing device inputs input activation to the layer of the neural network to obtain output activation including a plurality of output channels from the layer; a step in which the computing device independently calculates a first quantization scale for a first output channel and a second quantization scale for a second output channel among the plurality of output channels; and a step in which the computing device generates the command file that causes the computing hardware to execute the function of the layer.

[0049] At this time, the first quantization scale may be larger than the second quantization scale, and the variance of data included in the first output channel may be larger than the variance of data included in the second output channel.

[0050] At this time, the first quantization scale may be greater than the second quantization scale, and the maximum value of data included in the first output channel may be greater than the maximum value of data included in the second output channel.

[0051] According to the present invention, a technology for independently quantizing each output channel by independently setting a quantization scale for each output channel can be provided in order to minimize the influence of quantization errors that may occur in each output channel of a convolutional layer.

[0052] FIG. 1 illustrates the main structure of computing devices that execute a method for neural network operations according to one embodiment of the present invention.

[0053] FIG. 2 illustrates a concept in which a user computing device obtains a command file to be executed by an NPU according to one embodiment of the present invention.

[0054] FIG. 3 is a flowchart illustrating a method for managing quantization errors of a neural network provided according to one embodiment of the present invention.

[0055] FIG. 4 illustrates a first layer, which is a specific layer of a neural network provided according to one embodiment of the present invention, input activation input to the first layer, and output activation output from the first layer.

[0056] Figure 5 illustrates a four-dimensional weight tensor provided for the operation of the first layer of Figure 4.

[0057] Figure 6 illustrates a process of generating output data by performing a convolution operation on the input activation of Figure 4 with the kth output channel weight tensor among the weight tensors of Figure 5.

[0058] FIG. 7 and FIG. 8 illustrate a data distribution, which is a distribution of data values ​​included in output activation output by a first layer, which is a specific layer of a neural network provided according to one embodiment of the present invention.

[0059] FIG. 9 illustrates an example of a structure of a neural network to which a neural network operation method executed by dedicated computing hardware is applied, according to one embodiment of the present invention.

[0060] FIG. 10 illustrates an example of the structure of a table including information on group-specific quantization scales included in a command file provided to a computing hardware dedicated to neural network operations provided according to one embodiment of the present invention.

[0061] FIG. 11 is a flowchart illustrating a method for managing quantization errors of a neural network provided according to another embodiment of the present invention.

[0062] FIG. 12 is a flowchart illustrating a method for executing a neural network operation on a computing hardware dedicated to neural network operation provided according to one embodiment of the present invention.

[0063] Hereinafter, embodiments of the present invention will be described with reference to the attached drawings. However, the present invention is not limited to the embodiments described herein and may be implemented in various other forms. The terminology used herein is intended to aid understanding of the embodiments and is not intended to limit the scope of the present invention. Furthermore, the singular forms used below also include the plural forms, unless the meanings clearly indicate the opposite.

[0064] FIG. 1 illustrates the main structure of computing devices that execute a method for neural network operations according to one embodiment of the present invention.

[0065] The user computing device (1) presented in FIG. 1 may be, for example, a desktop computer, a laptop computer, a smartphone, and a tablet.

[0066] A computing device (1) may include a DRAM (Dynamic Random Access Memory) (130), an NPU (110), a bus (700) connecting the DRAM (130) and the NPU (110), and other hardware (99) connected to the bus (700), a main processor (160), and a storage unit (170).

[0067] The NPU (110) may also be referred to as a hardware accelerator. The NPU (110) may be configured to operate only on integer data.

[0068] In addition, the computing device (1) may further include a power supply unit, a communication unit, a user interface, and peripheral units that are not shown. The bus (700) may be shared by the NPU (110), other hardware (99), and the main processor (160).

[0069] The above storage unit (10) may be integrally connected to the computing device (1) or may be detachably connected.

[0070] The above NPU (110) may include a DMA (Direct Memory Access) part (20), a control unit (40), an internal memory (30), an input buffer (850), a data operation unit (810), and an output buffer (840).

[0071] Some or all of the data temporarily stored in the internal memory (30) may be provided from the DRAM (130) via the bus (700). At this time, in order to move the data stored in the DRAM (130) to the internal memory (30), the control unit (40) and the DMA unit (20) may control the internal memory (30) and the DRAM (130).

[0072] In this specification, DRAM (130) may also be referred to as external memory.

[0073] Data stored in the internal memory (30) can be provided to the data operation unit (810) through the input buffer (850).

[0074] The output values ​​generated by the operation of the above data operation unit (810) can be stored in the internal memory (30) via the output buffer (840). The output values ​​stored in the internal memory (30) can also be written to the DRAM (130) under the control of the control unit (40) and the DMA unit (20).

[0075] The control unit (40) can comprehensively control the operation of resources within the NPU (110), such as the DMA unit (20), internal memory (30), and the data operation unit (810).

[0076] In one implementation example, the data operation unit (810) may perform a first operation function during a first time period and a second operation function during a second time period. For example, the data operation unit (810) may perform a first operation function according to the operation rules of a first layer of a neural network during a first time period and a second operation function according to the operation rules of a second layer of a neural network during a second time period.

[0077] In Fig. 1, one data operation unit (810) is shown within the NPU (110). However, in a modified embodiment not shown, the data operation units (810) shown in Fig. 3 may be provided in multiple units within the NPU (110) to perform operations requested by the control unit (40) in parallel.

[0078] In one implementation example, the data operation unit (810) may output the output data sequentially in a given order over time, rather than outputting the output data all at once.

[0079] The computing device (2) for developers presented in Fig. 1 may be, for example, a server, a desktop computer, or a laptop computer. The computing device (2) may include a DRAM (230), a bus (2700), other hardware (299), a main processor (260), and a storage unit (270).

[0080] FIG. 2 illustrates a concept in which a user computing device obtains a command file to be executed by an NPU according to one embodiment of the present invention.

[0081] In this specification, the user computing device (1) may be referred to as a first computing device, and the developer computing device (2) may be referred to as a second computing device.

[0082] In one example, a user computing device (1) can obtain a command file to be executed by an NPU from a developer computing device (2) through a predetermined communication channel.

[0083] In another example, a user computing device (1) can obtain a command file to be executed by an NPU from a developer computing device (2) through a predetermined communication channel via a relay device (3). The relay device (3) may be a production device that is used in the production process of the user computing device (1).

[0084] FIG. 3 is a flowchart illustrating a method for managing quantization errors of a neural network provided according to one embodiment of the present invention.

[0085] FIG. 4 illustrates a first layer, which is a specific layer of a neural network provided according to one embodiment of the present invention, input activation input to the first layer, and output activation output from the first layer.

[0086]

[0087] *Figure 5 illustrates a four-dimensional weight tensor (60) provided for the convolution operation of the first layer of Figure 4.

[0088] Figure 6 illustrates a process of generating output data by performing a convolution operation on the input activation of Figure 4 with the kth output channel weight tensor among the weight tensors of Figure 5.

[0089] FIG. 7 and FIG. 8 illustrate a data distribution, which is a distribution of data values ​​included in output activation output by a first layer, which is a specific layer of a neural network provided according to one embodiment of the present invention.

[0090] The following description is provided with reference to Figures 3 to 8.

[0091] In step (S110), the computing device (2) can input an input activation (50) to a first layer (601), which is a specific layer of a predetermined neural network, and obtain an output activation (70) including a plurality of output channels (70[1], 70[2], 70[3], 70[4]) from the first layer (601).

[0092] In one embodiment, the neural network may already have its structure and internal parameters completed and determined.

[0093] At this time, in a preferred embodiment, the first layer (601) may be a layer that performs a normal convolution operation.

[0094] At this time, the input activation (50) may be a three-dimensional tensor and may have a width (w) dimension, a height (h) dimension, and an input channel (ci) dimension. In Fig. 4, the input activation (50) is exemplified as having a total of three input channels. The first input channel, the second input channel, and the third input channel are indicated by reference numbers 50[1], 50[2], and 50[3], respectively.

[0095] At this time, the output activation (70) may be a three-dimensional tensor and may have a width (w) dimension, a height (h) dimension, and an output channel (co) dimension. In Fig. 4, the output activation (70) is exemplified as having four output channels. The first output channel, the second output channel, the third output channel, and the fourth output channel are indicated by reference numbers 70[1], 70[2], 70[3], and 70[4], respectively.

[0096] Referring to FIG. 5, the four-dimensional weight tensor (60) used for the convolution operation of the first layer (601) may be composed of a total of K output channel weight tensor parts. In this case, K is the total number of output channels constituting the output activation (70).

[0097] In the examples of FIGS. 4 and 5, since K=4, the 4-dimensional weight tensor (60) is composed of k-th output channel weight tensor parts (60[k]) (k=1, 2, 3, and 4).

[0098] Each of the above kth output channel weight tensor parts (60[k]) is a three-dimensional tensor.

[0099] The above kth output channel weight tensor part (60[k]) is used to generate the kth output channel (70[k]) of the output activation (70).

[0100] Referring to FIG. 6, the kth output channel weight tensor (60[k]) and the input activation (50) are convolved to produce the kth original output channel (70 O [k]) is produced. Since the above convolution includes a multiplication operation, the first bit number representing the input activation (50) and the kth output channel weight tensor part (60[k]) is greater than the kth original output channel (70). O The second bit number representing [k] is larger. The kth original output channel (70 O [k]) is assigned to the kth output channel (70[k]) and the kth quantization scale (S coBy quantizing to [k], the kth output channel (70[k]) of the output activation (70) can be produced. The number of bits representing the kth output channel (70[k]) can be the same as the first number of bits.

[0101] According to the prior art, the kth quantization scale (S co [k]) has the same value for all output channels, and according to the present invention, the kth quantization scale (S co [k]) can be independently selected for at least two different output channels.

[0102] Since step (S110) is executed on a general computing device (2) capable of high-precision calculations such as FP32, FP64, or FP128, the input activations input to the first layer, the output activations output from the first layer, and the parameters assigned to the first layer can be in a state where there is practically no quantization error. The assumption that the quantization error of data can be practically ignored due to high-precision calculations is valid for all of the data values ​​presented in step (S120) and step (S130) described below, as well as in FIG. 7.

[0103] In the example presented in Fig. 6, for example, if the input activation (50) and the kth output channel weight tensor (60[k]) are expressed as INT8, the kth original output channel (70 O [k]) can be expressed as INT24. And again, the above kth output channel (70[k]) can be expressed as INT8.

[0104] The above convolution is a normal convolution. The normal convolution has the characteristic that multiple input channels are operated together to produce a single output channel. In one embodiment, the multiple input channels are quantized using the same quantization scale. The first layer (601) of FIG. 4 may be a layer that performs the normal convolution operation.

[0105] Referring again to FIG. 3, in step (S120), the computing device (2) can calculate a data distribution, which is a distribution of data values ​​included in the output activation (70), for each output channel. The data distribution is illustrated in FIG. 7.

[0106] The process of calculating the above data distribution for each output channel can be accomplished, for example, by counting the number of data having a specific value among the data included in the kth output channel (70[k]).

[0107] Referring to FIG. 7, the distribution of data values ​​included in the first output channel (70[1]), the second output channel (70[2]), the third output channel (70[3]), and the fourth output channel (70[4]) of the output activation (70) are presented sequentially from top to bottom of the drawing. Reference numeral 71[k] in FIG. 7 represents the k-th output channel distribution (71[k]), which is the distribution of data values ​​included in the k-th output channel (70[k]) (k=1, 2, 3, 4).

[0108] In the four graphs presented in Figure 7, the horizontal axis represents the value of data included in each output channel, and the vertical axis represents the number of data having each data value in each output channel.

[0109] In this specification, r is determined in relation to the kth output channel (70[k]). max is r max It can be written as [k].

[0110] Above r max [k] can be selected according to a predetermined rule based on the values ​​of data included in the kth output channel (70[k]).

[0111] In Fig. 7, the value of the data having the largest absolute value among the values ​​of the data included in the kth output channel (70[k]) is a temporary r for the kth output channel (70[k]). max Inr max,t An example determined by [k] is presented.

[0112] In another embodiment, the above r max,,t [k] may be determined based on the mountain (σ[k]) of values ​​included in the kth output channel (70[k]). For example, the above r max,t [k] can also be determined as a value proportional to the above variance (σ[k]). However, r max,t The rule for determining [k] is not limited to the method presented in Fig. 7 and the above-described embodiment, and if it is a value that can well describe the boundary of the data value included in the kth output channel (70[k]), the above r max,t [k] can be selected.

[0113] Referring again to FIG. 3, in step (S130), the computing device (2) can divide the plurality of output channels into a plurality of groups based on the data distributions (71[k]) produced for each output channel (70[k]).

[0114] The number of the above-described groups may be a value predetermined by a predetermined rule or a value selected by the user.

[0115] Figure 7 shows an example in which the first output channel (70[1]) and the second output channel (70[2]) are classified into the first group (G[1]), and the third output channel (70[3]) and the fourth output channel (70[4]) are classified into the second group (G[2]).

[0116] Fig. 8 is the same as Fig. 7. However, Fig. 8 shows an example in which the first output channel (70[1]) is classified into the first group (G[1]), the second output channel (70[2]) is classified into the second group (G[2]), the third output channel (70[3]) is classified into the first group (G[3]), and the fourth output channel (70[4]) is classified into the fourth group (G[4]).

[0117] In the present invention, one group may be composed of one or more output channels.

[0118] The statistical values ​​of the above output channels (70[k]) can be used as a criterion for dividing the above multiple output channels into multiple groups.

[0119] In the first embodiment of defining a group, the maximum value (mv[k]) among the absolute values ​​of the values ​​included in each output channel (70[k]) can be selected (k=1, 2, 3, 4). Then, the maximum value (mv[k]) is the temporary r of the corresponding output channel (70[k]). max Inr max,t [k] can be set. Then, the above set r max,t [k] can be sorted in order of size, and the plurality of output channels can be divided into multiple groups based on the sorted sizes.

[0120] In the second embodiment of defining a group, the variance (σ[k]) of the values ​​included in each output channel (70[k]) can be determined (k=1, 2, 3, 4). Then, the value obtained by multiplying the variance (σ[k]) by a predetermined constant value p1 ((p1·σ[k])) is the temporary r of the corresponding output channel (70[k]). max Inr max,t [k] can be set. Then, the above set r max,t [k] can be sorted in order of size, and the plurality of output channels can be divided into multiple groups based on the sorted sizes.

[0121] In the third embodiment of defining a group, the variance (σ[k]) of the values ​​included in each output channel (70[k]) can be determined (k=1, 2, 3, 4). Then, a boundary value (p2·σ[k]), which is a value obtained by multiplying each variance (σ[k]) by a predetermined constant value p2, can be determined. Then, among the values ​​included in each output channel (70[k]), the largest value that is less than or equal to the boundary value (p2·σ[k]) is determined as the temporary r of the corresponding output channel (70[k]). max Inr max,t [k] can be set. Then, the above set r max,t[k] can be sorted in order of size, and the plurality of output channels can be divided into multiple groups based on the sorted sizes. In this third embodiment, among the data of each output channel (70[k]), data that deviates significantly from the trend of the data distribution (71[k]) of the output channel (70[k]) is mentioned above. max,t It has the effect of excluding [k] from the candidates.

[0122] The first to third embodiments defining the above group are examples to help understand the present invention, and it is easily understood that other embodiments can be applied to the present invention.

[0123] At this time, the maximum value among the variances of the data distributions of each of the output channels belonging to the first group among the plurality of groups may be smaller than the minimum value among the variances of the data distributions of each of the output channels belonging to the second group among the plurality of groups. In this case, in a preferred embodiment, the first quantization scale assigned to the first group may be smaller than the second quantization scale assigned to the second group.

[0124] Referring again to FIG. 3, in step (S140), the computing device (2) can independently determine a quantization scale for each group and assign it to each group.

[0125] At this time, in order to determine the quantization scale for each group, r is set for the output channels belonging to the group. max,t [k]s (temporary r max The maximum value among the max,G[ ] can be set to . And the quantization scale for that group is S G[ ] can be obtained as in [Formula 3].

[0126] [Formula 3]

[0127] S G [ ] =2*(q bit -1)-1 / r max,G[ ]

[0128] In the example presented in Fig. 7, r is set to the output channels (70[1], 70[2]) belonging to the first group (G[1]). max,t [k]s (temporary r max The maximum value among the ) is r max,t [2] Therefore, r max,G[1] =r max,t [2] is set. And the quantization scale S for the first group (G[1]) G [1] is S G [1] =2*(q bit -1)-1 / r max,G[1] It is determined that r is set in the output channels (70[3], 70[4]) belonging to the second group (G[2]). max,t [k]s (temporary r max The maximum value among the ) is r max,t [4] Therefore, r max,G[2] =r max,t [4] is set. And the quantization scale S for the second group (G[2]) G [2] is S G[2] =2*(q bit -1)-1 / r max,G[2] It is decided that.

[0129] In the example presented in Fig. 8, r is set to the output channels (70[1]) belonging to the first group (G[1]). max,t [k]s (temporary r max The maximum value among the ) is r max,t [1] Therefore, r max,G[1] =r max,t is set to [1]. And the quantization scale S for the first group (G[1]) G [1] is S G [1] =2*(q bit -1)-1 / r max,G[1] is determined. The quantization scales for the other second group (G[2]), third group (G[3]), and fourth group (G[4]) can also be determined in a similar manner.

[0130] Quantization scale S for each group G[g] Once this is determined, the final quantization scale S for all output channels belonging to the above group CO [k] As expected, S G [g] can be set to the same value (where k is the index of the output channel belonging to group G[g]).

[0131] For example, all output channels belonging to a first group among the plurality of groups may be assigned a first quantization scale, and all output channels belonging to a second group among the plurality of groups may be assigned a second quantization scale.

[0132] And the first quantization scale and the second quantization scale can have different values.

[0133] Referring again to FIG. 3, in step (S150), the computing device (2) can generate a command file that causes the dedicated computing hardware (1) for processing neural network operations to execute the function of the first layer (601).

[0134] At this time, the computing device (2) is assigned a quantization scale (S) to each output channel (70[k]) of the output activation (70) of the first layer (601). CO [k] ) may be included in the above command file.

[0135] At this time, the dedicated computing hardware (1) may be configured to execute a convolution operation between the first input activation inputted to the first layer and a predetermined first output channel weight tensor unit to produce a first intermediate value (= first original output channel), and to execute a convolution operation between the first input activation inputted to the first layer and a predetermined second output channel weight tensor unit to produce a second intermediate value (= second original output channel).

[0136] At this time, the first output channel weight tensor part is a weight tensor used to obtain the first output channel among the output activations of the first layer, and the second output channel weight tensor part is a weight tensor used to obtain the second output channel among the output activations of the first layer.

[0137] Next, the dedicated computing hardware (1) may be configured to quantize the first intermediate value (= first original output channel) with a first quantization scale assigned to a group to which the first output channel belongs among the plurality of groups to produce the first output channel, and to quantize the second intermediate value (= second original output channel) with a second quantization scale assigned to a group to which the second output channel belongs among the plurality of groups to produce the second output channel. At this time, if the group to which the first output channel belongs and the group to which the second output channel belongs are different from each other, the first quantization scale and the second quantization scale may have different values.

[0138] Consequently, according to one embodiment of the present invention, when the function of the first layer (601) is executed in the dedicated computing hardware (1), data included in two output channels belonging to the output activation output by the first layer may be quantized by different quantization scales.

[0139] FIG. 9 illustrates an example of a structure of a neural network to which a neural network operation method executed by dedicated computing hardware is applied, according to one embodiment of the present invention.

[0140] According to one embodiment of the present invention, the output activation output by the first layer (601) may be provided as the input activation of the second layer (602) belonging to the neural network. At this time, the second layer (602) is a convolution layer that performs a depth-wise convolution operation.

[0141] Between the first layer (601) and the second layer (602), there may be additional layers that perform other activation functions not shown. The activation functions include, but are not limited to, various types such as sigmoid, ReLU, and Mul. However, no additional convolutional layers are included between the first layer (601) and the second layer (602).

[0142] The above depth-wise convolution operation uses only one input channel among the input activations input to the corresponding layer in order to generate one output channel among the output activations output by the corresponding layer performing the depth-wise convolution operation. Therefore, if the second layer (602) is a convolution layer performing the depth-wise convolution operation, it is not necessary for all input channels belonging to the input activations input to the second layer (602) to be quantized by the same quantization scale. Therefore, as described above, data included in two output channels belonging to the output activation (70) output by the first layer (601) may be quantized by different quantization scales, but the output activation (70) may be input to the second layer (602) performing the depth-wise convolution operation.

[0143] In contrast, if it is assumed that the second layer (602) is a convolution layer that performs a normal convolution operation, the computing hardware (1) may not operate properly in the neural network structure presented in FIG. 9 by the command file generated by the method presented in FIG. 3. This is because, although it is preferable for the normal convolution operation to receive input activation quantized by one quantization scale, according to the method presented in FIG. 3, the channels of the activation that the first layer (601) provides to the second layer (602) may be quantized by different quantization scales.

[0144] In one embodiment, the step (S110) may include a step of independently inputting a plurality of different test input activations to the first layer to independently obtain a plurality of output activations. In addition, the step (S120) may be configured to calculate the data distribution included in the plurality of output activations for each output channel. That is, the characteristics of the data distribution of each output channel of the output activation output by the first layer can be observed using a plurality of different test input activations.

[0145] FIG. 10 illustrates an example of the structure of a table including information on group-specific quantization scales included in a command file provided to a computing hardware dedicated to neural network operations provided according to one embodiment of the present invention.

[0146] In step (S150) of FIG. 3, the command file for executing the function of the first layer (601) generated by the computing device (2) may include information such as the table presented in FIG. 10.

[0147] The above table may include a layer ID field indicating layer IDs of layers constituting a neural network, a group ID field indicating group IDs identifying multiple groups constituting each layer, a configuration channel ID field indicating configuration channel IDs identifying output channels constituting each group, and a group quantization scale field indicating a group quantization scale, which is a quantization scale assigned to each group. Each row of the above table may be referred to as a 'record'. That is, the first row of the above table is the first record.

[0148] At this time, the layer indicated by the layer ID included in the above table may be a normal convolution layer.

[0149] In the example presented in Fig. 10, the layer with a layer ID of 1 may be, for example, the first layer (601). Fig. 10 shows an example in which the first layer with a layer ID of 1 and the third layer with a layer ID of 3 each consist of 10 output channels.

[0150] Information about the above group ID may not be explicitly included in the above table. Specific values ​​of the layer ID and group quantization scale may be recorded within the above table, and the ID (configuration channel ID) of the output channel to which the group quantization scale is to be applied may be provided.

[0151] In one embodiment, the configuration channel ID may be presented in bitmap format. In this case, each bitmap included in a specific layer may have the same size as the total number of output channels that the specific layer has. For example, in the first row, the layer ID is '1', the bitmap representing the configuration channel ID is '1100101000', and the group quantization scale assigned thereto is S G[1]The above bitmap consists of 10 elements, of which the first, second, fifth, and seventh elements have a value of '1' and the rest have a value of '0'. Therefore, among the output channels of the layer whose layer ID is '1', the first, second, fifth, and seventh output channels have the group quantization scale of S G[1] is quantized using .

[0152] The command file including the above table can be stored in the memory of the computing hardware (1) dedicated to the neural network operation. The table can be used by the computing hardware (1) dedicated to the neural network operation. The computing hardware (1) can check the ID of the output channel that it needs to generate, then find bitmap information including the checked output channel ID, and check the value of the group quantization scale corresponding to the found bitmap. The value of the output channel can be quantized using the checked group quantization scale.

[0153] FIG. 11 is a flowchart illustrating a method for managing quantization errors of a neural network provided according to another embodiment of the present invention.

[0154] In step (S210), the computing device (2) can input input activation to the first layer (601) of the neural network to obtain output activation (70) including a plurality of output channels (70[k]) from the first layer (601).

[0155] In step (S220), the computing device (2) can independently calculate a first quantization scale for a first output channel (70[1]) among the plurality of output channels and a second quantization scale for a second output channel (70[2]).

[0156] In step (S230), the computing device (3) can generate a command file that causes the dedicated computing hardware (1) for processing neural network operations to execute the function of the first layer.

[0157] At this time, the computing device (2) may be configured to include information about the first quantization scale assigned to the first output channel of the first layer and information about the second quantization scale assigned to the second output channel in the command file.

[0158] And the computing hardware (1) may be configured to execute a convolution operation between the first input activation input to the first layer (601) and a predetermined first output channel weight tensor section to produce a first intermediate value (= first original output channel), and to execute a convolution operation between the first input activation and a predetermined second output channel weight tensor section to produce a second intermediate value (= second original output channel).

[0159] At this time, the first output channel weight tensor section is a weight tensor used to obtain the first output channel, and the second output channel weight tensor section is a weight tensor used to obtain the second output channel.

[0160] The above computing hardware (1) may be configured to quantize the first intermediate value (= first original output channel) with the first quantization scale to produce the first output channel, and to quantize the second intermediate value (= second original output channel) with the second quantization scale to produce the second output channel.

[0161] At this time, the first quantization scale may be determined based on the distribution of data of the first group including data included in the first output channel, and the second quantization scale may be determined based on the distribution of data of the second group including data included in the second output channel.

[0162] FIG. 12 is a flowchart illustrating a method for executing a neural network operation on a computing hardware dedicated to neural network operation provided according to one embodiment of the present invention.

[0163] In step (S310), the computing hardware (1) can obtain a first input activation input to a specific layer of a predetermined neural network.

[0164] In step (S320), the computing hardware (1) can execute a convolution operation of the first input activation and a predetermined first output channel weight tensor portion assigned to the layer to calculate a first intermediate value, and execute a convolution operation of the first input activation and a predetermined second output channel weight tensor portion assigned to the layer to calculate a second intermediate value.

[0165] At this time, the first output channel weight tensor unit and the second output channel weight tensor unit may be weight tensors used to obtain the first output channel and the second output channel among the output activations of the layer, respectively.

[0166] In step (S330), the computing hardware (1) can quantize the first intermediate value with a first quantization scale stored in a first location of the memory of the computing hardware (1) to produce the first output channel, and can quantize the second intermediate value with a second quantization scale stored in a second location of the memory to produce the second output channel.

[0167] At this time, the first quantization scale and the second quantization scale may be included in a command file downloaded by the computing hardware from another device.

[0168] In step (S330), the computing hardware (1) can find a record corresponding to the specific layer and the first output channel among the tables stored in its memory. At this time, the group quantization scale specified in the record is the first quantization scale.

[0169] And the above command file may be generated by a computing device (2) that is configured to execute the quantization error management method of the above neural network.

[0170] In this specification, the ‘quantization error management method of a neural network’ may also be referred to as a ‘method for generating instruction code for executing a neural network.’

[0171] By utilizing the embodiments of the present invention described above, those skilled in the art will be able to easily implement various changes and modifications without departing from the essential characteristics of the present invention. The content of each claim may be combined with other claims that are not in a citation relationship within the scope of this specification, as long as it is understood.

[0172] <Sasa>

[0173] The national research and development projects that supported this invention are as follows. [Project ID] 20200031, [Project Number] 2020-0-01310, [Ministry Name] Ministry of Science and ICT, [Project Management (Specialized) Agency Name] National IT Industry Promotion Agency, National Research Foundation of Korea, [Research Project Name] Development of Next-Generation Intelligent Semiconductor Technology, [Research Project Name] Development of a Multisensory-Based Context-Predicting Mobile Artificial Intelligence Processor, [Contribution Ratio] 1 / 1, [Project Implementing Agency Name] Open Edge Technology Co., Ltd., [Research Period] April 1, 2020 - December 31, 2024

Claims

1. A step of a computing device inputting input activation to a specific layer of a predetermined neural network to obtain an output activation including a plurality of output channels from the specific layer, wherein the output activation is an activation provided by the specific layer to a second layer; A step in which the computing device calculates a data distribution, which is a distribution of data values ​​included in the output activation, for each output channel; A step of the computing device dividing the plurality of output channels into a plurality of groups based on the data distributions produced for each output channel; A step in which the computing device independently determines a quantization scale for quantizing each portion of the output activation corresponding to each group and assigns the quantization scale to each group; and A step of generating a command file that causes the computing device to execute the function of the specific layer by dedicated computing hardware that processes neural network operations; Includes, The computing device is configured to include information about a quantization scale assigned to each output channel of the output activation of the specific layer in the command file, The value of the first quantization scale assigned to the first group among the plurality of groups is different from the value of the second quantization scale assigned to the second group among the plurality of groups, All output channels belonging to the above first group are assigned the above first quantization scale, Characterized in that the second quantization scale is assigned to all output channels belonging to the second group. How to manage quantization errors in neural networks.

2. In paragraph 1, Information about the quantization scale includes a table including a configuration channel ID field that identifies output channels included in each of the plurality of groups, and a group quantization scale field that indicates the quantization scale assigned to each of the plurality of groups. How to manage quantization errors in neural networks.

3. In paragraph 1, The maximum value among the variances of the data distributions of each of the output channels belonging to the first group is characterized in that it is smaller than the minimum value among the variances of the data distributions of each of the output channels belonging to the second group. How to manage quantization errors in neural networks.

4. In paragraph 3, The first quantization scale is characterized in that it is smaller than the second quantization scale. How to manage quantization errors in neural networks.

5. In paragraph 1, The above obtaining step includes a step of independently inputting a plurality of different input activations to the specific layer and independently obtaining a plurality of output activations, The above-mentioned calculating step is configured to calculate the data distribution included in the plurality of output activations for each output channel. How to manage quantization errors in neural networks.

6. In paragraph 1, The above specific layer is a layer that performs a normal convolution operation. How to manage quantization errors in neural networks.

7. In paragraph 6, The second layer, which receives the output activation of the specific layer among the layers constituting the neural network, is characterized in that it is a depth-wise convolution layer. How to manage quantization errors in neural networks.

8. In paragraph 1 or paragraph 6, The above computing hardware is configured to execute a convolution operation between the first input activation input to the specific layer and the weight tensor of a predetermined first output channel to produce an intermediate value. The above first output channel weight tensor part is a weight tensor used to obtain the first part corresponding to the first output channel among the output activations of the specific layer, The computing hardware is configured to quantize the intermediate value with a quantization scale assigned to a group to which the first output channel belongs among the plurality of groups to produce the first part corresponding to the first output channel. How to manage quantization errors in neural networks.

9. A step of a computing device inputting input activation to a first layer of a neural network to obtain an output activation including a plurality of output channels from the first layer, wherein the output activation is an activation provided by the first layer to a second layer; A step in which the computing device independently calculates a first quantization scale for quantizing a first portion of the output activation corresponding to a first output channel among the plurality of output channels and a second quantization scale for quantizing a second portion of the output activation corresponding to a second output channel among the plurality of output channels; and A step of generating a command file that causes the computing device to execute the function of the first layer by dedicated computing hardware that processes neural network operations; Includes, The computing device is configured to include in the command file information about a first quantization scale assigned to the first output channel of the first layer and information about a second quantization scale assigned to the second output channel, The value of the first quantization scale is different from the value of the second quantization scale, The above computing hardware is configured to execute a convolution operation between the first input activation input to the first layer and a predetermined first output channel weight tensor to produce a first intermediate value, and to execute a convolution operation between the first input activation and a predetermined second output channel weight tensor to produce a second intermediate value. The first output channel weight tensor section is a weight tensor used to obtain a first part corresponding to the first output channel among output activations generated by the computing hardware for the first layer, and the second output channel weight tensor section is a weight tensor used to obtain a second part corresponding to the second output channel among output activations generated by the computing hardware for the first layer, The computing hardware is configured to quantize the first intermediate value with the first quantization scale to produce the first part corresponding to the first output channel, and to quantize the second intermediate value with the second quantization scale to produce the second part corresponding to the second output channel. How to manage quantization errors in neural networks.

10. In paragraph 9, The above calculating steps are: A step in which the computing device calculates a data distribution, which is a distribution of data values ​​included in the output activation obtained from the first layer, for each output channel; A step of the computing device dividing the plurality of output channels into a plurality of groups based on the data distributions produced for each output channel; and A step in which the computing device independently determines a quantization scale for each of the groups and assigns the quantization scale to each of the groups; Includes, The above first quantization scale is a quantization scale assigned to the first group to which the first output channel belongs among the plurality of groups, The above second quantization scale is a quantization scale assigned to the second group to which the second output channel belongs among the plurality of groups. How to manage quantization errors in neural networks.

11. A neural network operation method in which dedicated computing hardware for processing neural network operations executes neural network operations. A step in which the above computing hardware acquires a first input activation input to a specific layer of a predetermined neural network; A calculation step in which the computing hardware calculates a first intermediate value by executing a convolution operation of the first input activation and a predetermined first output channel weight tensor portion assigned to the specific layer, and calculates a second intermediate value by executing a convolution operation of the first input activation and a predetermined second output channel weight tensor portion assigned to the specific layer, wherein the first output channel weight tensor portion and the second output channel weight tensor portion are weight tensors used to obtain a first portion corresponding to a first output channel among output activations of the specific layer and a second portion corresponding to a second output channel among output activations, respectively; and An output activation calculation step in which the computing hardware quantizes the first intermediate value with a first quantization scale stored in a first location of a memory of the computing hardware to produce a first part corresponding to the first output channel, and quantizes the second intermediate value with a second quantization scale stored in a second location of the memory to produce a second part corresponding to the second output channel; Includes, The value of the first quantization scale is different from the value of the second quantization scale, The above output activation is an activation that the specific layer provides to the second layer. Neural network computational methods.

12. In paragraph 11, The above output activation calculation step is: A step of checking a first record corresponding to the specific layer and the first output channel and a second record corresponding to the specific layer and the second output channel among the tables stored in the memory included in the computing hardware; and A step of setting the value of the group quantization scale field included in the confirmed first record to the first quantization scale, and setting the value of the group quantization scale field included in the confirmed second record to the second quantization scale; including, Neural network computational methods.

13. In paragraph 12, In the step of acquiring the first input activation, the computing hardware is configured to acquire a layer ID that identifies the specific layer. Neural network computational methods.

14. In paragraph 11, The above first quantization scale and the above second quantization scale are included in a command file downloaded by the computing hardware from another device, The above command file is generated by a computing device that is configured to execute a quantization error management method of a given neural network. The above method for managing quantization errors of the neural network is as follows: A step in which the computing device inputs input activation to the specific layer of the neural network and obtains output activation including a plurality of output channels from the specific layer; A step in which the computing device independently calculates a first quantization scale for a first output channel and a second quantization scale for a second output channel among the plurality of output channels; and A step of generating a command file that causes the computing device to execute the function of the specific layer; characterized by including, Neural network computational methods.

Images