Quantization method, quantization system and quantization device for artificial neural network

Abstract

The invention provides a quantization method, a quantization system and quantization equipment for an artificial neural network. The quantification method for the artificial neural network can comprise the steps of estimating a sample scaling factor of first sample parameters, wherein the first sample parameters are part of first parameters in the artificial neural network; determining a predicted scaling factor based on the sample scaling factor; and quantizing the first parameter based on the predicted scaling factor.

Description

Technical Field

[0001] The present invention relates to an artificial neural network, and more particularly, to a quantization method, a quantization device and a quantization system for an artificial neural network. Background Art

[0002] An artificial neural network may refer to a computing device or a method executed by a computing device to implement a set of interconnected artificial neurons (or neuron models). Artificial neurons can generate output data by performing simple operations on input data, and the output data can be sent to other artificial neurons. As an example of an artificial neural network, a deep neural network or deep learning may have a multi-layer structure.

[0003] When dynamic quantization is used in deep learning inference operations, the input of each layer of the model may require a scale factor. When quantization operations are performed by obtaining a scale factor for each input of each layer, deep learning inference may require a lot of computation. Summary of the invention

[0004] The inventive concept provides a quantization method, a quantization system and a quantization device for an artificial neural network, in which high accuracy and low computational complexity of the artificial neural network can be achieved.

[0005] According to one or more embodiments, a quantization method for an artificial neural network is provided, the quantization method comprising: estimating a sample scaling factor of a first sample parameter, the first sample parameter being a portion of first parameters within the artificial neural network; determining a prediction scaling factor based on the sample scaling factor; and quantizing the first parameter based on the prediction scaling factor.

[0006] According to one or more embodiments, a quantization system for an artificial neural network is provided, the quantization system comprising: at least one processor; and a storage medium, the storage medium being configured to store commands executable by the at least one processor to perform a quantization process of the artificial neural network, wherein the quantization process of the artificial neural network comprises: estimating a sample scaling factor of a first sample parameter, the first sample parameter being a portion of first parameters within the artificial neural network; determining a prediction scaling factor based on the sample scaling factor; and quantizing the first parameter based on the prediction scaling factor.

[0007] According to one or more embodiments, a quantization device for an artificial neural network is provided, the quantization device comprising: a scaling factor estimator, the scaling factor estimator being configured to estimate a sample scaling factor of a first sample parameter, the first sample parameter being a part of first parameters within the artificial neural network, and the scaling factor estimator being configured to determine a predicted scaling factor based on the sample scaling factor; and a quantizer, the quantizer being configured to quantize the first parameter based on the predicted scaling factor. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Example embodiments will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0009] Figure 1 is a block diagram illustrating a quantization system according to an example embodiment;

[0010] Figure 2 is a diagram illustrating an example of an artificial neural network according to an example embodiment;

[0011] Figure 3A and Figure 3B is a diagram illustrating a quantization method according to an example embodiment;

[0012] Figure 4 is a flow chart illustrating a quantization method for an artificial neural network according to an example embodiment;

[0013] Figure 5 is a block diagram illustrating a quantization apparatus for an artificial neural network according to an example embodiment;

[0014] Figure 6 is a diagram illustrating an exemplary embodiment of the Figure 5 A block diagram of a processing device inside a quantization device;

[0015] Figure 7 is a diagram illustrating an exemplary embodiment of the Figure 5 A block diagram of a quantization with scale factor estimation (QSE) unit within a quantization device;

[0016] Figure 8 is a block diagram illustrating a computing system according to an example embodiment; and

[0017] Fig. 9 is a block diagram illustrating a portable computing system according to an example embodiment. DETAILED DESCRIPTION

[0018] Hereinafter, example embodiments of the inventive concept will be described in detail with reference to the accompanying drawings.

[0019] Figure 1 is a block diagram illustrating a quantization system 100 according to an example embodiment. An artificial neural network may refer to a computing system that attempts to simulate a biological neural network in an animal brain. In an artificial neural network, the execution of a task can be learned by considering multiple samples (or examples), which is different from classical algorithms such as rule-based programming that execute tasks according to predefined conditions. An artificial neural network may have a structure in which artificial neurons (or neurons) are connected to each other, and the connection between each neuron may be referred to as a synapse. A neuron may process a received signal and may send the processed signal to another neuron through a synapse. The output of a neuron may be referred to as an activation. Neurons and / or synapses may have variable weights, and the influence of a signal processed by a neuron may be increased or decreased according to the weight. In addition, a bias associated with an individual neuron is applied.

[0020] A deep neural network or deep learning architecture may have a layer structure, and the output of a particular layer may be the input of a subsequent layer. In such a multi-layer structure, each layer may be trained according to multiple samples. An artificial neural network (such as a deep neural network) may be implemented by multiple processing nodes corresponding to artificial neurons, respectively, and may require high computational complexity to obtain satisfactory results, such as high accuracy results. Therefore, many computing resources may be required.

[0021] To reduce computational complexity, artificial neural networks can be quantized. Quantization can refer to the process of mapping input values ​​to values ​​that are smaller than the number of input values, such as mapping real numbers to integers by rounding. In artificial neural networks, quantization can include the process of converting a floating-point decimal neural network to an integer neural network. For example, in artificial neural networks, quantization can be used in activations, weights of layers, etc. A floating-point decimal number can include a sign, an exponent, and a significand, where an integer can include an integer portion. In some embodiments, the integer portion of an integer can include a sign bit. Reference Figure 1 , the input data set IN is expressed in floating-point decimal numbers, and the output data set OUT that undergoes the quantization process can be expressed in integers. An artificial neural network using floating-point decimal numbers can have high accuracy and high computational complexity, whereas an artificial neural network using integers can have reduced accuracy and lower computational complexity.

[0022] In an artificial neural network, due to the trade-off relationship between the accuracy of the result and the computational complexity, the quantization of the artificial neural network may result in a reduction in accuracy, and the degree of the reduction in accuracy may depend on the method of quantization. Hereinafter, as described below with reference to the accompanying drawings, the quantization system 100 according to the example embodiment may provide quantization as required while minimizing the reduction in accuracy, and thus, may provide a quantized neural network with reduced complexity and sufficiently high performance.

[0023] The quantization system 100 may be any type of system that performs quantization according to example embodiments, and may also be referred to as a quantization device. For example, the quantization system 100 may be a computing system including at least one processor and at least one memory. As non-limiting examples, the quantization system 100 may be a mobile computing system such as a laptop computer, a smart phone, etc., and a fixed computing system such as a desktop computer and a server, etc. Figure 1 As shown, the quantization system 100 may include an estimator 120 and a quantizer 140, and each of the estimator 120 and the quantizer 140 may be implemented by a logic block implemented by logic synthesis, a software block executed by a processor, or a combination thereof. In some embodiments, each of the estimator 120 and the quantizer 140 may be a process as a set of multiple commands executed by a processor, or may be stored in a memory accessible to the processor.

[0024] In this document, the scaling factor estimator 120 and the quantizer 140 may each be an analog circuit and / or a digital circuit including one or more of the following: logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, etc., and may be configured to run software and / or firmware to perform the corresponding functions or operations described above.

[0025] refer to Figure 1 , the quantization system 100 may receive an input data set IN, and may generate a quantized output data set OUT. The input data set IN may include a plurality of input data, and the quantized output data set OUT may include a plurality of output data. Figure 1As shown, a plurality of matrix-type data as input data sets IN may be input to the quantization system 100 according to the example embodiment, and the quantization system 100 may output a plurality of matrix-type data as a quantized output data set OUT. The plurality of data according to the example embodiment may include a plurality of data for each pixel extracted from the image. Each input data constituting the input data set IN may include a floating-point decimal number. Each output data constituting the output data set OUT to be output by the quantization system 100 may include an integer. As described below, in the quantization system 100 according to the example embodiment, the calculation for determining the scaling factor for all inputs in the quantization operation may be omitted, and therefore, the total amount of calculation may be reduced.

[0026] The estimator 120 may receive an input data set IN and may determine a prediction scaling factor such as an activation, a weight, etc. to provide to the quantizer 140. A scaling factor may be required to quantize a floating point decimal number variable. The scaling factor may indicate a value for mapping a range of values ​​of the data being quantized to a quantization range corresponding to a maximum value and a minimum value that can be represented by the number of bits used in the quantization process. Figure 3A and Figure 3B A detailed description of the scaling factor is provided in detail. The process of quantizing a variable can be classified into dynamic quantization and static quantization. The quantization system 100 according to the example embodiment can quantize multiple pieces of data by using a dynamic quantization method, in which weights, activations, etc. in an artificial neural network are quantized in real time in an inference step after learning. For convenience in this specification, multiple pieces of data to be quantized such as weights, activations, etc. in an artificial neural network can be referred to as parameters. When the scaling factor is calculated or estimated for all parameters to be quantized, the scaling factor can have high accuracy, but may require excessive calculation. As described below, the estimator 120 according to the example embodiment can estimate the scaling factor for part of the parameter to be quantized, and can determine a predicted scaling factor from the estimated scaling factor to provide the determined predicted scaling factor to the quantizer 140. That is, the estimator 120 can estimate the scaling factor only for part of the parameter to be quantized, and skip estimating the scaling factor for other parts of the first parameter. Since the quantization system 100 can estimate scaling factors for only part of the parameters instead of all the parameters, determine a predicted scaling factor based on the scaling factors estimated from the part of the parameters, and use the predicted scaling factor to perform quantization on the parameters, the calculation for determining the scaling factor for all inputs in the quantization operation during deep learning inference can be omitted, and therefore the total amount of calculation can be reduced.

[0027] The quantizer 140 may receive the prediction scaling factors corresponding to the parameters from the estimator 120, and may quantize the parameters based on the prediction scaling factors to generate a quantized output data set OUT. Figure 7 etc. describe the detailed quantization process of the quantizer 140 in detail.

[0028] Figure 2 is a diagram illustrating an example of an artificial neural network according to example embodiments.

[0029] refer to Figure 2 The artificial neural network may have a structure including an input layer L1, hidden layers (L2, L3) and an output layer Ln (where n is a natural number of 3 or more), and may generate an output signal based on received input data (e.g., I 1 ,I 2 ) to perform operations, and output data (e.g., 1 , O 2 ).

[0030] The artificial neural network may be a deep neural network or an n-layer neural network including one or more hidden layers. Figure 2 As shown, the artificial neural network may be a deep neural network including an input layer L1, hidden layers L2 and L3, and an output layer Ln. The deep neural network may include a convolutional neural network (CNN), a recurrent neural network (RNN), a deep belief network, a restricted Boltzmann machine, etc., and example embodiments are not limited thereto.

[0031] When the artificial neural network has a deep neural network (DNN) structure, the artificial neural network includes more layers from which effective information can be extracted, so that the artificial neural network can process more complex data sets than in other types of artificial neural networks according to the prior art. Figure 2 The structures shown differ in the various structural layers.

[0032] Each of the layers L1 to Ln included in the artificial neural network may include a plurality of artificial nodes, which are also referred to as neurons, units or similar terms. Figure 2 As shown, the input layer L1 may include two nodes ND, and the hidden layers L2 and L3 may include three nodes ND. However, this is only an example, and each layer included in the artificial neural network may include various numbers of nodes ND, and the number of nodes ND included in each layer may be different from each other.

[0033] The nodes included in each layer included in the artificial neural network can be connected to each other to exchange data. For example, one node NDD can receive data from other nodes ND to perform calculations, and can output the results of the calculations to other nodes ND.

[0034] The input and output of each node ND may be referred to as activation. Activation may be an output value of one node ND, and may be an input value of a node ND included in the next layer. Each node ND may determine its own activation based on the activation and weight received from the node ND included in the previous layer. The weight is a network parameter used to calculate the activation in each node ND, and may be a value assigned to a connection relationship between nodes ND. For example, in the second layer L2, a node ND may determine its own activation based on the activation (a 11 、a 12 ), weight (w 21 、w 22 、w 23 、w 24 、w 25 、w 26 ) and bias (b 21 、b 22 、b 23 ) to determine their own activation. Each node ND may be a computational unit that receives input and outputs activation, and may perform input-output mapping.

[0035] The artificial neural network may include an activation function between layers. The activation function may convert the output of a previous layer into the input of a next layer. For example, the activation function may be a nonlinear function such as a rectified linear unit (ReLU), a parameterized rectified linear unit (PReLU), a hyperbolic tangent (tanh), a sigmoid function, and may nonlinearly convert the output of the second layer L2 between the second layer L2 and the third layer L3. The activation may be a value obtained by applying the activation function to the weighted sum of the activations received from the previous layer.

[0036] Then, refer to Figure 1 and Figure 2 , the objects to be quantized by the quantization system 100 may include activations and weights of an artificial neural network. For example, the activations (a received from the first layer L1 and input to the second layer L2 11 、a 12 ) and weights (w 21 、w 22 、w 23 、w 24 、w 25 、w 26 ) can be data in floating point decimal format. The quantization system 100 can activate (a 11、a 12 ) and weight (w 21 、w 22 、w 23 、w 24 、w 25 、w 26 ) is quantized into data in integer format. The weighted sum can be calculated using the quantized activations and weights, so the size of the deep learning model can be reduced and the amount of calculation can be reduced. Subsequently, as the activation function is applied to the value of the calculated weighted sum, the activation (a 21 、a 22 、a 23 ) may be data in a floating point decimal format again. That is, the object to be quantized by the quantization system 100 may be a parameter of a specific layer or may be a parameter corresponding to each layer.

[0037] Figure 3A and Figure 3B is a view illustrating a quantization method according to example embodiments.

[0038] Specifically, Figure 3A The affine quantization method is illustrated, and Figure 3B The scale quantization method is illustrated.

[0039] refer to Figure 3A In the affine quantization method, the maximum and minimum values ​​of the data before being quantized can respectively correspond linearly to the maximum or minimum value of the data format to be quantized. Figure 3A As shown, the maximum value (α) of the data before being quantized may be 7, and the minimum value (β) of the data before being quantized may be -5. When the data is quantized into an 8-bit integer format, the quantized data may have a value ranging from -128 to 127. According to the affine quantization method, the maximum value (α) of the data before being quantized may correspond to 127, and the minimum value (β) of the data before being quantized may correspond to -128. The data value between the maximum value (α) and the minimum value (β) may correspond linearly to the data between 127 and -128. Therefore, when the maximum value (α) of the data before being quantized and the minimum value (β) of the data before being quantized have different values, the value 0 before being quantized may correspond to another value that is not 0 after being quantized. In the process of linearly corresponding values, the ratio of the range of the value of the data after being quantized to the range of the value of the data before being quantized may be a scaling factor. That is, in the affine quantization method, the scaling factor may be calculated as the following [Formula 1].

[0040] [Formula 1]

[0041]

[0042] In [Equation 1], b represents the number of bits of integer data after quantization. Figure 3A The values ​​shown are Figure 3A The scaling factor of the affine quantization method can be

[0043] refer to Figure 3B In the scaling quantization method, similar to the affine quantization method, the maximum and minimum values ​​of the data before being quantized can respectively correspond linearly to the maximum and minimum values ​​of the data format to be quantized. However, in the scaling quantization method, in order to make the value of 0 before quantization correspond to the value of 0 after quantization, the relationship (α=-β) can be satisfied. That is, in the scaling quantization method, the maximum value (α) of the absolute value of the data before quantization can be obtained, and α and -α can respectively correspond linearly to the maximum and minimum values ​​of the data format to be quantized. In the process of linearly corresponding values, the ratio of the range of the value of the data after quantization to the range of the value of the data before quantization can be a scaling factor. That is, in the scaling quantization method, the scaling factor can be calculated as follows [Formula 2].

[0044] [Formula 2]

[0045]

[0046] In [Formula 2], b represents the number of bits of integer data after quantization. Figure 3B The values ​​shown are Figure 3B The scaling factor of the scaling quantization method can be

[0047] exist Figure 3A and Figure 3B In the two quantization methods of , the scaling factor may be multiplied by the data before being quantized, and the value obtained by the multiplication may be within the range of the data after being quantized. The quantization method may include a rounding operation that expresses the result of multiplying the scaling factor by the data to be quantized in an integer format and a clipping operation that corrects the value out of the range, as described below with reference to Figure 7 Described in detail. Figure 3A and Figure 3B In the two quantization methods of , a process of calculating the maximum and minimum values ​​of the data may be required to calculate or estimate the scaling factor. When various data sets are input and the scaling factor is calculated for all data sets, excessive calculations may be required. As described below, according to example embodiments, calculations for determining scaling factors for all inputs in a quantization operation during deep learning inference may be omitted, so the total amount of calculations may be reduced.

[0048] Figure 4 is a flow chart illustrating a quantization method for an artificial neural network according to example embodiments.

[0049] For example, Figure 4 The quantification method can be used to Figure 1 In the following, reference will be made to the quantization system 100. Figure 1 right Figure 4 Give a description.

[0050] refer to Figure 4 In operation S410, an operation of estimating a sample scaling factor of a first sample parameter may be performed. The first sample parameter may be part of a first parameter. The first parameter may include an input variable input to the first layer or a weight of the first layer, which input variable or weight is to be quantized. The input variable input to the first layer may be an activation output from a previous layer of the first layer.

[0051] The first sample parameter as part of the first parameter may be a parameter for determining a predicted scaling factor to be described below. The first sample parameter may be selected by sampling a portion (not a predefined sample) of the input first parameter. For example, when the input first parameter includes A data sets, the first sample parameter may include the first B data sets of the input first parameter (where B is a natural number less than the natural number A). That is, in an example embodiment, the first sample parameter may refer to a series (e.g., continuous) of first parameters input among the first parameters. The first B data sets of the input first parameter may be the first sample parameter, and an operation of estimating or calculating a scaling factor for each first sample parameter may be performed on the first B data sets of the input first parameter. The scaling factor corresponding to the first sample parameter may be obtained by using the reference above. Figure 1 The method described in Figure 3 is used for estimation. For example, when the input data set IN is Figure 1 When the first sample parameter is input as shown, the maximum or minimum value in the data set can be obtained, and the scaling factor can be calculated or estimated based on this, as described above with reference to FIG3. In order to obtain the maximum or minimum value in the data set, a comparator can be used or a comparison operation can be performed, as described below with reference to Figure 7 As described in detail.

[0052] refer to Figure 4, in operation S420, an operation of determining a prediction scaling factor based on a sample scaling factor may be performed. In the previous operation S410, after estimating the sample scaling factor for the first sample parameter, a prediction scaling factor that can represent all first parameters may be determined based on the sample scaling factor. As described above, when calculating scaling factors for all parameters, excessive calculations may be required. Therefore, in the quantization method of the present disclosure, a prediction scaling factor may be utilized in the quantization calculation. In the method of determining a prediction scaling factor based on a sample scaling factor, various formulas may be utilized, and different formulas may be selected according to the characteristics or relationships of a data set. The method of determining a prediction scaling factor according to an example embodiment may use the following [Formula 3].

[0053] [Formula 3]

[0054]

[0055] [Formula 3] is a calculation formula to be executed by a computer and / or a program, and is a calculation formula indicating a calculation method of a prediction scaling factor PSF when the sample scaling factor SF is sequentially input to the quantizer 140. In [Formula 3], specifically, the prediction scaling factor can be calculated by reflecting an exponential moving average. [Formula 3] shows a case where the proportional coefficient is 1 / 2 in the calculation method of the prediction scaling factor PSF, and the embodiment is not limited to [Formula 3]. Another example formula based on [Formula 3] is shown in Formula 4 below.

[0056] [Formula 4]

[0057] PSF=(1-k)·PSF+k·SF

[0058] [Formula 4] is a calculation formula to be executed by a computer and / or a program, and is a calculation formula indicating a calculation method of the prediction scaling factor PSF when the sample scaling factor SF is sequentially input to the quantizer 140. k is a real number between 0 and 1, and the larger the k value, the greater the influence of the previous observation value can be reduced. When k is 1 / 2, [Formula 3] holds.

[0059] A method of determining a prediction scaling factor according to another exemplary embodiment may use the following [Equation 5].

[0060] [Formula 5]

[0061]

[0062] [Formula 5] is a calculation formula to be executed by a computer and / or a program, and is a calculation formula indicating a calculation method of a prediction scaling factor PSF when the sample scaling factors SF are sequentially input to the quantizer 140. In [Formula 5], specifically, the prediction scaling factor can be calculated by reflecting the arithmetic mean.

[0063] A method of determining a prediction scaling factor according to another exemplary embodiment may use the following [Equation 6].

[0064] [Formula 6]

[0065] PSF=min(PSF,SF)

[0066] [Formula 6] is a calculation formula to be executed by a computer and / or a program, and is a calculation formula indicating a calculation method of the prediction scaling factor PSF when the sample scaling factors SF are sequentially input to the quantizer 140. In [Formula 6], specifically, the prediction scaling factor can be calculated by reflecting the minimum value of the sample scaling factor.

[0067] A method of determining a prediction scaling factor according to another exemplary embodiment may use the following [Equation 7].

[0068] [Formula 7]

[0069] PSF=max(PSF,SF)

[0070] [Formula 7] is a calculation formula to be executed by a computer and / or a program, and is a calculation formula indicating a calculation method of the prediction scaling factor PSF when the sample scaling factors SF are sequentially input to the quantizer 140. In [Formula 7], specifically, the prediction scaling factor can be calculated by reflecting the maximum value of the sample scaling factor.

[0071] Formulas such as [Formula 3] to [Formula 7] may be calculated by sequentially calculating sample scaling factors for sample parameters input in time order, and cumulatively reflecting the sequentially calculated sample scaling factors in the predicted scaling factor value. In this case, the sample scaling factor calculated (or estimated) first may be determined as the initial predicted scaling factor. In addition, after calculating the sample scaling factor corresponding to the sample parameter, the predicted scaling factor value may also be calculated based on all sample scaling factors.

[0072] By formulas such as [Formula 3] to [Formula 7], in operation S420, an operation of determining a prediction scaling factor based on a sample scaling factor may be performed. However, this is merely an example formula, and the method of determining a prediction scaling factor based on a sample scaling factor is not limited thereto. For example, the prediction scaling factor value calculated from the sample scaling factor may be adjusted by adding or multiplying a suitable coefficient to a formula such as [Formula 3] to [Formula 7]. In addition, the value of the prediction scaling factor may also be calculated by combining a plurality of formulas including one or more of the above-mentioned formulas. By utilizing a suitable formula, a quantization operation may be performed in which accuracy may be guaranteed while reducing the amount of computation of the artificial neural network.

[0073] refer to Figure 4In operation S430, an operation of quantizing the first parameter based on the predicted scaling factor may be performed. The quantization operation may be performed by Figure 1 The operation of the quantizer 140 is performed, and may also be performed by the quantizer 140 described below including Figure 7 The method for quantizing the first parameter may include the affine quantization method or the scaling quantization method described above with reference to FIG. 3 . Specifically, the predicted scaling factor calculated in the previous operation S420 may be the calculated factor, rather than the process of calculating the scaling factor for all parameters, and may be used in the quantization process of the first parameter. Instead of calculating the scaling factor for each first parameter, the first parameter may be quantized using the calculated predicted scaling factor. According to the above embodiment, by using the predicted scaling factor, the calculation for determining the scaling factor for all inputs in the quantization operation during deep learning inference can be omitted, and therefore, the total amount of calculation can be reduced. In addition, the computing resources for implementing an artificial neural network based on a quantized artificial neural network with high accuracy can be reduced, and the scope of application of the artificial neural network can be expanded.

[0074] Figure 5 is a block diagram illustrating a quantization apparatus for an artificial neural network according to example embodiments.

[0075] refer to Figure 5 , a quantization device 500 for an artificial neural network may include a memory 510 and a processor 520. Figure 1 and Figure 4 right Figure 5 Description is made and redundant description with reference to previous drawings is omitted.

[0076] The memory 510 of the quantization device 500 may store a program for quantization of an artificial neural network according to an example embodiment, and may store activations or data quantized by a quantization method. In addition, the memory 510 may store Figure 1 The memory 510 may store a series of commands related to the execution of the quantization. The memory 510 may communicate with the processor 520 to exchange data with each other.

[0077] The processor 520 may be one or more of a central processing unit (CPU), a graphics processing unit (GPU), and a neural processing unit (NPU). However, this is an example, and the processor 520 is not limited to the above description. Figure 5The illustrated processor 520 may include a data fetcher 522, a vector processing unit 524, and an inner product array 526. The data fetcher 522 may receive data such as weights, activations, biases, etc. from the memory 510, and may send the data to the vector processing unit 524. The vector processing unit 524 may include a quantization and scaling factor estimation (QSE) unit 524_1 therein. Figure 5 The illustration is an example, and the QSE unit 524_1 may operate independently outside the vector processing unit 524. The QSE unit 524_1 may obtain a prediction scaling factor corresponding to a parameter such as an input weight, an activation, etc., and may perform quantization based on the prediction scaling factor. Figure 7 The detailed operation of the QSE unit 524_1 is described.

[0078] The vector processing unit 524 may send the quantized parameters to the data extractor 522, and the data extractor 522 may send the quantized parameters to the inner product array 526. The inner product array 526 may calculate a weighted sum based on data such as input quantized weights, quantized activations, quantized biases, and the like. Figure 5 The quantization device 100 is illustrated to include N processing units 526_1, 526_2, ..., 526_N. However, this is an example, and the quantization device 100 may also include a different number of processing units PE. The processing unit PE may calculate a weighted sum between the quantized weight and the quantized activation, and will be referred to below. Figure 6 The detailed operation of the processing unit PE is described in detail. When the weighted sum is calculated based on unquantized parameters, the inner product array 526 requires a lot of calculations, and may require more time and many resources in the reasoning operation of deep learning. According to an example embodiment, the calculation for determining the scaling factor for all inputs in the quantization operation during deep learning reasoning can be omitted, and therefore, the total amount of calculations can be reduced.

[0079] Data such as activation output through the inner product array 526 may be sent to the vector processing unit 524, and the vector processing unit 524 may receive additional data from the data fetcher 522 to perform a quantization operation of the next layer. In addition, the vector processing unit 524 may send the output activation data sent from the inner product array 526 to the memory 510 so as to store the output activation data.

[0080] Despite Figure 5 It is not shown in the figure, but may additionally include a configuration of a buffer or a register for storing data therein, and may additionally include a configuration such as an adder or a multiplexer that selects or adds data.

[0081] Figure 6 is a diagram illustrating an exemplary embodiment of the Figure 5 A block diagram of a processing device inside a quantization device 500.

[0082] refer to Figure 5 and Figure 6 , the inner product array 526 included in the device 500 may include a plurality of processing devices PE 526_1, 526_2, ..., 526_N. Each processing device PE (e.g., Figure 6 The illustrated 526_N) may include a plurality of multipliers MPR and an adder tree. The processing device PE 526_N may perform the calculation shown in the following [Equation 8].

[0083] [Formula 8]

[0084]

[0085] In [Equation 8], I represents input quantized activation, W represents input quantized weight, Bias represents bias, and y represents output. The processing device PE 526_N may receive X quantized activations, quantized weights corresponding to the quantized activations, and bias. X may represent the number of channels of the layer on which calculations are performed, and may have different values ​​for each layer. In addition, Figure 6 It is illustrated that there are X multipliers MPR. However, this is an example, and the processing device PE 526_N may include a different number of multipliers MPR according to hardware specifications. The quantized activations and quantized weights multiplied by the multipliers may be added together with the bias by the adder tree. As a result, the processing device PE 526_N may calculate a weighted sum by receiving the X quantized activations, the quantized weights, and the bias values.

[0086] Despite Figure 6 Although not shown in the figure, the processing device PE 526_N according to an example embodiment may include a register or a memory for storing data according to an embodiment, and may additionally include a component for applying an activation function.

[0087] Figure 7 is a diagram illustrating an exemplary embodiment of the Figure 5 Block diagram of the QSE unit inside the quantization device.

[0088] refer to Figure 5 and Figure 7 , the vector processing unit 524 inside the quantization device 500 may include a QSE unit 524_1. The QSE unit 524_1 may include a scaling factor estimator 710 and a quantizer 720. The scaling factor estimator 710 and the quantizer 720 may be Figure 1 The example of the estimator 120 and the quantizer 140 of FIG. 1 and the example of ... Figure 1 and Figure 4 Redundant description of the description.

[0089] The QSE unit 524_1 may perform scaling factor estimation and quantization of parameters such as activations, weights, and the like. The QSE unit 524_1 may receive a first parameter and may output a quantized parameter. In this case, each parameter may refer to a data set, and each data set may include N data. For example, in the case of an image having 3×3 pixels, one image data set may include 9 pieces of data corresponding to 9 pixels, and in the quantization operation in the first layer, N may be 9. In this case, each first parameter may include 9 pieces of data, and the QSE unit 524_1 may receive input data 0_1, 0_2, ..., and 0_N constituting one input parameter, and may output quantized output data Q_1, Q_2, ..., and Q_N. The quantized output data Q_1, Q_2, ..., and Q_N may constitute one quantized output parameter. That is, Figure 7 The illustrated input and output operations are operations corresponding to one parameter (ie, one data set). For example, when processing 1000 images (each image having 3×3 pixels), with respect to one image, the first layer may receive 9 pieces of input data 0_1, 0_2, ..., and 0_N, and may output quantized output data Q_1, Q_2, ..., and Q_N. In addition, the input and output of the corresponding data may be performed for 1000 images.

[0090] The scaling factor estimator 710 inside the QSE unit 524_1 may estimate a sample scaling factor for a first sample parameter. The first sample parameter may be a portion of the first parameter. In an embodiment, the QSE unit 524_1 may estimate the scaling factor only for the portion of the first parameter to be quantized and skip estimating the scaling factor for other portions of the first parameter. Figure 4 As described, the first sample parameter according to the example embodiment may refer to a specific number of first parameters among the first parameters. The first sample parameter may be a continuous first parameter and may be a group of first parameters that appear first. For each first sample parameter, the scaling factor estimator 710 may calculate the maximum and minimum values ​​of the sample parameter by using a plurality of comparators CPR to estimate the sample scaling factor for the corresponding sample parameter. As described above with reference to Figure 3A and Figure 3B As described, the scaling factor estimator 710 may calculate the sample scaling factor based on the maximum value and the minimum value calculated by the plurality of comparators CPR.

[0091] The scaling factor estimator 710 may include a register 711 configured to store therein a sample scaling factor and a predicted scaling factor. The scaling factor estimator 710 may calculate the sample scaling factor for each first sample parameter by using the above-mentioned plurality of comparators CPR until B first sample parameters (e.g., B data sets) are input to the scaling factor estimator 710. The value of B may be preset. The calculated sample scaling factor may be delivered to the quantizer 720 and may be stored in the register 711. The scaling factor estimator 710 may calculate the predicted scaling factor based on equations such as [Equation 3] to [Equation 6] and the calculated sample scaling factor, as described above with reference to Figure 4 Just as described.

[0092] After all first sample parameters (e.g., B data sets) are input to the QSE unit 524_1, the register 711 of the scaling factor estimator 710 may calculate and store the predicted scaling factor. Subsequently, when additional first parameters are input to the QSE unit 524_1, the scaling factor estimator 710 may not perform a calculation for calculating a maximum value or a minimum value by a comparison calculation between multiple pieces of data of each additional first parameter. That is, after the calculation of the predicted scaling factor is completed, the scaling factor estimator 710 may output the predicted scaling factor stored in the register 711 to the quantizer 720 without estimating the scaling factor based on the input of the first parameter. The scaling factor calculation may not be performed for all parameters (e.g., all data sets), and the quantization calculation may be performed using the predicted scaling factor. Therefore, according to an example embodiment, the calculation for determining the scaling factor for all inputs in the quantization operation during deep learning reasoning may be omitted, and therefore, the total amount of calculation may be reduced and reasoning may be accelerated.

[0093] After the input data O_1, O_2, ..., and O_N and the scaling factor SF from the scaling factor estimator 710 are input to the quantizer 720, the quantizer 720 may output quantized output data Q_1, Q_2, ..., and Q_N. In the portion where the first sample parameter is input to the quantizer 720, the calculated sample scaling factor may be input to the quantizer 720 from the scaling factor estimator 710. Figure 7It is shown that the scaling factor SF is input to the quantizer 720. However, this is a comprehensive representation, and after the calculation of the predicted scaling factor is completed, as described above, the predicted scaling factor stored in the register 711 may be input to the quantizer 720. That is, when the first sample parameter is input, the sample scaling factor corresponding to each first sample parameter may be sequentially input to the quantizer 720 through a series of calculations of the scaling factor estimator 710. After all sample parameters are input, the predicted scaling factor may be calculated and stored in the register 711, and the predicted scaling factor may be input to the quantizer 720.

[0094] The quantizer 720 may multiply the prediction scaling factor by each input data O_1, O_2, ..., and O_N through the multiplier MPR. Subsequently, the quantizer 720 may round the multiplied value and may output the final quantized output data Q_1, Q_2, ..., and Q_N through a clipping calculation. The rounding calculation unit in the quantizer 720 may indicate the output data in the form of an integer.

[0095] The calculation of the clipping calculation unit can be expressed as [Equation 9].

[0096] [Formula 9]

[0097]

[0098] In [Equation 9], x is an input data value, l is a minimum value of quantized data to be expressed, and u is a maximum value of quantized data to be expressed. The clipping calculation unit in the quantizer 720 can adjust the data value so that the output data can be between the maximum value and the minimum value.

[0099] According to the above-described embodiment, the calculation for estimating the scaling factor for all inputs in the quantization operation during deep learning inference can be omitted, and therefore the total amount of calculation can be reduced. In addition, the computational resources for implementing an artificial neural network with high accuracy based on quantization can be reduced, and the application range of the artificial neural network can be expanded.

[0100] In this article, the scaling factor estimator 710 and the quantizer 710 can each be an analog circuit and / or a digital circuit including one or more of the following: logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, etc., and can be configured to run software and / or firmware to perform the corresponding functions or operations mentioned above.

[0101] Figure 8 is a block diagram illustrating a computing system 2000 according to an example embodiment.

[0102] In some embodiments, Figure 1 The quantization system 100 and / or Figure 5 The quantization device 500 may be implemented in Figure 8 In the computing system 2000. Figure 8 As shown, computing system 2000 may include system memory 2100, processor 2300, storage device 2500, input / output (IO) device 2700, and communication connection 2900. For example, components included in computing system 2000 may be communicatively connected to each other via a bus.

[0103] The system memory 2100 may include a program 2120. The program 2120 may enable the processor 2300 to perform quantization of an artificial neural network according to an example embodiment. For example, the program 2120 may include a plurality of commands that can be executed by the processor 2300, and the plurality of commands included in the program 2120 may be executed by the processor 2300 so that quantization of the artificial neural network may be performed. The system memory 2100 as a non-limiting example may include a volatile memory such as a static random access memory (SRAM) and a dynamic random access memory (DRAM), or a non-volatile memory such as a flash memory, etc.

[0104] The processor 2300 may include at least one core for executing any command set (e.g., Intel Architecture-32 (IA-32), 64-bit extended IA-32, x86-64, PowerPC, Sparc, MIPS, ARM, IA-64, etc.) The processor 2300 may execute commands stored in the system memory 2100 and may execute the program 2120 so that quantization of the artificial neural network may be performed.

[0105] Even if the power supplied to the computing system 2000 is blocked, the storage device 2500 does not lose the stored data. For example, the storage device 2500 may include a non-volatile memory such as an electrically erasable programmable read-only memory (EEPROM), a flash memory, a phase change random access memory (PRAM), a resistive random access memory (RRAM), a nano floating gate memory (NFGM), a polymer random access memory (PoRAM), a magnetic random access memory (MRAM), a ferroelectric random access memory (FRAM), or a storage medium such as a magnetic tape, an optical disk, a magnetic disk, etc. In some embodiments, the storage device 2500 may be detachable from the computing system 2000.

[0106] In some embodiments, the storage device 2500 may store a program 2120 for quantization of an artificial neural network according to example embodiments, and before the program 2120 is executed by the processor 2300, the program 2120 or at least a portion thereof may be loaded from the storage device 2500 into the system memory 2100. In some embodiments, the storage device 2500 may store a file written in a program language, and the program 2120 or at least a portion thereof generated by a compiler or the like may be loaded from the file into the system memory 2100.

[0107] In some embodiments, the storage device 2500 may store data to be processed by the processor 2300 and / or data processed by the processor 2300. For example, the storage device 2500 may store activations or data quantized according to the above quantization method, and may store Figure 1 The quantized output data set OUT, or the bias or data generated during the quantization can be stored.

[0108] The input / output device 2700 may include an input device such as a keyboard, a pointing device, etc., and may include an output device such as a display device, a printer, etc. For example, a user may trigger the processor 2300 to run the program 2120 by inputting Figure 2 The input data (for example, I 1 ,I 2 ), or can be checked through the input / output device 2700 Figure 1 The quantified output data set OUT and / or error messages.

[0109] Communication connection 2900 may provide access to a network external to computing system 2000. For example, a network may include multiple computing systems and communication links, and the communication links may include wired links, optical links, wireless links, or links in any other format.

[0110] Fig. 9 is a block diagram illustrating a portable computing system according to an example embodiment.

[0111] In some embodiments, the quantized artificial neural network according to the example embodiments may be implemented in a portable computing device 3000. The portable computing device 3000 as a non-limiting example may be any portable electronic device supplied with power by a battery or a self-contained power source, such as a mobile phone, a tablet personal computer (PC), a wearable device, and an Internet of Things (IoT) device.

[0112] like Fig. 9As shown, the portable computing device 3000 may include a storage subsystem 3100, an input / output (IO) device 3300, a processing unit 3500, and a network interface 3700. The storage subsystem 3100, the input / output device 3300, the processing unit 3500, and the network interface 3700 may communicate with each other via a bus 3900. In some embodiments, at least two of the storage subsystem 3100, the input / output device 3300, the processing unit 3500, and the network interface 3700 may be included in one package as a system on a chip (SoC).

[0113] The storage subsystem 3100 may include a random access memory (RAM) 3120 and a storage device 3140. The RAM 3120 and / or the storage device 3140 may store commands to be executed by the processing unit 3500 and data to be processed by the processing unit 3500. For example, the RAM 3120 and / or the storage device 3140 may store variables such as signals, weights, biases of an artificial neural network, and may also store parameters of artificial neurons (or computing nodes) of the artificial neural network. In some embodiments, the storage device 3140 may include a non-volatile memory.

[0114] The processing unit 3500 may include a CPU 3520, a GPU 3540, a digital signal processor (DSP) 3560, and an NPU 3580. Fig. 9 Unlike in the embodiment of the present invention, in some embodiments, the processing unit 3500 may include only at least part of the CPU 3520, the GPU 3540, the DSP 3560, and the NPU 3580.

[0115] The CPU 3520 may control the overall operation of the portable computing device 3000 and perform a specific job itself, or may instruct other components of the processing unit 3500 to perform a specific job, for example, in response to an external input received through the input / output device 3300. The GPU 3540 may generate data of an image to be output through a display device included in the input / output device 3300, or may encode data received from a camera included in the input / output device 3300. The DSP 3560 may process a digital signal, for example, a digital signal provided from the network interface 3700, so that valid data may be generated.

[0116] The NPU 3580 as execution hardware for the artificial neural network may include a plurality of computing nodes corresponding to at least a portion of the artificial neurons constituting the artificial neural network, and at least a portion of the plurality of computing nodes may process signals in parallel. According to example embodiments, a quantized artificial neural network (such as a deep neural network) has high accuracy and low computational complexity, and thus may be easily implemented in Fig. 9In the portable computing device 3000, the quantized artificial neural network has a fast processing speed and can be implemented by, for example, a simple and small NPU 3580.

[0117] The input / output device 3300 may include input devices such as a touch input device, a sound input device, a camera, etc., and output devices such as a display device, a sound output device, etc. The network interface 3700 may provide the portable computing device 3000 with access to mobile communication networks such as Long Term Evolution (LTE), fifth generation (5G), etc., and may also provide access to local area networks such as Wireless Fidelity (WiFi).

[0118] While the inventive concept has been particularly shown and described with reference to embodiments thereof, it should be understood that various changes in form and details may be made therein without departing from the spirit and scope of the appended claims and their equivalents.

Claims

1. A quantization method for an artificial neural network, the quantization method comprising: estimating a sample scaling factor of a first sample parameter, the first sample parameter being a portion of first parameters within the artificial neural network; determining a prediction scaling factor based on the sample scaling factor; as well as The first parameter is quantized based on the prediction scaling factor.

2. The quantization method according to claim 1, wherein: The first parameters include input variables of a first layer input into the artificial neural network and weights of the first layer.

3. The quantification method according to claim 1, wherein: Determining the prediction scaling factor includes determining the prediction scaling factor based on an exponential moving average of the sample scaling factors.

4. The quantification method according to claim 1, wherein: Determining the prediction scaling factor includes determining the prediction scaling factor based on an arithmetic mean of the sample scaling factors.

5. The quantization method according to claim 1, wherein: Determining the prediction scaling factor includes determining the prediction scaling factor based on a maximum value or a minimum value of the sample scaling factors.

6. The quantization method according to claim 1, wherein: Quantifying the first parameter includes: The first parameter is quantized by affine quantization or scaling quantization.

7. The quantification method according to claim 6, wherein: Quantifying the first parameter further includes: The first parameters are quantized by using the prediction scaling factors instead of their respective scaling factors.

8. A quantization system for an artificial neural network, the quantization system comprising: at least one processor; as well as a storage medium configured to store commands executable by the at least one processor to perform a quantization process of the artificial neural network, Wherein, the quantization process of the artificial neural network includes: estimating a sample scaling factor of a first sample parameter, the first sample parameter being a portion of first parameters within the artificial neural network; determining a prediction scaling factor based on the sample scaling factor; and The first parameter is quantized based on the prediction scaling factor.

9. The quantification system according to claim 8, wherein: The first parameters include input variables of a first layer input into the artificial neural network and weights of the first layer.

10. The quantification system according to claim 8, wherein: Determining the prediction scaling factor includes determining the prediction scaling factor based on an exponential moving average of the sample scaling factors.

11. The quantification system according to claim 8, wherein: Determining the prediction scaling factor includes determining the prediction scaling factor based on an arithmetic mean of the sample scaling factors.

12. The quantification system according to claim 8, wherein: Determining the prediction scaling factor includes determining the prediction scaling factor based on a maximum value or a minimum value of the sample scaling factors.

13. The quantification system according to claim 8, wherein: Quantizing the first parameter includes: quantizing the first parameter by affine quantization or scaling quantization.

14. The quantification system according to claim 8, wherein: The at least one processor is configured to perform the quantization process of the artificial neural network in real time as initial data is input to the at least one processor.

15. A quantization device for an artificial neural network, the quantization device comprising: a scaling factor estimator configured to estimate a sample scaling factor of a first sample parameter, the first sample parameter being a portion of first parameters within the artificial neural network, and the scaling factor estimator configured to determine a predicted scaling factor based on the sample scaling factor; as well as A quantizer is configured to quantize the first parameter based on the prediction scaling factor.

16. The quantization device according to claim 15, wherein: The first parameters include input variables of a first layer input into the artificial neural network and weights of the first layer.

17. The quantization device according to claim 15, wherein: The prediction scaling factor is determined based on an exponential moving average or an arithmetic mean of the sample scaling factors.

18. The quantization device according to claim 15, wherein: The prediction scaling factor is determined based on a maximum value or a minimum value of the sample scaling factors.

19. The quantization device according to claim 15, wherein: The quantizer is configured to quantize the first parameter by affine quantization or scaling quantization.

20. The quantization device according to claim 19, wherein: The quantizer is configured to quantize the first parameter by using the prediction scaling factor instead of the respective scaling factors of the first parameters.

Images