Neural network device and method for operating a neural network

Abstract

A neural network device and a method for operating a neural network are disclosed. A neural network device includes: a training device configured to: receive training image data, and pre-train a neural network based on the training image data, wherein the parameter value of at least one layer of the pre-trained neural network is an M-bit floating point type, and M is a natural number greater than 3; a quantization parameter calculator configured to: obtain the parameters of the pre-trained neural network, and quantize the parameters of the pre-trained neural network so that the quantized parameters have a mixed data type; analyze the statistical distribution of the M-bit floating point parameter values; obtain the quantization level of each parameter that statistically covers the distribution range of the parameter value based on the analyzed statistical distribution; and quantize the M-bit floating point weight to the N-bit floating point weight using the quantization parameter obtained based on the obtained quantization level of each parameter, where N is a natural number greater than 1 and less than M.

Description

[0001] This application is based on and claims the benefit of priority from Korean Patent Application No. 10-2019-0080543 filed on July 4, 2019, in the Korean Intellectual Property Office (KIPO), the disclosure of which is incorporated herein in its entirety by reference. Technical Field

[0002] The present disclosure relates to a neural network, and more particularly, to a neural network device using mixed data and a method for quantizing parameters of a neural network. Background Art

[0003] A neural network refers to a computing architecture that models a biological brain. Recently, with the development of neural network technology, various electronic systems have been actively studied to analyze input image data and extract effective information using a neural network device. A neural network device may use a large amount of calculation for complex input image data. In order for a neural network device to analyze high-quality image data and extract information in real time, a technology that can efficiently process neural network operations may be used.

[0004] Because typical neural network devices for image recognition may require a large number of operations and / or process the operations inefficiently, the image recognition performance of typical neural network devices for image recognition may be inefficient when implemented by low-power, high-performance embedded systems (such as, mobile devices (e.g., smart phones) with limited resources). Summary of the invention

[0005] According to an embodiment, a neural network device includes: a training device configured to: receive training image data, and pre-train a neural network based on the training image data, wherein a parameter value of at least one layer of the pre-trained neural network is an M-bit floating point type, and M is a natural number greater than 3; a quantization parameter calculator configured to: obtain parameters of the pre-trained neural network, and quantize the parameters of the pre-trained neural network so that the quantized parameters have a mixed data type; and a processor configured to apply the quantized parameters to the pre-trained neural network to obtain an optimized neural network, wherein the quantization parameter calculator is further configured to: analyze a statistical distribution of parameter values ​​of the M-bit floating point type; obtain a quantization level of each parameter that statistically covers a distribution range of the parameter values ​​based on the analyzed statistical distribution; and quantize an M-bit floating point weight into an N-bit floating point weight using a quantization parameter obtained based on the obtained quantization level of each parameter. N is a natural number greater than 1 and less than M.

[0006] The processor is also configured to: receive image data to be recognized, and input the image data to be recognized into the optimized neural network, so as to execute the optimized neural network to perform image recognition on the image data to be recognized.

[0007] The quantization parameter calculator is further configured to quantize the M-bit floating-point input data of each layer of the at least one layer of the optimized neural network into N-bit fixed-point asymmetric input data using the quantization parameter obtained based on the obtained quantization level of each parameter when the optimized neural network is executed.

[0008] The quantization parameter calculator is further configured to analyze the statistical distribution by obtaining statistics of parameter values ​​of M-bit floating-point input data, weights and biases during pre-training of the neural network, and wherein the processor is further configured to obtain a quantized neural network by applying N-bit fixed-point asymmetric input data and N-bit floating-point weights to the neural network.

[0009] The quantization parameter calculator is also configured to: obtain a first step size, a second step size, a first zero point, and a second zero point based on the obtained quantization level of each parameter, the maximum value and the minimum value of the analyzed statistical distribution, wherein the first step size is associated with the M-bit floating-point input data and corresponds to the first gap between the obtained quantization levels of each parameter, the second step size corresponds to the second gap between the quantization levels of the output data obtained by performing a node operation of the M-bit floating-point input data and weights, the first zero point indicates a zero value of the M-bit floating-point input data, and the second zero point indicates a zero value of the output data; obtain new weights based on the obtained first step size and the obtained second step size; and obtain a bias associated with the output data based on the obtained new weights.

[0010] The quantization parameter calculator is further configured to obtain asymmetric input data of the N-bit fixed-point type by applying the obtained first step size and the obtained second step size to the input data of the M-bit floating-point type.

[0011] The quantization parameter calculator is further configured to obtain the N-bit floating point type weight by applying the obtained first step size and the obtained second step size to the M-bit floating point type weight.

[0012] The N-bit floating point weights include an exponent bias.

[0013] The quantization parameter calculator is further configured to differently apply the exponential bias to a plurality of layers included in the neural network.

[0014] The quantization parameter calculator is further configured to obtain the weight of the N-bit floating point type using a multiple of 2 that is greater than a maximum value of the weight of the N-bit floating point type and is closest to the maximum value.

[0015] The quantization parameter calculator is further configured to obtain a new bias associated with the output data based on the unquantized bias, the obtained new weight, the obtained first zero point, and the obtained second step size.

[0016] The quantization parameter calculator is further configured to: analyze a statistical distribution of a parameter value of each of a plurality of nodes included in the at least one layer of the neural network; and quantize a weight associated with each of the plurality of nodes based on the analyzed statistical distribution.

[0017] The quantization parameter calculator includes: a logic circuit configured to quantize M-bit floating-point input data and weights into N-bit fixed-point asymmetric input data and N-bit floating-point weights using the quantization parameters; a control circuit configured to control the logic circuit; an input register and a weight register configured to store the quantization parameters; and a result register configured to store the N-bit fixed-point asymmetric input data and the N-bit floating-point weights.

[0018] The processor includes: a register configured to receive N-bit fixed-point asymmetric input data and N-bit floating-point weights from a quantization parameter calculator, and store the N-bit fixed-point asymmetric input data and the N-bit floating-point weights; a logic circuit configured to apply the N-bit fixed-point asymmetric input data and the N-bit floating-point weights to a neural network to obtain a quantized neural network; and a control circuit configured to control the logic circuit.

[0019] The statistical distribution is approximated by a normal distribution or a Laplace distribution, and wherein the quantization parameter calculator is further configured to obtain a quantization level of each parameter based on any one or any combination of a mean, variance, standard deviation, maximum value and minimum value of the parameter values ​​obtained from the analyzed statistical distribution.

[0020] The quantization parameter calculator is further configured to obtain a quantization level of each parameter based on any one or any combination of a mean value, a variance, a standard deviation, a maximum value and a minimum value of the parameter value to reduce the probability of a quantization error.

[0021] According to an embodiment, a neural network device includes: a training device configured to receive training image data and pre-train a neural network based on the training image data, wherein a parameter value of at least one layer of the pre-trained neural network is an M-bit floating point type, and M is a natural number greater than 3; a quantization parameter calculator configured to obtain parameters of the pre-trained neural network and quantize the parameters of the pre-trained neural network so that the quantized parameters have a mixed data type; and a processor configured to apply the quantized parameters to the pre-trained neural network to obtain an optimized neural network, wherein the quantization parameter calculator includes: a logic circuit configured to quantize an M-bit floating point weight into an N-bit floating point weight using a quantization parameter, and N is a natural number greater than 1 and less than M; a control circuit configured to control the logic circuit; an input register and a weight register configured to store the quantization parameter; and a result register configured to store the N-bit floating point weight.

[0022] The processor is also configured to: receive image data to be recognized, and input the image data to be recognized into the optimized neural network, so as to execute the optimized neural network to perform image recognition on the image data to be recognized.

[0023] The logic circuit is further configured to: when the optimized neural network is executed, use the quantization parameter to quantize the M-bit floating-point input data of each layer of the at least one layer of the optimized neural network into N-bit fixed-point asymmetric input data.

[0024] The quantized parameters include N-bit fixed-point asymmetric input data and N-bit floating-point weights.

[0025] According to an embodiment, a method of operating a neural network includes: receiving training image data, and pre-training the neural network based on the training image data, wherein a parameter value of at least one layer of the pre-trained neural network is an M-bit floating point type, and M is a natural number greater than 3; analyzing the statistical distribution of the M-bit floating point parameter values; obtaining a quantization level of each parameter that statistically covers a distribution range of the parameter values ​​based on the analyzed statistical distribution; quantizing the M-bit floating point weights into N-bit floating point weights using a quantization parameter obtained based on the obtained quantization level of each parameter, and N is a natural number greater than 1 and less than M; and applying the N-bit floating point weights to the pre-trained neural network to obtain an optimized neural network.

[0026] The method further includes: receiving image data to be recognized, and inputting the image data to be recognized into the optimized neural network, so as to execute the optimized neural network to perform image recognition on the image data to be recognized.

[0027] When performing the optimization neural network, the M-bit floating-point input data of each layer of the at least one layer of the optimization neural network is quantized into N-bit fixed-point asymmetric input data using a quantization parameter obtained based on the obtained quantization level of each parameter.

[0028] The step of analyzing the statistical distribution includes obtaining statistics of parameter values ​​of M-bit floating point input data, weights and biases during pre-training of the neural network.

[0029] The step of quantizing the weight of the M-bit floating point type includes: obtaining a first step length, a second step length, a first zero point, and a second zero point based on the obtained quantization level of each parameter, the maximum value and the minimum value of the analyzed statistical distribution, wherein the first step length is associated with the M-bit floating point input data and corresponds to a first gap between the obtained quantization levels of each parameter, the second step length corresponds to a second gap between the quantization levels of the output data obtained by performing a node operation of the M-bit floating point input data and the weight, the first zero point indicates a zero value of the M-bit floating point input data, and the second zero point indicates a zero value of the output data; obtaining a new weight based on the obtained first step length and the obtained second step length; and obtaining a bias associated with the output data based on the obtained new weight.

[0030] The statistical distribution is approximated by a normal distribution or a Laplace distribution, and wherein a quantization level of each parameter is obtained based on any one or any combination of a mean value, a variance, a standard deviation, a maximum value, and a minimum value of parameter values ​​obtained from the analyzed statistical distribution.

[0031] According to an embodiment, a neural network device includes: a quantization parameter calculator configured to quantize parameters of a pre-trained neural network so that the quantized parameters have a mixed data type; and a processor configured to apply the quantized parameters to the neural network. The quantization parameter calculator is further configured to: analyze the statistical distribution of parameter values ​​of M-bit floating point type, the parameter values ​​are associated with at least one layer of the neural network, M is a natural number greater than 3; obtain a quantization level of each parameter that statistically covers the distribution range of the parameter value based on the analyzed statistical distribution; and quantize the input data and weights of the M-bit floating point type into asymmetric input data of N-bit fixed point type and weights of N-bit floating point type using the quantization parameter obtained based on the obtained quantization level of each parameter, N is a natural number greater than 1 and less than M.

[0032] According to an embodiment, a neural network device includes: a quantization parameter calculator configured to quantize parameters of a pre-trained neural network so that the quantized parameters have a mixed data type; and a processor configured to apply the quantized parameters to the neural network. The quantization parameter calculator includes: a logic circuit configured to quantize M-bit floating-point input data and weights into N-bit fixed-point asymmetric input data and N-bit floating-point weights using a quantization parameter; a control circuit configured to control the logic circuit; an input register and a weight register configured to store the quantization parameter; and a result register configured to store the N-bit fixed-point asymmetric input data and the N-bit floating-point weight.

[0033] According to an embodiment, a method for quantizing parameters of a neural network includes: analyzing the statistical distribution of M-bit floating-point parameter values, the parameter values ​​are associated with at least one layer of the neural network, and M is a natural number greater than 3. The method also includes: obtaining a quantization level of each parameter that statistically covers the distribution range of the parameter value based on the analyzed statistical distribution; using a quantization parameter obtained based on the obtained quantization level of each parameter, quantizing the M-bit floating-point input data and weights into N-bit fixed-point asymmetric input data and N-bit floating-point weights, where N is a natural number greater than 1 and less than M. The method also includes: applying the N-bit fixed-point asymmetric input data and the N-bit floating-point weights to the neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flowchart illustrating a method of quantizing parameters of a neural network according to an embodiment.

[0035] Figure 2 is a block diagram illustrating a neural network device according to an embodiment.

[0036] Figure 3 and Figure 4 is a diagram for describing an example of a deep learning neural network structure driven by a neural network device according to an embodiment.

[0037] Figure 5A is a diagram showing a method according to an embodiment of the present invention. Figure 2 A block diagram of an example of a quantization parameter calculator in FIG.

[0038] Figure 5B is a diagram showing a method according to an embodiment of the present invention. Figure 2 A block diagram of one of the processors in FIG.

[0039] Figure 6 is a block diagram of a hardware configuration of a neural network device according to an embodiment.

[0040] Figure 7 is a diagram showing examples of floating-point values ​​and fixed-point values.

[0041] Figure 8 is a diagram showing an example of the relationship between the decimal length and precision of a fixed-point value.

[0042] Fig. 9 is a graph showing an example of the statistical distribution of parameters used in a layer.

[0043] Fig.10 is a graph showing another example of the statistical distribution of parameters used in a layer.

[0044] Fig.11 is a graph showing another example of the statistical distribution of parameters used in a layer.

[0045] Fig.12 is a diagram showing a method according to an embodiment of the present invention. Figure 2 An illustration of a neural network device that quantizes input data and weights.

[0046] Fig.13A Quantized input data provided from the quantization parameter calculator is shown.

[0047] Fig. 13B The quantization weight provided from the quantization parameter calculator is shown.

[0048] Fig.14 is a diagram showing the quantization according to the embodiment Figure 1 Flowchart of the operations on input data and weights in .

[0049] Fig.15 is a diagram illustrating an example of a quantization operation of a quantization parameter calculator according to an embodiment.

[0050] Fig.16 is a diagram illustrating another example of a quantization operation of a quantization parameter calculator according to an embodiment.

[0051] Fig.17 is a diagram showing a method according to an embodiment of the present invention. Figure 2 Schematic diagram of example operations of a processor in a neural network device.

[0052] Fig.18 It is shown Fig.17 Block diagram of the multiplier and arithmetic shifter in the operation of the processor.

[0053] Fig.19 is a block diagram illustrating a neural network system according to an embodiment. DETAILED DESCRIPTION

[0054] The embodiment provides a neural network device capable of increasing dynamic range and accuracy.

[0055] Embodiments provide a method of quantizing parameters of a neural network that can increase dynamic range and accuracy.

[0056] In detail, the quantization parameter calculator quantizes the input data (input image data) of the M-bit floating point type and the weight into the asymmetric input data of the N-bit fixed point type and the weight of the N-bit floating point type, respectively, based on the quantization level that minimizes the quantization error in advance. Here, the quantization of the input data of the M-bit floating point type and the quantization of the weight of the M-bit floating point type can be performed simultaneously or at different times. In other words, the quantization of the input data of the M-bit floating point type and the quantization of the weight of the M-bit floating point type can also be performed separately. It should be understood that when the quantized weights are applied to the neural network, the neural network is also quantized accordingly into a quantized neural network without the quantization of the input data. The processor applies the quantized input data and the quantized weights to the neural network. Therefore, the neural network device can increase the dynamic range and accuracy by separately quantizing and applying the quantized data to the neural network, and can improve the calculation speed and reduce power consumption. Therefore, the neural network to which the quantized data is applied can achieve high recognition performance (e.g., high recognition rate and / or recognition speed) when used to recognize image data.

[0057] Hereinafter, embodiments will be described more fully with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals always represent the same elements. Repeated descriptions may be omitted.

[0058] Figure 1 is a flowchart illustrating a method of quantizing parameters of a neural network according to an embodiment.

[0059] Reference Figure 1 , in operation S100, a statistical distribution of M-bit floating-point parameter values ​​is analyzed from data of a pre-trained floating-point neural network. The parameter value is associated with at least one layer of the neural network, and M is a natural number greater than 3. In other words, the parameter value of at least one layer of the pre-trained neural network is an M-bit floating-point type. A neural network is a machine learning scheme that uses many artificial neurons connected by connecting lines to simulate the operating model of software and / or hardware of a biological system among various machine learning schemes. A neural network uses artificial neurons that have the functions of simplified biological neurons, and the artificial neurons are connected by connecting lines with predetermined connection strengths to perform human learning or recognition. Recently, deep learning is being studied to overcome the limitations of neural networks. The following will refer to Figure 3 Describe the deep learning neural network architecture.

[0060] In one example, the pre-trained neural network can be obtained by the following steps: receiving training image data, and pre-training the neural network based on the training image data, wherein the parameter value of at least one layer of the pre-trained neural network is an M-bit floating point type, and M is a natural number greater than 3. For example, the above steps can be performed by a training device. The training device can be implemented in various ways (for example, a processor, etc.).

[0061] In operation S200, a quantization level of each parameter that statistically covers the distribution range of floating point parameter values ​​is determined based on the statistical distribution. Figures 9 to 11 Describe the determination of the quantification level.

[0062] In operation S300, the M-bit floating point weight is quantized into the N-bit floating point weight using a quantization parameter generated based on a quantization level. N is a natural number greater than 1 and less than M.

[0063] In operation S400, the quantized N-bit floating point weights are applied to a neural network, and a quantized neural network (also referred to as an optimized neural network) is generated.

[0064] In addition, optionally, the following steps may be performed: receiving image data to be recognized, and inputting the image data to be recognized into an optimized neural network to execute the optimized neural network to perform image recognition on the image data to be recognized. For example, the above steps may be performed by a processor. The output data (e.g., output activation) generated by the previous layer of the optimized neural network is output to the next layer of the previous layer as input data (e.g., input activation) of the next layer. For example, the first layer generates the output data of the first layer based on the image data to be recognized, and the second layer receives the output data of the first layer as input data.

[0065] As described above, the neural network is pre-trained or constructed such that the input data of at least one layer as a parameter value is an M-bit floating point type. In this case, when the optimization neural network is executed, the M-bit floating point type input data of each layer of at least one layer of the optimized neural network is quantized into an asymmetric input data of an N-bit fixed point type using a quantization parameter obtained based on the obtained quantization level of each parameter. For example, the above steps may be performed by a quantization parameter calculator. When the at least one layer of the optimized neural network includes the first layer of the optimized neural network, the M-bit floating point type input data of the first layer of the optimized neural network may correspond to the image data to be recognized of the M-bit floating point type.

[0066] In conventional schemes for quantizing floating-point input data and weights, it is difficult to improve both precision and dynamic range. In order to increase precision, the dynamic range is degraded, and in order to increase the dynamic range, the precision is degraded. However, according to an example, the M-bit floating-point input data and weights are quantized into N-bit fixed-point asymmetric input data and N-bit floating-point weights in advance, and the quantized N-bit fixed-point asymmetric input data and the quantized N-bit floating-point weights are applied to a neural network to generate a quantized neural network. Therefore, both precision and dynamic range can be improved.

[0067] Figure 2 is a block diagram illustrating a neural network device according to an embodiment.

[0068] Reference Figure 2 , the neural network device 100 may include a plurality of processors 110 , a task manager TKMNG 120 , a quantization parameter calculator QPCL 130 , and a memory MEM 160 .

[0069] The neural network device 100 may be driven by a plurality of processors 110. For example, the plurality of processors 110 may include Figure 2 The heterogeneous processors shown in . According to an example, the plurality of processors 110 may include at least two homogeneous processors. Various services (e.g., tasks TK or applications) such as image classification services, user authentication services, and / or advanced driver assistance system (ADAS) services may be executed and processed by the plurality of processors 110. Task TK may include any one or any combination of a plurality of operations and a plurality of arithmetic operations. For example, task TK may represent an application (such as an image classification service, a user authentication service based on biometric information, an ADAS service, etc.). For example, a plurality of operations may include various operations (such as a convolution operation, a rectified linear unit (RELU) operation, etc.).

[0070] According to an embodiment, at least one of the plurality of processors 110 and the quantization parameter calculator QPCL 130 may be implemented as various numbers of hardware structures, software structures, and / or firmware structures that perform the various functions described herein, respectively. For example, they may use a direct circuit structure (such as a memory, a processor, a logic circuit, a lookup table, etc.) that can perform various functions under the control of one or more microprocessors or other control devices. In addition, they may be specifically implemented by a module, a program, or a portion of code, which contains one or more executable instructions for performing a specified logical function and executed by one or more microprocessors or other control devices. In addition, they may include a microprocessor (such as a central processing unit (CPU)) that performs various functions, or may be implemented by a microprocessor (such as a central processing unit (CPU)) that performs various functions. They may be combined into a single component that performs all operations or functions to be performed by them, respectively.

[0071] According to an embodiment, the plurality of processors 110 may include a central processing unit (CPU) 111 , a graphics processing unit (GPU) 112 , a neural processing unit (NPU) 113 , a digital signal processor (DSP) 114 , an image signal processor (ISP) 115 , and dedicated hardware (DHW) 116 .

[0072] For example, the specialized hardware 116 may include a visual processing unit (VPU), a visual intellectual property (VIP), etc. Each processor may be referred to as a processing element (PE).

[0073] although Figure 2Computing resources are shown only as an example of multiple processors 110, but multiple processors 110 may also include communication resources (such as a direct memory access unit (DMA) for controlling access to memory 160, connectivity for supporting various internal communications and / or external communication connections, etc.).

[0074] The task manager 120 receives a task TK to be executed from an external device or receives a task TK to be executed through a user, manages or schedules the received task TK, and distributes the task TK to a plurality of processors 110. For example, the task manager 120 may distribute operations included in the task TK to a plurality of processors 110, and generate path information indicating a calculation path of the task TK. The calculation path of the task TK may include: a sequence of operations included in the task TK and a driving sequence of processors for executing the operations included in the task TK.

[0075] The quantization parameter calculator 130 may analyze the statistical distribution of M-bit floating-point parameter values ​​associated with at least one layer of the neural network from the data of the pre-trained floating-point neural network, may determine the quantization level of each parameter that statistically covers the distribution range of the floating-point parameter values ​​based on the statistical distribution, and use the quantization parameter generated based on the quantization level to quantize the M-bit floating-point input data and weight into N-bit fixed-point asymmetric input data and N-bit floating-point weight, respectively, and may provide the processor 110 with a mixed quantization parameter MXPR including the quantized N-bit fixed-point asymmetric input data and the quantized N-bit floating-point weight. That is, the quantization parameter calculator 130 may perform Figure 1 In one non-limiting example, the quantization parameter calculator 130 may analyze the statistical distribution of parameter values ​​of each of a plurality of nodes included in at least one layer of the neural network, and quantize the input data and weights associated with each of the plurality of nodes based on the analyzed statistical distribution.

[0076] Any one or any combination of processors 110 may apply quantized N-bit fixed-point asymmetric input data and quantized N-bit floating-point weights to the neural network. That is, any one or any combination of processors 110 may execute Figure 1 Operation S400 in .

[0077] Memory 160 may store various data processed by neural network device 100. In an example, memory 160 may include at least one volatile memory (such as dynamic random access memory (DRAM), synchronous DRAM (SDRAM), static random access memory (SRAM), etc.) and / or at least one non-volatile memory (such as electrically erasable programmable read-only memory (EEPROM), flash memory, phase change random access memory (PRAM), resistive random access memory (RRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), nano floating gate memory (NFGM), or polymer random access memory (PoRAM), etc.).

[0078] although Figure 2 Only data / signal transmission flows between some elements in neural network device 100 are shown, but all elements in neural network device 100 may be connected to each other via at least one bus, and thus all elements in neural network device 100 may communicate with each other via at least one bus.

[0079] Neural network device 100 may also include software elements (e.g., frameworks, kernels or device drivers, middleware, application programming interfaces (APIs), applications or apps, etc.) At least a portion of the software elements may be referred to as an operating system (OS).

[0080] Figure 3 and Figure 4 is a diagram for describing an example of a deep learning neural network structure driven by a neural network device according to an embodiment.

[0081] Reference Figure 3 A general neural network may include an input layer IL, multiple hidden layers HL1, HL2, ..., HLn and an output layer OL.

[0082] The input layer IL may include i input nodes x 1 、x 2 , …, x i , where i is a positive integer. Input data (eg, vector input data) IDAT of length i may be input to the input node x 1 、x 2 , …, x i , so that each element of the input data IDAT is input to the input node x 1 、x 2 , …, x i The corresponding node in .

[0083] The plurality of hidden layers HL1, HL2, ..., HLn may include n hidden layers (where n is a positive integer) and may include a plurality of hidden nodes h 11 、h 1 2 、h 1 3 ,…,h 1 m 、h 2 1 、h 2 2 、h 2 3 ,…,h 2 m 、h n 1 、h n 2 、h n 3 ,…,h n m For example, the hidden layer HL1 may include m hidden nodes h 1 1 、h 1 2 、h 1 3 ,…,h 1 m , the hidden layer HL2 may include m hidden nodes h 2 1 、h 2 2 、h 2 3 ,…,h 2 m , the hidden layer HLn may include m hidden nodes h n 1 、h n 2 、h n 3 ,…,h n m (where m is a positive integer).

[0084] The output layer OL may include j output nodes y 1 ,y 2 ,...,y j (where j is a positive integer). Output node y 1 ,y 2 ,...,y j Each of may correspond to a corresponding category in the categories to be classified. The output layer OL may output an output value ODAT (e.g., a category score or, simply, a score) associated with the input data IDAT for each category. The output layer OL may be referred to as a fully connected layer and may indicate, for example, the probability that the input data IDAT corresponds to a car.

[0085] Figure 3 The structure of the neural network shown in can be represented by information about branches (e.g., branches w111, wi1m, wn11, and wnmj) (or connections) between nodes shown as lines and weight values ​​assigned to each branch. Nodes within one layer may not be connected to each other, but nodes of different layers may be completely or partially connected to each other.

[0086] Each node (for example, node h 1 1 ) can receive the previous node (for example, node x 1 ), may perform a computation, operation, or calculation on the received output, and may output the result of the computation, operation, or calculation as a signal to the next node (e.g., node h 2 1 ) output. Each node can calculate a value to be output by applying a function (e.g., a nonlinear function) to the input.

[0087] Generally, the structure of the neural network is set in advance, and the weight values ​​for the connections between nodes are appropriately set using the data with known answers to which category the data belongs. The data with known answers is called "training data", and the process of determining the weight values ​​is called "training". The neural network "learns" during the training process. The set of independently trainable structures and weight values ​​is called a "model", and the process of predicting which category the input data belongs to by the model with determined weight values ​​and then outputting the predicted value is called a "testing" process.

[0088] Because each node (for example, node h 1 1 ) is connected to all nodes in the previous layer (e.g., including node x in layer IL 1 、x 2 , …, x i ), then as the size of the input image data increases, the number of weight values ​​increases dramatically, so Figure 3 The general neural network shown in may not be suitable for processing input image data. Therefore, a convolutional neural network implemented by combining a filtering technique with a general neural network has been studied so that a two-dimensional image (eg, input image data) can be efficiently trained by the convolutional neural network.

[0089] Reference Figure 4 , a convolutional neural network may include multiple layers CONV1, RELU1, CONV2, RELU2, POOL1, CONV3, RELU3, CONV4, RELU4, POOL2, CONV5, RELU5, CONV6, RELU6, POOL3 and FC.

[0090] Unlike general neural networks, each layer of a convolutional neural network can have three dimensions of width, height, and depth, so the data input to each layer can be volume data with three dimensions of width, height, and depth. For example, if Figure 4 If the input image in has a size of 32 widths (eg, 32 pixels) and 32 heights and three color channels R, G, and B, the input data IDAT corresponding to the input image may have a size of 32×32×3. Figure 4 The input data IDAT in may be referred to as input volume data or input activation volume.

[0091] Each of the convolution layers CONV1, CONV2, CONV3, CONV4, CONV5, and CONV6 may perform a convolution operation on the input volume data. In image processing, a convolution operation refers to an operation that processes image data based on a mask having a weight value and obtains an output value by multiplying the input value by the weight value and adding all the multiplied values. The mask may be referred to as a filter, a window, or a kernel.

[0092] The parameters of each convolutional layer may consist of a set of learnable filters. Each filter may be small in space (along width and height) but may extend through the entire depth of the input volume. For example, during a forward pass, each filter may be slid (more precisely, convolved) across the width and height of the input volume, and a dot product may be calculated between the entries of the filter and the input at any position. As the filter slides across the width and height of the input volume, a two-dimensional activation map may be generated that gives the response of the filter at each spatial position. As a result, an output volume may be generated by stacking these activation maps along the depth dimension. For example, if input volume data having a size of 32×32×3 passes through a convolutional layer CONV1 having four filters with zero padding, the output volume data of the convolutional layer CONV1 may have a size of 32×32×12 (e.g., the depth of the volume data increases).

[0093] Each of the RELU layers RELU1, RELU2, RELU3, RELU4, RELU5, and RELU6 may perform a rectified linear unit operation corresponding to an activation function defined by, for example, a function f(x)=max(0, x) (e.g., for all negative inputs x, the output is zero). For example, if input volume data having a size of 32×32×12 passes through the RELU layer RELU1 to perform the rectified linear unit operation, the output volume data of the RELU layer RELU1 may have a size of 32×32×12 (e.g., the size of the volume data is maintained).

[0094] Each of the pooling layers POOL1, POOL2, and POOL3 may perform a downsampling operation on the input volume data along the spatial dimensions of width and height. For example, four input values ​​arranged in a 2×2 matrix structure (formation) may be converted into one output value based on a 2×2 filter. For example, the maximum value of the four input values ​​arranged in a 2×2 matrix structure may be selected based on 2×2 maximum pooling, or the average value of the four input values ​​arranged in a 2×2 matrix structure may be obtained based on 2×2 average pooling. For example, if input volume data having a size of 32×32×12 passes through a pooling layer POOL1 having a 2×2 filter, the output volume data of the pooling layer POOL1 may have a size of 16×16×12 (e.g., the width and height of the volume data are reduced, and the depth of the volume data is maintained).

[0095] A convolutional layer (e.g., CONV1) and a RELU layer (e.g., RELU1) can form a pair of CONV layers / RELU layers in a convolutional neural network, multiple pairs of CONV layers / RELU layers can be repeatedly arranged in the convolutional neural network, and pooling layers can be periodically inserted in the convolutional neural network, thereby reducing the spatial size of the image and extracting the characteristics of the image.

[0096] The output layer or fully connected layer FC may output the result (e.g., class score) of the input volume data IDAT for each class. For example, as the convolution operation and the downsampling operation are repeated, the input volume data IDAT corresponding to the two-dimensional image may be converted into a one-dimensional matrix or a one-dimensional vector. For example, the fully connected layer FC may represent the probability that the input volume data IDAT corresponds to a car, a truck, an airplane, a boat, and a horse, respectively.

[0097] The type and number of layers included in the convolutional neural network may not be limited to reference Figure 4 The example described above is not limited to the example described above and may be changed. In addition, the convolutional neural network may further include other layers (such as a softmax layer for converting a score value corresponding to a predicted result into a probability value, a bias adding layer for adding at least one bias, etc.).

[0098] In this way, the deep learning neural network may include multiple layers, and the fixed-point format may be determined independently for each of the multiple layers. The nodes in each layer may perform quantization based on the same fixed-point format. When the artificial neural network (ANN) includes multiple layers, a virtual overflow detection circuit (not shown) included in the ANN may generate multiple virtual overflow information corresponding to the multiple layers, respectively, and the quantization parameter calculator 130 may determine the fixed-point format for the next quantization for each of the multiple layers based on the multiple virtual overflow information.

[0099] Figure 5Ais a diagram showing a method according to an embodiment of the present invention. Figure 2 A block diagram of an example of a quantization parameter calculator in FIG.

[0100] Reference Figure 5A , the quantization parameter calculator 130 may include a control circuit CTRL 131 , a logic circuit LOG 132 , an embedded memory eMEM 133 , an input register REGI 134 , a weight register REGW 135 , and a result register REGR 136 .

[0101] The control circuit 131 controls the overall operation of the quantization parameter calculator 130. The control circuit 131 may control the flow of instructions and data for quantizing parameters of a neural network.

[0102] The embedded memory 133 may store instructions and data of the quantization parameter calculator 130. Input data and weight values ​​to be quantized may be stored in the input register 134 and the weight register 135, respectively.

[0103] The logic circuit 132 can analyze the statistical distribution of M-bit floating-point parameter values ​​associated with at least one layer of the neural network from the data of the pre-trained floating-point neural network, determine the quantization level of each parameter that statistically covers the distribution range of the floating-point parameter values ​​based on the statistical distribution, and use the quantization parameter generated based on the quantization level to quantize the M-bit floating-point input data and weights into N-bit fixed-point asymmetric input data and N-bit floating-point weights, respectively, based on the input data and weight values ​​stored in the input register 134 and the weight register 135.

[0104] The logic circuit 132 may store the quantization parameter in the input register 134 and the weight register 135, and may store the quantized N-bit fixed-point type asymmetric input data and the quantized N-bit floating-point type weight in the result register 136. The logic circuit 132 may be connected to the input register 134 and the weight register 135, and may also be connected to the result register 136.

[0105] The control circuit 131 may control the result register 136 to provide the quantized N-bit fixed-point asymmetric input data and the quantized N-bit floating-point weight to one of the processors 110 .

[0106] The input register 134 and the weight register 135 may be implemented using a memory having a faster operating speed than the embedded memory 133 , and the input data and the weight may be loaded from the embedded memory 133 to the input register 134 and the weight register 135 .

[0107] Figure 5B is a diagram showing a method according to an embodiment of the present invention. Figure 2 A block diagram of one of the processors in FIG.

[0108] Figure 5B An example of a neural processor 113 is shown; other processors may have a similar or identical configuration to the neural processor 113 .

[0109] Reference Figure 5B , the neural processor (or processor) 113 may include a control circuit CTRL 171 , a logic circuit LOG 172 , an embedded memory eMEM 173 and a register REG 174 .

[0110] The control circuit 171 controls the overall operation of the processor 113. The control circuit 171 may control the flow of instructions and data for quantizing the parameters of the neural network.

[0111] The embedded memory 173 may store instructions and data of the processor 113. The register 174 may store quantized N-bit fixed-point type asymmetric input data and quantized N-bit floating-point type weights provided from the result register 136.

[0112] The logic circuit 172 may be connected to the register 174 and may apply the quantized N-bit fixed-point type asymmetric input data and the quantized N-bit floating-point type weight to the neural network to generate a quantized neural network QNN.

[0113] Figure 6 is a block diagram of a hardware configuration of a neural network device according to an embodiment.

[0114] Reference Figure 6 , the neural network device 100 may include a processor 113 , a quantization parameter calculator 130 , and a memory 160 .

[0115] Neural network device 100 may correspond to a computing device having various processing functions (such as generating a neural network, training (or learning) a neural network, quantizing a floating-point neural network into a mixed-data neural network including fixed-point data and floating-point data, or retraining a neural network).

[0116] The processor 113 may repeatedly train (learn) a given initial neural network to generate a pre-trained neural network 161. The initial neural network may have floating point parameters (e.g., parameters with 32-bit floating point precision) to ensure the processing accuracy of the neural network. The floating point parameters may include, for example, various types of data input to / output from the neural network (such as input / output activations, weights, and biases of the neural network). As the repeated training of the neural network proceeds, the floating point parameters of the neural network may be adjusted to calculate a more accurate output for a given input.

[0117] Compared with fixed point, floating point can use a relatively large amount of calculation and high memory access frequency. Therefore, in mobile devices (such as smart phones, tablet PCs, wearable devices, etc.) and embedded devices with relatively low processing performance, the processing of neural networks with floating point parameters may not be smooth. Therefore, in order to drive the neural network with an acceptable loss of accuracy while substantially reducing the amount of calculation in such devices, the floating point parameters processed in the neural network can be quantized.

[0118] The neural network device 100 may perform quantization 163 to convert the parameters of the trained neural network into a predetermined fixed-point type and floating-point type in consideration of the processing performance of the device (e.g., a mobile device, an embedded device, etc.) on which the neural network will be deployed. In addition, the neural network device 100 may send the quantized neural network QNN to the device on which the neural network will be deployed. The device on which the neural network will be deployed may be, for example, an autonomous driving vehicle, a robot, a smart phone, a tablet device, an augmented reality (AR) device, an Internet of Things (IoT) device, etc. that performs image recognition using a neural network.

[0119] The quantization parameter calculator 130 may obtain data of the pre-trained neural network 161 using the floating point data stored in the memory 160. The pre-trained neural network data may be data repeatedly trained using floating point type parameters. The neural network training may be repeatedly performed by receiving the training set data as input, and then the neural network training may be repeatedly performed again using the test set data, but is not limited thereto. The training set data is input data for training the neural network, the test set data is input data that does not overlap with the training set data, and the test set data may train the neural network trained using the training set data while measuring the performance of the neural network.

[0120] The quantization parameter calculator 130 may analyze the statistical distribution of M-bit floating point parameter values ​​associated with at least one layer of the neural network from the data of the pre-trained neural network 161. The quantization parameter calculator 130 may analyze the statistical distribution by obtaining statistics for each channel of floating point parameter values ​​of weights, input activations, and output activations used in each layer during pre-training of the neural network. In a non-limiting example, the quantization parameter calculator 130 may analyze the statistical distribution by obtaining statistics of parameter values ​​of M-bit floating point input data, weights, and biases.

[0121] The quantization parameter calculator 130 may determine the quantization level of each parameter that statistically covers the distribution range of the floating-point parameter value based on the statistical distribution, and use the quantization parameter generated based on the quantization level to quantize the M-bit floating-point input data and weight into N-bit fixed-point asymmetric input data and N-bit floating-point weight (163). The quantization parameter calculator 130 may provide the processor 113 with a mixed parameter MXPR including the quantized N-bit fixed-point asymmetric input data and the quantized N-bit floating-point weight.

[0122] The processor 113 may apply the quantized N-bit fixed-point asymmetric input data and the quantized N-bit floating-point weights to the pre-trained neural network 161 to generate a quantized neural network QNN.

[0123] The memory 160 may store data sets related to the neural network (e.g., data of an untrained initial neural network, data of a neural network generated in a training process, data of a neural network that has completed training, and data of a quantized neural network) that have been processed or will be processed by the processor 113 and / or the quantization parameter calculator 130. In addition, the memory 160 may store various programs related to a training algorithm and a quantization algorithm of a neural network to be executed by the processor 113 and / or the quantization parameter calculator 130.

[0124] Figure 7 is a diagram showing examples of floating-point values ​​and fixed-point values.

[0125] Reference Figure 7 , the floating point value 510 is represented as "a×2 b ”, where “a” is the fraction part and “b” is the exponent part, where a and b are natural numbers. The floating point value 510 is represented by 32 bits including a 1-bit sign part, an 8-bit exponent part, and a 23-bit fraction part.

[0126] In addition, the fixed-point value 520 is represented by "Qm.n", where m and n are natural numbers. In the expression "Qm.n", "m" represents the number of digits indicating the exponent part, and "n" represents the number of digits indicating the fractional part. Therefore, the bit width of the fixed-point value is obtained by summing the 1-bit sign part, the m-bit exponent part, and the n-bit fractional part (1+m+n). Because the bits indicating the fractional part in the fixed-point bit are n bits, the fractional length is n. For example, "Q3.4" is a total of 8-bit fixed-point values ​​including a 1-bit sign part, a 3-bit exponent part, and a 4-bit fractional part, "Q1.30" is a total of 32-bit fixed-point values ​​including a 1-bit sign part, a 1-bit exponent part, and a 30-bit fractional part, and "Q15.16" is a total of 32-bit fixed-point values ​​including a 1-bit sign part, a 15-bit exponent part, and a 16-bit fractional part.

[0127] Figure 8 is a diagram showing an example of the relationship between the decimal length and precision of a fixed-point value.

[0128] Reference Figure 8 , assuming that the total bit width allocated to the fixed-point value is 3 bits, the fixed-point expression 530 of Q2.0 with a decimal length of 0 is compared with the fixed-point expression 540 of Q1.1 with a decimal length of 1.

[0129] For Q2.0, because the exponent part is 2 bits and the fraction part is 0 bits, fixed-point values ​​from -4 to 3 can be represented, and the interval between possible fixed-point values ​​is 1. For Q1.1, because the exponent part is 1 bit and the fraction part is 1 bit, fixed-point values ​​from -2 to 1.5 can be represented, and the interval between possible fixed-point values ​​is 0.5.

[0130] From the comparison, it can be seen that although 3 bits are allocated to both the fixed-point expression 530 of Q2.0 and the fixed-point expression 540 of Q1.1, Q2.0 can express a wider range of fixed-point values ​​than Q1.1, but because the intervals between the fixed-point values ​​are wider, Q2.0 has lower precision. On the other hand, Q1.1 can express a narrower range of fixed-point values ​​than Q2.0, but because the intervals between the fixed-point values ​​are narrower, Q1.1 has higher precision. Therefore, it can be seen that the precision of the fixed-point value depends on the decimal length of the fixed-point value (i.e., the number of decimal places allocated to the fixed-point value).

[0131] Since the neural network device 100 quantizes M-bit floating-point input data into N-bit fixed-point input data and the quantized fixed-point input data is asymmetric, the neural network device 100 can improve accuracy and dynamic range.

[0132] Fig. 9 is a graph showing an example of the statistical distribution of parameters used in a layer.

[0133] Reference Fig. 9 , after repeatedly training a neural network with floating-point parameters, a distribution of intrinsic floating-point values ​​(i.e., parameter values) is generated for the layers. Figure 5A The quantization parameter calculator 130 analyzes the statistical distribution of at least one layer from the pre-trained neural network data based on the statistics of the floating-point parameter values ​​of each channel. In detail, the quantization parameter calculator 130 obtains the statistics of the floating-point parameter values ​​of the weights, input activations, and output activations from the pre-trained neural network data, and normalizes the statistics of the layer using the probability density function (PDF) of the normal (Gaussian) distribution 550. However, although for ease of explanation, Fig. 9An example is shown in which the quantization parameter calculator 130 normalizes statistics using the PDF of the normal distribution 550, but the quantization parameter calculator 130 is not limited thereto. In other words, the quantization parameter calculator 130 may analyze statistics by using various types of statistical distributions or probability distributions other than the normal distribution 550. Fig. 9 The normal distribution 550 of may be the distribution of floating point activation values ​​in a layer or the distribution of floating point weight values ​​in a layer.

[0134] The quantization parameter calculator 130 may determine a quantization level based on the normal distribution 550 so that the parameters of the layer are quantized into a point expression with a decimal length. In detail, the quantization parameter calculator 130 obtains a statistical maximum value Xmax and a statistical minimum value Xmin of the parameters in the normal distribution 550, and determines a quantization level K that can statistically cover floating-point values ​​in a range between the statistical maximum value Xmax and the statistical minimum value Xmin. In a non-limiting example, the quantization parameter calculator 130 may obtain a quantization level for each parameter based on any one or any combination of a mean value, a variance, a standard deviation, a maximum value, and a minimum value of the parameter values ​​obtained from the analyzed statistical distribution.

[0135] The statistical maximum value Xmax and the statistical minimum value Xmin are values ​​based on the range in which the quantization error is minimized. The statistical maximum value Xmax and the statistical minimum value Xmin can be defined in various ways. For example, the statistical maximum value Xmax and the statistical minimum value Xmin can be the actual maximum parameter value and the actual minimum parameter value of the parameter in the normal distribution 550. Alternatively, the statistical maximum value Xmax and the statistical minimum value Xmin can be the upper and lower limits obtained based on the mean value, variance or standard deviation σ1 of the normal distribution 550. For example, in a non-limiting example, the statistical maximum value Xmax may correspond to 3σ1, however, the present invention is not limited thereto.

[0136] Fig.10 is a graph showing another example of the statistical distribution of parameters used in a layer.

[0137] Reference Fig.10 ,replace Fig. 9 The quantization parameter calculator 130 uses the PDF of the Laplace distribution 560 to normalize the statistics of each layer. Fig. 9 As described above, the quantization parameter calculator 130 determines the quantization level of the parameter of the corresponding layer by obtaining the statistical maximum value Xmax and the statistical minimum value Xmin applicable to the Laplace distribution 560 .

[0138] Fig.11 is a graph showing another example of the statistical distribution of parameters used in a layer.

[0139] Reference Fig.11 , the quantization parameter calculator 130 normalizes the statistics of each layer using the PDF of the normal distribution 570. Fig. 9 As described above, the quantization parameter calculator 130 determines the quantization level of the parameter of the corresponding layer by obtaining the statistical maximum value Ymax and the statistical minimum value Ymin applicable to the normal distribution 570. The statistical maximum value Ymax and the statistical minimum value Ymin may be upper and lower limits obtained based on the mean value, variance or standard deviation σ2 of the normal distribution 570.

[0140] Fig.12 is a diagram showing a method according to an embodiment of the present invention. Figure 2 An illustration of a neural network device that quantizes input data and weights.

[0141] exist Fig.12 In the example, assume that M is 32 and N is 8.

[0142] Reference Fig.12 , the quantization parameter calculator 130 may quantize the input data IDTA and the weight WDTA of the M-bit floating point type (32-bit floating point (FLP)) into the asymmetric input data QIDTA of the N-bit fixed point type (8-bit asymmetric (ASM) fixed point (FXP)) and the weight QWDTA of the N-bit floating point type (8-bit FLP), respectively. The processor 113 may apply the quantized asymmetric input data QIDTA of the N-bit fixed point type (8-bit ASM FXP) and the quantized weight QWDTA of the N-bit floating point type (8-bit FLP) to the neural network to generate the quantized neural network QNN.

[0143] Fig.13A Quantized input data provided from the quantization parameter calculator is shown.

[0144] Reference Fig.13A , the quantization parameter calculator 130 may quantize the M-bit floating point input data IDTA into the N-bit fixed point asymmetric input data QIDTA. The M-bit floating point input data IDTA may be represented as e×2 f , the N-bit fixed-point asymmetric input data QIDTA can be expressed as Qm.n.

[0145] The sum of the fractional part e and the exponent part f of the M-bit floating-point input data IDTA corresponds to M-1, and m+n of the N-bit fixed-point asymmetric input data QIDTA corresponds to N-1. Although it is assumed that M is 32 and N is 8, the values ​​of M and N are not limited to this. Fig. 9 The statistical maximum value Xmax and the statistical minimum value Xmin are asymmetric with respect to each other, so the quantized input data QIDTA is asymmetric. When the quantized input data QIDTA is represented asymmetrically, the input data IDTA can be represented more accurately and the accuracy can be improved.

[0146] Fig. 13B The quantization weight provided from the quantization parameter calculator is shown.

[0147] Reference Fig. 13B , the quantized weight Wnew may include a fractional part a' and an exponential part b', and may also include an exponential bias bse.

[0148] When the quantization parameter calculator 130 quantizes the weight of the M-bit floating point type into the weight of the N-bit floating point type, the sum of the number of bits in the fraction part a' and the number of bits in the exponent part b' corresponds to N-1.

[0149] Because the quantized weight Wnew includes the exponential bias bse, the weight WDAT can be represented with a high dynamic range.

[0150] Before the negative weights and the positive weights are quantized, the exponential bias BSE may be applied differently to the negative weights and the positive weights. In a non-limiting example, the exponential bias BSE may be applied differently to a plurality of layers included in a neural network.

[0151] Fig.14 is a diagram showing the quantization according to the embodiment Figure 1 Flowchart of the operations on input data and weights in .

[0152] Reference Figure 5A as well as Figures 6 to 14 , in order to quantize the input data and weights ( Figure 1 In operation S300 of the present invention, in operation S310, the quantization parameter calculator 130 may determine a first step size, a second step size, a first zero point, and a second zero point based on the quantization level, the maximum value of the statistical distribution, and the minimum value of the statistical distribution. The first step size is associated with the input data and corresponds to a gap between quantization levels, the second step size corresponds to a gap between quantization levels of output data obtained by performing a node operation of the input data and the weight, the first zero point indicates a zero value of the input data, and the second zero point indicates a zero value of the output data.

[0153] In operation S320, the quantization parameter calculator 130 may determine a new weight based on the first step size and the second step size. In operation S330, the quantization parameter calculator 130 may determine a bias associated with the output data based on the new weight.

[0154] The quantization parameter calculator 130 is provided by Based on the statistical maximum X max and statistical minimum X min Determine the first step length Δ x , the first step length Δ x is associated with the input data x and corresponds to the gaps between quantization levels.

[0155] Here, K corresponds to the quantization level.

[0156] The input data x can be quantized by Expression 1 below.

[0157] [Expression 1]

[0158] x=Δ x (q x -z x )

[0159] Here, q x represents the quantized input data, z x Indicates the first zero point.

[0160] The quantization parameter calculator 130 is provided by Use the maximum value Y of the output data y max And the minimum value Y of the output data y min , to determine the second step length Δ y , which is used to analyze the statistics of the next layer.

[0161] The output data y can be quantized by the following Expression 2.

[0162] [Expression 2]

[0163] y=Δ y (q y -z y )=∑ c，k {w c，k x c，k}+bias

[0164] Here, c represents the c-th layer, k represents the k-th node in the c-th layer, bias represents the bias associated with the node, and z y represents the second zero point, q y Represents quantized output data.

[0165] The quantization parameter calculator 130 calculates the following expression 3 based on the first step length Δ x and the second step length Δ y Generate new weights W new .

[0166] [Expression 3]

[0167]

[0168] Here, it can be expressed by the following expression 4

[0169] [Expression 4]

[0170]

[0171] Here, if through Indicates the new bias new , then the following expression 5 is derived, and the quantization parameter calculator 130 generates the quantized offset

[0172] [Expression 5]

[0173] Here, round is a rounding function for rounding a predetermined number. In Expression 5, a rounding function may be used instead of the rounding function, which is a function that takes a real number as input and gives as output the maximum integer less than or equal to the real number.

[0174] Therefore, the quantization parameter calculator 130 may calculate the value of the new weight W based on the bias, new , the first zero point z x and the second step length Δ y To generate a new bias associated with the output data new In addition, the quantized output q can be generated by the following expression 6 y .

[0175] [Expression 6]

[0176]

[0177] If the new weight q is quantized by expression 7 w , then Expression 6 is derived as Expression 8.

[0178] [Expression 7]

[0179] q w =-1 符号 ×2 3位指数-bse ×4 bits with hidden bits in the mantissa

[0180] Here, bse represents exponential bias.

[0181] [Expression 8]

[0182]

[0183] The quantization parameter calculator 130 may stretch the weights before quantizing them to achieve higher accuracy. The stretched weights are expressed by Expression 9 below.

[0184] [Expression 9]

[0185]

[0186] Here, Sw corresponds to a multiple of 2, and w stretched Corresponding to the expanded weight, the multiple of 2 is greater than the new weight w new The maximum value of and is closest to the maximum value The quantization parameter calculator 130 generates the weight of the quantized N-bit floating point type by using a multiple of 2 which is greater than a maximum value of the weight of the quantized N-bit floating point type and is closest to the maximum value.

[0187] The quantization parameter calculator 130 may analyze statistical distribution of parameter values ​​of each node included in at least one layer, and may quantize input data and weight of each node based on the statistical distribution.

[0188] Fig.15 is a diagram illustrating an example of a quantization operation of a quantization parameter calculator according to an embodiment.

[0189] Reference Fig.15 , the quantization parameter calculator 130 receives input data of a 32-bit floating point type in operation S612, collects samples from the input data in operation S614, and calculates a quantization step size based on a statistical distribution of the collected samples in operation S616. The quantization parameter calculator 130 receives a weight of a 32-bit floating point type in operation S618, calculates a new weight based on the quantization step size in operation S622, and calculates a new bias in operation S624. The quantization parameter calculator 130 calculates a zero point based on the quantization step size in operation S626, calculates a minimum value of a statistical distribution representing a true zero point in operation S632, and calculates quantized asymmetric input data of an 8-bit fixed point type in step S634. The quantization parameter calculator 130 calculates an exponential bias based on the new weight in operation S628, and calculates a quantized 8-bit floating point type weight in operation S636.

[0190] Fig.16 is a diagram illustrating another example of a quantization operation of a quantization parameter calculator according to an embodiment.

[0191] Fig.16 and Fig.15 The difference is that operation S627 of extending the weight is added between operations S622 and S628, and the detailed description will be omitted.

[0192] Fig.17 is a diagram showing a method according to an embodiment of the present invention. Figure 2 Schematic diagram of example operations of a processor in a neural network device.

[0193] Fig.17 Example operations of processor 113 for one layer in neural network device 100 are shown.

[0194] Reference Fig.17 , one layer in the neural network device 100 may include a plurality of multipliers 711 to 71m (m is an integer greater than 2), a plurality of arithmetic shifters 721 to 72m, and a plurality of adders 731 to 73p and 741 (p is an integer greater than 2).

[0195] The multiplier 711 performs multiplication on the quantized input data QIDTA1 and the fractional part a'1 of the quantized weight, and provides the result of the multiplication to the arithmetic shifter 721. The arithmetic shifter 721 shifts the output of the multiplier 711 by the sum of the exponential part b'1 of the quantized weight and the exponential bias bse1 to output the shifted result. The multiplier 71m performs multiplication on the quantized input data QIDTAm and the fractional part a'm of the quantized weight, and provides the result of the multiplication to the arithmetic shifter 72m. The arithmetic shifter 72m shifts the output of the multiplier 71m by the sum of the exponential part b'm of the quantized weight and the exponential bias bsem to output the shifted result. Adders 731 to 73p and 741 add the outputs of the arithmetic shifters 721 to 72m and the bias to provide the quantized output data QODTA. In order to improve the accuracy, the quantization parameter calculator 130 may quantize the bias to an M-bit asymmetric fixed-point type.

[0196] Fig.18 It is shown Fig.17 Schematic diagram of the operation of the multiplier and arithmetic shifter in the processor.

[0197] Reference Fig.18 The multiplier 711 performs fixed-point multiplication on the quantized N-bit fixed-point input data QIDTA and the fractional part a' of the quantized N-bit floating-point weight to provide a fixed-point output MOUT of FXP. The arithmetic shifter 721 shifts the output MOUT of the multiplier 711 by the exponential part b' of the quantized N-bit floating-point weight to provide a fixed-point output SOUT. The arithmetic shifter 721 further shifts the output MOUT of the multiplier 711 by the exponential bias bse of the weight to provide a fixed-point output SOUT.

[0198] Fig.19 is a block diagram illustrating a neural network system according to an embodiment.

[0199] Reference Fig.19 , the neural network system 1000 may include a first electronic device 1100 and a second electronic device 1200. The deep learning system may be driven by the first electronic device 1100 and the second electronic device 1200.

[0200] The first electronic device 1100 may include a plurality of heterogeneous processors 1110, a task manager TKMNG 1120, a quantization parameter calculator 1130, and a memory MEM 1160. The second electronic device 1200 may include a plurality of heterogeneous processors 1210, a task manager TKMNG 1220, a quantization parameter calculator QPCL 1230, and a memory MEM 1260. The plurality of heterogeneous processors 1110 and 1210 may include CPUs 1111 and 1211, GPUs 1112 and 1212, NPUs 1113 and 1213, DSPs 1114 and 1214, ISPs 1115 and 1215, and dedicated hardware 1116 and 1216, respectively.

[0201] Fig.19 The plurality of heterogeneous processors 1110 and 1210, the task managers 1120 and 1220, the quantization parameter calculators 1130 and 1230, and the memories 1160 and 1260 in the embodiment may be connected with the reference Figures 1 to 18 Basically the same as described.

[0202] In an example, some of the heterogeneous processors (e.g., 1111, 1112, 1113, 1114, 1115, and 1116) may be included in the first electronic device 1100, and other heterogeneous processors (e.g., 1211, 1212, 1213, 1214, 1215, and 1216) may be included in the second electronic device 1200. The first electronic device 1100 may be an electronic device that directly interacts with a user (e.g., is directly controlled by a user). The second electronic device 1200 may be physically separated from the first electronic device 1100, and may interoperate with the first electronic device 1100.

[0203] In an example, the first electronic device 1100 can be any computing device and / or mobile device, such as a personal computer (PC), a laptop computer, a mobile phone, a smart phone, a tablet computer, a personal digital assistant (PDA), a portable multimedia player (PMP), a digital camera, a portable camera, a music player, a video player, a portable game console, a navigation device, a wearable device, an Internet of Things (IoT) device, an Internet of Everything (IoE) device, a virtual reality (VR) device, an augmented reality (AR) device, etc.

[0204] In an example, the second electronic device 1200 may be any computing device and / or mobile device interoperable with the first electronic device 1100. For example, the second electronic device 1200 may be a companion device that depends on the first electronic device 1100, such as a wearable device (e.g., a smart watch). Alternatively, the second electronic device 1200 may be an internal server (e.g., a home gateway) that controls an IoT device and / or an IoE device, or an external server (e.g., a cloud server).

[0205] The inventive concepts may be applied to various devices and systems including neural network and / or machine learning systems.

[0206] As is conventional in the field of the inventive concept, embodiments are described and shown in the accompanying drawings according to functional blocks, units and / or modules. It will be appreciated by those skilled in the art that these blocks, units and / or modules are physically implemented by electronic (or optical) circuits (such as, logic circuits, discrete components, microprocessors, hard-wired circuits, memory elements, wiring connections, etc.), wherein the electronic (or optical) circuits can be formed using semiconductor-based manufacturing techniques or other manufacturing techniques. In the case where blocks, units and / or modules are implemented by microprocessors or the like, they can be programmed using software (e.g., microcode) to perform the various functions discussed herein, and can be optionally driven by firmware and / or software. Optionally, each block, unit and / or module can be implemented by dedicated hardware, or implemented as a combination of dedicated hardware for performing some functions and processors (e.g., one or more programmed microprocessors and associated circuits) for performing other functions. In addition, without departing from the scope of the inventive concept, each block, unit and / or module of the embodiment can be physically divided into two or more interacting and discrete blocks, units and / or modules. Furthermore, without departing from the scope of the inventive concept, the blocks, units and / or modules of the embodiments may be physically combined into more complex blocks, units and / or modules.

[0207] The above content is an explanation of the embodiment and should not be interpreted as limiting the embodiment. Although the embodiment has been described, it will be readily appreciated by those skilled in the art that many modifications are possible in the embodiment without substantially departing from the inventive concept.

Claims

1. A neural network device, comprising: A training device is configured to: receive training image data, and pre-train a neural network based on the training image data, wherein a parameter value of at least one layer of the pre-trained neural network is an M-bit floating point type, and M is a natural number greater than 3; A quantization parameter calculator configured to: obtain parameters of a pre-trained neural network, and quantize the parameters of the pre-trained neural network so that the quantized parameters have a mixed data type; and a processor configured to apply the quantized parameters to the pre-trained neural network to obtain an optimized neural network, The quantization parameter calculator is further configured as follows: Analyze the statistical distribution of M-bit floating-point parameter values; obtaining a quantization level for each parameter that statistically covers the distribution range of parameter values ​​based on the analyzed statistical distribution; and Using a quantization parameter obtained based on the obtained quantization level of each parameter, quantizing the M-bit floating point type weight into an N-bit floating point type weight, where N is a natural number greater than 1 and less than M, The quantization parameter calculator is further configured as follows: obtaining a first step length of a first gap between quantization levels of each parameter associated with the input data of the M-bit floating point type and corresponding to the obtained, obtaining a second step length corresponding to a second gap between quantization levels of output data obtained by performing a node operation of input data of M-bit floating point type and weights, A new weight is obtained based on the obtained first step length and the obtained second step length.

2. The neural network device according to claim 1, wherein: The processor is also configured to: receive image data to be recognized, and input the image data to be recognized into the optimized neural network, so as to execute the optimized neural network to perform image recognition on the image data to be recognized.

3. The neural network device according to claim 2, wherein: The quantization parameter calculator is further configured to quantize the M-bit floating-point input data of each layer of the at least one layer of the optimized neural network into N-bit fixed-point asymmetric input data using the quantization parameter obtained based on the obtained quantization level of each parameter when the optimized neural network is executed.

4. The neural network device according to claim 1, wherein: The quantization parameter calculator is further configured to analyze the statistical distribution by obtaining statistics of the parameter values ​​of the M-bit floating point type input data, weights and biases during pre-training of the neural network, and The processor is further configured to obtain a quantized neural network by applying N-bit fixed-point asymmetric input data and N-bit floating-point weights to the neural network.

5. The neural network device according to claim 1, wherein: The quantization parameter calculator is also configured to: Based on the obtained quantitative level of each parameter, the maximum and minimum values ​​of the analyzed statistical distribution, the first step length, the second step length, the first zero point and the second zero point are obtained, Wherein, the first zero point indicates the zero value of the input data of the M-bit floating point type, and the second zero point indicates the zero value of the output data; A bias associated with the output data is obtained based on the obtained new weight.

6. The neural network device according to claim 5, wherein: The quantization parameter calculator is further configured to obtain asymmetric input data of the N-bit fixed-point type by applying the obtained first step size and the obtained second step size to the input data of the M-bit floating-point type.

7. The neural network device according to claim 5, wherein: The quantization parameter calculator is further configured to obtain the N-bit floating point type weight by applying the obtained first step size and the obtained second step size to the M-bit floating point type weight.

8. The neural network device according to claim 7, wherein: The N-bit floating point weights include an exponent bias.

9. The neural network device according to claim 8, wherein: The quantization parameter calculator is further configured to differently apply the exponential bias to a plurality of layers included in the neural network.

10. The neural network device according to claim 7, wherein: The quantization parameter calculator is further configured to obtain the weight of the N-bit floating point type using a multiple of 2 that is greater than a maximum value of the weight of the N-bit floating point type and is closest to the maximum value.

11. The neural network device according to claim 5, wherein: The quantization parameter calculator is further configured to obtain a new bias associated with the output data based on the unquantized bias, the obtained new weight, the obtained first zero point, and the obtained second step size.

12. The neural network device according to claim 1, wherein: The quantization parameter calculator is also configured to: analyzing a statistical distribution of parameter values ​​of each of a plurality of nodes included in the at least one layer of the neural network; and Based on the analyzed statistical distribution, a weight associated with each of the plurality of nodes is quantified.

13. The neural network device according to claim 3, wherein: The quantization parameter calculator includes: The logic circuit is configured to: quantize the M-bit floating point input data and weights into N-bit fixed point asymmetric input data and N-bit floating point weights using a quantization parameter; A control circuit configured as a control logic circuit; The input register and the weight register are configured to: store quantization parameters; and The result register is configured to store N-bit fixed-point asymmetric input data and N-bit floating-point weights.

14. The neural network device according to claim 3, wherein: The processors include: The register is configured to: receive N-bit fixed-point asymmetric input data and N-bit floating-point weights from a quantization parameter calculator, and store the N-bit fixed-point asymmetric input data and the N-bit floating-point weights; A logic circuit configured to: apply N-bit fixed-point asymmetric input data and N-bit floating-point weights to a neural network to obtain a quantized neural network; and The control circuit is configured as a control logic circuit.

15. The neural network device according to claim 1, wherein: The statistical distribution is approximated by a normal distribution or a Laplace distribution, and The quantization parameter calculator is further configured to obtain a quantization level of each parameter based on any one or any combination of a mean value, a variance, a standard deviation, a maximum value and a minimum value of the parameter values ​​obtained from the analyzed statistical distribution.

16. The neural network device according to claim 15, wherein: The quantization parameter calculator is further configured to obtain a quantization level of each parameter based on any one or any combination of a mean value, a variance, a standard deviation, a maximum value and a minimum value of the parameter value to reduce the probability of a quantization error.

17. A neural network device, comprising: A training device is configured to: receive training image data, and pre-train a neural network based on the training image data, wherein a parameter value of at least one layer of the pre-trained neural network is an M-bit floating point type, and M is a natural number greater than 3; A quantization parameter calculator configured to: obtain parameters of a pre-trained neural network, and quantize the parameters of the pre-trained neural network so that the quantized parameters have a mixed data type; and a processor configured to apply the quantized parameters to the pre-trained neural network to obtain an optimized neural network, Among them, the quantization parameter calculator includes: The logic circuit is configured to: quantize the M-bit floating point weight into an N-bit floating point weight using a quantization parameter, where N is a natural number greater than 1 and less than M; A control circuit configured as a control logic circuit; The input register and the weight register are configured to: store quantization parameters; and The result register is configured to store N-bit floating-point weights. The quantization parameter calculator is further configured as follows: Analyze the statistical distribution of M-bit floating-point parameter values, obtaining a quantification level for each parameter that statistically covers the distribution range of parameter values ​​based on the analyzed statistical distribution, obtaining a first step length of a first gap between quantization levels of each parameter associated with the input data of the M-bit floating point type and corresponding to the obtained, obtaining a second step length corresponding to a second gap between quantization levels of output data obtained by performing a node operation of input data of M-bit floating point type and weights, A new weight is obtained based on the obtained first step length and the obtained second step length.

18. A method of operating a neural network, the method comprising: Receiving training image data, and pre-training a neural network based on the training image data, wherein a parameter value of at least one layer of the pre-trained neural network is an M-bit floating point type, where M is a natural number greater than 3; Analyze the statistical distribution of M-bit floating-point parameter values; obtaining a quantization level for each parameter that statistically covers the distribution range of parameter values ​​based on the analyzed statistical distribution; quantizing the weight of the M-bit floating point type into a weight of the N-bit floating point type using a quantization parameter obtained based on the obtained quantization level of each parameter, N being a natural number greater than 1 and less than M; and Apply N-bit floating point weights to a pre-trained neural network to obtain an optimized neural network, The step of quantizing the M-bit floating point weight to the N-bit floating point weight includes: obtaining a first step length of a first gap between quantization levels of each parameter associated with the input data of the M-bit floating point type and corresponding to the obtained, obtaining a second step length corresponding to a second gap between quantization levels of output data obtained by performing a node operation of input data of M-bit floating point type and weights, A new weight is obtained based on the obtained first step length and the obtained second step length.

19. The method according to claim 18, wherein: The method further comprises: The image data to be recognized is received, and the image data to be recognized is input into the optimized neural network, so that the optimized neural network is executed to perform image recognition on the image data to be recognized.

20. The method according to claim 19, wherein: When performing the optimization neural network, the M-bit floating-point input data of each layer of the at least one layer of the optimization neural network is quantized into N-bit fixed-point asymmetric input data using a quantization parameter obtained based on the obtained quantization level of each parameter.

Images