Method and apparatus for neural network quantization

Abstract

According to the method and apparatus for quantizing neural networks, a quantized neural network is generated by: performing learning of the neural network; for each layer of the first neural network, obtaining the weight difference between the initial weights and the updated weights determined through learning in each cycle; analyzing the statistics of the weight difference of each layer; determining one or more layers from the layers to be quantized with lower bit precision based on the analyzed statistics; and generating a second neural network by quantizing the one or more layers determined with lower bit precision.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to Korean Patent Application No. 10-2019-0002986, filed on January 9, 2019, with the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure relates to methods and apparatus for adjusting the precision of some selected layers of a neural network to a lower bit. Background Technology

[0004] A neural network is a computational architecture that, after extensive training, can provide a computationally intuitive mapping between input and output patterns. Devices used to process neural networks perform numerous operations on complex inputs, making it difficult to analyze large amounts of input data and extract desired information in real time. Summary of the Invention

[0005] The present invention is provided in a simplified form to describe the selection of ideas further described below in the detailed embodiments. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to help determine the scope of the claimed subject matter.

[0006] In one general aspect, a method for quantizing a neural network is provided, the method comprising: performing multiple cycles of feedforward and backpropagation learning on a first neural network having a first bit precision; for each layer of the first neural network, obtaining a weight difference between initial weights and updated weights determined through learning in each cycle; analyzing statistics of the weight difference for each layer; determining one or more layers from the layers to be quantized with a second bit precision lower than the first bit precision based on the analyzed statistics; and generating a second neural network by quantizing the one or more layers determined with the second bit precision.

[0007] Statistics can include the mean square of the weight differences at each level.

[0008] The method may include: sorting the layers according to the magnitude of the analyzed statistics, wherein determining one or more layers to be quantified includes identifying layers from the sorted layers that have relatively small magnitudes of the analyzed statistics.

[0009] Determining one or more layers to be quantized may include: when quantizing some of the sorted layers with second-bit precision, using a binary search algorithm to identify one or more layers to be quantized in response to the accuracy loss of the neural network compared to the first neural network being within a threshold.

[0010] Accuracy loss can include the recognition rate of neural networks.

[0011] Determining one or more layers to be quantified may include: selecting a certain number of layers from the sorted layers as the one or more layers, in ascending order of the magnitude of the analyzed statistics.

[0012] The determination of one or more layers to be quantified may include: not selecting the layer with the smallest size of the analyzed statistic among the sorted layers as one or more layers to be quantified.

[0013] The first neural network may have layers with fixed-point parameters of a first bit precision, and the first neural network is quantized from a third neural network with layers of floating-point parameters of a third bit precision, wherein the third bit precision is higher than the first bit precision, and the quantized second neural network may include one or more defined layers with fixed-point parameters of a second bit precision and other layers with fixed-point parameters of a first bit precision.

[0014] The method may include: in response to a layer of a first neural network having floating-point parameters with a first bit precision, quantizing layers other than the one or more layers into layers with fixed-point parameters of a fourth bit precision, wherein the fourth bit precision is lower than the first bit precision and higher than the second bit precision, wherein the quantized second neural network may include one or more determined layers with fixed-point parameters of the second bit precision and layers with fixed-point parameters of the fourth bit precision.

[0015] In another general aspect, an apparatus for quantizing a neural network is provided, the apparatus including a processor configured to: perform multiple cycles of feedforward and backpropagation learning on a first neural network having a first bit precision; obtain a weight difference between the initial weights of each layer in the first neural network and the updated weights determined by learning over each cycle; analyze statistics of the weight differences of each layer; determine one or more layers from the layers to be quantized with a second bit precision lower than the first bit precision based on the analyzed statistics; and generate a second neural network by quantizing the one or more layers determined with the second bit precision.

[0016] Statistics can include the mean square of the weight differences at each level.

[0017] The processor can be configured to sort the layers in order of the magnitude of the analyzed statistics, and identify the layers with relatively small analyzed statistics as one or more layers to be quantified.

[0018] The processor can be configured to use a binary search algorithm to determine one or more layers to be quantized when quantizing some of the sorted layers with second-bit precision, in response to the accuracy loss of the neural network compared to the first neural network being within a threshold.

[0019] Accuracy loss can include the recognition rate of neural networks.

[0020] The processor can be configured to determine a certain number of layers as one or more layers from the sorted layers, in ascending order of the magnitude of the analyzed statistics.

[0021] The processor can be configured to not identify the layer with the smallest size of the analyzed statistic among the sorted layers as one or more layers to be quantified.

[0022] The first neural network may have layers with fixed-point parameters of a first bit precision, and the first neural network is quantized from a third neural network with layers of floating-point parameters of a third bit precision, wherein the third bit precision is higher than the first bit precision, and the quantized second neural network may include one or more defined layers with fixed-point parameters of a second bit precision and other layers with fixed-point parameters of a first bit precision.

[0023] The processor can be configured to: in response to a layer of a first neural network having floating-point parameters with a first bit precision, quantize layers other than the one or more layers to layers with fixed-point parameters with a fourth bit precision, wherein the fourth bit precision is lower than the first bit precision and higher than the second bit precision, and the quantized second neural network may include the determined one or more layers with fixed-point parameters with the second bit precision and the layers with fixed-point parameters with the fourth bit precision.

[0024] The device may include a memory storing instructions that, when executed, configure the processor to perform learning, obtain weight differences, analyze statistics, determine one or more layers, and generate a second neural network.

[0025] In another general aspect, a neural network apparatus is provided, comprising: a memory configured to store weights and instructions for layers of a first neural network; and a processor configured to execute instructions to: train the first neural network having a first bit precision; obtain, after training, the difference between the weights of the layers of the first neural network and the corresponding updated weights of the layers; analyze statistics of the differences for each layer; based on the statistics, determine at least one layer from the layers to be quantized with a second bit precision lower than the first bit precision; generate a second neural network by quantizing the at least one layer with the second bit precision; and output the second neural network to the apparatus.

[0026] The processor can be configured to: train the first neural network for a certain number of cycles; and after each cycle, determine the change of the corresponding weights based on the difference between the weights of each layer before and after each cycle of training.

[0027] Other features and aspects will become apparent from the following detailed description, drawings and claims. Attached Figure Description

[0028] Figure 1 This is a diagram illustrating an example of a neural network architecture.

[0029] Figure 2 This is a diagram illustrating an example of operations performed in a neural network.

[0030] Figure 3 This is a diagram illustrating an example of the hardware configuration of a neural network quantization device.

[0031] Figure 4 An example is shown where the learned neural network is quantized by a neural network quantization device and deployed by a hardware accelerator.

[0032] Figure 5 Examples of floating-point and fixed-point values ​​are shown.

[0033] Figure 6 An example is shown showing the relationship between decimal length and accuracy in fixed-point systems.

[0034] Figure 7 This is a diagram illustrating an example of neural network learning.

[0035] Figure 8A This is a diagram illustrating an example of weights being updated through backpropagation learning of a neural network.

[0036] Figure 8B This is a diagram showing an example of the weight differences at each layer.

[0037] Figure 8C This is a graph showing an example of the analysis results of the statistics of the weight differences at each layer.

[0038] Figure 9 This is a diagram illustrating an example of ranking each layer based on the weight difference statistic.

[0039] Figure 10 This is a diagram illustrating an example of selecting a layer from a sorted set of layers to be quantized with lower bit precision.

[0040] Figure 11 This is a diagram illustrating an example of a method for determining which layers to quantize with lower bit precision.

[0041] Figure 12 This is a diagram illustrating an example of a method for determining which layers to quantize with lower bit precision.

[0042] Figure 13 This is a diagram illustrating an example of quantizing a neural network, including layers with floating-point parameters, into a neural network including layers with fixed-point parameters.

[0043] Figure 14 This is a diagram illustrating an example of quantizing a neural network, including layers with floating-point parameters, into a neural network including layers with fixed-point parameters.

[0044] Figure 15 This is a diagram illustrating an example of a quantized neural network.

[0045] Figure 16 This is a graph illustrating an example of the correlation between the number of layers quantized with lower bit precision and the accuracy loss.

[0046] Figure 17 This is a graph showing the results of comparing the use of the weight difference statistic as a criterion for determining which layer to quantize in lower bits with the use of other methods.

[0047] Figure 18 This is a diagram showing an example of an electronic device.

[0048] Figure 19 This is a diagram illustrating an example of a method for quantizing a neural network.

[0049] Throughout the accompanying drawings and detailed embodiments, unless otherwise described or provided, the same reference numerals shall be understood to refer to the same elements, features, and structures. The drawings need not be drawn to scale, and for clarity, illustration, and convenience, the relative sizes, proportions, and depictions of elements in the drawings may be exaggerated. Detailed Implementation

[0050] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding the disclosure of this application. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but can be changed as will become apparent upon understanding the disclosure of this application, except for operations that must be performed in a specific order. Furthermore, for clarity and conciseness, descriptions of features known in the art may be omitted.

[0051] The features described herein may be embodied in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein are provided merely to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein, which will become apparent upon understanding the disclosure of this application.

[0052] Throughout the specification, when an element such as a layer, region, or substrate is described as being "on," "connected to," or "coupled to" another element, it may be directly "on," "connected to," or "coupled to" the other element, or one or more other elements may be present in between. Conversely, when an element is described as being "directly on," "directly connected to," or "directly coupled to" another element, no other elements may be present in between.

[0053] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts are not limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or part from another. Therefore, without departing from the teachings of the examples described herein, the first component, assembly, region, layer, or part mentioned in the examples may also be referred to as the second component, assembly, region, layer, or part.

[0054] The terminology used herein is for the purpose of describing various examples only and is not intended to limit this disclosure. Unless the context clearly indicates otherwise, the articles “a,” “an,” and “the” are intended to include plural forms. The terms “comprising,” “including,” and “having” indicate the presence of the stated features, number, operation, component, element, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, number, operation, component, element, and / or combinations thereof. Throughout this disclosure, the expression “at least one of a, b, or c” indicates only a, only b, only c, both a and b, both a and c, both b and c, all a, b, and c, or variations thereof. The use of the term “may” in this document with respect to examples or embodiments (e.g., what an example or embodiment may include or implement) means that there is at least one example or embodiment that includes or implements such a feature, and that all examples and embodiments are not limited thereto.

[0055] It will be apparent upon understanding the disclosure of this application that the features of the examples described herein can be combined in various ways. Furthermore, it will be apparent upon understanding the disclosure of this application that, while the examples described herein have various configurations, other configurations are also possible.

[0056] Figure 1This is a diagram illustrating an example of the architecture of neural network 1.

[0057] refer to Figure 1 Neural network 1 can be a deep neural network (DNN) or an n-layer neural network. A DNN or an n-layer neural network can correspond to the following neural networks: for example, convolutional neural networks (CNN), recurrent neural networks (RNN), deep belief networks, restricted Boltzmann machines, fully connected networks (FCN), deep convolutional networks (DCN), long short-term memory (LSTM) networks, and gated recurrent units (GRU). For example, neural network 1 can be implemented as a CNN, but is not limited to this. Figure 1 When a neural network is used to represent a CNN, neural network 1 corresponds to some layers of the CNN. Therefore, neural network 1 can correspond to convolutional layers, pooling layers, or fully connected layers of a CNN.

[0058] In a convolutional layer, the first feature map FM1 is the input feature map, and the second feature map FM2 is the output feature map. A feature map is a dataset that represents various features of the input or output data. Feature maps FM1 and FM2 can be high-dimensional matrices with two or more dimensions, and each feature map can include activation parameters. When feature maps FM1 and FM2 are, for example, three-dimensional feature maps, each feature map in FM1 and FM2 can have a width W (or number of columns), a height H (or number of rows), and a depth D. In the example, the depth D can correspond to the number of channels.

[0059] In the convolutional layer, a convolution operation is performed between the first feature map FM1 and the weight map WM to generate the second feature map FM2. The weight map WM filters the first feature map FM1 and can be referred to as a filter or a kernel. The depth of the weight map WM (i.e., the number of channels in the weight map WM) is equal to the product of the depth of the first feature map FM1 and the depth of the second feature map FM2, that is, equal to the product of the number of channels in the first feature map FM1 and the number of channels in the second feature map FM2. In the example, the weight map WM is shifted to slide over the entire region of the first feature map FM1, acting as a sliding window. Each time the weight map WM is shifted, each weight included in the weight map WM is multiplied by the feature value in the first feature map FM1 located at the corresponding position in the region of the first feature map FM1 that overlaps with the weight map WM. The products of all multiplications are summed to obtain the corresponding feature value in the second feature map FM2 located at the position corresponding to the position in the weight map WM. When the first feature map FM1 and the weight map WM are convolved with each other, channels of the second feature map FM2 are generated. Although Figure 1A weight map WM is shown, but in reality, multiple weight maps are convolved with a first feature map FM1 to generate multiple channels of a second feature map FM2. Zero padding can be added around the boundaries of the first feature map FM1 and the second feature map FM2 if needed. However, a detailed discussion of zero padding is omitted for simplicity.

[0060] The second feature map FM2 of a convolutional layer can be the input feature map of the next layer. For example, the second feature map FM2 can be the input feature map of a pooling layer.

[0061] Figure 2 This is a diagram illustrating an example of the operations performed in neural network 2.

[0062] refer to Figure 2 The neural network 2 has a structure including an input layer, a hidden layer and an output layer, performs operations based on the received input data (e.g., I1 and I2), and generates output data (e.g., O1 and O2) based on the results of the operations.

[0063] As mentioned above, neural network 2 can be a DNN or an n-layer neural network including two or more hidden layers. Figure 2 In the example shown, neural network 2 is a DNN that includes an input layer "Layer 1", two hidden layers "Layer 2" and "Layer 3", and an output layer "Layer 4". When neural network 2 is implemented as a DNN, it includes multiple hidden layers for efficient information processing, enabling it to handle more complex datasets than a neural network with a single hidden layer. Although neural network 2 is shown as having four layers, this is just an example, and neural network 2 can include more or fewer layers, or more or fewer channels. In other words, neural network 2 can include layers with... Figure 2 The layers shown have various structures with different configurations.

[0064] Each layer in Neural Network 2 includes multiple channels. A channel corresponds to multiple artificial nodes, also referred to as neurons, processing elements (PEs), units, or other terms. For example, as... Figure 2 As shown, each of layers 1 and 4 includes two channels (nodes), while each of layers 2 and 3 includes three channels (nodes). However, this is just an example, and each layer in neural network 2 may include other numbers of channels (nodes).

[0065] In Neural Network 2, channels in one layer connect to channels in another layer to process data. For example, a channel can receive data from another channel, perform operations on that data, and output the results to the other channels.

[0066] The input to each channel can be called the input activation, and the output of each channel can be called the output activation. In other words, activation is a parameter that serves as the output of one channel and simultaneously as the input to one or more other channels included in the next layer. Each channel determines its own activation based on weights and the activations received from the channels included in the previous layer. Weights are parameters used to compute the output activation in each channel and are values ​​assigned to the connection between two channels.

[0067] Each channel is processed by a computing unit or processing element that receives input activation and outputs output activation, and the input and output activations of each channel are mapped to input and output feature maps, respectively. For example, when "σ" represents the activation function, This represents the weights from the k-th channel in layer (i-1) to the j-th channel in layer i. k i-1 " indicates the output activation of the k-th channel in the (i-1)-th layer (which is also the input activation of the j-th channel in the i-th layer)," This represents the bias of the j-th channel included in the i-th layer, and When indicating the output activation of the j-th channel included in the i-th layer, the output activation can be calculated according to Equation 1 below.

[0068]

[0069] like Figure 2 As shown, the output activation of the first channel CH1 of the second layer "layer 2" is controlled by... To represent. Furthermore, according to Equation 1, Has value However, Equation 1 above is merely an example used to describe the activations, weights, and biases used to process data in neural network 2, and neural network 2 is not limited to this. For example, activations could be values ​​obtained by passing through rectified linear units (ReLUs) or by applying an activation function to the sum of activations received from the previous layer.

[0070] As described above, in neural network 2, large datasets are exchanged between multiple interconnected channels and undergo a large number of computations while traversing layers. The following describes methods and apparatus for reducing the number of computations required to process complex input data and simultaneously reducing the accuracy loss of neural network 2.

[0071] Figure 3 This is a diagram illustrating an example of the hardware configuration of the neural network quantization device 10.

[0072] refer to Figure 3The neural network quantization device 10 includes a processor 110 and a memory 120. Figure 3 In the neural network quantization device 10, for ease of description, only those shown are... Figure 3 Examples of related constituent elements. However, besides Figure 3 In addition to the constituent elements shown, the neural network quantization device 10 may also include common constituent elements.

[0073] The neural network quantization device 10 is a computing device with various processing functions, such as functions for generating floating-point neural networks, training floating-point neural networks, quantizing floating-point neural networks to obtain fixed-point neural networks, and retraining fixed-point neural networks. For example, the neural network quantization device 10 can be implemented in various types of devices, such as servers, mobile devices, smartphones, embedded devices, wearable smart devices (e.g., rings, watches, glasses, eyeglass-type devices, bracelets, anklets, belts, necklaces, earrings, headbands, helmets, devices embedded in clothing, or eyeglass displays (EGDs)) that use neural networks to perform speech recognition, image recognition, and image classification; computing devices (e.g., servers, laptops, notebooks, mini-notebooks, netbooks, ultra-mobile personal computers (UMPCs), tablet PCs, phablets, mobile internet devices (MIDs), personal digital assistants (PDAs), enterprise digital assistants (EDAs), ultra-mobile personal computers (UMPCs), portable laptop PCs); and electronic products (e.g., robots, digital cameras, digital camcorders, portable game consoles, MP3 players). The device can be used for various applications, including but not limited to: media players, portable / personal multimedia players (PMPs), handheld e-readers, GPS navigation devices, personal navigation devices, portable navigation devices (PNDs), handheld game consoles, e-books, televisions (TVs), high-definition televisions (HDTVs), smart TVs, smart appliances, smart home devices, or security devices for door control, voice authentication systems, augmented reality (AR) devices, Internet of Things (IoT) devices, autonomous vehicles, robotic devices, or medical devices. The neural network quantization device 10 can be applied to vehicles and vehicle management systems, such as autonomous vehicles, automated or autonomous driving systems, intelligent vehicles, advanced driver assistance systems (ADAS), and navigation systems that assist vehicles in safely maintaining their lanes. Examples described herein can be used for road guidance information in vehicle navigation devices, such as augmented reality head-up displays (AR). (3DHUD). Furthermore, the neural network quantization device 10 can be a dedicated hardware accelerator installed in the aforementioned device, and the neural network quantization device 10 can be a hardware accelerator as a dedicated module for driving the neural network, such as a neural processing unit (NPU), tensor processing unit (TPU), neural engine, but not limited thereto. The above examples are non-limiting, and other examples (e.g., applications in training, gaming, healthcare, public safety, tourism, and marketing) are considered to be fully within the scope of this disclosure. These devices perform one or more functions, such as speech recognition, image recognition, and image classification, and the neural network quantization device 10 can also be provided for other types of devices.

[0074] Processor 110 performs functions for controlling the neural network quantization device 10. For example, processor 110 controls all functions of the neural network quantization device 10 by executing one or more programs stored in memory 120. Processor 110 is included in the reference Figures 3-4 as well as Figure 18 At least one of the described devices may include a reference. Figures 3-4 as well as Figure 18 At least one of the described devices, or the execution reference Figures 11-15 and Figure 19 At least one of the described methods. Processor 110 refers to a data processing device configured as hardware, wherein the hardware has circuitry in its physical structure for performing desired operations. For example, the desired operations may include code or instructions included in a program. For example, processor 110 may be embodied as a microprocessor, processor core, multi-core processor, multiprocessor, central processing unit (CPU), graphics processing unit (GPU), application processor (AP), application-specific integrated circuit (ASIC), and field-programmable gate array (FPGA) included in neural network device 100, but is not limited thereto. Furthermore, processor 110 may execute programs stored in memory 120 to control all operations of neural network device 100. For example, processor 110 may be implemented as a microprocessor (MCU), wherein the CPU, memory (read-only memory (ROM) or random access memory (RAM)), etc., are mounted on a single chip, but is not limited thereto. Further details regarding processor 110 are provided below.

[0075] Memory 120 is hardware used to store various data processed in the neural network quantization device 10. For example, memory 120 can store data already processed in the neural network quantization device 10 and data to be processed in the neural network quantization device 10. Furthermore, memory 120 can store applications and drivers to be executed by the neural network quantization device 10. Memory 120 can be DRAM, but is not limited to it. Memory 120 can include one or both of volatile and non-volatile memory. Examples of non-volatile memory include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), flash memory, phase-change random access memory (RAM) (PRAM), magnetic RAM (MRAM), resistive RAM (RRAM), and ferroelectric RAM (FeRAM). Examples of volatile memory include dynamic RAM (DRAM), static RAM (SRAM), synchronous DRAM (SDRAM), PRAM, MRAM, RRAM, and ferroelectric RAM (FeRAM). In one example, memory 120 may include any one or a combination of any two or more of the following: hard disk drive (HDD), solid-state drive (SSD), high-density flash memory (CF) card, secure digital card (SD) card, micro secure digital card (microSD) card, mini secure digital card (miniSD) card, extreme digital picture (xD-Picture) card, and memory stick. Further details about memory 120 are provided below.

[0076] Processor 110 generates a trained neural network by repeatedly training an initial neural network. To ensure accurate computation, in this example, the initial neural network has floating-point parameters, for example, parameters with 32-bit floating-point precision. The parameters include various types of data input to and output from the neural network, such as the neural network's input activations, weights, biases, and output activations. As the neural network is repeatedly trained, its floating-point parameters are adjusted or updated to produce more accurate outputs for a given input. However, this disclosure is not limited to this, and neural network learning can be performed using a quantized neural network after performing neural network quantization as described below.

[0077] Compared to fixed-point parameters, floating-point parameters require a relatively large amount of computation and relatively frequent memory accesses. Therefore, processing neural networks with floating-point parameters may not be optimal in mobile devices such as smartphones, tablets, and wearables, and in embedded devices with relatively low processing power. Therefore, in order to drive neural networks within an acceptable loss of accuracy while significantly reducing the amount of computation in the device, the floating-point parameters of the trained neural network are quantized. In this application, "parameter quantization" refers to the conversion from floating-point parameters to fixed-point parameters. Therefore, in the example, quantizing any layer included in the neural network can mean quantizing the parameters of that layer.

[0078] In this example, the neural network quantization device 10 performs quantization of the floating-point parameters of the trained neural network to fixed-point parameters with a specific number of bits, taking into account the processing performance of the device (e.g., a mobile device or embedded device) deploying the neural network. The neural network quantization device 10 then sends the quantized neural network to the device deploying the neural network. The device deploying the neural network can be any of the devices listed above, such as autonomous vehicles using neural networks to perform speech or image recognition, smartphones, tablets, augmented reality (AR) devices, or Internet of Things (IoT) devices, but is not limited to these.

[0079] In the example, processor 110 retrieves data from a pre-trained floating-point neural network stored in memory 120. The pre-trained neural network is implemented using data that has been repeatedly trained with floating-point parameters. The neural network can be trained by first repeatedly inputting training set data and then repeatedly inputting test set data, but training is not limited to this. The training set data is the input data used to perform initial training of the neural network, while the test set data is input data independent of the training set data used to perform additional training of the neural network to improve its accuracy. During both the initial training and additional training, the performance of the neural network is measured, and training continues until the desired accuracy is achieved.

[0080] The process of quantizing each layer of a neural network to a fixed-point type, performed by processor 110, is described in detail with reference to the accompanying drawings.

[0081] Memory 120 stores datasets related to neural networks that have been processed by processor 110 or are to be processed by processor 110, such as data of an untrained initial neural network, data of a neural network generated during training, data of a neural network that has been trained, and data of a quantized neural network. Furthermore, memory 120 stores various programs related to the training and quantization algorithms of the neural networks to be executed by processor 110.

[0082] Figure 4An example is shown where the learned neural network is quantized by a neural network quantization device and deployed by a hardware accelerator.

[0083] refer to Figure 4 As mentioned above, in neural network quantization devices such as PCs or servers ( Figure 3 In 10), the processor ( Figure 3 The processor 110 (which may be a GPU) enables a floating-point neural network 410 (e.g., a 32-bit floating-point neural network) to learn. Due to the floating-point parameters of the neural network 410 being learned, it cannot be efficiently processed in a low-power or low-performance hardware accelerator. Therefore, the processor 110 of the neural network quantization device 10 quantizes the floating-point neural network 410 into a fixed-point neural network 420, for example, a 16-bit or low fixed-point type. The hardware accelerator is dedicated hardware for driving the neural network 420. Since the hardware accelerator is implemented with relatively low-power or low-performance hardware, it can be optimized for fixed-point operations rather than floating-point operations. The hardware accelerator can be, for example, a neural processing unit (NPU), a tensor processing unit (TPU), or a neural engine, which are dedicated devices for driving neural networks, but the hardware accelerator is not limited to these.

[0084] In the example, the hardware accelerator for driving the quantized neural network 420 can be implemented in a separate device independent of the neural network quantization device 10. However, the hardware accelerator is not limited to this, and can be implemented in the same device as the neural network quantization device 10.

[0085] Figure 5 Examples of floating-point and fixed-point values ​​are shown.

[0086] refer to Figure 5 The floating-point value 510 is represented as "a×2". b The floating-point value 510 is represented by 32 bits, including a 1-bit sign part, an 8-bit exponent part, and a 23-bit fractional part.

[0087] Furthermore, the fixed-point value 520 is represented by “Qm.n”, where m and n are natural numbers. In the expression “Qm.n”, “m” represents the number of bits indicating the exponent part, and “n” represents the number of bits indicating the fractional part. Therefore, the bit width of the fixed-point value is obtained by summing the 1-bit sign part, the m-bit exponent part, and the n-bit fractional part (1+m+n). Since there are n bits indicating the fractional part in the fixed-point bits, the fractional length is n. For example, “Q3.4” is a fixed-point value with a total of 8 bits, including 1 bit for the sign part, 3 bits for the exponent part, and 4 bits for the fractional part; “Q1.30” is a fixed-point value with a total of 32 bits, including 1 bit for the sign part, 1 bit for the exponent part, and 30 bits for the fractional part; and “Q15.16” is a fixed-point value with a total of 32 bits, including 1 bit for the sign part, 15 bits for the exponent part, and 16 bits for the fractional part.

[0088] Figure 6 An example is shown showing the relationship between the decimal length of a fixed-point value and its accuracy.

[0089] refer to Figure 6 Assuming the total bit width allocated to the fixed-point value is 3 bits, the fixed-point representation 610 of Q2.0 with a fractional length of 0 is compared with the fixed-point representation 620 of Q1.1 with a fractional length of 1.

[0090] For Q2.0, since the exponent part is 2 bits and the fractional part is 0 bits, it can represent fixed-point values ​​from -4 to 3, and the interval between possible fixed-point values ​​is 1. For Q1.1, since the exponent part is 1 bit and the fractional part is 1 bit, it can represent fixed-point values ​​from -2 to 1.5, and the interval between possible fixed-point values ​​is 0.5.

[0091] The comparison shows that although both Q2.0's fixed-point representation 610 and Q1.1's fixed-point representation 620 are allocated 3 bits, Q2.0 can represent a wider range of fixed-point values ​​than Q1.1, but its accuracy is lower because the intervals between fixed-point values ​​are wider. Q1.1 can represent a narrower range of fixed-point values ​​than Q2.0, but has higher accuracy because the intervals between fixed-point values ​​are narrower. Therefore, it can be seen that the accuracy of a fixed-point value depends on the fractional length of the fixed-point value, that is, on the number of fractional bits allocated to the fixed-point value.

[0092] Figure 7 This is a diagram illustrating an example of the learning process of a neural network 710.

[0093] refer to Figure 7The neural network 710 may include an input layer, N layers (layer 1, layer 2, ..., layer N-1 and layer N, where N is a natural number), and an output layer. In this respect, weights are assigned to each of the N layers. In other words, weight 1 (i.e., W1) can be assigned to layer 1, weight 2 (i.e., W2) can be assigned to layer 2, ..., and weight N-1 (i.e., W...) can be assigned to layer N-1. N-1 And weights N (i.e., W) can be assigned to layer N. N ).exist Figure 2 The text describes how multiple channels can be set within a layer. Figure 7 The weights assigned to each layer as described in the text can correspond to values ​​representing the weights of one or more channels included in each layer.

[0094] The layer identifiers in the neural network are used to distinguish the various layers in the accompanying drawings, and the interpretation of the embodiments does not limit the layer identifiers. Therefore, even when layers represented by the same identifier exist in different drawings, this is merely for ease of explanation, and the identifiers may refer to the same layer or different layers.

[0095] In the example, Figure 3 The processor 110 repeatedly performs feedforward learning and backpropagation learning on the neural network 710 for multiple cycles, for example, K cycles, where K is a natural number.

[0096] The neural network 710 can achieve its recognition accuracy or recognition rate through feedforward learning, and propagate the error of the neural network 710 to each layer through backpropagation learning. The error of the neural network 710 can represent the error associated with the biases or weights included in each layer. Therefore, since the layer errors can be corrected through repeated feedforward and backpropagation learning (cycle 1 to cycle K), the recognition accuracy or recognition rate of the neural network 710 can be improved.

[0097] In the example, the layers (i.e., the parameters of the layers) of the learned neural network 710 are quantized to a lower precision. For example, when the neural network 710 is a model with 32-bit floating-point precision, the learned neural network corresponds to a model with 32-bit floating-point precision. To adjust the neural network, the neural network 710 with 32-bit floating-point precision can be quantized to a fixed-point neural network with 16 bits, 8 bits, or fewer bits. In this respect, even when quantizing the neural network, the neural network can operate efficiently with reduced accuracy loss. In the following description, a method for performing quantization on multiple layers in the neural network during the neural network quantization process is described.

[0098] Figure 8A This is a diagram illustrating an example of weights updated through backpropagation learning of a neural network. (Reference) Figure 8A Table 810 shows the initial weights of each layer and Table 820 shows the updated weights of each layer that are updated through backpropagation learning.

[0099] exist Figure 7 In the process of learning the initial neural network ( Figure 7 Table 710 includes layers 1 to N. According to Table 810, layer 1 has weight 1 (i.e., W1), layer 2 has weight 2 (i.e., W2), ..., layer N-1 has weight N-1 (i.e., W... N-1 , and layer N has weight N (i.e. W) N ).

[0100] Figure 3 The processor 110 enables the neural network 710 to learn repeatedly for multiple cycles, and the processor 110 determines the updated weights based on the backpropagation learning of each cycle.

[0101] According to Table 820, as a result of backpropagation learning at period 1, the updated weights of layer 1 are determined to be W. 1[1] The updated weights of layer 2 are determined to be W. 2[2] The updated weights of layer N-1 are determined to be W. N-1[1] And the updated weight N of layer N is determined to be W. N[1] Similarly, as a result of backpropagation learning up to the Kth cycle, processor 110 determines the updated weights for each layer.

[0102] Figure 8B This is a diagram showing an example of how the weight differences are described for each layer.

[0103] Weight difference W diff This represents the difference between the initial weights of each layer of the neural network before learning and the updated weights of each layer determined through backpropagation learning in each cycle.

[0104] For details, please refer to Figure 8B This shows the relationship with the weight difference W. diff The relevant table is 830. For example, regarding layer 1, when performing learning for cycle 1, the updated weights W of layer 1... 1[1] The difference between the initial weights W1 and the initial weights W1 of layer 1 corresponds to the weight difference obtained from learning layer 1 as a result of period 1. Similarly, Figure 3 The processor 110 also obtains the weight difference of other layers and other cycles.

[0105] Figure 8C This is a graph showing an example of the results of an analysis of the statistics used to describe the weight differences at each layer.

[0106] In neural network quantization, as the weight difference W of each layer is indicated... diff A decrease in the value of the statistical measure may result in quantization being performed with lower bit precision. In this regard, the weight difference W between layers... diff The statistics may include the mean square of the weighted differences at all periods of each layer, but this disclosure is not limited thereto, and the statistics may be of different types, such as mean, variance or standard deviation.

[0107] refer to Figure 8C The weight difference W for each layer is shown. diff Analysis of statistical measures. Specifically, regarding layer 1, the squared weight difference (W) at computation cycle 1 of processor 110. 1[1] -W1) 2 The square of the weight difference at period 2 (W) 1[2] -W1) 2 ... the square of the weight difference at period K (W) 1[K] -W1) 2 The mean of the values ​​is used to obtain W as a statistic. diff _m.s._ layer 1, that is, the mean square of the weight differences of layer 1. Similarly, processor 110 obtains the statistics, i.e., the mean square, of the weight differences of other layers.

[0108] The statistics of the weight difference of each layer can be used to determine and select the layers to be quantized with lower bit precision in neural network quantization.

[0109] Figure 9 This is a diagram illustrating an example of ranking the layers based on the weight difference statistic.

[0110] refer to Figure 9 , Figure 3 The processor 110 can sort the layers of the neural network according to the magnitude of the weight difference statistic (e.g., mean square) corresponding to each layer to determine the layers to be quantized with lower bit precision. In other words, the processor 110 can be based on W... diff _m.s._ layer 1, W diff _m.s._ layer 2、...、W diff _m.s._ layers N (they are Figure 8C The weight differences (statistics) of each layer in the analysis are sorted in order of magnitude. The sorting method can be either ascending or descending.

[0111] Although Figure 9 The N layers (where N is a natural number) are shown sorted in ascending order according to the size of the weight difference statistic (e.g., mean square), but the identifier of each layer is indicated only for illustrative purposes, and this disclosure is not limited thereto.

[0112] Processor 110 can first as Figure 9 The layers are sorted as described in the description to select the layers to be quantized with lower bit precision, but this disclosure is not limited thereto. The processor 110 may select layers that meet certain conditions based on weight difference statistics (e.g., mean square) without sorting the layers.

[0113] Figure 10 This is a diagram illustrating an example of selecting a layer from a sorted set of layers to be quantized with lower bit precision.

[0114] refer to Figure 10 , Figure 3 The processor 110 identifies one or more layers (layers 100, 90, ..., 40) from all sorted layers whose analyzed weight difference statistics are relatively small as those to be quantized. (See reference) Figure 11 and Figure 12 To further describe in detail the method of selecting layers with relatively small weight difference statistics.

[0115] In the example, processor 110 may not determine the layer (layer 100) with the smallest analyzed statistic size as one or more layers to be quantized. In the example, when the layer (layer 100) with the smallest analyzed statistic size is quantized with lower bit precision, the number of classes to be represented may be reduced. However, this disclosure is not limited thereto, and processor 110 may quantize layers including the layer (layer 100) with the smallest analyzed statistic size.

[0116] exist Figure 10 In this context, all sorted layers can correspond to quantized layers of the neural network. In this respect, a layer quantized with lower bit precision can be a layer quantized with "A-bit precision," where A is a natural number, while other layers can be layers quantized with "B-bit precision," where B is a natural number. A-bit precision is a lower precision than B-bit precision. In this example, the layer quantized with lower bit precision can correspond to the layer with the lowest precision among all quantized layers, but this disclosure is not limited thereto.

[0117] As mentioned above, the quantization of a layer can be the quantization of parameters such as the biases or weights included in the layer, which can represent the determination or change of the decimal length of a fixed-point parameter.

[0118] The following description describes a method for determining and selecting layers to be quantized with lower bit precision from all layers or all sorted layers.

[0119] Figure 11This diagram illustrates an example of a method for determining which layers to quantize with lower bit precision. It can be performed in the order and manner shown. Figure 11 The operations are described, but the order of some operations or some operations may be changed or omitted without departing from the spirit and scope of the illustrative examples described. They can be executed in parallel or concurrently. Figure 11 Many of the operations shown. Figure 11 One or more blocks, and combinations thereof, can be implemented by a hardware-based special-purpose computer (e.g., a processor) that performs the specified function, or by a combination of special-purpose hardware and computer instructions. In the example, Figure 11 The operation can be performed by Figure 3 The neural network quantization device 10 is used to perform this. In addition to the following... Figure 11 Beyond the description, for Figures 1 to 10 The description also applies to Figure 11 And it is incorporated into this article by reference. Therefore, the above description need not be repeated here.

[0120] In operation 1101, Figure 3 The processor 110 is based on the analyzed weight difference statistics (e.g., such as...) Figures 8A to 8C The processor 110 sorts the layers of the neural network according to the mean square (e.g., the weight difference statistic). In other words, the processor 110 can sort the layers in ascending or descending order based on the magnitude of the weight difference statistic (e.g., the mean square).

[0121] In operation 1102, in the example, processor 110 selects the layer with half of the lower statistics among the layers as candidate layers to be quantized with lower bit precision.

[0122] In operation 1103, when quantizing the selected candidate layer, processor 110 determines whether the accuracy loss of the quantized neural network is equal to or less than a threshold. When the accuracy loss is equal to or less than the threshold, processor 110 executes operation 1105. When the accuracy loss is greater than the threshold, processor 110 executes operation 1104.

[0123] In operation 1104, when the accuracy loss of the quantized neural network exceeds a threshold, processor 110 again determines candidate layers to be quantized with lower bit precision. In the example, in operation 1104, processor 110 uses the selected candidate layers to update all layers. Therefore, in operation 1102, processor 110 can again determine the number of candidate layers from all updated layers (half the number of candidate layers in the previous operation 1102).

[0124] In operation 1105, when the accuracy loss of the quantized neural network is equal to or less than a threshold, the processor 110 determines the selected candidate layer as the layer to be quantized with lower bit precision.

[0125] In operation 1106, processor 110 generates a quantized neural network by quantizing the determined layers with lower bit precision.

[0126] refer to Figure 11 In the example, the binary search algorithm is described as an implementation for determining the layer to be quantized with lower bit precision. Figure 3 The processor 110 determines one or more layers to be quantized by using a binary search algorithm to determine whether the accuracy loss of the neural network, including some layers quantized with A-bit (lower bit) precision among the sorted layers, is within a threshold compared to the neural network including layers not quantized with A-bit precision.

[0127] Although not in Figure 11 As shown, however, in operation 1103, when the accuracy loss is greater than a threshold, the following operation can be optionally added: determining whether the number of candidate layers is equal to or less than a specific number (e.g., 5), which can be predefined by the user. When the optional operation is added, even when the accuracy loss is greater than the threshold, the processor 110 can determine that the number of candidate layers is equal to or less than that number (e.g., 5) and perform operation 1105.

[0128] Figure 12 This diagram illustrates an example of a method for determining which layers to quantize with lower bit precision. It can be performed in the order and manner shown. Figure 12 The operations are described, but the order of some operations or some operations may be changed or omitted without departing from the spirit and scope of the illustrative examples described. They can be executed in parallel or concurrently. Figure 12 Many of the operations shown. Figure 12 One or more blocks, and combinations thereof, can be implemented by a hardware-based special-purpose computer (e.g., a processor) performing the specified function, or by a combination of special-purpose hardware and computer instructions. In the example, Figure 12 The operation can be performed by Figure 3 The neural network quantization device 10 is used to perform this. In addition to the following... Figure 12 Beyond the description, for Figures 1 to 11 The description also applies to Figure 12 And it is incorporated into this article by reference. Therefore, the above description need not be repeated here.

[0129] In operation 1201, Figure 3 The processor 110 is configured to quantize the number of layers with lower bit precision.

[0130] In operation 1202, Figure 3 The processor 110 is based on the analyzed weight difference statistics (e.g., such as...) Figures 8A to 8CThe processor 110 sorts the layers of the neural network in ascending or descending order according to the magnitude of the weight difference statistic (e.g., mean square).

[0131] In operation 1203, processor 110 identifies a certain number of layers with lower statistics from all sorted layers as the layers to be quantized.

[0132] In operation 1204, processor 110 quantizes the determined layers with lower bit precision, thereby generating a quantized neural network.

[0133] refer to Figure 12 ,and Figure 11 The methods used in employing the binary search algorithm differ. Figure 3 The processor 110 determines a certain number of layers from the sorted layers as layers to be quantized with lower bit precision, according to the ascending order of the magnitude of the analyzed statistics.

[0134] exist Figure 11 and Figure 12 In the following, a statistic based on the weight difference of each layer is described (e.g., in...). Figures 8A to 8C The method described herein (mean square) is used to select the layer to be quantized with lower bit precision. However, this disclosure is not limited to this, and the method to determine the layer to be quantized with lower bit precision can be achieved by using the weight difference statistic in different ways to select the layer with the lower statistic.

[0135] Figure 13 This is a diagram illustrating an example of quantizing a neural network 1301, which includes layers with floating-point parameters, into a neural network that includes layers with fixed-point parameters.

[0136] refer to Figure 13 The neural network 1301 may include N layers with 32-bit floating-point parameters. Figure 3 The processor 110 performs the following process 1310: it analyzes the statistics of the weight differences of each layer through feedforward and backpropagation learning of the neural network 1301, and determines one or more layers 1315 to be quantized with "A-bit precision" of less than 32 bits based on the analyzed weight difference statistics. Therefore, some layers 1315 determined from all layers of the neural network 1301 are quantized with A-bit precision.

[0137] The processor 110 performs the following process 1320: quantizes other layers 1325 of all layers of the neural network 1301 that were not determined to be quantized with A-bit precision into layers with fixed-point parameters of “B-bit precision”, wherein B-bit precision is less than 32-bit precision and more than A-bit precision.

[0138] When the bit precision for quantizing all layers of neural network 1301 is determined, processor 110 executes process 1330 to quantize each layer with the determined bit precision, thereby generating quantized neural network 1302. Quantized neural network 1302 includes layer 1315 with fixed-point parameters of A bit precision and other layers 1325 with fixed-point parameters of B bit precision.

[0139] Figure 14 This is a diagram illustrating an example of quantizing a neural network 1401, which includes layers with floating-point parameters, into a neural network that includes layers with fixed-point parameters.

[0140] refer to Figure 14 Neural network 1401 may include N layers with 32-bit floating-point parameters. Neural network 1402 is obtained by quantizing neural network 1401 and has N layers with 8-bit floating-point parameters.

[0141] Figure 3 The processor 110 analyzes the statistics of the weight differences in each layer through feedforward and backpropagation learning of the neural network 1402 quantized with 8-bit precision, and determines one or more layers 1410 in each layer to be quantized with "A-bit precision" less than 8 bits based on the analyzed weight difference statistics. Therefore, the layers 1410 determined from all layers of the neural network 1402 are quantized with A-bit precision.

[0142] Therefore, the neural network 1401 is quantized into a neural network including a layer 1410 with fixed-point parameters of A-bit precision and other layers with fixed-point parameters of 8-bit precision.

[0143] exist Figure 13 and Figure 14 The bit precision values ​​(e.g., 32 bits or 8 bits) of the neural networks described herein are merely exemplary for illustrative purposes, and this disclosure is not limited thereto.

[0144] Figure 15 This is a diagram illustrating an example of a quantized neural network 1501.

[0145] refer to Figure 15 The quantized neural network 1501 may include layers with 4-bit precision (lower bit precision) and layers with 8-bit precision. In this respect, the layers with 4-bit precision are layers with lower bit precision and correspond to layers determined based on analyzed weight difference statistics, wherein, as described above, the analyzed weight difference statistics are obtained by analyzing the statistics of the weight differences of each layer through feedforward and backpropagation learning. As described above, the number of layers with 4-bit precision can be determined to reduce the accuracy loss of the neural network 1501.

[0146] Figure 15 The identifiers are defined for ease of explanation only, and this disclosure is not limited thereto.

[0147] Figure 16 This is a graph illustrating an example of the correlation between the number of layers quantized with lower bit precision and accuracy loss. In this example, the accuracy loss could be related to the recognition rate of the neural network.

[0148] Figure 16 An example of simulation results for Inception V3 is shown, illustrating the correlation of accuracy loss when some layers with lower weight difference statistics (mean square) among all weighted layers are quantized with lower bit precision. Figure 16 In the example, some layers with lower weight difference statistics (mean square) are quantized with 4-bit precision, while other layers are 8-bit precision.

[0149] As the number of layers quantized with lower bit precision increases, the accuracy loss increases because parameter loss occurs with the increase in the number of quantized layers. The accuracy loss is only 1% when only 25 out of all 95 layers with 8-bit precision are additionally quantized with lower bit (4-bit) precision. Correspondingly, despite the accuracy loss being only 1%, processing speed can be effectively increased while the computational load on the processor 110 relative to the neural network is reduced. Therefore, by quantizing some layers with lower bit precision at a tolerable level of accuracy loss, high recognition accuracy or recognition rate of the neural network can be ensured while maintaining effective computational load and processing speed in devices deploying the neural network.

[0150] Figure 17 This is a graph showing an example of a comparison between using the weight difference statistic and using other methods as criteria for determining layers to be quantized with lower bit precision.

[0151] refer to Figure 17 The comparison results are shown between the use of weight range 1701, the use of accuracy 1702, and the use of the mean square of weight difference 1703 as criteria for determining the layers to be quantized with lower bit precision.

[0152] In the 1701 example, the weight ranges of each layer are sorted in ascending order, and some layers with smaller weight ranges are quantized with lower bit (4-bit) precision. However, as...

[0153] Figure 17As shown, it can be seen that the accuracy loss of 1701 when using the weight range is greater compared to other cases 1702 and 1703. This is because as the weight range decreases, the weight value to be represented decreases, and therefore it is possible to represent it with lower bit precision. However, even when the weight range is small, when the maximum value of the weight is large, the number of integer bits used to represent the weight increases accordingly.

[0154] One example of using accuracy in 1702 is a method that involves quantizing each layer of the neural network one by one with lower bit (4-bit) precision, calculating its recognition accuracy (or recognition rate), and determining which layers to quantize with lower bit precision based on a range from minimum to maximum accuracy loss. Figure 17 As shown, the accuracy loss of case 1702 using accuracy is similar to the accuracy loss of case 1703 using the mean square of weight differences. However, case 1702 using accuracy consumes a very large amount of processing time because the accuracy loss is calculated one by one while all layers included in the sequential quantization neural network are computed.

[0155] Unlike the examples of 1701 and 1702, the example of 1703, which uses the mean square of the weight difference, can determine a certain number of layers to be quantized with lower bit precision with little loss of accuracy, faster processing speed, and relatively few operations.

[0156] Figure 18 This is a diagram showing an example of an electronic device.

[0157] refer to Figure 18 Electronic device 1800 uses a neural network to analyze input data in real time to extract information, determines a situation based on the extracted information, and controls the components of the electronic device 1800 installed therein based on the determined situation. For example, electronic device 1800 can be applied to robotic devices (e.g., drones and advanced driver assistance systems (ADAS), smart TVs, smartphones, medical devices, mobile devices, image display devices, measuring devices, and IoT devices), and can also be installed in reference systems. Figure 3 Any of the various types of electronic devices and applications mentioned above.

[0158] In the example, electronic device 1800 includes a processor 1810, RAM 1820, neural network device 1830, memory 1840, sensor module 1850, communication (Tx / Rx) module 1860, and input / output module 1870. Although not explicitly stated... Figure 18As shown, however, electronic device 1800 may also include one or more other components, such as a security module and a power control device. Some hardware components of electronic device 1800 may be mounted on at least one semiconductor chip. Neural network device 1830 may include the aforementioned neural network-specific hardware accelerator or a device including a neural network-specific hardware accelerator.

[0159] Processor 1810 controls all operations of electronic device 1800. Processor 1810 can be a single-core processor or a multi-core processor. Processor 1810 processes or executes programs and / or data stored in memory 1840. In one example, processor 1810 controls the functionality of neural network device 1830 by executing programs stored in memory 1840. For example, processor 1810 can be implemented by a CPU, GPU, or AP. In addition to processor 1810, for... Figure 3 The description of processor 110 also applies to Figure 18 And it has been incorporated into this article by reference. Therefore, the above description need not be repeated here.

[0160] RAM 1820 temporarily stores programs, data, or instructions. For example, programs and / or data stored in memory 1840 can be temporarily stored in RAM 1820 under the control of boot code or processor 1810. RAM 1820 can be implemented using memory such as dynamic RAM (DRAM) or static RAM (SRAM). Further details about RAM 1820 are provided below.

[0161] The neural network device 1830 can perform learning on the neural network, execute operations of the neural network based on received input data, and generate information signals based on the results of the operations. The neural network may include CNNs, RNNs, deep belief networks, restricted Boltzmann machines, or any of the aforementioned neural networks, but this disclosure is not limited thereto.

[0162] The neural network device 1830 can have various processing functions, such as generating neural networks, enabling neural networks to learn or train neural networks, quantizing floating-point neural networks into fixed-point neural networks, or retraining neural networks. In other words, the neural network device 1830 is hardware that enables neural networks to learn and perform processing using the aforementioned neural networks quantized into fixed-point forms, and it can correspond to the aforementioned hardware accelerators dedicated to neural networks.

[0163] The information signal can be any type of recognition signal, such as a speech recognition signal, object recognition signal, image recognition signal, or biometric information recognition signal. In one example, the neural network device 1830 receives frame data included in a video stream as input data and generates a recognition signal for an object included in an image indicated by the frame data based on the frame data. However, the neural network device 1830 is not limited to this; instead, the neural network device 1830 can receive other types of input data and generate recognition signals based on the input data, depending on the type or function of the electronic device 1800 in which the electronic device 1800 is installed.

[0164] Memory 1840 is a storage device used to store data (e.g., operating system (OS), various programs, and individual data entries). In one example, memory 1840 stores intermediate results generated during the operation of neural network device 1830, such as output feature maps. In one example, memory 1840 stores compressed output feature maps. Furthermore, memory 1840 may store quantized neural network data used by neural network device 1830, such as parameters, weight maps, or weight lists. Further details of memory 1840 are provided below.

[0165] The memory 1840 may be DRAM, but is not limited to it. The memory 1840 may include one or both of volatile and non-volatile memory. Examples of non-volatile memory include ROM, PROM, EPROM, EEPROM, flash memory, PRAM, MRAM, RRAM, and FeRAM. Examples of volatile memory include DRAM, SRAM, SDRAM, PRAM, MRAM, RRAM, and FeRAM. In one example, the memory 1840 may include any one or any combination of two or more of HDD, SSD, CF, SD, microSD, miniSD, and Memory Stick.

[0166] Sensor module 1850 collects information relating to the surrounding environment of the electronic device 1800 in which the electronic device is installed. Sensor module 1850 senses or receives signals (e.g., image signals, voice signals, magnetic signals, biometric signals, or touch signals) from outside the electronic device and converts the sensed or received signals into data. For this purpose, sensor module 1850 can be any one or any combination of two or more of various types of sensing devices (e.g., microphone, imaging device, image sensor, light detection and ranging (LIDAR) sensor, ultrasonic sensor, infrared sensor, biosensor, or touch sensor).

[0167] Sensor module 1850 provides converted data as input data to neural network device 1830. In one example, sensor module 1850 includes an image sensor that generates a video stream by capturing images of the external environment of the electronic device and provides consecutive data frames of the video stream to neural network device 1830 in the order of the input data. However, sensor module 1850 is not limited to this; rather, sensor module 1850 may provide other types of data to neural network device 1830.

[0168] The communication module 1860 includes various wired or wireless interfaces capable of communicating with external devices. For example, the communication module 1860 may include a local area network (LAN), a wireless local area network (WLAN) (e.g., Wi-Fi), a wireless personal area network (WPAN) (e.g., Bluetooth), a wireless universal serial bus (USB), ZigBee, near field communication (NFC), radio frequency identification (RFID), power line communication (PLC), or a communication interface capable of connecting to a mobile cellular network (e.g., 3G, 4G, or LTE).

[0169] In one example, the communication module 1860 receives quantized neural network data from an external device. The external device can be, for example... Figure 3 The device, such as the neural network quantization device 10, trains a neural network based on a large amount of data, quantizes the trained neural network into a fixed-point neural network, and provides the quantized neural network data to the electronic device 1800. The received quantized neural network data is stored in the memory 1840.

[0170] Input / output module 1870 is a physical structure including one or more hardware components that provide the ability to present a user interface, display information, output information, and / or receive user input. Input / output module 1870 outputs the results it receives from electronic device 1800. However, input / output module 1870 is not limited to the examples described above, and in these examples, any display effectively connected to electronic device 1800 (e.g., computer monitor and eyeglass display (EGD)) can be used without departing from the spirit and scope of the illustrative examples described.

[0171] Figure 19 This is a diagram illustrating an example of a method for quantizing neural networks. It can be performed in the order and manner shown. Figure 19 However, without departing from the spirit and scope of the illustrative examples described, the order of some operations may be changed or some operations may be omitted. Figure 19 Many of the operations shown can be performed in parallel or concurrently. Figure 19One or more blocks, and combinations thereof, can be implemented by a hardware-based special-purpose computer (e.g., a processor) performing the specified function, or by a combination of special-purpose hardware and computer instructions. In the example, Figure 19 The operation can be performed by Figure 3 The neural network quantization device 10 is used to perform this. In addition to the following... Figure 19 Beyond the description, for Figures 1 to 18 The description also applies to Figure 19 And it is incorporated into this article by reference. Therefore, the above description need not be repeated here.

[0172] In operation 1901, processor 110 repeatedly performs multiple cycles of feedforward and backpropagation learning on a first neural network with a first bit precision.

[0173] In operation 1902, processor 110 obtains the weight difference between the initial weights and the updated weights of each layer in the first neural network, which are determined through backpropagation learning in each cycle.

[0174] In operation 1903, processor 110 analyzes the statistics of the weight difference of each layer.

[0175] In operation 1904, processor 110 determines one or more layers to be quantized with a second bit precision, which is lower than the first bit precision, based on analyzed statistics.

[0176] In operation 1905, processor 110 quantizes the determined layer with a second bit precision, thereby generating a second neural network including the quantized layer.

[0177] Perform the operations described in this application Figure 3 The neural network quantization device 10 in the middle Figure 4 The neural processing unit (NPU) in, and Figure 18The electronic device 1800 and neural network device 1830 are implemented by hardware components configured to perform the operations described in this application, which are executed by hardware components. Examples of hardware components that may be used to perform the operations described in this application, where appropriate, include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more hardware components performing the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). The processor or computer may be implemented by one or more processing elements (e.g., logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond to and execute instructions in a defined manner to achieve a desired result). In one example, the processor or computer includes one or more memories storing instructions or software executed by the processor or computer, or connected to one or more memories storing instructions or software executed by the processor or computer. Hardware components implemented by a processor or computer can execute instructions or software, such as an operating system (OS) and one or more software applications running on the OS, to perform the operations described in this application. The hardware component can also access, manipulate, process, create, and store data in response to the execution of instructions or software. For brevity, the singular terms "processor" or "computer" may be used in the description of the examples described in this application, but in other examples multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors or another processor and another controller. One or more processors or a processor and a controller may implement a single hardware component, or two or more hardware components. The hardware components can have any one or more different processing configurations, examples of which include single processor, discrete processor, parallel processor, single instruction single data (SISD) multiprocessing, single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0178] Perform the operations described in this application Figure 1 , Figure 2 , Figures 7-15 and Figure 19 The methods illustrated are executed by computing hardware, such as one or more processors or computers that execute instructions or software as described above to perform the operations described in this application (operations implemented by the methods). For example, a single operation or two or more operations may be executed by a single processor, two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors or a processor and a controller, and one or more other operations may be executed by one or more other processors or another processor and another controller. One or more processors or a processor and a controller may execute a single operation or two or more operations.

[0179] Instructions or software for controlling a processor or computer to implement hardware components as described above and to execute methods are written as computer programs, code segments, instructions, or any combination thereof, for individually or collectively instructing or configuring the processor or computer to operate as a machine or special-purpose computer to perform operations performed by the hardware components and the methods described above. In examples, the instructions or software include at least one of the following: applets, dynamic link libraries (DLLs), middleware, firmware, device drivers, and applications that store output status information. In one example, the instructions or software include machine code that is directly executed by the processor or computer, such as machine code generated by a compiler. In another example, the instructions or software include higher-level code that is executed by the processor or computer using an interpreter. A programmer of ordinary skill in the art can readily write the instructions or software based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions in the specification, which disclose algorithms for performing operations performed by the hardware components and the methods described above.

[0180] Instructions or software used to control computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, along with any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), flash memory, card-type storage (e.g., multimedia card, secure digital (SD) card, or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store instructions or software, and any associated data, data files, and data structures, in a non-transitory manner, and to provide instructions or software, and associated data, data files, and data structures, to a processor or computer such that the processor or computer can execute the instructions. In one example, the instructions or software, and any associated data, data files, and data structures, are distributed across a networked computer system, causing one or more processors or computers to store, access, and execute the instructions and software, and any associated data, data files, and data structures, in a distributed manner.

[0181] Although this disclosure includes specific examples, it will be apparent upon understanding the disclosure of this application that various changes in form and detail may be made to these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered descriptive only and not for limiting purposes. The description of features or aspects in each example is to be considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order and / or if components in the described system, architecture, device, or circuit are combined in a different manner and / or replaced or supplemented by other components or their equivalents. Therefore, the scope of this disclosure is not limited by specific embodiments but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents are to be construed as being included in this disclosure.

Claims

1. A processor-implemented method of neural network quantization of a neural network comprising a plurality of layers, the neural network being for performing speech recognition, image recognition, and image classification, the method comprising: performing a plurality of cycles of forward propagation and backpropagation learning on a first neural network having a first bit precision; for each layer in the first neural network, obtaining a weight difference between an initial weight and an updated weight determined by the learning of each cycle; analyzing a statistic of the weight difference of each layer, wherein the statistic comprises a mean square of the weight difference of each layer; ordering the layers in an order of a size of the analyzed statistic; determining, from the layers, one or more layers to be quantized at a second bit precision lower than the first bit precision based on the analyzed statistic, wherein the determining of the one or more layers to be quantized comprises: identifying, from the ordered layers, a layer having a relatively small size of the analyzed statistic; and generating a second neural network by quantizing the determined one or more layers at the second bit precision.

2. The method of claim 1, wherein, The determining of the one or more layers to be quantized comprises: when quantizing some of the ordered layers at the second bit precision, in response to an accuracy loss of the neural network being within a threshold compared to the first neural network, using a binary search algorithm to identify the one or more layers to be quantized.

3. The method of claim 2, wherein, The accuracy loss comprises a recognition rate of the neural network.

4. The method of claim 1, wherein, The determining of the one or more layers to be quantized comprises: determining a number of layers from the ordered layers as the one or more layers in an ascending order of the size of the analyzed statistic.

5. The method of claim 1, wherein, The determining of the one or more layers to be quantized comprises: not determining a layer having a smallest size of the analyzed statistic from the ordered layers as the one or more layers to be quantized.

6. The method of claim 1, wherein, the first neural network has layers of fixed-point parameters of the first bit precision, and the first neural network is quantized from a third neural network having layers of floating-point parameters of a third bit precision, wherein the third bit precision is higher than the first bit precision, and the quantized second neural network comprises the determined one or more layers having fixed-point parameters of the second bit precision and other layers having fixed-point parameters of the first bit precision.

7. The method of claim 1, further comprising: in response to the first neural network having layers of floating-point parameters of the first bit precision, quantizing layers other than the one or more layers as layers of fixed-point parameters of a fourth bit precision, wherein the fourth bit precision is lower than the first bit precision and higher than the second bit precision, wherein the quantized second neural network comprises the determined one or more layers having fixed-point parameters of the second bit precision and layers having fixed-point parameters of the fourth bit precision.

8. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method defined in claim 1.

9. An apparatus for neural network quantization of a neural network comprising a plurality of layers, the neural network being used for performing speech recognition, image recognition, and image classification, the apparatus comprising: a processor configured to: perform a plurality of cycles of feedforward and backpropagation learning on a first neural network having a first bit precision; obtain, for each layer in the first neural network, a weight difference between an initial weight and an updated weight determined by the learning of each cycle; analyze a statistic of the weight difference for each layer, wherein the statistic comprises a mean square of the weight difference for each layer; rank the layers in an order according to a size of the analyzed statistic; determine, from the layers, one or more layers to be quantized at a second bit precision lower than the first bit precision based on the analyzed statistic, wherein the processor is configured to determine, as the one or more layers to be quantized, a layer having a relatively small size of the analyzed statistic among the ranked layers; and generate a second neural network by quantizing the determined one or more layers at the second bit precision.

10. The apparatus of claim 9, wherein, the processor is further configured to, when quantizing some of the ranked layers at the second bit precision, determine the one or more layers to be quantized using a binary search algorithm in response to an accuracy loss of the neural network being within a threshold compared to the first neural network.

11. The apparatus of claim 10, wherein, the accuracy loss comprises a recognition rate of the neural network.

12. The apparatus of claim 9, wherein, the processor is further configured to determine, as the one or more layers, a number of layers from the ranked layers in an ascending order of the size of the analyzed statistic.

13. The apparatus of claim 9, wherein, the processor is further configured to not determine, as the one or more layers to be quantized, a layer having a smallest size of the analyzed statistic among the ranked layers.

14. The apparatus of claim 9, wherein the first neural network has layers of fixed-point parameters of the first bit precision, and the first neural network is quantized from a third neural network having layers of floating-point parameters of a third bit precision, wherein the third bit precision is higher than the first bit precision, and the quantized second neural network comprises the determined one or more layers having fixed-point parameters of the second bit precision and other layers having fixed-point parameters of the first bit precision.

15. The apparatus of claim 9, wherein, the processor is further configured to, in response to the first neural network having layers of floating-point parameters of the first bit precision, quantize layers other than the one or more layers as layers of fixed-point parameters of a fourth bit precision, wherein the fourth bit precision is lower than the first bit precision and higher than the second bit precision, and the quantized second neural network comprises the determined one or more layers having fixed-point parameters of the second bit precision and the layers having fixed-point parameters of the fourth bit precision.

16. The apparatus of claim 9, further comprising a memory storing instructions that, when executed, configure the processor to perform the learning, obtain the weight difference, analyze the statistic, determine the one or more layers, and generate the second neural network.

17. A neural network apparatus for neural network quantization of a neural network comprising a plurality of layers, the neural network being used to perform speech recognition, image recognition, and image classification, the neural network apparatus comprising: a memory configured to store weights of layers of a first neural network and instructions; and a processor configured to execute the instructions to: train the first neural network having a first bit precision; obtain, after training, weight differences between the weights of the layers of the first neural network and respective updated weights of the layers; analyze statistics of the weight differences of each layer, wherein the statistics comprise a mean square of the weight differences of each layer; sort the layers by sizes of the analyzed statistics; determine, based on the statistics, at least one layer from the layers to be quantized at a second bit precision lower than the first bit precision, wherein the instructions cause the processor to determine, as the one or more layers to be quantized, the layers from the sorted layers having relatively small sizes of the analyzed statistics; generate a second neural network by quantizing the at least one layer at the second bit precision; output the second neural network to a device.

18. The neural network device of claim 17, wherein, the processor is further configured to: train the first neural network for a number of cycles; and obtain changes in respective weights after each cycle based on differences in weights of each layer before and after each cycle of the cycles of training.

Images