Method and apparatus for recognizing an image and method and apparatus for training a neural network

Abstract

Disclosed are a method and apparatus for recognizing an image and a method and apparatus for training a neural network. The method includes obtaining image data to be recognized as input data of a neural network, generating output data of a current layer by performing an operation of the current layer on input data of the current layer for the current layer among at least one layer within the neural network, obtaining local quantization data by performing primary quantization on global recovery data based on local maximum values of each output channel of the current layer, storing the local quantization data in a memory, obtaining global quantization data by performing secondary quantization on local recovery data based on global maximum values corresponding to the current layer, providing the global quantization data as input data of a next layer to the next layer to perform an operation of the next layer, and outputting an image recognition result based on output data generated by an operation of a final layer of the neural network.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2019-0126298, filed on October 11, 2019, with the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0002] This disclosure relates to methods and apparatus for recognizing images, as well as methods and apparatus for training neural networks for recognizing images. Background Technology

[0003] Neural networks are widely used in image recognition technology. A neural network is a computing system based on a computational architecture that uses the biological brain as a model. According to the latest developments in neural network technology, neural network devices are used in various types of electronic systems to analyze input image data and extract useful information.

[0004] Neural network devices perform large computational loads on complex input image data. Therefore, techniques for efficiently processing network operations are needed to allow neural network devices to analyze input image data and extract information in real time. In particular, low-power and high-performance embedded systems (such as smartphones) have limited resources, thus requiring techniques to minimize accuracy loss in various processing operations while reducing the amount of resources required to process complex input image data. Summary of the Invention

[0005] Various embodiments of this disclosure provide a method and apparatus for quantizing data from a neural network. A computer-readable recording medium is also provided, on which a program for performing the above methods on a computer is recorded.

[0006] Other aspects will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the disclosed embodiments.

[0007] According to one aspect of this disclosure, a method for recognizing an image includes: obtaining image data to be recognized as input data of a neural network; generating output data of a current layer by performing an operation on the input data of the current layer for a current layer in at least one layer of the neural network, wherein the output data of the current layer includes multiple output channels; obtaining locally quantized data by performing an initial quantization on globally recovered data based on the local maximum value of each output channel of the current layer, wherein the globally recovered data is recovered from the output data generated by the operation of the current layer based on the global maximum value corresponding to the previous layer; storing the locally quantized data in a memory; obtaining globally quantized data by performing a secondary quantization on the locally recovered data based on the global maximum value corresponding to the current layer, wherein the locally recovered data is recovered from the locally quantized data based on the local maximum value of each output channel; providing the globally quantized data as input data to the next layer to perform the operation of the next layer; and outputting an image recognition result based on the output data generated by the operation of the final layer of the neural network.

[0008] The steps to obtain local quantization data include: obtaining global recovery data by recovering the output data generated by the operation of the current layer for each output channel based on the global maximum value corresponding to the previous layer, wherein the global maximum value corresponding to the previous layer is generated based on the data obtained by recovering the output data generated by the operation of the previous layer.

[0009] The steps to obtain local quantized data include: performing initial quantization on the global recovery data of each output channel based on the local maximum value, which is the maximum value of the global recovery data of each output channel, to obtain local quantized data, so that the maximum value of the global recovery data of each output channel is quantized to the same value.

[0010] The steps to obtain global quantization data include: obtaining local quantization data by recovering local quantization data for each output channel based on the local maximum value of the global quantization data for each output channel, so that the local quantization data is restored to the data before the initial quantization.

[0011] The steps to obtain global quantized data include: performing secondary quantization on the local recovery data of each output channel based on the global maximum value, which is the maximum value of the local recovery data, so that the local recovery data are quantized to the same scale.

[0012] The steps for obtaining local quantized data also include: recovering the output data generated by the operations of the current layer based on the global maximum value corresponding to the previous layer; and obtaining global recovered data by performing pooling operations, permutation normalization, or activation using an activation function on the recovered data.

[0013] The method further includes: storing a local maximum value obtained by calculating the maximum value of the global recovery data of each output channel in all output channels of the current layer; and storing the global maximum value corresponding to the current layer in a register, wherein the global maximum value corresponding to the current layer is obtained by calculating the maximum value among the local maximum values.

[0014] For each output channel, the steps to obtain local quantized data are executed sequentially.

[0015] When the current layer is the first layer of the neural network, the output data of the first layer is generated by performing the operations of the first layer on the image data to be recognized. When the current layer is a layer other than the first layer of the neural network, the output data of the current layer is generated by performing the operations of the current layer on the global quantized data obtained based on the output data of the previous layer.

[0016] According to another aspect of this disclosure, a method for training a neural network for image recognition includes: obtaining training image data; performing training on the neural network based on the training image data, wherein, in each iteration of training, for the current layer in at least one layer within the neural network, the step of obtaining input data for the next layer based on the output data of the current layer includes: obtaining locally quantized data by performing an initial quantization on globally recovered data based on the local maximum value of each output channel of the current layer, the output data of the current layer including multiple output channels, the globally recovered data being recovered from the output data generated by the operation of the current layer based on the global maximum value corresponding to the previous layer; storing the locally quantized data in a memory; obtaining globally quantized data by performing a secondary quantization on the locally recovered data based on the global maximum value corresponding to the current layer, the locally recovered data being recovered from the locally quantized data based on the local maximum value of each output channel; and providing the globally quantized data as input data to the next layer to perform the operation of the next layer.

[0017] The operations of the current layer and the next layer are based on backpropagation, and the output data of the current layer operation and the input data of the next layer operation include either the loss gradient or the weight gradient.

[0018] When the current layer is the first layer of the neural network, the output data of the first layer is generated by performing the operations of the current layer on the training image data; when the current layer is a layer other than the first layer of the neural network, the output data of the current layer is generated by performing the operations of the current layer on the global quantized data obtained based on the output data of the previous layer.

[0019] According to another aspect of this disclosure, an image recognition device includes: a memory; and a processor configured to perform inference on a neural network based on acquired image data by executing at least one program to perform image recognition. The processor is further configured to: acquire image data to be recognized as input data to the neural network; generate output data for the current layer by performing operations on the input data of the current layer for the current layer in at least one layer within the neural network, wherein the output data of the current layer includes a plurality of output channels; obtain locally quantized data by performing initial quantization on globally recovered data based on the local maximum values ​​of each output channel of the current layer, the globally recovered data being recovered from the output data generated by the operations of the current layer based on the global maximum values ​​corresponding to the previous layer; store the locally quantized data in the memory; obtain globally quantized data by performing secondary quantization on the locally recovered data based on the global maximum values ​​corresponding to the current layer, the locally recovered data being recovered from the locally quantized data based on the local maximum values ​​of each output channel; provide the globally quantized data as input data to the next layer to perform operations of the next layer; and output an image recognition result based on the output data generated by the operations of the final layer of the neural network.

[0020] When the current layer is the first layer of the neural network, the output data of the first layer is generated by performing the operations of the first layer on the image data to be recognized. When the current layer is a layer other than the first layer of the neural network, the output data of the current layer is generated by performing the operations of the current layer on the global quantized data obtained based on the output data of the previous layer.

[0021] The processor is also configured to obtain globally recovered data by recovering the output data generated by the operation of the current layer for each output channel based on the global maximum value corresponding to the previous layer, the global maximum value corresponding to the previous layer being generated based on the data obtained by recovering the output data generated by the operation of the previous layer.

[0022] The processor is also configured to obtain locally quantized data by performing an initial quantization of the global recovery data of each output channel based on a local maximum value, which is the maximum value of the global recovery data of each output channel, such that the maximum value of the global recovery data of each output channel is quantized to the same value.

[0023] The processor is also configured to obtain local recovery data by recovering the local quantized data of each output channel based on the local maximum value of the global recovery data for each output channel, such that the local quantized data is restored to the data before the initial quantization.

[0024] The processor is also configured to obtain global quantized data by performing secondary quantization on the local recovery data of each output channel based on a global maximum value that is the maximum value of the local recovery data, so that the local recovery data are quantized to the same scale.

[0025] The processor is also configured to: recover the output data generated by the operations of the current layer based on the global maximum value corresponding to the previous layer; and obtain the global recovered data by performing pooling operations, permutation normalization, or activation using an activation function on the recovered data.

[0026] The processor is also configured to: store a local maximum value obtained by calculating the maximum value of the global recovery data of each output channel in all output channels of the current layer; and store the global maximum value corresponding to the current layer in a register, the global maximum value corresponding to the current layer being obtained by calculating the maximum value among the local maximum values.

[0027] The processor is also configured to sequentially acquire local quantization data for each output channel.

[0028] According to another aspect of this disclosure, an apparatus for training a neural network for image recognition includes: a memory; and a processor configured to: acquire training image data and perform training on the neural network based on the training image data. The processor is further configured to: in each iteration of training, for the current layer in at least one layer within the neural network, acquire input data for the next layer based on the output data of the current layer, wherein the acquisition step includes: acquiring locally quantized data by performing an initial quantization on globally recovered data based on the local maximum values ​​of each output channel of the current layer, the output data of the current layer including multiple output channels, the globally recovered data being recovered from the output data generated by the operation of the current layer based on the global maximum value corresponding to the previous layer; storing the locally quantized data in the memory; acquiring globally quantized data by performing a secondary quantization on the locally recovered data based on the global maximum value corresponding to the current layer, the locally recovered data being recovered from the locally quantized data based on the local maximum values ​​of each output channel; and providing the globally quantized data as input data to the next layer to perform the operation of the next layer.

[0029] The processor is also configured to perform operations of the current layer and operations of the next layer based on backpropagation, wherein the output data of the current layer operation and the input data of the next layer operation include either the loss gradient or the weight gradient.

[0030] According to one aspect of this disclosure, a method for quantizing data of a neural network includes: obtaining locally quantized data by performing an initial quantization on globally recovered data based on the local maximum values ​​of each output channel of the current layer, wherein the globally recovered data is recovered from output data generated by operations of the current layer based on the global maximum values ​​corresponding to the previous layer; storing the locally quantized data in memory to perform operations of the next layer; obtaining globally quantized data by performing a secondary quantization on the locally recovered data based on the global maximum values ​​corresponding to the current layer, wherein the locally recovered data is recovered from the locally quantized data based on the local maximum values ​​of each output channel; and providing the globally quantized data as input data for operations of the next layer.

[0031] According to another aspect of this disclosure, a computer-readable recording medium has thereon a program for performing the method on a computer.

[0032] According to another aspect of this disclosure, an apparatus for quantizing data of a neural network includes a memory and a processor. The memory stores at least one program, and the processor is configured to: quantize the neural network data by executing the at least one program, wherein the processor is further configured to: obtain locally quantized data by performing an initial quantization on globally recovered data based on the local maximum value of each output channel of the current layer, the globally recovered data being recovered from the output data generated by the operation of the current layer based on the global maximum value corresponding to the previous layer; store the locally quantized data in the memory to perform the operation of the next layer; obtain globally quantized data by performing a secondary quantization on the locally recovered data based on the global maximum value corresponding to the current layer, the locally recovered data being recovered from the locally quantized data based on the local maximum value of each output channel; and provide the globally quantized data as input data for the operation of the next layer. Attached Figure Description

[0033] The above and other aspects, features, and advantages of the disclosed specific embodiments will become clearer from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0034] Figure 1 This is a diagram illustrating an example of a neural network according to embodiments of the present disclosure;

[0035] Figure 2 This is a diagram illustrating an example of the architecture of a neural network according to embodiments of the present disclosure;

[0036] Figure 3 This is a diagram illustrating examples of forward and backward propagation according to embodiments of the present disclosure;

[0037] Figure 4 This is a diagram illustrating an example of the operation of a convolutional layer in forward and backward propagation according to embodiments of the present disclosure;

[0038] Figure 5 This is a diagram illustrating examples of a neural network learning apparatus and a neural network inference apparatus for recognizing images according to embodiments of the present disclosure;

[0039] Figure 6 This is a diagram illustrating an example of data from a quantized neural network according to embodiments of the present disclosure;

[0040] Figure 7 This is a flowchart illustrating an example of a method for quantizing data of a neural network according to embodiments of the present disclosure;

[0041] Figure 8 This is a flowchart illustrating an example of storing local maximum values ​​and global maximum values ​​according to embodiments of the present disclosure;

[0042] Figure 9 This is a diagram illustrating an example of data from a quantized neural network according to embodiments of the present disclosure;

[0043] Figure 10 This is a diagram illustrating examples of obtaining local and global maximum values ​​according to embodiments of the present disclosure;

[0044] Figure 11 This is a diagram illustrating examples of primary and secondary quantization for each output channel according to embodiments of the present disclosure;

[0045] Figure 12 This illustrates the use according to embodiments of the present disclosure. Figure 7 A diagram illustrating the data quantization algorithm for the neural network of the method shown;

[0046] Figure 13 These are illustrations used to describe examples of data from a quantized neural network according to embodiments of the present disclosure; and

[0047] Figure 14 This is a block diagram of an example neural network device for recognizing images according to an embodiment. Detailed Implementation

[0048] Referring now to the embodiments, examples of which are shown in the accompanying drawings, wherein the same reference numerals always denote the same elements. In this respect, the embodiments may take different forms and should not be construed as limited to the description set forth herein. Therefore, the embodiments are described below only by reference to the accompanying drawings to explain various aspects. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items. When a statement such as “at least one of…” follows a column of elements, it modifies the entire column of elements, not individual elements within the column.

[0049] While the terminology used in the embodiments is selected from commonly used terms, the terms may vary according to the intent of those skilled in the art, judicial precedent, or the introduction of new technologies. Furthermore, in specific cases, the applicant may voluntarily choose the terms, in which case the meaning of the terms is disclosed in the corresponding descriptive section. Therefore, the terms used in the specification should not be defined by their simple names, but rather by their meanings and the context throughout the specification.

[0050] Throughout this disclosure, if a part is assumed to include a component, the term "comprising" means that the corresponding component may also include other components, unless a specific meaning contrary to the meaning of the corresponding component is described. Terms such as "unit" as used in embodiments indicate a unit for performing at least one function or operation, and may be implemented in hardware, software, or a combination of hardware and software.

[0051] In the following description, embodiments will be illustrated with reference to the accompanying drawings. However, embodiments may be implemented in various forms and are not limited to the examples described herein.

[0052] In the following description, embodiments will be described in detail with reference to the accompanying drawings.

[0053] Figure 1 This is a diagram illustrating an example of a neural network according to embodiments of the present disclosure.

[0054] Figure 1 The neural network 1 shown can be, but is not limited to, an example of a deep neural network (DNN). For example, neural network 1 can be a DNN that includes an input layer layer1, multiple hidden layers layer2, layer3, layer4 and layer5, and an output layer layer6.

[0055] When implementing neural network 1 using a DNN, neural network 1 can include many layers capable of processing useful information. Therefore, compared to using a neural network consisting of a single layer, neural network 1 can handle more complex datasets. Simultaneously, neural network 1... Figure 1 The diagram is shown as comprising six layers, but this illustration is merely an example. In other words, neural network 1 can include more than... Figure 1 The diagram shows the number of layers, both large and small. In other words, neural network 1 may include layers similar to... Figure 1 The layers shown are of various structures with different layers.

[0056] Figure 2 This is a diagram illustrating an example of the architecture of a neural network according to embodiments of the present disclosure.

[0057] Although Figure 2The diagram shows some convolutional layers in neural network 2, but in addition to the convolutional layers shown, neural network 2 may also include pooling layers, fully connected layers, etc.

[0058] Neural Network 2 can be implemented as an architecture with multiple layers and output feature maps. In one example, Neural Network 2 could be a neural network for object recognition (such as image recognition). In Neural Network 2, filters called kernels are used to operate on the input image, resulting in a feature map being output to the next layer. Another filter is used to operate on the output feature map, which serves as the input feature map to the next layer, producing a new output feature map. By repeating this process, Neural Network 2 can ultimately output a recognition result regarding the features of the input image.

[0059] For example, when an input image with a size of 24×24 pixels is input to... Figure 2 In Neural Network 2, by utilizing the operation of the first filter, the input image can be output as a feature map with four channels, each channel having a feature map size of 20×20 pixels. That is, a feature map can be output for each of the four output channels. Subsampling (or pooling) operations can be used to output the four-channel output feature maps as four-channel feature maps with a size of 10×10 pixels. Subsequently, the size of the feature maps can be reduced by repeating the operation of the corresponding filters and subsampling (or pooling) operations, finally outputting features with a size of 1×1 pixels each. In Neural Network 2, the operations and subsampling (or pooling) operations can be repeatedly performed in several layers to filter the input image and output robust features that represent the entire input image, and the recognition result of the input image as a non-limiting example is obtained through the final output features. In Neural Network 2, at least one layer can perform the process of quantizing the output data of the current layer to provide it as input data to the next layer. In such processing, floating-point data can be quantized to fixed-point format, or fixed-point data can be processed and then quantized back to fixed-point format. Based on the above, it is understood that the example embodiments discussed herein include one or more of such layers, levels, edges, etc., in various combinations of operations of the neural network 2.

[0060] Reference Figure 3 and Figure 4 This will describe forward propagation and backward propagation.

[0061] Figure 3 This is a diagram illustrating examples of forward and backward propagation according to embodiments of the present disclosure.

[0062] exist Figure 3 The image shows a portion of convolutional layer 3. Figure 3 Convolutional layer 3 can include input features Figure X Filter F and output feature map O.

[0063] Input features Figure X The filter F and output feature map O can correspond to the input activation (such as the output activation from the previous layer), weights, and output activation, respectively. The input features are processed through forward propagation. Figure X The filter F is convolved with the filter to produce an output feature map O. This convolutional forward propagation is also known as the inference operation.

[0064] For example, a sliding window scheme can be used to optimize the filter F and the input features. Figure X Perform convolution operations. More specifically, multiply the input activations in the first window by weights and sum the products. Move the first window along any axis (e.g., the x-axis, y-axis, or z-axis) to form a second window. Multiply the input activations in the second window by weights and sum the products. By continuously performing convolution operations in this manner, an output feature map O is generated.

[0065] Although Figure 3 An input feature is shown in the figure. Figure X and one There are multiple filters F, but they can handle multiple input features. Figure X Convolution operations are performed with multiple filters F to generate an output feature map O.

[0066] Once the final output feature map is generated, it can be compared with the expected result to generate the loss function. The training neural network can then be processed to minimize the loss function.

[0067] To minimize the loss function, the loss gradient... It can propagate backwards. The loss gradient can represent the gradient of activation.

[0068] Loss gradient for backpropagation Convolution is performed with a filter that has elements rearranged from filter F, producing the loss gradient used for backpropagation to the previous layer.

[0069] loss gradient and input features Figure X Perform convolution operations to generate weight gradients.

[0070] Figure 4 This is a diagram illustrating an example of the operation of a convolutional layer in forward and backward propagation according to embodiments of the present disclosure.

[0071] Input features can be obtained through forward propagation Figure XThe input features are convolved with filter F to produce an output feature map O. More specifically, once the input features are used... Figure X By using the activation and weights of filter F to perform a multiplication-accumulation (MAC) operation, the activation of the output feature map O can be generated.

[0072] For example, Equation 1 can be used to generate the input feature map. and filter Output feature map

[0073] [Equation 1]

[0074] o 11 =x 11 f 11 +x 12 f 12 +x 21 f 21 +x 22 f 22

[0075] o 12 =x 12 f 11 +x 13 f 12 +x 22 f 21 +x 23 f 22

[0076] o 21 =x 21 f 11 +x 22 f 12 +x 31 f 21 +x 32 f 22

[0077] o 22 =x 22 f 11 +x 23 f 12 +x 32 f 21 +x 33 f 22

[0078] x ii Indicator Input Features Figure X Activation, f ii Indicates the weights of filter F, o ii The result activation of the output feature map O is indicated (where i is an integer greater than 0).

[0079] The backpropagation loss gradient can be used The filter F′ with rearranged elements is convolved to produce the loss gradient used for backpropagation to the previous layer. More specifically, once the loss gradient is used By using the elements of the filter F' and the weights of the filter F' to perform the MAC operation, the loss gradient can be generated. The elements of filter F. In one example, filter F' can be generated by rotating the elements of filter F by 180°.

[0080] For example, Equation 2 can be used to obtain the backpropagation loss gradient. and filters with rearranged elements Generate the loss gradient for backpropagation to the previous layer.

[0081] [Equation 2]

[0082]

[0083]

[0084]

[0085]

[0086]

[0087]

[0088]

[0089]

[0090]

[0091] The backpropagation loss gradient can be used and input features Figure X Perform convolution operations to generate weight gradients. More specifically, once the loss gradient is used Elements and input features Figure X The activation of the MAC operation generates the weight gradient. Element.

[0092] For example, this can be achieved by using Equation 3 from the input feature map. and the loss gradient backpropagated from the current layer Generate weight gradients for backpropagation to the previous layer.

[0093] [Equation 3]

[0094]

[0095]

[0096]

[0097]

[0098] Figure 5 This is an illustration used to describe examples of a neural network learning apparatus and a neural network inference apparatus for recognizing images according to embodiments of the present disclosure.

[0099] Reference Figure 5 The neural network learning device 51 can be a computing device with various processing functions, such as the ability to create and train (or learn) neural networks, quantify neural network data, or retrain neural networks. For example, the neural network learning device 51 can be implemented using various types of devices, such as personal computers (PCs), server devices, mobile devices, etc.

[0100] The neural network learning device 51 can create a trained neural network 52 by repeatedly training (learning) a given initial neural network based on training image data. In each iteration of training, floating-point data can be quantized to fixed-point format during training processing, or fixed-point data can be processed and then quantized back to fixed-point format. Such quantization processing can be performed on at least one layer within the neural network.

[0101] The neural network learning device 51 can pass the trained neural network 52 to a hardware accelerator such as the neural network inference device 53. The neural network inference device 53 performs inference on the trained neural network 52 based on acquired image data to perform image recognition. The neural network inference device 53 can be included in a mobile device, embedded device, etc. The neural network inference device 53, as hardware for driving the neural network 54, can be implemented with relatively low power or low performance, and therefore can be appropriately implemented for fixed-point operations rather than floating-point operations. The functionality of the neural network inference device 53 is not limited to driving the neural network 54, and can include the functionality of the neural network learning device 51. In one example, the neural network inference device 53 can perform the training operations described herein, and the neural network learning device 51 can perform the inference operations described herein.

[0102] The neural network inference device 53 may correspond to, but is not limited to, a dedicated module for driving neural networks (such as a neural processor (NPU), tensor processor (TPU), neural engine, etc.). The neural network inference device 53 may be included in, but is not limited to, autonomous vehicles, robots, smartphones, tablet devices, augmented reality (AR) devices, Internet of Things (IoT) devices, etc., that use neural networks to perform speech recognition, image recognition, etc.

[0103] Figure 6 This is a diagram illustrating an example of data from a quantized neural network according to embodiments of the present disclosure.

[0104] When driving or training a neural network, the following processes can be repeatedly performed: the calculator 61 (also known as the MAC array) performs the operation of the current layer, quantizes the data output as the result of the operation, stores the quantized data in the memory 62, and reads the data stored in the memory 62 and inputs the read data into the calculator 61 for use in the operation of the next layer.

[0105] In forward propagation, the output data generated by the operation may include activations. In backpropagation, the output data generated by the operation may include loss gradients and weight gradients.

[0106] Meanwhile, before quantization, the output data generated by the operation can be stored in on-chip memory used as a cache.

[0107] Neural network inference devices, including those in smartphones, tablets, and wearable devices with relatively low processing power, may have limited on-chip memory space. Finally, in order to drive or train neural networks with such resource-constrained inference devices, data needs to be quantized while significantly reducing the space used by on-chip memory.

[0108] In the following text, reference will be made to Figures 7 to 8 Methods for describing data from quantized neural networks.

[0109] Figure 7 This is a flowchart illustrating an example of a method for quantizing data of a neural network according to an embodiment of the present disclosure.

[0110] Figure 7 The flowchart shown may include information that will be referenced later. Figure 14 The neural network device 1400 is described as performing operations in a time-sequential manner. The following description applies to the neural network device 1400.

[0111] In operation 710, the processor 1420 of the neural network device 1400 can perform initial (first) quantization on the global recovery data based on the local maximum value of each output channel of the current layer, thereby obtaining locally quantized data. The global recovery data is recovered from the output data of the operation of the current layer based on the global maximum value corresponding to the previous layer.

[0112] The previous and current layers can be, but are not limited to, the input, hidden, and output layers of a DNN. The previous and current layers can be, but are not limited to, the layers included in a convolutional neural network. The previous and current layers can be, but are not limited to, convolutional layers, pooling layers, flattening layers, normalization layers, fully connected layers, etc.

[0113] When the neural network device 1400 drives the neural network via forward propagation, the output data generated by the operations of the current layer may include activations. Optionally, when the neural network device 1400 trains the neural network via backpropagation, the output data generated by the operations of the current layer may include loss gradients or weight gradients.

[0114] Processor 1420 can recover the output data generated by the operations of the current layer based on the global maximum value corresponding to the previous layer, thereby generating globally recovered data. Data recovery can be represented as the data being dequantized to the scale before quantization.

[0115] The processor 1420 can recover the output data generated by the operations of the current layer based on the global maximum value corresponding to the previous layer, and perform sub-operations (such as pooling operations, permutation normalization, activation using activation functions, etc.) to generate globally recovered data.

[0116] Processor 1420 can add noise to the output data generated by the operations of the current layer to generate global recovery data. By performing initial quantization on the global recovery data generated by adding noise to the output data, the learning performance of the neural network can be improved. The noise can be, but is not limited to, Gaussian noise, uniform noise, white noise, etc.

[0117] The global maximum value corresponding to the previous layer can be a value generated based on the output data produced by the operations of the previous layer. For example, the global maximum value corresponding to the previous layer could be the maximum value in the output data recovered when the output data produced by the operations of the previous layer is recovered.

[0118] The output data generated by the operations of the current layer can be recovered based on Equation 4. To recover the output data generated by the operations of the current layer, the global maximum value corresponding to the previous layer can be used as max(|x|) in Equation 4, and the output data of the operations of the current layer can be used as maxq. n (x).

[0119] [Equation 4]

[0120]

[0121] Processor 1420 can recover output data generated by operations of the current layer for each output channel of the current layer. When recovering output data generated by operations of the current layer for each output channel of the current layer, global recovery data can be generated for each output channel of the current layer. The global recovery data generated for each output channel of the current layer can be quantized based on a separate reference value for each output channel of the current layer.

[0122] The processor 1420 can sequentially recover the output data generated by the operation of the current layer for each output channel of the current layer. For example, when the current layer includes three output channels, the processor 1420 can recover the output data of the second output channel after recovering the output data of the first output channel, and then recover the output data of the third output channel.

[0123] Processor 1420 can generate locally quantized data by performing initial quantization on the global recovered data based on the local maximum value of each output channel of the current layer. Initial quantization based on the local maximum value of each output channel of the current layer can represent quantization based on a separate reference value for each output channel of the current layer. The local maximum value can be the maximum value of the global recovered data for each output channel of the current layer.

[0124] The global recovered data can be quantized based on Equations 5 and 6. For the initial quantization of the global recovered data, the local maximum value of each output channel of the current layer can be used as Equation 5, and the global recovered data can be used as x. In Equations 5 and 6, n can indicate the number of bits used for the quantization level.

[0125] [Equation 5]

[0126]

[0127] [Equation 6]

[0128]

[0129] Processor 1420 can sequentially perform initial quantization on the global recovery data for each output channel of the current layer. For example, when the current layer includes three output channels, processor 1420 can quantize the global recovery data of the second output channel after quantizing the global recovery data of the first output channel, and then quantize the global recovery data of the third output channel. By sequentially quantizing the global recovery data for each output channel, the amount of on-chip memory resources used for quantization processing can be reduced.

[0130] The processor 1420 can sequentially perform recovery and initial quantization of the output data generated by the operations of the current layer for each output channel of the current layer. For example, when the current layer includes three output channels, the processor 1420 can recover and quantize the output data of the first output channel, then recover and quantize the output data of the second output channel, and then recover and quantize the output data of the third output channel.

[0131] In operation 720, processor 1420 may store locally quantized data in memory to perform operations at the next level. The memory may be the memory 1410 of the neural network device 1400, or it may be on-chip memory included in processor 1420.

[0132] The next layer can be, but is not limited to, the input layer, hidden layer, and output layer of a DNN. The next layer can be, but is not limited to, layers included in a convolutional neural network. The next layer can be, but is not limited to, convolutional layers, pooling layers, flattening layers, normalization layers, fully connected layers, etc.

[0133] In operation 730, processor 1420 can perform secondary quantization on the locally recovered data based on the global maximum value corresponding to the current layer, thereby obtaining global quantized data. The locally recovered data is recovered from the locally quantized data based on the local maximum value of each output channel.

[0134] Processor 1420 can recover locally quantized data based on the local maxima of each output channel, thereby generating locally recovered data. Locally quantized data can be recovered for each output channel based on the same local maxima as those in the initial quantization. The locally quantized data can be dequantized to its original scale by being recovered based on the local maxima of each output channel. For example, the locally recovered data can be the same as the globally recovered data.

[0135] Locally quantized data can be recovered based on Equation 4. To recover locally quantized data, the local maximum value of each output channel of the current layer can be used as max(|x|) in Equation 4, and the locally quantized data can be used as maxq. n (x).

[0136] The processor 1420 can sequentially recover local quantized data for each output channel of the current layer. For example, when the current layer includes three output channels, the processor 1420 can recover the local quantized data of the second output channel after recovering the local quantized data of the first output channel, and then recover the local quantized data of the third output channel.

[0137] The processor 1420 can perform secondary quantization on the locally recovered data based on the global maximum value corresponding to the current layer, thereby generating globally quantized data.

[0138] The global maximum value corresponding to the current layer can be the maximum value in the globally recovered data. The global maximum value corresponding to the current layer can be calculated based on the local maximum value of each output channel of the current layer. For example, the global maximum value corresponding to the current layer could be the maximum of the local maximum values ​​among all output channels of the current layer.

[0139] The local recovered data can be quantized a second time based on Equations 5 and 6. To perform the second quantization on the local recovered data, the global maximum value corresponding to the current layer can be used as max(|x|) in Equation 5.

[0140] The processor 1420 can sequentially perform secondary quantization on the local recovery data for each output channel of the current layer. For example, when the current layer includes three output channels, the processor 1420 can quantize the local recovery data of the second output channel after quantizing the local recovery data of the first output channel, and then quantize the local recovery data of the third output channel.

[0141] The processor 1420 can sequentially perform recovery and secondary quantization of the locally quantized data for each output channel of the current layer. For example, when the current layer includes three output channels, the processor 1420 can recover and quantize the locally quantized data of the first output channel, then recover and quantize the locally quantized data of the second output channel, and then recover and quantize the locally quantized data of the third output channel.

[0142] In operation 740, processor 1420 can provide global quantized data as input data for the next layer of operation.

[0143] Figure 8 This is a flowchart illustrating an example of storing local maximum values ​​and global maximum values ​​according to an embodiment of the present disclosure.

[0144] In operation 810, processor 1420 may store the local maximum values ​​of all output channels of the current layer in a register of processor 1420. For example, processor 1420 may store the local maximum values ​​in a register file.

[0145] The processor 1420 can obtain a local maximum by calculating the maximum value of the global recovery data for each output channel in all output channels of the current layer.

[0146] Processor 1420 may store local maxima sequentially in registers. For example, when the current layer includes three output channels, processor 1420 may store the local maximum calculated from the global recovery data of the first output channel, store the local maximum calculated from the global recovery data of the second output channel, and then store the local maximum calculated from the global recovery data of the third output channel.

[0147] The processor 1420 can read local maximum values ​​stored in registers to recover local quantized data in subsequent processing.

[0148] In operation 820, processor 1420 can store the global maximum value (the maximum value among local maximum values) corresponding to the current layer in a register.

[0149] The processor 1420 can obtain the global maximum value corresponding to the current layer by calculating the maximum value among the local maximum values ​​of all output channels of the current layer.

[0150] The processor 1420 can read the global maximum value stored in the register to perform secondary quantization on the locally recovered data in subsequent processing.

[0151] Figure 9 This is a diagram illustrating an example of data from a quantized neural network according to embodiments of the present disclosure.

[0152] In the neural network, calculator 910 (also referred to as a "MAC array") performs convolution operations. The calculator described below can be included in the processing unit of processor 1420. Calculator 910 can execute the operations of the next layer after performing the operations of the current layer. The output data produced by the operations of the current layer can be used for the operations of the next layer after primary and secondary quantization.

[0153] When the neural network operates via forward propagation, the output data 940 generated by the operations of the current layer may include activations. When the neural network is trained via backpropagation, the output data 940 generated by the operations of the current layer may include loss gradients or weight gradients.

[0154] Processor 1420 can recover output data 940 based on the global maximum value corresponding to the previous layer, thereby generating globally recovered data 950. Processor 1420 can recover output data 940 based on the global maximum value corresponding to the previous layer, and perform sub-operations such as pooling, permutation normalization, and activation using an activation function, thereby generating globally recovered data 950. Processor 1420 can add noise to output data 940 to generate globally recovered data.

[0155] Processor 1420 can restore output data 940 for each output channel of the current layer. Processor 1420 can restore output data 940 sequentially for each output channel of the current layer. For example, when the current layer includes n output channels (n is a natural number greater than 1), processor 1420 can restore the output data 941 of the first output channel, then restore the output data 942 of the second output channel, and finally restore the output data 943 of the nth output channel.

[0156] Processor 1420 can calculate local maximum values ​​(Local Max#1, Local Max#2, ..., Local Max#n) from global recovery data 950. For example, when the current layer includes n output channels, processor 1420 can calculate the maximum value of global recovery data 951 for the first output channel as the local maximum value (Local Max#1), ​​calculate the maximum value of global recovery data 952 for the second output channel as the local maximum value (Local Max#2), and finally calculate the maximum value of global recovery data 953 for the nth output channel as the local maximum value (Local Max#n). The calculated local maximum values ​​(Local Max#1, Local Max#2, ..., Local Max#n) can be stored in register 930 of processor 1420.

[0157] Processor 1420 can calculate the global maximum value (Global Max) corresponding to the current layer from the local maximum values ​​Local Max#1, Local Max#2, ..., Local Max#n. For example, processor 1420 can calculate the maximum value among the local maximum values ​​Local Max#1, Local Max#2, ..., Local Max#n as the global maximum value (Global Max) corresponding to the current layer. The calculated global maximum value (Global Max) can be stored in register 930.

[0158] The processor 1420 can perform initial quantization on the global recovery data 950 based on the local maximum values ​​Local Max#1, Local Max#2, ..., Local Max#n, thereby generating locally quantized data 960.

[0159] Processor 1420 can perform initial quantization on global recovery data 950 for each output channel of the current layer. Processor 1420 can sequentially perform initial quantization on global recovery data 950 for each output channel of the current layer. For example, when the current layer includes n output channels, processor 1420 can quantize the global recovery data 951 of the first output channel based on the local maximum value Local Max#1, then quantize the global recovery data 952 of the second output channel based on the local maximum value Local Max#2, and finally quantize the global recovery data 953 of the nth output channel based on the local maximum value Local Max#n.

[0160] Local quantization data 960 can be stored in memory 920 for use in the next layer of operation. Memory 920 can be memory 1410 of neural network device 1400, or on-chip memory included in processor 1420.

[0161] Processor 1420 can load local quantization data 970 from memory 920 and recover the loaded local quantization data 970 based on the local maximum value of each output channel of the current layer, thereby generating local recovery data 980. The local maximum value of each output channel of the current layer can be read from register 930.

[0162] Processor 1420 can recover local quantized data 970 for each output channel of the current layer. Processor 1420 can recover local quantized data 970 sequentially for each output channel of the current layer. For example, when the current layer includes n output channels, processor 1420 can recover the local quantized data 971 of the first output channel based on the local maximum value Local Max#1, then recover the local quantized data 972 of the second output channel based on the local maximum value Local Max#2, and finally recover the local quantized data 973 of the nth output channel based on the local maximum value Local Max#n.

[0163] Local recovery data 980 can be the same as global recovery data 950. For example, when the current layer includes n output channels, the global recovery data 951 of the first output channel can be the same as local recovery data 981, the global recovery data 952 of the second output channel can be the same as local recovery data 982, and finally, the global recovery data 953 of the nth output channel can be the same as local recovery data 983.

[0164] Processor 1420 can perform secondary quantization on local recovered data 980 based on the global maximum value (Global Max) corresponding to the current layer, thereby generating globally quantized data 990. The global maximum value (Global Max) corresponding to the current layer can be read from register 930.

[0165] Processor 1420 can perform secondary quantization on the locally recovered data 980 for each output channel of the current layer. Processor 1420 can sequentially perform secondary quantization on the locally recovered data 980 for each output channel of the current layer. For example, when the current layer includes n output channels, processor 1420 can quantize the locally recovered data 981 of the first output channel based on the global maximum value Global Max, then quantize the locally recovered data 982 of the second output channel based on the global maximum value Global Max, and finally quantize the locally recovered data 983 of the nth output channel based on the global maximum value Global Max.

[0166] The processor 1420 can perform secondary quantization on the partially recovered data 980 for all output channels, thereby quantizing the partially recovered data 980 to the same scale.

[0167] The global quantization data 990 can be input into the calculator 910 for use in the next layer of operation.

[0168] Figure 10 This is a diagram illustrating an example of obtaining local and global maximum values ​​according to embodiments of the present disclosure.

[0169] When the current layer includes four output channels, a local maximum value can be calculated for each of the four output channels.

[0170] The processor 1420 can calculate the local maximum value (Local Max#1) of the first output channel from the global recovery data 1010 of the first output channel. For example, the processor 1420 can calculate the maximum value 6 of the global recovery data 1010 of the first output channel as the local maximum value (Local Max#1).

[0171] Similarly, the processor 1420 can calculate the local maximum value Local Max#2 of the second output channel, the local maximum value Local Max#3 of the third output channel, and the local maximum value Local Max#4 of the fourth output channel from the global recovery data 1020 of the second output channel, the global recovery data 1030 of the third output channel, and the global recovery data 1040 of the fourth output channel.

[0172] Processor 1420 can calculate the global maximum value (Global Max) corresponding to the current layer from the local maximum values ​​Local Max#1, Local Max#2, Local Max#3, and Local Max#4. For example, processor 1420 can calculate the maximum value of 18 among the local maximum values ​​Local Max#1, Local Max#2, Local Max#3, and Local Max#4 as the global maximum value (Global Max).

[0173] Figure 11 This is a diagram illustrating examples of primary and secondary quantization for each output channel according to embodiments of the present disclosure.

[0174] More specifically, Figure 11 An example is shown where the output data 1111 and 1121 generated from the operation of the current layer, which includes two output channels, are first quantized and then second quantized to 8 bits.

[0175] Processor 1420 can recover the output data 1111 of the first channel, thereby generating global recovered data 1112 of the first channel. Processor 1420 can recover the output data 1111 of the first channel based on the global maximum value Global Max@Previous Layer corresponding to the previous layer. For example, the output data 1111 can be recovered based on Equation 4, and more specifically, the value of the output data 1111 can be recovered to be 144 / 2 based on the global maximum value Global Max@Previous Layer 64 corresponding to the previous layer. 8 The data is used to generate global recovery data with a value of 8. For example, when max(|x|) is 64 and maxq n (x) is 144 / 2 8 At that time, Equation 4 can be used to generate global recovery data x with a value of 8.

[0176] Processor 1420 can calculate the local maximum value (Local Max#1) of the first channel. For example, processor 1420 can calculate the maximum value 8 of the global recovery data 1112 as the local maximum value (Local Max#1).

[0177] The processor 1420 can perform initial quantization on the global recovery data 1112 of the first channel based on the local maximum value (Local Max#1) of the first channel, thereby generating locally quantized data 1113 of the first channel. For example, the global recovery data 1112 can be initially quantized based on equations 5 and 6. More specifically, the data with a value of 2 in the global recovery data 1112 can be initially quantized based on the local maximum value (Local Max#1) of the first channel, thereby generating a value of 160 / 2. 8 Locally quantized data. For example, when x is 2 and max(|x|) is 8, Equation 5 can be used to generate a value of 160 / 2. 8 local quantization data maxq n (x).

[0178] Similarly, processor 1420 can recover the output data 1121 of the second channel, thereby generating global recovered data 1122 for the second channel. Processor 1420 can recover the output data 1121 based on the global maximum value Global Max@PreviousLayer corresponding to the previous layer. For example, the value of output data 1121 can be recovered to be 384 / 2 based on the global maximum value Global Max@Previous Layer 64 corresponding to the previous layer. 8 The data is used to generate global recovery data with a value of 128. For example, when max(|x|) is 64 and maxq n (x) is 384 / 2 8At that time, Equation 4 can be used to generate global recovery data x with a value of 128.

[0179] Similarly, processor 1420 can calculate the local maximum value (Local Max#2) of the second channel. For example, processor 1420 can calculate the maximum value (128) of the global recovery data (1122) as the local maximum value (Local Max#2).

[0180] Similarly, the processor 1420 can perform initial quantization on the global recovery data 1122 of the second channel based on the local maximum value (Local Max#2) of the second channel, thereby generating locally quantized data 1123 of the second channel. For example, the data with a value of 32 in the global recovery data 1122 can be initially quantized based on the local maximum value (Local Max#2) of the second channel (128), thereby generating a value of 160 / 2. 8 Locally quantized data. For example, when x is 32 and max(|x|) is 128, Equation 5 can be used to generate a value of 160 / 2. 8 local quantization data maxq n (x).

[0181] The processor 1420 can calculate the global maximum value (Global Max@Current Layer) corresponding to the current layer from the local maximum value (Local Max#1) of the first channel and the local maximum value (Local Max#2) of the second channel. For example, the processor 1420 can calculate the maximum value of 128 in the local maximum values ​​(Local Max#1) of the first channel and the second channel as the global maximum value (Global Max@CurrentLayer) corresponding to the current layer.

[0182] The local maximum value (Local Max#1) of the first channel, the local maximum value (Local Max#2) of the second channel, and the global maximum value (Global Max@CurrentLayer) corresponding to the current layer can be stored in the registers of the processor 1420.

[0183] Processor 1420 can recover the local quantized data 1113 of the first channel, thereby generating local recovered data 1114 of the first channel. Processor 1420 can recover the local quantized data 1113 of the first channel based on the local maximum value Local Max#1 of the first channel. For example, the local quantized data 1113 can be recovered based on Equation 4, and more specifically, the value of the local quantized data 1113 can be recovered to be 144 / 2 based on the local maximum value Local Max#1 of the first channel. 8The data is used to generate locally recovered data with a value of 1. For example, when max(|x|) is 8 and maxq n (x) is 144 / 2 8 At that time, Equation 4 can be used to generate local recovery data x with a value of 1.

[0184] The processor 1420 can perform secondary quantization on the local restored data 1114 of the first channel based on the global maximum value (Global Max@Current Layer) corresponding to the current layer, thereby generating globally quantized data 1115 of the first channel. For example, the local restored data 1114 can be secondary quantized based on Equations 5 and 6. More specifically, the data with a value of 4 in the local restored data 1114 can be secondary quantized based on the global maximum value (Global Max@Current Layer) 128 corresponding to the current layer, thereby generating data with a value of 132 / 2. 8 Globally quantized data. For example, when x is 4 and max(|x|) is 128, Equation 5 can be used to generate a value of 132 / 2. 8 global quantization data maxq n (x).

[0185] Similarly, processor 1420 can recover the local quantization data 1123 of the second channel, thereby generating local recovered data 1124 of the second channel. Processor 1420 can recover the local quantization data 1123 of the second channel based on the local maximum value (Local Max#2) of the second channel. For example, the value of the local quantization data 1123 can be recovered to be 144 / 2 based on the local maximum value (Local Max#2) of the second channel (128). 8 The data is used to generate local recovery data with a value of 16. For example, when max(|x|) is 128 and maxq n (x) is 144 / 2 8 At that time, Equation 4 can be used to generate local recovery data x with a value of 16.

[0186] The processor 1420 can perform secondary quantization on the local recovery data 1124 of the second channel based on the global maximum value (Global Max@Current Layer) corresponding to the current layer, thereby generating globally quantized data 1125 for the second channel. For example, based on the global maximum value (Global Max@Current Layer) 128 corresponding to the current layer, the data with a value of 64 in the local recovery data 1124 can be secondary quantized to generate a value of 192 / 2. 8 The global quantization data. For example, when x is 64 and max(|x|) is 128, Equation 5 can be used to generate a value of 192 / 2. 8global quantization data maxq n (x).

[0187] Figure 12 This illustrates the use according to embodiments of the present disclosure. Figure 7 The diagram illustrates the data quantization algorithm for the neural network method shown in the figure.

[0188] Reference Figure 12 In the quantization algorithm, the backpropagation loss gradient is quantized.

[0189] In operation 1201, processor 1420 can recover the local quantization data loss_gradient for each input channel of the current layer. G This generates locally recovered data, `loss_gradient`. In this case, the input channels of the current layer can correspond to the output channels in the forward propagation. The processor 1420 can then use local maxima `local_maxc`. in To recover the local quantized data loss_gradient G .

[0190] In operation 1202, processor 1420 can quantize the local recovery data loss_gradient of each input channel of the current layer, thereby generating the globally quantized data loss_gradient. G The processor 1420 can be based on the global maximum value global_max. i To quantize the local recovery data loss_gradient, thereby generating the globally quantized data loss_gradient. G .

[0191] In operation 1203, processor 1420 can perform global quantization on the data loss_gradient. G Perform convolution operation.

[0192] In operation 1204, processor 1420 can recover the output data loss_gradient generated by the operation of the current layer for each output channel of the current layer. G In this case, the output channels of the current layer can correspond to the input channels in the forward propagation. This can be based on the global maximum value `global_max`. i To recover the output data loss_gradient G .

[0193] In operation 1205, processor 1420 may perform sub-operations on the recovered data (such as permutation normalization, pooling operations, activation using activation functions, etc.) to generate global recovered data.

[0194] In operation 1206, processor 1420 can calculate the local maximum value local_maxc for each output channel of the current layer. out .

[0195] In operation 1207, processor 1420 can quantize the global recovery data loss_gradient for each output channel of the current layer, thereby generating locally quantized data loss_gradient. G Processor 1420 can be based on local maximum local_maxc out To quantize the global recovery data loss_gradient, thereby generating the locally quantized data loss_gradient. G .

[0196] In operation 1208, processor 1420 can obtain the local maximum value local_max from all output channels for the current layer. C1 local_max C2 ... to calculate the global maximum value global_max 1-1 .

[0197] Figure 13 This is a diagram illustrating an example of data from a quantized neural network according to embodiments of the present disclosure.

[0198] In a neural network, the calculator (also known as a "MAC array") 1310 performs convolution operations. The calculator 1310 can execute the operations of the next layer after performing the operations of the current layer. After quantization, the output data produced by the operations of the current layer can be used in the operations of the next layer.

[0199] When the neural network operates via forward propagation, the output data 1330 generated by the operations of the current layer may include activations. When the neural network is trained via backpropagation, the output data 1330 generated by the operations of the current layer may include loss gradients or weight gradients.

[0200] Processor 1420 can predict the global maximum value used to recover output data 1330. For example, processor 1420 can predict the global maximum value based on the output data produced by the operation of the previous layer. For example, processor 1420 can predict the global maximum value based on aspects of data changes during quantization.

[0201] Processor 1420 can recover output data 1330 based on the predicted global maximum value, thereby generating globally recovered data 1340. Processor 1420 can recover output data 1330 based on the predicted global maximum value and perform sub-operations (such as pooling operations, permutation normalization, activation using an activation function, etc.) to generate globally recovered data 1340. Processor 1420 can recover output data 1330 based on the predicted global maximum value and add noise to the recovered data to generate globally recovered data 1340.

[0202] Processor 1420 can restore the output data 1330 of each output channel of the current layer. Processor 1420 can restore the output data 1330 of each output channel of the current layer sequentially. For example, when the current layer includes n output channels, processor 1420 can restore the output data 1331 of the first output channel, restore the output data 1332 of the second output channel, and finally restore the output data 1333 of the nth output channel.

[0203] The processor 1420 can calculate local maxima from the global recovery data 1340. For example, when the current layer includes n output channels, the processor 1420 can calculate the maximum value of the global recovery data 1341 of the first output channel as the local maximum value of the first output channel, calculate the maximum value of the global recovery data 1342 of the second output channel as the local maximum value of the second output channel, and finally calculate the maximum value of the global recovery data 1343 of the nth output channel as the local maximum value of the nth output channel.

[0204] Processor 1420 can generate locally quantized data 1350 by quantizing global recovery data 1340 based on local maximum values.

[0205] Processor 1420 can quantize the global recovery data 1340 of each output channel of the current layer. Processor 1420 can sequentially quantize the global recovery data 1340 of each output channel of the current layer. For example, when the current layer includes n output channels, processor 1420 can quantize the global recovery data 1341 of the first output channel based on the local maximum value of the first output channel, quantize the global recovery data 1342 of the second output channel based on the local maximum value of the second output channel, and finally quantize the global recovery data 1343 of the nth output channel based on the local maximum value of the nth output channel.

[0206] Processor 1420 can compensate for local quantization data 1350 based on the difference between the local maximum value and the predicted global maximum value of each output channel of the current layer, thereby generating global quantization data 1360. For example, when the local maximum value and the predicted global maximum value of each output channel are equal to each other, processor 1420 can set the local quantization data 1350 as global quantization data 1360. For example, when the local maximum value of each output channel is less than the predicted global maximum value, processor 1420 can perform compensation in such a way that the local quantization data 1350 is reduced proportionally to the difference between the local maximum value and the predicted global maximum value, thereby generating global quantization data 1360.

[0207] Processor 1420 can compensate for the local quantization data 1350 of each output channel of the current layer. Processor 1420 can sequentially compensate for the local quantization data 1350 of each output channel of the current layer. For example, when the current layer includes n output channels, processor 1420 can compensate for the local quantization data 1351 of the first output channel based on the difference between a first local maximum and a predicted global maximum, compensate for the local quantization data 1352 of the second output channel based on the difference between a second local maximum and a predicted global maximum, and finally compensate for the local quantization data 1353 of the nth output channel based on the difference between the nth local maximum and the predicted global maximum.

[0208] Because the processor 1420 generates global quantized data 1360 without storing local maxima in registers, the amount of resources used by registers in quantization can be reduced.

[0209] Global quantization data 1360 can be stored in and read from memory 1320, and input to calculator 1310 for use in the next layer of operation. Memory 1320 can be memory 1410 of neural network device 1400, or it can be on-chip memory included in processor 1420.

[0210] Figure 14 This is a block diagram of an example neural network device for recognizing images according to embodiments of the present disclosure.

[0211] Reference Figure 14 The neural network device 1400 for image recognition may include a memory 1410 and a processor 1420. Although in Figure 14 It is not shown in the diagram, but the neural network device 1400 can be connected to an external memory. Figure 14 The neural network device 1400 shown may include components associated with the current embodiment. Therefore, it will be apparent to those skilled in the art that the neural network device 1400 may also include components other than those shown in the present embodiment. Figure 14 Other general components besides those shown.

[0212] Neural network device 1400 can be a reference Figure 1 and Figure 2 Devices for implementing the aforementioned neural network. For example, the neural network device 1400 can be implemented using various types of devices (such as personal computers (PCs), server devices, mobile devices, embedded devices, etc.). Specifically, the neural network device 1400 may be included in, but is not limited to, smartphones, tablets, augmented reality (AR) devices, Internet of Things (IoT) devices, autonomous vehicles, robotic devices, or medical devices that perform image recognition using neural networks. The neural network device 1400 may correspond to a dedicated hardware (HW) accelerator installed on such a device, and may be an HW accelerator (such as a neural processor (NPU), tensor processor (TPU), or neural engine) that serves as a dedicated module for driving the neural network. The neural network device 1400 may be... Figure 5 The neural network learning device 51 or the neural network inference device 53.

[0213] The memory 1410 stores various types of data processed in the neural network device 1400. For example, the memory 1410 may store data that is being processed or to be processed in the neural network device 1400 for data processing. In addition, the memory 1410 may store applications or drivers that will be driven by the neural network device 1400.

[0214] For example, memory 1410 may include random access memory (RAM) (such as dynamic random access memory (DRAM) or static random access memory (SRAM)), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM, Blu-ray disc, optical disc storage device, hard disk drive (HDD), solid-state drive (SSD), or flash memory.

[0215] Processor 1420 controls the overall functions of neural network device 1400 in executing neural network functions. For example, processor 1420 typically controls neural network device 1400 to process data by executing programs stored in memory 1410. Processor 1420 may be implemented as a central processing unit (CPU), graphics processing unit (GPU), or application processor (AP) included in neural network device 1400, but is not limited thereto.

[0216] Processor 1420 may read data (e.g., image data, feature map data, or kernel data) from memory 1410 or write data (e.g., image data, feature map data, or kernel data) to memory 1410 and execute a neural network using the read / written data. When executing the neural network, processor 1420 may drive processing units disposed therein to repeatedly perform operations between the input feature map and the kernel, thereby generating data associated with the output feature map. Here, the operation count of the operation may be determined based on various factors such as the number of channels in the input feature map, the number of channels in the kernel, the size of the input feature map, the size of the kernel, and the precision of the value.

[0217] For example, the processing unit may include logic circuitry for arithmetic operations. More specifically, the processing unit may include a calculator implemented using a combination of multipliers, adders, and accumulators. A multiplier may include a combination of multiple sub-multipliers, and an adder may also include a combination of multiple sub-adders.

[0218] Processor 1420 may also include on-chip memory responsible for registers (or register files) and caches for processing operations, and an allocator for allocating various operands (such as pixel values ​​of an input feature map or weights of a filter). For example, the allocator may allocate operands (such as pixel values ​​and weights) required for an operation to be performed by the processing unit from data stored in memory 1410 to the on-chip memory. The allocator may then reassign the operands allocated to the on-chip memory to the processing unit for the operation.

[0219] Furthermore, the foregoing embodiments of this disclosure can be written as programs executable on a computer and can be implemented on a general-purpose digital computer using a computer-readable recording medium. The structure of the data used in the foregoing embodiments of this disclosure can be recorded on a computer-readable recording medium using various methods. Examples of computer-readable recording media include magnetic storage media (e.g., ROM, floppy disk, hard disk, etc.), optical recording media (e.g., CD-ROM or DVD), etc.

[0220] It should be understood that the embodiments described herein should be considered in a descriptive sense only and not for limiting purposes. The description of features or aspects within each embodiment should generally be considered applicable to other similar features or aspects in other embodiments of this disclosure. Although one or more embodiments have been described with reference to the accompanying drawings, those skilled in the art will understand that various changes in form and detail may be made therein without departing from the spirit and scope defined by the claims.

Claims

1. A method of identifying an image, the method comprising: obtaining image data to be identified as input data of a neural network; for a current layer among at least one layer within the neural network, generating output data of the current layer by performing an operation of the current layer on input data of the current layer, wherein the output data of the current layer includes a plurality of output channels; based on local maximum values of each of the output channels of the current layer, obtaining local quantization data by performing primary quantization on global recovery data recovered from the output data generated by the operation of the current layer based on global maximum values corresponding to a previous layer preceding the current layer; storing the local quantization data in a memory; based on global maximum values corresponding to the current layer, obtaining global quantization data by performing secondary quantization on local recovery data recovered from the local quantization data based on the local maximum values of each of the output channels; providing the global quantization data as input data of a next layer succeeding the current layer to the next layer to perform an operation of the next layer; and outputting an image identification result based on output data generated by the operation of a final layer of the neural network.

2. The method of claim 1, wherein, The step of obtaining the local quantization data includes obtaining the global recovery data by recovering, for each of the output channels, the output data generated by the operation of the current layer based on global maximum values corresponding to the previous layer, the global maximum values corresponding to the previous layer being generated based on data obtained by recovering output data generated by an operation of the previous layer.

3. The method of claim 1, wherein, The step of obtaining the local quantization data includes obtaining the local quantization data by performing the primary quantization on the global recovery data of each of the output channels based on the local maximum values being maximum values of the global recovery data of each of the output channels, such that the maximum values of the global recovery data of each of the output channels are quantized to the same value.

4. The method of claim 1, wherein, The step of obtaining the global quantization data includes obtaining the local recovery data by recovering the local quantization data of each of the output channels based on the local maximum values being maximum values of the global recovery data of each of the output channels, such that the local quantization data is recovered to data before the primary quantization.

5. The method of claim 1, wherein, The step of obtaining the global quantization data includes obtaining the global quantization data by performing the secondary quantization on the local recovery data of each of the output channels based on the global maximum values being maximum values of the local recovery data, such that the local recovery data is quantized to the same scale.

6. The method of claim 1, wherein, The step of obtaining the local quantization data further includes: recovering the output data generated by the operation of the current layer based on global maximum values corresponding to the previous layer; and obtaining the global recovery data by performing a pooling operation, permutation normalization, or activation using an activation function on the recovered data. 7.The method of claim 1, further comprising: storing local maximum values obtained by calculating maximum values of the global recovery data of each of all of the output channels of the current layer; and storing the global maximum values corresponding to the current layer in a register, the global maximum values corresponding to the current layer being obtained by calculating a maximum value among the local maximum values. The step of obtaining the local quantization data is sequentially performed for each of the output channels.

8. The method of claim 1, wherein, ​ 9. The method of claim 1, wherein, When the current layer is the first layer of the neural network, the output data of the first layer is generated by performing the operation of the first layer on the training image data, and when the current layer is a layer other than the first layer of the neural network, the output data of the current layer is generated by performing the operation of the current layer on the global quantized data obtained based on the output data of the previous layer. 10.A method of training a neural network for recognizing images, the method comprising: obtaining training image data; performing training on the neural network based on the training image data, wherein, in each iteration of the training, for a current layer in at least one layer within the neural network, the step of obtaining input data of a next layer following the current layer based on output data of the current layer comprises: obtaining local quantized data by performing primary quantization on global recovered data based on local maxima of each output channel of the current layer, the output data of the current layer comprising a plurality of output channels, the global recovered data recovered from output data generated by the operation of the current layer based on global maxima corresponding to a previous layer preceding the current layer; storing the local quantized data in a memory; obtaining global quantized data by performing secondary quantization on local recovered data based on global maxima corresponding to the current layer, the local recovered data recovered from the local quantized data based on the local maxima of each output channel; and providing the global quantized data as the input data of the next layer to the next layer to perform the operation of the next layer.

11. The method of claim 10, wherein, the operation of the current layer and the operation of the next layer are based on backpropagation, and the output data of the operation of the current layer and the input data of the operation of the next layer comprise any one of loss gradients and weight gradients.

12. The method of claim 10, wherein, When the current layer is the first layer of the neural network, the output data of the first layer is generated by performing the operation of the current layer on the training image data; and when the current layer is a layer other than the first layer of the neural network, the output data of the current layer is generated by performing the operation of the current layer on the global quantized data obtained based on the output data of the previous layer. 13.A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the method of any one of claims 1 to 12. 14.An apparatus for recognizing images, the apparatus comprising: a memory; and a processor configured to perform neural network inference based on obtained image data by executing at least one program to perform image recognition, wherein the processor is further configured to: obtain image data to be recognized as input data of the neural network; for a current layer in at least one layer within the neural network, generate output data of the current layer by performing the operation of the current layer on input data of the current layer, wherein the output data of the current layer comprises a plurality of output channels; obtain local quantized data by performing primary quantization on global recovered data based on local maxima of each output channel of the current layer, the global recovered data recovered from output data generated by the operation of the current layer based on global maxima corresponding to a previous layer preceding the current layer; store the local quantized data in the memory; obtain global quantized data by performing secondary quantization on local recovered data based on global maxima corresponding to the current layer, the local recovered data recovered from the local quantized data based on the local maxima of each output channel; and provide the global quantized data as the input data of the next layer to the next layer to perform the operation of the next layer. obtaining global quantization data by performing secondary quantization on local restored data based on a global maximum value corresponding to the current layer, the local restored data being restored from the local quantization data based on a local maximum value for each output channel; providing the global quantization data as input data for a next layer following the current layer to the next layer to perform an operation of the next layer; and outputting an image recognition result based on output data resulting from the operation of the final layer of the neural network.

15. The apparatus of claim 14, wherein, When the current layer is a first layer of the neural network, the output data of the first layer is generated by performing the operation of the first layer on the image data to be recognized, and when the current layer is a layer other than the first layer of the neural network, the output data of the current layer is generated by performing the operation of the current layer on the global quantization data obtained based on the output data of the previous layer.

16. The apparatus of claim 14, wherein, The processor is further configured to obtain global restored data by restoring the output data resulting from the operation of the current layer for each output channel based on a global maximum value corresponding to the previous layer, the global maximum value corresponding to the previous layer being generated based on data obtained by restoring the output data resulting from the operation of the previous layer.

17. The apparatus of claim 14, wherein, The processor is further configured to obtain the local quantization data by performing primary quantization on the global restored data for each output channel based on a local maximum value that is a maximum value of the global restored data for each output channel, such that the maximum value of the global restored data for each output channel is quantized to the same value.

18. The apparatus of claim 14, wherein, The processor is further configured to obtain the local restored data by restoring the local quantization data for each output channel based on a local maximum value that is a maximum value of the local quantization data for each output channel, such that the local quantization data is restored to data before the primary quantization.

19. The apparatus of claim 14, wherein, The processor is further configured to obtain the global quantization data by performing secondary quantization on the local restored data for each output channel based on a global maximum value that is a maximum value of the local restored data, such that the local restored data is quantized to the same scale.

20. The apparatus of claim 14, wherein, The processor is further configured to: restore the output data resulting from the operation of the current layer based on a global maximum value corresponding to the previous layer; and obtain the global restored data by performing a pooling operation, permutation normalization, or activation using an activation function on the restored data.

21. The apparatus of claim 14, wherein, The processor is further configured to: store the local maximum value obtained by calculating a maximum value of the global restored data in each of all output channels of the current layer; and store the global maximum value corresponding to the current layer in a register, the global maximum value corresponding to the current layer being obtained by calculating a maximum value among the local maximum values.

22. The apparatus of claim 14, wherein, The processor is further configured to sequentially obtain the local quantization data for each output channel.

23. An apparatus of training a neural network for recognizing an image, the apparatus comprising: a memory; and a processor configured to obtain training image data, perform training on the neural network based on the training image data, wherein the processor is further configured to, in each iteration of the training, for a current layer in at least one layer within the neural network, obtain input data for a next layer following the current layer based on output data of the current layer, wherein the processor is further configured to obtain the output data of the next layer by performing an operation of the next layer on the input data for the next layer. The step of obtaining the input data of the next layer based on the output data of the current layer comprises: obtaining, based on the local maximum value of each output channel of the current layer, local quantization data by performing primary quantization on global recovery data, the output data of the current layer comprising a plurality of output channels, the global recovery data being recovered from the output data resulting from the operation of the current layer based on a global maximum value corresponding to a previous layer before the current layer; storing the local quantization data in a memory; obtaining, based on the global maximum value corresponding to the current layer, global quantization data by performing secondary quantization on local recovery data, the local recovery data being recovered from the local quantization data based on the local maximum value of each output channel; and providing the global quantization data as the input data of the next layer to the next layer to perform the operation of the next layer.

24. The apparatus of claim 23, wherein, The processor is further configured to: perform the operation of the current layer and the operation of the next layer based on backpropagation, and the output data of the operation of the current layer and the input data of the operation of the next layer comprise any one of a loss gradient and a weight gradient.

Images