Method and apparatus for processing using neural networks

Abstract

A method and apparatus for processing using a neural network are provided. The method includes generating an integral image for each channel in a first layer of the neural network based on a region and of pixel values in a first output feature map of the channel in the first layer, generating an accumulated integral image by performing an accumulation operation on the integral images generated for the respective channels, obtaining a pre-output feature map of a second layer by performing a convolution operation between an input feature map of the second layer and a weight kernel after the first layer, and removing a bias in the weight kernel by subtracting an accumulated value of the accumulated integral image from a pixel value of the pre-output feature map to obtain a second output feature map of the second layer.

Description

[0001] Cross-reference to related applications

[0002] This application is based on the benefit of Korean Patent Application No. 10-2019-0168142, filed on December 16, 2019, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0003] The following description relates to methods and apparatuses that utilize neural network processing. Background Technology

[0004] Neural networks are computing systems based on computational architectures. Neural network technology can analyze input data and extract useful information from it.

[0005] Neural network devices typically require extensive computation on complex input data. To enable a typical neural network device to handle such computations, it must often perform operations that read or write large amounts of data from or to memory, which can consume significant amounts of energy due to frequent memory accesses. Low-power and high-performance systems, such as mobile devices or Internet of Things (IoT) devices, typically have limited resources and therefore require techniques to reduce the energy consumption needed to process large amounts of data. Summary of the Invention

[0006] This summary is provided to introduce some concepts in a simplified form, which will be further described below in detail in the specific embodiments. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help define the scope of the claimed subject matter.

[0007] In one general aspect, a method for processing a neural network includes: generating an integral map for each channel in the first layer based on the sum of pixel values ​​in a region of a first output feature map of a channel in the first layer of the neural network; generating a cumulative integral map by performing a summation operation on the integral maps generated for each channel; obtaining a pre-output feature map of the second layer by performing a convolution operation between an input feature map of the second layer after the first layer and a weight kernel; and obtaining a second output feature map of the second layer by removing the offset in the weight kernel by subtracting the summation value of the cumulative integral map from the pixel values ​​of the pre-output feature map.

[0008] The method may also include performing recognition of the second output feature map.

[0009] The weight kernel can include weights obtained through asymmetric quantization of the neural network.

[0010] The integral image may include data obtained by summing the pixel values ​​in the region from the reference pixel of the first output feature image to the first output pixel of the first output feature image, and setting the value of the pixel in the integral image corresponding to the first output pixel.

[0011] The reference pixel can be set as one of the four corner pixels of the first output feature map.

[0012] An integral image can be generated by performing a pixel-by-pixel summation operation on the integral images generated for each channel.

[0013] The cumulative integral graph generated from the first layer can correspond to data about the offset of the weight kernels of the second layer, generated by the asymmetric quantization of the neural network.

[0014] Obtaining the second output feature map may include: determining the bounding box of the input feature map mapped to the weight kernel to obtain the pre-output pixels of the pre-output feature map of the second layer; obtaining the pixel values ​​of the pixels in the accumulated integral map that are set to correspond to the four corner pixels of the bounding box; calculating the offset in the pixel values ​​of the pre-output pixels based on the obtained pixel values; and obtaining the second output pixel of the second output feature map by subtracting the calculated offset from the pixel values ​​of the pre-output pixels.

[0015] The four corner pixels of the bounding box can include the top-right pixel, top-left pixel, bottom-right pixel, and bottom-left pixel of the bounding box. Obtaining the pixel value of the pixel set as the cumulative integral image can include: obtaining the pixel value of the first integral image pixel corresponding to the top-right pixel, the pixel value of the second integral image pixel corresponding to the top-left pixel, the pixel value of the third integral image pixel corresponding to the bottom-right pixel, and the pixel value of the fourth integral image pixel corresponding to the bottom-left pixel from the cumulative integral image.

[0016] When the reference pixel used to generate the cumulative integral map is the lower left pixel of the first output feature map, calculating the offset may include: calculating the offset in the pixel value of the pre-output pixel by subtracting the pixel values ​​of the second integral map pixel and the third integral map pixel from the sum of the pixel values ​​of the first integral map pixel and the pixel values ​​of the fourth integral map pixel.

[0017] A non-transitory computer-readable recording medium may store instructions that, when executed by a processor, cause the processor to control the execution of the above-described method.

[0018] In another aspect, a neural processing device includes one or more processors configured to: generate an integral map for each channel in a first layer of a neural network based on the sum of pixel values ​​in a region of a first output feature map of a channel in a first layer; generate a cumulative integral map by performing an accumulation operation on the integral maps generated for each channel; obtain a pre-output feature map of a second layer by performing a convolution operation between an input feature map of a second layer after the first layer and a weight kernel; and obtain a second output feature map of the second layer by removing the offset in the weight kernel by subtracting the accumulated value of the cumulative integral map from the pixel values ​​of the pre-output feature map.

[0019] The weight kernel can include weights obtained through asymmetric quantization of the neural network.

[0020] The integral image may include data obtained by summing the pixel values ​​in the region from the reference pixel of the first output feature image to the first output pixel of the first output feature image, and setting the value of the pixel in the integral image corresponding to the first output pixel.

[0021] The reference pixel can be set as one of the four corner pixels of the first output feature map.

[0022] An integral image can be generated by performing a pixel-by-pixel summation operation on the integral images generated for each channel.

[0023] The cumulative integral graph generated from the first layer can correspond to data about the offset of the weight kernels of the second layer, generated by the asymmetric quantization of the neural network.

[0024] The one or more processors may also be configured to: determine the bounding box of the input feature map mapped to the weight kernel to obtain the pre-output pixel of the pre-output feature map of the second layer; obtain the pixel value of the pixel of the accumulated integral map corresponding to the four corner pixels of the bounding box; calculate the offset in the pixel value of the pre-output pixel based on the obtained pixel value; and obtain the second output pixel of the second output feature map by subtracting the calculated offset from the pixel value of the pre-output pixel.

[0025] The four corner pixels of the bounding box may include the top-right pixel, top-left pixel, bottom-right pixel, and bottom-left pixel of the bounding box. The one or more processors may also be configured to: obtain from the accumulated integral image the pixel values ​​of the first integral image pixel corresponding to the top-right pixel, the second integral image pixel corresponding to the top-left pixel, the third integral image pixel corresponding to the bottom-right pixel, and the fourth integral image pixel corresponding to the bottom-left pixel.

[0026] The one or more processors may also be configured to: when the reference pixel used to generate the cumulative integral map is the lower left pixel of the first output feature map, calculate the offset in the pixel value of the pre-output pixel by subtracting the pixel values ​​of the second integral map pixel and the third integral map pixel from the sum of the pixel values ​​of the first integral map pixel and the pixel values ​​of the fourth integral map pixel.

[0027] The one or more processors may also be configured to generate an accumulated integral map based on a first output feature map of the first layer read from a memory. The memory may store the generated accumulated integral map, and when a pre-output feature map of the second layer is obtained, the one or more processors may also be configured to obtain a second output feature map of the second layer by reading the generated accumulated integral map from the memory and performing offset removal.

[0028] In another general aspect, a neural processing device includes one or more processors configured to: obtain an accumulated integral map; obtain multiple channels of a pre-output feature map of the second layer by performing convolution operations between multiple channels of an input feature map of a second layer after a first layer in a neural network and weight kernels; and generate multiple channels of an output feature map of the second layer by subtracting a selected accumulated value of the accumulated integral map from the corresponding pixel value of the pre-output feature map for each channel of the output feature map.

[0029] The one or more processors may also be configured to generate a corresponding integral map based on the region sum of corresponding pixel values ​​in each of the multiple channels of the first output feature map in the first layer of the computational neural network.

[0030] An accumulated integral graph can be generated by performing an accumulation operation on each integral graph.

[0031] Other features and aspects will become clear from the following detailed description, drawings, and claims. Attached Figure Description

[0032] Figure 1 It is a diagram of the architecture of a neural network according to one or more embodiments.

[0033] Figure 2 It is a diagram of operations performed in a neural network according to one or more embodiments.

[0034] Figure 3A and Figure 3B This is a diagram illustrating an example of convolution operations in a neural network.

[0035] Figure 4 This is a diagram illustrating floating-point and fixed-point arithmetic.

[0036] Figure 5A and Figure 5B This is a diagram of an example method for quantizing floating-point numbers to fixed-point numbers according to one or more embodiments.

[0037] Figure 6 It is a graph of the weight offset of a convolution operation resulting from asymmetric quantization weights according to one or more embodiments.

[0038] Figure 7 This is a block diagram of the hardware configuration of a neural processing device according to one or more embodiments.

[0039] Figure 8 It is a graph that removes weight offsets by using an integral graph according to one or more embodiments.

[0040] Figure 9 It is a graph that generates an integral map based on an output feature map according to one or more embodiments.

[0041] Figure 10 It is a graph that generates a cumulative integral graph by using an integral graph according to one or more embodiments.

[0042] Figure 11A and Figure 11B It is a graph that generates an output feature map by removing offsets from a pre-output feature map using an accumulated integral map, according to one or more embodiments.

[0043] Figure 12 It is a block diagram of an electronic system according to one or more embodiments.

[0044] Figure 13 This is a flowchart of a method for processing a neural network in a neural processing device according to one or more embodiments.

[0045] Throughout the accompanying drawings and detailed embodiments, the same reference numerals refer to the same elements. The drawings may not be drawn to scale, and for clarity, illustrative purposes, the relative sizes, proportions, and depictions of the elements in the drawings may be enlarged. Detailed Implementation

[0046] The following detailed description is provided to help the reader gain a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, after understanding the disclosure of this application, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent. For example, the order of operations described herein is merely illustrative and is not limited to those orders of operations set forth herein, but will be apparent after understanding the disclosure of this application that they can be changed, except for operations that must be performed in a certain order. Furthermore, for clarity and conciseness, descriptions of features that become apparent after understanding the disclosure of this application may be omitted.

[0047] The features described herein may be implemented in various forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided merely to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein that will become apparent upon understanding the disclosure of this application.

[0048] The term “and / or” as used in this document includes any one of the associated listed items and any combination of any two or more.

[0049] Although terms such as “first,” “second,” and “third” may be used in this document to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts are not limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or part from another. Therefore, without departing from the teachings of the examples described herein, the first component, assembly, region, layer, or part mentioned in the examples may also be referred to as the second component, assembly, region, layer, or part.

[0050] The terminology used herein is for the purpose of describing various examples only and is not intended to limit this disclosure. Unless the context clearly indicates otherwise, the articles “a,” “an,” and “the” are also intended to include the plural form. The terms “comprising,” “including,” and “having” indicate the presence of the stated features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0051] The features of the examples described herein can be combined in various ways that will become apparent after understanding the disclosure of this application. Furthermore, although the examples described herein have various configurations, other configurations will become apparent after understanding the disclosure of this application.

[0052] Figure 1 It is a diagram of the architecture of a neural network 1 according to one or more embodiments.

[0053] exist Figure 1In this document, neural network 1 may have a deep neural network (DNN) or an n-layer neural network architecture. A DNN or n-layer neural network may correspond to a convolutional neural network (CNN), a recurrent neural network (RNN), a deep belief network, or a restricted Boltzmann machine. For example, neural network 1 may be implemented as a CNN, but the type of neural network 1 is not limited to this. It should be noted that the use of the term "may" (e.g., what an example or embodiment may include or implement) with respect to examples or embodiments means that there exists at least one example or embodiment that includes or implements this feature, while all examples and embodiments are not limited thereto.

[0054] Neural Network 1 can be implemented as a multi-layered computational architecture with respect to the input image, feature maps, and output. In Neural Network 1, the input image can be convolved with filters called weight kernels, and the result is an output feature map (OFM). The generated output feature map can be convolved with the kernels again as the input feature map (IFM) for the next layer, and the next layer outputs a new output feature map. As a result of repeatedly performing convolution operations, Neural Network 1 can output the recognition result of the features of the input image.

[0055] For example, when inputting a 24×24 pixel image into... Figure 1 In Neural Network 1, the input image can be output as a 4-channel feature map, each with a size of 20×20 pixels, through convolution operations with kernels. Then, the size of each 20×20 feature map is reduced by repeated convolution operations with other kernels, ultimately outputting features with a size of 1×1 pixels. In this state, Neural Network 1 repeatedly performs convolution and pooling operations (or subsampling) in many layers to filter strong features from the input image features, such as those representing the entire image, and outputs the filtered features. Therefore, the recognition result of the input image can be generated by outputting the final features.

[0056] Figure 2 It is a diagram of operations performed in neural network 2 according to one or more embodiments.

[0057] exist Figure 2 In the neural network 2, there is a structure including an input layer, one or more hidden layers and an output layer, and it can perform operations based on the received input data (e.g., I1 and I2) and generate output data (e.g., 01 and 02) based on the results of the operations.

[0058] As mentioned above, neural network 2 can be a DNN with two or more hidden layers or an n-layer neural network. For example, such as Figure 2As shown, neural network 2 can be a DNN comprising an input layer (Layer 1), two hidden layers (Layer 2 and Layer 3), and an output layer (Layer 4). When neural network 2 is implemented as a DNN architecture, it can include more layers capable of processing information compared to a neural network with a single layer, and therefore can handle more complex datasets. Although neural network 2 is shown as having four layers, this is just an example, and neural network 2 can include more or fewer layers or more or fewer channels, each with one or more artificial nodes. In other words, neural network 2 can include... Figure 2 The diagram shows various layered structures with different structures.

[0059] Each layer in Neural Network 2 can include multiple channels. Each channel represents one or more artificial nodes, also referred to as corresponding neurons, processing elements (PEs), or units. For example, as... Figure 2 As shown, Layer 1 can include two channels (nodes), and Layer 2 and Layer 3 can each include three channels. However, this is just an example, and each layer in neural network 2 can include a variety of numbers of channels (and corresponding nodes).

[0060] In Neural Network 2, each layer includes channels (nodes) that can connect to other channels (nodes), and all of these channels (nodes) are capable of processing data. For example, a channel (node) can receive data from other channels (nodes) to perform calculations and output the results to other channels (nodes).

[0061] The input and output of each channel or node can be referred to as the input stimulus (e.g., the stimulus from another stimulus layer) and the output stimulus, respectively. In other words, the stimulus can be a parameter corresponding to the output of a channel (or node) and simultaneously to the inputs of channels (and corresponding nodes) included in the next layer. Each channel (or node) can determine its own stimulus output based on the input stimulus and pre-determined connection weights for the inputs from channels (and corresponding nodes) included in the previous layer. The weights are parameters used to calculate the output stimulus in each channel (node), and the weights can be values ​​assigned to the connection relationships between channels (or corresponding nodes). Inputs or input stimuli correspond to IFM, while output stimuli correspond to OFM.

[0062] Each channel (or each node, or multiple nodes per channel) can be processed by a computing unit or processing element that outputs an output excitation based on the received input, and the input-output of each channel (node) can be mapped to each other. For example, "σ" is the excitation function. It is the weight from the k-th channel (or node) included in the (i-1)-th layer to the j-th channel (or node) included in the i-th layer. It is the bias of the j-th channel (node) included in the i-th layer, and when When the excitation is applied to the j-th channel (or node) in the i-th layer, the excitation... The calculation can be performed using Formula 1 below.

[0063] Formula 1:

[0064]

[0065] like Figure 2 As shown, the excitation of the first channel CH1 (or the first node) of the second layer 2 can be generated by... This indicates that... Furthermore, according to Formula 1, It can have The value of . However, Equation 1 is merely an example used to describe the excitations and weights used to process data in neural network 2, and this disclosure is not limited thereto. The excitation can be the value obtained by applying the excitation function to the sum of excitations received from the previous layer and then passing it through a modified linear unit (ReLU).

[0066] As described above, in Neural Network 2, large datasets are exchanged between multiple interconnected channels (or nodes) and undergo extensive computational processing through multiple incremental layers. In this state, since the data can correspond to floating-point values ​​of various precisions, Neural Network 2 can perform quantization of the neural network's parameters (e.g., activations or weights) to reduce precision loss while minimizing the computational load required to process complex input data such as floating-point values.

[0067] Figure 3A and Figure 3B It is a diagram of convolution operations of a neural network according to one or more embodiments.

[0068] exist Figure 3A In the input feature map 201, X channels exist, and each channel's input feature map can have a size of H rows and W columns, where X, W, and H are natural numbers. Each weight kernel 202 has a size of R rows and S columns, and each weight kernel 202 can have a number of channels corresponding to the number X of channels in the input feature map 201, and there can be a total of Y weight kernels. The output feature map 203 can have a total of Y channels, where R, S, and Y are natural numbers. The output feature map 203 is generated by a three-dimensional convolution operation between the input feature map 201 and the weight kernels 202, and according to the convolution operation, it can have Y channels.

[0069] Below Figure 3BThe process of generating an output feature map through a convolution operation between an input feature map and a weight kernel is further described. This is because the operation is repeated between the input feature map 201 in all channels and all channels of each weight kernel 202. Figure 3B The two-dimensional convolution operation described in the text can generate all channels of each output feature map 203.

[0070] exist Figure 3B In this embodiment, it is assumed that the input feature map 210 has a size of 6×6 pixels, the weight kernel 220 has a size of 3×3 pixels, and the output feature map 230 has a size of 4×4 pixels. However, this disclosure is not limited to this, and the neural network can be implemented with feature maps and kernels of various sizes. Furthermore, the values ​​defined in the input feature map 210, weight kernel 220, and output feature map 230 are exemplary values, and this embodiment is not limited thereto.

[0071] The weight kernel 220 performs convolution operations by sliding across the input feature map 210 in units of windows or blocks, each 3×3 pixels in size, according to a set of strides. The windows in the input feature map 210 mapped to the weight kernel 220 can be referred to by the term "bounding box".

[0072] The convolution operation can be represented as an operation to obtain each pixel value of the output feature map 230 by summing all the following values, which are obtained by multiplying each pixel value of a specific window of the input feature map 210 by each element or weight of the corresponding position in the weight kernel 220. Specifically, the weight kernel 220 is first convolved with the first window or first bounding box 211 of the input feature map 210. In other words, each of the pixel values ​​1, 2, 3, 4, 5, 6, 7, 8, and 9 of the first window 211 is multiplied by the weights of the weight kernel 220, namely -1, -3, +4, +7, -2, -1, -5, +3, and +1, respectively, to obtain the results -1, -6, 12, 28, -10, -6, -35, 24, and 9. Next, by summing all the obtained values ​​-1, -6, 12, 28, -10, -6, -35, 24, and 9, 15 is calculated, and the output pixel value 231 in the first row and first column of the output feature map 230 is determined to be 15. In this state, the output pixel value 231 in the first row and first column of the output feature map 230 corresponds to the first window 211. Similarly, since a convolution operation is performed between the second window or second bounding box 212 of the input feature map 210 and the weight kernel 220, the output pixel value 232 in the first row and second column of the output feature map 230 is determined to be 4. Finally, since a convolution operation is performed between the sixteenth window or sixteenth bounding box 213 of the last window of the input feature map 210 and the weight kernel 220, the output pixel value 233 in the fourth row and fourth column of the output feature map 230 is determined to be 11.

[0073] In other words, the convolution operation between an input feature map 210 and a weight kernel 220 can be processed by repeatedly performing the multiplication of the values ​​of each corresponding element in the input feature map 210 and the original kernel 220 and the addition of the multiplication results, and the output feature map 230 is generated as the result of the convolution operation.

[0074] As described above, the three-dimensional convolution operation is performed by repeatedly performing the above two-dimensional convolution operation on the input feature maps of multiple channels and multiple kernels to generate output feature maps of multiple channels.

[0075] Figure 4 This is a diagram illustrating floating-point and fixed-point arithmetic.

[0076] exist Figure 4 In the example floating-point 410, a floating-point value can be represented as "a×2". b In this form, "a" represents the fraction or significant bit, and "b" represents the exponent. A single-precision (FP32) floating-point value can be represented as 32 bits, including 1 sign bit, 8 exponent bits, and 23 fraction bits.

[0077] Next, in the exemplary fixed-point 420, the fixed-point can be represented as 1+m+n bits, where m and n are natural numbers. In this state, 1+m+n bits may include 1 sign bit, m exponent bits, and n fraction bits. The bit length representing the fixed-point can be varied in various ways depending on factors such as the precision of the neural network and the processor performance.

[0078] Figure 5A and Figure 5B This is a diagram of a method for quantizing floating-point numbers to fixed-point numbers according to one or more embodiments.

[0079] exist Figure 5A In the distribution diagram 510, the weights in floating-point format can be quantized to have a symmetric distribution about 0. This can be called symmetric weight quantization, where the absolute values ​​of the upper threshold Wmax and the lower threshold Wmin of the dynamic range of the weights quantized due to symmetric weight quantization are the same.

[0080] and Figure 5A Unlike symmetric weight quantization, floating-point weights can be quantized with an asymmetric distribution about 0, as shown in Figures 521 or 522. In other words, with asymmetric weight quantization, the mean m of the weight distribution can be non-zero because the absolute values ​​of the upper limit threshold Wmax and the lower limit threshold Wmin of the dynamic range of the weights quantized due to asymmetric weight quantization can be different from each other.

[0081] Unlike convolution operations in neural networks performed using symmetrically quantized weights, when performing convolution operations using asymmetrically quantized weights, it may be desirable to remove the offset from the result of the convolution operation (i.e., the output feature map). In other words, when performing a convolution operation between asymmetrically quantized weights and the input feature map or input activation, as much residual data as the offset of the asymmetricly quantized weights exists in the result of the convolution operation (i.e., the output feature map). Therefore, it may be desirable to remove the residual data from the result of the convolution operation.

[0082] Figure 6 This is a graph showing an example weight offset in a convolution operation due to asymmetric quantization of the weights.

[0083] exist Figure 6 In this context, the convolution operation between the input feature map or input stimulus and the asymmetric quantized weights can be expressed as Equation 2.

[0084] Formula 2:

[0085]

[0086] The offset or residual data term is included in the convolution operation based on asymmetric quantization weights, as shown in Equation 2. Therefore, when subtracting the corresponding offset or residual data term from the result of the convolution operation... When the weights are shifted, they can be approximated as the result of a convolution operation based on symmetric quantized weights.

[0087] Figure 7 This is a block diagram of the hardware configuration of a neural processing device 100 according to one or more embodiments.

[0088] exist Figure 7 In this context, the neural processing device 100 may include at least one processor 110 and at least one memory 120. Figure 7 In the neural processing device 100 shown, only some elements relevant to one or more embodiments are illustrated. Therefore, in addition to Figure 7 In addition to the components shown, the neural processing device 100 may also include other or additional components.

[0089] The neural processing device 100 can correspond to a computing device for performing neural network operations. For example, the neural processing device 100 can correspond to a personal computer (PC), server device, or mobile device, or an accelerator for performing neural network operations in such a device. Furthermore, the neural processing device 100 can be configured in or incorporated into devices such as autonomous vehicles, robots, smartphones, tablets, augmented reality (AR) devices, and Internet of Things (IoT) devices, which use neural networks to perform speech recognition or image recognition. However, the neural processing device 100 is not limited to these categories and can correspond to various types of devices or processing devices for performing neural network operations in such devices.

[0090] Processor 110 is a hardware component that performs control functions for controlling neural processing device 100. For example, processor 110 can typically control neural processing device 100 by processing or executing instructions and / or data stored in neural processing device 100. Processor 110 may be implemented as a central processing unit (CPU), graphics processing unit (GPU), application processor (AP), neural processing unit (NPU), tensor processing unit (TPU) disposed in neural processing device 100, but this disclosure is not limited thereto.

[0091] Memory 120 is hardware used to store various neural network data that has been processed or is to be processed in processor 110. For example, memory 120 may store input / output feature map data or convolutional data processed in the neural network. In addition, memory 120 may store various applications (e.g., convolutional processing applications) to be driven by processor 110 based on execution instructions for them.

[0092] The memory 120 may correspond to a storage device such as random access memory (RAM), read-only memory (ROM), hard disk drive (HDD), solid-state drive (SSD), compact flash memory (CF), secure digital (SD), micro-secure digital (Micro-SD), mini-secure digital (Mini-SD), extreme digital (xD), or memory stick, but the type of memory 120 is not limited to this and can be a variety of types.

[0093] Processor 110 can generate an integral image by removing the aforementioned weight offsets or residual data from the result of the convolution operation.

[0094] Specifically, the processor 110 determines the layer of the neural network (e.g., Layer). i-1 The output feature map of the channels included in the output feature map (where i is a natural number) is used to generate an integral map for each channel based on the sum of the pixel values ​​in the output feature map. In this state, the integral map represents data obtained by the following operation: [The text abruptly ends here, likely due to an incomplete translation or a missing section.] i-1 The value obtained by summing the pixel values ​​in the region from the reference pixel of the output feature map to another output pixel of the output feature map is set as the value of the integral map pixel corresponding to the position of the output pixel.

[0095] The processor 110 generates a cumulative integral image by performing an accumulation operation on the integral images generated for each channel. In this state, the cumulative integral image can be generated by performing a pixel-by-pixel accumulation operation on the integral images generated for each channel.

[0096] Processor 110 passes through the next layer (e.g., Layer) i The input feature map is convolved with the weight kernel to obtain the next layer. i The pre-output feature map. In this state, according to one or more embodiments, the "pre-output feature map" can be defined as representing the output feature map from which the weight offset has not yet been removed.

[0097] Processor 110 obtains the next layer by subtracting the accumulated values ​​included in the accumulated integral map from the pixel values ​​of the pre-output feature map to remove the offset present in the weight kernel. i The output feature map.

[0098] When used in the next layer i When the weight kernel of the convolution operation is obtained through asymmetric quantization, the processor 110 can be based on the previous layer. i-1 The output feature map (i.e., the cumulative integral map) is stripped of weight offsets.

[0099] The memory 120 can store the integral image and the accumulated integral image generated for each layer, and when the processor 110 completes the convolution operation for the current layer, it can provide the accumulated integral image stored for the previous layer to the processor 110.

[0100] Figure 8 It is a graph that removes weight offsets by using an integral graph according to one or more embodiments.

[0101] exist Figure 8 middle, Figure 7 The processor 110 can access the Layer i-1 801 performs convolution operations to obtain the output feature map 810 for each channel.

[0102] Processor 110 according to Layer i-1 The output feature map 810 of the included channels in 801 generates an integral map for each channel based on the sum of the pixel values ​​in the output feature map 810. As described above, the integral map includes data obtained through the following operation: by applying data to the Layer... i-1 The value obtained by summing the pixel values ​​around the reference pixel of the output feature map 810 or in the region from the reference pixel to other output pixels of the output feature map 810 is set as the value of the integral map pixel corresponding to the position of the output pixel. In this state, the reference pixel can be set to one of the four corner pixels of each output feature map 810, which will be referred to below. Figure 9 Further detailed description.

[0103] Processor 110 generates Layer by performing an accumulation operation on the integral map generated for each channel. i-1 The accumulated integral image 815 is shown in Figure 801. In this state, the accumulated integral image can be generated by performing a pixel-by-pixel accumulation operation on the integral images generated for each channel, as will be discussed below. Figure 10 Further detailed description.

[0104] In addition to generating integral plots and accumulating integral plots 815, processor 110 also performs layer processing. i The 802 input feature map is convolved with the weight kernel to obtain the layer. i The pre-output feature map 820 of 802. In this state, due to Layer i 802 is Layer i-1 Layers after 801, therefore Layer i The input feature map of 802 corresponds to Layer i-1 The output feature map of layer 801 is shown in image 810. The weight kernel can include weights obtained through asymmetric quantization of the neural network. Therefore, from layer...i-1 The cumulative integral graph 815 generated by 801 can correspond to the Layer generated due to the asymmetric quantization of the neural network. i Data related to the offset of the weight kernel of 802.

[0105] Next, processor 110 can use the cumulative integral map 815 to remove the offset present in the weight kernel from the pre-output feature map 820. In other words, processor 110 subtracts the accumulated value included in the cumulative integral map 815 from the pixel values ​​of the pre-output feature map 820. As mentioned above, since the cumulative integral map 815 corresponds to data about the offset, the accumulated value included in the cumulative integral map 815 is related to the offset.

[0106] More specifically, in order to obtain Layer i The pre-output feature map of 802 and the pre-output pixels of 820 are used by processor 110 to determine the bounding boxes mapped to the input feature map of the weight kernel (see, for example, [link to relevant documentation]). Figure 3B The processor 110 obtains the pixel values ​​of the pixels in the accumulated integral map 815, which are set to correspond to the four corner pixels of the bounding box. Then, the processor 110 calculates the offset present in the pixel values ​​of the pre-output pixels using the obtained pixel values. The processor 110 then obtains the output pixel of the output feature map 825 by subtracting the previously calculated offset from the pixel values ​​of the pre-output pixels. (See below for reference...) Figure 11A and Figure 11B A more detailed description of the method using the corner pixels of the bounding box.

[0107] exist Figure 8 For ease of explanation, the description is only in Layer. i-1 801 and Layer i The example in the 802 two-layer relationship uses an integral graph to remove weight offsets. However, this disclosure is not limited to this, and the same method can be used by using the preceding layer (e.g., Layer 802). i-2 To obtain the layer from the integral graph. i-1 The output feature map 810 of 801. In other words, in one or more embodiments, similar or identical methods can be applied to multiple layers included in a neural network.

[0108] Figure 9 It is a graph that generates an integral map based on an output feature map according to one or more embodiments.

[0109] exist Figure 9 Although only three channels are discussed in this disclosure, this disclosure is not limited to this, and various numbers of channels can exist in the layer.

[0110] The output feature map OFM_ch1 910 can be a feature map with a size of 4×4, and the output pixels of the output feature map 910 can be referred to as OP1 to OP16.

[0111] exist Figure 9 In the example, for ease of explanation, it is assumed that the reference pixel used to generate the Integral Map_ch1 920 is pixel OP13 located at the lower left corner of the output feature map 910. However, since the reference pixel can be set to one of the four corner pixels of the output feature map 910, this disclosure is not limited thereto, and pixels of the output feature map 910, such as OP1, OP4, or OP16, can be set as the reference pixel.

[0112] For example, the following describes the calculation of integral image pixel IMP7 in integral image 920. The position of integral image pixel IMP7 in integral image 920 corresponds to the position of output pixel OP7 in output feature image 910. Figure 7 The processor 110 sums all pixel values ​​within the region 911 from reference pixel OP13 to output pixel OP7. The processor 110 sets the sum to the value of the integral map pixel IMP7 corresponding to the position of the output pixel OP7. In other words, IMP7 = OP5 + OP6 + OP7 + OP9 + OP10 + OP11 + OP13 + OP14 + OP15.

[0113] Processor 110 generates integral map 920 by calculating pixel values ​​for each integral map pixel in the same manner. Since the reference pixel is set to OP13 in output feature map 910, the pixel value of integral map pixel IMP13 of integral map 920 is the same as the pixel value of output pixel OP13, and the pixel value of integral map pixel IMP4 of integral map 920 corresponds to the sum of the pixel values ​​of all output pixels of output feature map 910.

[0114] The processor 110 not only generates the output feature map 910, but also generates an integral map for each channel in the same way for the output feature maps of the other channels.

[0115] Various examples include the sizes of the output feature map 910, the size of the integral map 920, and the positions of the reference pixels, based on the above-described generation method of the integral map.

[0116] Figure 10 It is a graph that generates a cumulative integral graph by using an integral graph according to one or more embodiments.

[0117] exist Figure 10 In, it can be obtained from a certain layer (e.g., Layer). i-1 The output layer generates an integral image 1010 for each channel.

[0118] When generating integral image 1010 Figure 7 The processor 110 can generate an accumulated integral map 1020, for example, by performing a pixel-by-pixel accumulation operation on the integral map 1010 generated for each channel. For example, the pixel value of the integral map pixel located at the upper right corner of the accumulated integral map 1020 corresponds to the value obtained by accumulating the pixel values ​​of the integral map pixels at the upper right corner of each integral map 1010. Therefore, the pixel value of each integral map pixel in the accumulated integral map 1020 can be obtained by calculating the accumulated pixel value of the corresponding integral map pixel in the integral map 1010.

[0119] When all accumulated values ​​of all pixels in the accumulated integral image 1020 are obtained, the processor 110 defines the accumulated integral image 1020 as the corresponding layer (e.g., Layer). i-1 The cumulative integral graph of 1020 is stored in the database. Figure 7 The accumulated integral graph 1020 stored in memory 120, as described above, can be used to generate the next layer (e.g., Layer 120). i The output feature map.

[0120] Figure 11A and 11B This is an example diagram of generating an output feature map by removing offsets from the pre-output feature map using an accumulated integral map.

[0121] Figure 7 The processor 110 determines the bounding boxes of the input feature maps mapped to the weight kernels in order to obtain a certain layer (e.g., Layer). i The pre-output pixels of the pre-output feature map. For example, suppose the location of the bounding box of the input feature map determined by processor 110 corresponds to... Figure 11A The region 1100 of the cumulative integral graph.

[0122] Processor 110 calculates the total accumulated value of the pixels in region 1100 of the accumulated integral image. In this state, the total accumulated value corresponds to the region value of region 1100.

[0123] In the process of calculating the total accumulated value, processor 110 first determines the position of the reference pixel. As assumed above, in Figure 11A and Figure 11B In this context, the reference pixel can be assumed to be the pixel at the bottom left.

[0124] Figure 11A The diagram shows regions 1100 and 1103 in the cumulative integral plot, and also shows a decomposition plot of regions 1100 and 1103, as well as regions 1101 and 1102, both of which include region 1103. Figure 11AIn the above formula 3, the region value of region 1100 can be calculated.

[0125] Formula 3:

[0126] val(x d y b )-val(x c y b )-val(x d y a )+val(x c y a )

[0127] In other words, the region value of region 1100 can be calculated by subtracting the region values ​​of region 1101 (xc, yb) and region 1102 (xd, ya) from the region value (xd, yb) and adding the region value of region 1103 (xc, ya). In this case, the region value of region 1103 (xc, ya) is added because the region value (xc, ya) is subtracted twice from regions 1101 and 1102.

[0128] When the region value of region 1100 is calculated in the manner described above, the weight offset included in the convolution operation performed within the bounding box corresponding to region 1100 can be calculated. Therefore, by subtracting the region value of region 1100 from the pre-output pixel value of the pre-output feature map, the processor 110 can obtain the output pixel value of the output feature map after removing the weight offset (residual), where the pre-output feature map is the result of performing a convolution operation within the corresponding bounding box of the input feature map.

[0129] exist Figure 11B In this paper, the method is described by applying it to feature maps. Figure 11A The method described in [the document / document].

[0130] First, processor 110 from Figure 7 The memory 120 obtains the information from the previous layer. i-1 The generated cumulative integral graph 1120.

[0131] Processor 110 in weighted core 1112 and Layer i A convolution operation is performed between the bounding boxes 1111 of the input feature map 1110. In this state, the bounding boxes 1111 can be, for example, with... Figure 11A The bounding box corresponding to region 1100.

[0132] As a result of the convolution operation, the processor 110 can obtain the Layer. iThe pre-output pixel Pre-OP2 of the pre-output feature map Pre-OFM1130. As mentioned above, the offset (residual) OFFSETOP2 due to the weights of asymmetric quantization may exist in the pre-output pixel Pre-OP2.

[0133] Processor 110 can use the accumulated integral map 1120 to calculate offset OFFSETOP2. Specifically, processor 110 determines the bounding box 1111 of the input feature map 1110 mapped to the weight kernel 1112, and obtains the pixel values ​​of the accumulated integral map 1120, which are set to the pixels corresponding to the four corner pixels of the bounding box 1111.

[0134] The four corner pixels of bounding box 1111 may include the top-right pixel, top-left pixel, bottom-right pixel, and bottom-left pixel of bounding box 1111. Processor 110 obtains the pixel value of the first integral map pixel IMP4 corresponding to the top-right pixel of bounding box 1111, the pixel value of the second integral map pixel IMP1 corresponding to the top-left pixel of bounding box 1111, the pixel value of the third integral map pixel IMP16 corresponding to the bottom-right pixel of bounding box 1111, and the pixel value of the fourth integral map pixel IMP13 corresponding to the bottom-left pixel of bounding box 1111 from the accumulated integral map 1120. In other words, processor 110 obtains the pixel values ​​of the first to fourth integral map pixels IMP4, IMP1, IMP16, and IMP13 for region 1121 of the accumulated integral map 1120 corresponding to bounding box 1111.

[0135] Then, since the reference pixel used to generate the cumulative integral image 1120 is the previous layer... i-1 The lower left pixel of the output feature map is used, so the processor 110 calculates the offset OFFSET in the pixel value of the pre-output pixel Pre-OP2 by subtracting the pixel values ​​of the second integral map pixel IMP1 and the third integral map pixel IMP16 from the sum of the pixel values ​​of the first integral map pixel IMP4 and the fourth integral map pixel IMP13. OP2 In other words, processor 110 can be used by applications Figure 11A The principle described in [the document] is used to calculate the offset. OP2 .

[0136] Therefore, the processor 110 can subtract the calculated offset OFFSET from the pixel value of the pre-output pixel Pre-OP2. OP2 To obtain Layer i The output feature map 1140 shows the output pixel OP2.

[0137] The processor 110 can obtain other output pixels OP1, OP3 and OP4 by using the above method, and finally generate an output feature map 1140 that has removed the offset caused by the weights of asymmetric quantization according to the above method.

[0138] In the described examples, for ease of explanation, although feature maps of a particular size, weight kernels of a particular size, and reference pixels at a particular location are illustrated in the description, the methods that can be performed on one or more embodiments are not limited thereto, and the examples include feature maps of various sizes, weight kernels of various sizes, and reference pixels at various locations in a similar or corresponding manner.

[0139] and Compared to the complexity of typical methods, the method according to one or more embodiments, which removes the offset of the output feature map by using an integral map, can reduce the complexity by up to three orders of magnitude, as shown in Equation 4 below, for example.

[0140] Formula 4:

[0141]

[0142] Figure 12 This is a block diagram of an electronic system 1200 according to one or more embodiments.

[0143] exist Figure 12 In this system, electronic system 1200 can analyze input data in real time based on neural networks to extract useful information, determine the status based on the extracted information, or control electronic devices equipped with electronic system 1200. For example, electronic system 1200 can be or can be applied to robotic devices such as drones or advanced driver assistance systems (ADAS), smart TVs, smartphones, medical devices, mobile devices, image display devices, measuring devices, IoT devices, etc. Furthermore, electronic system 1200 can be or can be installed on at least one of various types of electronic devices.

[0144] Electronic system 1200 may include a processor 1210, RAM 1220, neural network device 1230, memory 1240, sensor module 1250, and communication module 1260. Electronic system 1200 may also include an input / output module, a security module, or a power control device. Some hardware components of electronic system 1200 may be mounted on at least one semiconductor chip. Neural network device 1230 may include… Figure 7 The neural processing device 100 described herein may be a neural network-specific hardware accelerator or a device that includes a neural network-specific hardware accelerator. Figure 7 The neural processing device 100 can also be an electronic system 1200.

[0145] Processor 1210 controls the overall operation of electronic system 1200. Processor 1210 may include one processor core (single-core) or multiple processor cores (multi-core). Processor 1210 can process or execute instructions and / or data stored in memory 1240. In an embodiment, processor 1210 can control the function of neural network device 1230 by executing instructions stored in memory 1240. Processor 1210 may be implemented as a CPU, GPU, or AP.

[0146] RAM 1220 can temporarily store programs, data, or instructions. For example, instructions and / or data stored in memory 1240 can be temporarily stored in RAM 1220 according to the control or boot code of processor 1210. RAM 1220 can be implemented as a memory such as dynamic RAM (DRAM) or static RAM (SRAM).

[0147] The neural network device 1230 can perform neural network operations based on received input data and can generate information signals based on the results of the operations. The neural network may include CNNs, RNNs, deep belief networks, or restricted Boltzmann machines, but this disclosure is not limited thereto. The neural network device 1230, as hardware that processes data using a neural network, may correspond to, for example, […]. Figure 7 The neural processing device 100 described herein is a dedicated hardware accelerator for neural networks.

[0148] Information signals can include one of various types of recognition signals, such as speech recognition signals, object recognition signals, image recognition signals, biometric recognition signals, etc. For example, neural network device 1230 can receive frame data included in a video stream as input data and generate recognition signals about objects included in an image indicated by the frame data based on the frame data. However, this disclosure is not limited thereto, and depending on the type or function of electronic system 1200 or electronic device equipped with electronic system 1200, neural network device 1230 can receive various types of input data and generate recognition signals based on the input data.

[0149] The memory 1240, serving as a data storage location, can store operating system (OS) programs, various instructions, and various data. In an embodiment, the memory 1240 can store neural network data, such as floating-point or fixed-point input / output stimuli or weights generated during the operation of the neural network device 1230. In an embodiment, the memory 1240 can store parameters used for mixed-precision MAC operations. Furthermore, the memory 1240 can store parameters encoded using an extended bit format.

[0150] The memory 1240 may be DRAM, but this disclosure is not limited thereto. The memory 1240 may include at least one of volatile memory or non-volatile memory. Non-volatile memory may include ROM, PROM, EPROM, EEPROM, flash memory, PRAM, MRAM, RRAM, FRAM, etc. Volatile memory may include DRAM, SRAM, SDRAM, PRAM, MRAM, RRAM, FeRAM, etc. In embodiments, the memory 1240 may include at least one of HDD, SSD, CF, SD, Micro-SD, Mini-SD, xD, or Memory Stick.

[0151] In this example, sensor module 1250 can collect information about the vicinity of the electronic device on which electronic system 1200 is installed, or collect information about the vicinity of electronic system 1200 when electronic system 1200 is an electronic device. Sensor module 1250 can sense or receive signals from outside the electronic device, such as video signals, voice signals, magnetic signals, biosignals, touch signals, etc., and convert the sensed or received signals into data. For this purpose, sensor module 1250 can include at least one of various types of sensing devices, such as microphones, imaging devices, image sensors, light detection and ranging (LIDAR) sensors, ultrasonic sensors, infrared sensors, biosensors, touch sensors, etc.

[0152] Sensor module 1250 can provide the converted data as input data to neural network device 1230. For example, sensor module 1250 may include an image sensor that generates a video stream by capturing the external environment of the electronic device and sequentially provides consecutive data frames of the video stream as input data to neural network device 1230. However, this disclosure is not limited thereto, and sensor module 1250 can provide various types of data to neural network device 1230.

[0153] The communication module 1260 may be provided with various wired or wireless interfaces capable of communicating with external devices. For example, the communication module 1260 may include communication interfaces capable of connecting to: wireless local area networks (WLANs) such as Wi-Fi, wired local area networks (LANs), wireless personal area networks (WPANs) such as Bluetooth, wireless universal serial bus (USB), Zigbee, near field communication (NFC), radio frequency identification (RFID), power line communication (PLC), or mobile cellular networks such as third-generation (3G), fourth-generation (4G), long-term evolution (LTE), or 5G (5G).

[0154] In this embodiment, the communication module 1260 can receive data about a quantized neural network from an external device. In this configuration, the external device can be a device that performs neural network training based on a large amount of data, quantizes the trained neural network into a fixed-point type, and provides the quantized neural network data to the electronic system 1200. The received quantized neural network data can be stored in the memory 1240.

[0155] Figure 13 This is a flowchart of a method for processing a neural network in a neural processing device according to one or more embodiments. Figure 13 In China, due to Figure 7 The method for processing floating-point operations in the neural processing device 100 is related to the embodiment described in the above figures; therefore, although not repeated here, the description given above is also applicable. Figure 13 The method.

[0156] In operation 1301, Figure 7 The processor 110 generates an integral map for each channel based on the first output feature map of the channels included in the first layer of the neural network, and on the region sum of the pixel values ​​in the first output feature map.

[0157] In operation 1302, processor 110 generates an accumulated integral map by performing an accumulation operation on the integral maps generated for each channel.

[0158] In operation 1303, processor 110 obtains the pre-output feature map of the second layer by performing a convolution operation between the input feature map of the second layer and the weight kernel.

[0159] In operation 1304, processor 110 obtains the second output feature map of the second layer by subtracting the accumulated values ​​included in the accumulated integral map from the pixel values ​​of the pre-output feature map to remove the offset present in the weight kernel.

[0160] The neural processing device, neural processing device 100, processor, processor 110, 1210, memory, memory 120, 1240, RAM 1220, neural network device 1230, sensor module 1250, and communication module 1260 that perform the operations described in this application are implemented by hardware components configured to perform the operations described in this application. Examples of hardware components that can be used to perform the operations described in this application, where appropriate, include controllers, sensors, generators, drivers, memory, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more hardware components performing the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer may be implemented by one or more processing elements (e.g., logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond to and execute instructions in a defined manner to achieve a desired result). In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. Hardware components implemented by the processor or computer may execute instructions or software, such as an operating system (OS) and one or more software applications running on the OS, to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to the execution of instructions or software. For brevity, the singular terms “processor” or “computer” may be used in the description of the examples described in this application, but multiple processors or computers may be used in other examples, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or processors and controllers, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors or processors and controllers may implement a single hardware component, or two or more hardware components. Hardware components may have any or more different processing configurations, examples of which include single-processor, discrete processor, parallel processor, single-instruction single-data (SISD) multiple processing, single-instruction multiple-data (SIMD) multiple processing, multiple-instruction single-data (MISD) multiple processing, and multiple-instruction multiple-data (MIMD) multiple processing.

[0161] Perform the operations described in this application Figures 1 to 13 The methods illustrated are executed by computing hardware, such as one or more processors or a computer, wherein the computing hardware is implemented as described above to execute instructions or software to perform the operations performed by these methods as described in this application. For example, a single operation or two or more operations may be executed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors or a processor and a controller, and one or more other operations may be executed by one or more other processors or another processor and another controller. One or more processors or a processor and a controller may execute a single operation or two or more operations.

[0162] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above can be written as computer programs, code segments, instructions, or any combination thereof, for individually or collectively instructing or configuring one or more processors or computers to operate as machines or special-purpose computers to perform the operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code that is directly executed by one or more processors or computers, such as machine code generated by a compiler. In another example, the instructions or software include higher-level code that is executed by one or more processors or computers using an interpreter. The instructions or software can be written using any programming language based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions in the specification (which disclose algorithms for performing the operations performed by the hardware components and the methods described above).

[0163] Instructions or software used to control computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, along with any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include read-only memory (ROM), random access memory (RAM), flash memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and to provide instructions or software and any associated data, data files, and data structures to one or more processors or computers such that the one or more processors or computers can execute the instructions. In one example, instructions or software, along with any associated data, data files, and data structures, are distributed across a networked computer system, enabling one or more processors or computers to store, access, and execute the instructions and software, along with any associated data, data files, and data structures, in a distributed manner.

[0164] Although this disclosure includes specific examples, it will be clear upon understanding the disclosure of this application that various changes in form and detail may be made to these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered merely descriptive and not for limiting purposes. The description of a feature or aspect in each example should be considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order and / or if components in the described system, architecture, device, or circuit are combined in a different manner and / or replaced or supplemented by other components or their equivalents. Therefore, the scope of this disclosure is not limited by the specific embodiments but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents should be construed as included in this disclosure.

Claims

1. A processor-implemented method, the method comprising: An integral map is generated based on the sum of the regions of corresponding pixel values ​​in each of the multiple channels of the first output feature map from the first layer of the neural network, wherein the integral map includes data obtained by the following operation: setting the value obtained by summing the pixel values ​​included in the region from the reference pixel of the first output feature map to the first output pixel of the first output feature map as the value of the pixel corresponding to the first output pixel in the integral map; An accumulated integral graph is generated by performing an accumulation operation on each integral graph. Multiple channels of the pre-output feature map of the second layer are obtained by performing convolution operations between multiple channels of the input feature map of the second layer after the first layer in the neural network and the weight kernel, wherein the weight kernel includes weights obtained by asymmetric quantization of the neural network; as well as Multiple channels of the second output feature map of the second layer are generated by subtracting the selected accumulated value of the accumulated integral map from the corresponding pixel value of the pre-output feature map for each channel of the second output feature map of the second layer.

2. The method according to claim 1, further comprising: Perform recognition on the second output feature map.

3. The method according to claim 1, further comprising: The weight kernel is generated by performing asymmetric quantization on the weight kernel trained for the second layer.

4. The method according to claim 1, wherein, The reference pixel is set as one of the four corner pixels of the first output feature map.

5. The method according to claim 1, wherein, The accumulated integral image is generated by performing a pixel-by-pixel accumulation operation on each integral image.

6. The method according to claim 5, wherein, The accumulated integral map generated from the first layer corresponds to data regarding the following: the offset of the weight kernel of the second layer generated by asymmetric quantization of the weight kernel trained for the second layer.

7. The method according to claim 1, wherein, The plurality of channels for generating the second output feature map include: Determine the bounding boxes of the channels of the input feature map mapped to the weight kernel to obtain the channels of the pre-output pixels of the pre-output feature map of the second layer; Obtain the pixel values ​​of the accumulated integral map that are set to correspond to the four corner pixels of the bounding box; The offset of the pixel value of the pre-output pixel is calculated based on the obtained pixel value; and The second output pixel of the second output feature map is obtained by subtracting the calculated offset from the pixel value of the pre-output pixel.

8. The method according to claim 7, wherein, The four corner pixels of the bounding box include the top-right pixel, top-left pixel, bottom-right pixel, and bottom-left pixel of the bounding box. Obtaining the pixel value set as the pixel of the accumulated integral image includes: The pixel values ​​of the first integral image pixel corresponding to the top right pixel, the second integral image pixel corresponding to the top left pixel, the third integral image pixel corresponding to the bottom right pixel, and the fourth integral image pixel corresponding to the bottom left pixel are obtained from the accumulated integral image.

9. The method according to claim 8, wherein, When the reference pixel used to generate the accumulated integral map is the lower left pixel of the first output feature map, calculating the offset includes: calculating the offset of the pixel value of the pre-output pixel by subtracting the pixel values ​​of the second integral map pixel and the third integral map pixel from the sum of the pixel values ​​of the first integral map pixel and the pixel values ​​of the fourth integral map pixel.

10. A non-transitory computer-readable recording medium storing instructions, which, when executed by a processor, cause the processor to control the execution of the method according to claim 1.

11. A neural processing device, comprising: One or more processors are configured as follows: An integral map is generated based on the region sum of the corresponding pixel values ​​in each of the multiple channels of the first output feature map in the first layer of the computational neural network, wherein the integral map includes data obtained by the following operation: setting the value obtained by summing the pixel values ​​included in the region from the reference pixel of the first output feature map to the first output pixel of the first output feature map as the value of the pixel corresponding to the first output pixel in the integral map; An accumulated integral graph is generated by performing an accumulation operation on each integral graph. Multiple channels of the pre-output feature map of the second layer are obtained by performing convolution operations between multiple channels of the input feature map of the second layer after the first layer in the neural network and the weight kernel, wherein the weight kernel includes weights obtained by asymmetric quantization of the neural network; as well as Multiple channels of the second output feature map of the second layer are generated by subtracting the selected accumulated value of the accumulated integral map from the corresponding pixel value of the pre-output feature map for each channel of the second output feature map of the second layer.

12. The neural processing device according to claim 11, wherein, The one or more processors are further configured to generate the weight kernel by performing asymmetric quantization on the weight kernel trained for the second layer.

13. The neural processing device according to claim 11, wherein, The reference pixel is set as one of the four corner pixels of the first output feature map.

14. The neural processing device according to claim 11, wherein, The one or more processors are further configured to generate the accumulated integral image by performing a pixel-by-pixel accumulation operation on each integral image.

15. The neural processing device according to claim 14, wherein, The accumulated integral map generated from the first layer corresponds to data regarding the following: the offset of the weight kernel of the second layer generated by asymmetric quantization of the weight kernel trained for the second layer.

16. The neural processing device according to claim 11, wherein, The one or more processors are further configured to: Determine the bounding boxes of the channels of the input feature map mapped to the weight kernel to obtain the channels of the pre-output pixels of the pre-output feature map of the second layer; Obtain the pixel values ​​of the accumulated integral map that are set to correspond to the four corner pixels of the bounding box; The offset in the pixel value of the pre-output pixel is calculated based on the obtained pixel value; as well as The second output pixel of the second output feature map is obtained by subtracting the calculated offset from the pixel value of the pre-output pixel.

17. The neural processing device according to claim 16, wherein, The four corner pixels of the bounding box include the top-right pixel, top-left pixel, bottom-right pixel, and bottom-left pixel of the bounding box. The one or more processors are further configured to: The pixel values ​​of the first integral image pixel corresponding to the top right pixel, the second integral image pixel corresponding to the top left pixel, the third integral image pixel corresponding to the bottom right pixel, and the fourth integral image pixel corresponding to the bottom left pixel are obtained from the accumulated integral image.

18. The neural processing device according to claim 17, wherein, The one or more processors are further configured to: when the reference pixel used to generate the accumulated integral map is the lower left pixel of the first output feature map, calculate the offset in the pixel value of the pre-output pixel by subtracting the pixel value of the second integral map pixel and the pixel value of the third integral map pixel from the sum of the pixel value of the first integral map pixel and the pixel value of the fourth integral map pixel.

19. The neural processing device according to claim 11, wherein, The one or more processors are further configured to generate the accumulated integral map based on the first output feature map of the first layer read from memory. The memory stores the generated cumulative integral graph, and The one or more processors are further configured to: when the pre-output feature map of the second layer is obtained, to obtain the second output feature map of the second layer by reading the generated cumulative integral map from the memory and performing an offset to remove the weight kernel.

20. A neural processing device, comprising: One or more processors are configured as follows: An accumulated integral map is obtained for multiple channels in the first layer of a neural network, wherein the accumulated integral map is generated by performing an accumulation operation on the integral map generated for each of the multiple channels in the first layer, and the integral map includes data obtained by: setting the value obtained by summing the pixel values ​​included in the region from the reference pixel of the first output feature map of the first layer to the first output pixel of the first output feature map as the value of the pixel corresponding to the first output pixel in the integral map; Multiple channels of the pre-output feature map of the second layer are obtained by performing convolution operations between multiple channels of the input feature map of the second layer after the first layer in the neural network and the weight kernel, wherein the weight kernel includes weights obtained by asymmetric quantization of the neural network; as well as Multiple channels of the output feature map of the second layer are generated by subtracting the selected accumulated value of the accumulated integral map from the corresponding pixel value of the pre-output feature map for each channel of the output feature map of the second layer.

Images