Processor-implemented methods and apparatus for neural networks for image recognition

Abstract

A processor-implemented method and apparatus for a neural network for image recognition are disclosed. The processor-implemented method of the neural network includes: obtaining intermediate pooling results respectively corresponding to a plurality of sub-pooling kernels obtained via decomposing an original pooling kernel, by performing a pooling operation on input pixels in a current window included in an input feature map using the plurality of sub-pooling kernels; obtaining a final pooling result corresponding to the current window by post-processing the intermediate pooling results; and determining an output pixel value of an output feature map based on the final pooling result, wherein the current window is determined according to the original pooling kernel sliding in the input feature map according to a raster scan order.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2019-0119090 filed on September 26, 2019, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety by reference for all purposes. Technical Field

[0002] Provided are a neural processing device and method with neural network pooling processing. Background Art

[0003] The neural network may be a computing system implemented as, for example, various types of electronic systems by referring to machine learning settings that analyze input data and extract effective information. Summary of the Invention

[0004] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0005] In one general aspect, a processor-implemented method for a neural network includes: performing a pooling operation on input pixels in a current window included in an input feature map by using multiple sub-pooling kernels obtained by decomposing an original pooling kernel to obtain intermediate pooling results respectively corresponding to the multiple sub-pooling kernels; obtaining a final pooling result corresponding to the current window by post-processing the intermediate pooling results; and determining an output pixel value of an output feature map based on the final pooling result, wherein the current window is determined by sliding the original pooling kernel in the input feature map according to a raster scan order.

[0006] The plurality of sub-pooling kernels may be a plurality of 1-dimensional (1D) kernels respectively including row elements of the original pooling kernel, and a total number of the plurality of sub-pooling kernels obtained by decomposing the original pooling kernel may correspond to a height of the original pooling kernel.

[0007] An intermediate pooling result of the current window obtained by the sub-pooling kernel among the multiple sub-pooling kernels can be shared with at least one other window in the input feature map.

[0008] In response to obtaining all intermediate pooling results for the current window, a final pooling result may be obtained.

[0009] The intermediate pooling results corresponding to the same window may be respectively stored in memory cells including memory addresses of the same column and different rows in the shared row buffer.

[0010] The method may further include: receiving the value of a current input pixel included in the current window according to a raster scanning order of the input feature map, wherein the step of obtaining the intermediate pooling result includes: updating at least one partial pooling result affected by the received value of the current input pixel stored in at least one memory unit based on the received value of the current input pixel.

[0011] The step of obtaining a final pooling result corresponding to the current window may include: reading an intermediate pooling result for the current window from a memory unit of a shared row buffer; and obtaining a final pooling result corresponding to an output pixel value by performing a post-processing operation on the read intermediate pooling result according to a preset pooling type.

[0012] The shared row buffer can store the intermediate pooling results obtained for other windows in the input feature map in a total number of memory rows corresponding to the height of the original pooling kernel in a circular manner.

[0013] In response to the intermediate pooling result stored in a memory unit of the shared row buffer no longer being shared, the intermediate pooling result stored in the one memory unit can be reused for a subsequent intermediate pooling result to be stored obtained by another sub-pooling core to obtain a final pooling result corresponding to another window.

[0014] The method may further include obtaining hyperparameters of the neural network, the hyperparameters comprising information related to any one or any combination of any two or more of the size of the original pooling kernel, the step size, and the pooling type, wherein a shared row buffer storing the obtained intermediate pooling results may be addressed based on the obtained hyperparameters.

[0015] The pooling operation can be a pooling type operation based on maximum pooling, wherein each intermediate pooling result is the maximum value among the values ​​of the input pixels mapped to the corresponding sub-pooling kernel, and the final pooling result is the maximum value among the intermediate pooling results, or the pooling operation can be a pooling type operation based on average pooling, wherein each intermediate pooling result is the sum of the values ​​of the input pixels mapped to the corresponding sub-pooling kernel, and the final pooling result is the value obtained by dividing the sum of the intermediate pooling results by the size of the original pooling kernel.

[0016] In another general aspect, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors, cause the one or more processors to perform the method described above.

[0017] In another general aspect, a neural processing device includes: one or more processors configured to: perform a pooling operation on input pixels in a current window included in an input feature map by utilizing multiple sub-pooling kernels obtained by decomposing an original pooling kernel, obtain intermediate pooling results corresponding to the multiple sub-pooling kernels, respectively, obtain a final pooling result corresponding to the current window by post-processing the intermediate pooling results, and determine an output pixel value of the output feature map based on the final pooling result, wherein the current window is determined according to sliding the original pooling kernel in the input feature map according to a raster scanning order.

[0018] The neural processing device may also include: a memory configured to store instructions, which, when executed by the one or more processors, configure the one or more processors to perform the steps of obtaining an intermediate pooling result, obtaining a final pooling result, and determining an output pixel value.

[0019] The plurality of sub-pooling kernels may be a plurality of 1-dimensional (1D) kernels respectively including row elements of the original pooling kernel, and a total number of the plurality of sub-pooling kernels obtained by decomposing the original pooling kernel may correspond to a height of the original pooling kernel.

[0020] An intermediate pooling result of the current window obtained by the sub-pooling kernel among the multiple sub-pooling kernels is shared with at least one other window in the input feature map.

[0021] In response to obtaining all intermediate pooling results for the current window, a final pooling result may be obtained.

[0022] Intermediate pooling results corresponding to the same window may be respectively stored in memory cells including memory addresses of the same column and different rows in a shared row buffer included in the memory.

[0023] The one or more processors may also be configured to: receive a value of a current input pixel included in a current window according to a raster scanning order of an input feature map; and update at least one partial pooling result affected by the received value of the current input pixel stored in at least one memory unit based on the received value of the current input pixel.

[0024] The one or more processors may also be configured to read the intermediate pooling result for the current window from the memory unit of the shared row buffer, and obtain a final pooling result corresponding to the output pixel value by performing a post-processing operation on the read intermediate pooling result according to a preset pooling type.

[0025] The shared row buffer can store the intermediate pooling results obtained for other windows in the input feature map in a total number of memory rows corresponding to the height of the original pooling kernel in a circular manner.

[0026] In response to the intermediate pooling result stored in a memory unit of the shared row buffer no longer being shared, the intermediate pooling result stored in the one memory unit is reused for a subsequent intermediate pooling result to be stored obtained by another sub-pooling core to obtain a final pooling result corresponding to another window.

[0027] The one or more processors may also be configured to obtain hyperparameters of the neural network, the hyperparameters comprising information related to any one or any combination of any two or more of the size of the original pooling kernel, the step size, and the pooling type, wherein a shared row buffer included in the memory for storing the obtained intermediate pooling results may be addressed based on the hyperparameters.

[0028] The pooling operation can be a pooling type operation based on maximum pooling, wherein each intermediate pooling result is the maximum value among the values ​​of the input pixels mapped to the corresponding sub-pooling kernel, and the final pooling result is the maximum value among the intermediate pooling results, or the pooling operation can be a pooling type operation based on average pooling, wherein each intermediate pooling result is the sum of the values ​​of the input pixels mapped to the corresponding sub-pooling kernel, and the final pooling result is the value obtained by dividing the sum of the intermediate pooling results by the size of the original pooling kernel.

[0029] In another general aspect, a method implemented by a processor of a neural network includes: utilizing multiple sub-pooling kernels obtained by decomposing an original pooling kernel, obtaining intermediate pooling results corresponding to the multiple sub-pooling kernels respectively from input pixels in a current window to be pooled included in an input feature map; in response to completion of the intermediate pooling for the current window, obtaining a final pooling result corresponding to the current window from the intermediate pooling results, determining the current window as the original pooling kernel slides in the input feature map; and determining an output pixel value of the output feature map based on the final pooling result.

[0030] The original pooling kernel can slide according to the raster scan order.

[0031] The plurality of sub-pooling kernels may be a plurality of 1-dimensional (1D) kernels respectively including row elements of the original pooling kernel, and a total number of the plurality of sub-pooling kernels obtained by decomposing the original pooling kernel may correspond to a height of the original pooling kernel.

[0032] At least one sub-pooling kernel among the multiple sub-pooling kernels with respect to the current window may be shared as a sub-pooling kernel with respect to at least one other window in the input feature map.

[0033] Other features and aspects will be apparent from the following detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a diagram for describing the architecture of a neural network according to an example.

[0035] Figure 2 is a diagram for describing the concept of the pooling operation performed in the pooling layer of a neural network.

[0036] Figure 3 is a diagram for describing different pooling types for performing a pooling operation.

[0037] Figure 4 is a block diagram of a hardware configuration of a neural processing device for processing pooling of a neural network according to an example.

[0038] Figure 5 is a diagram for describing sub-pooling kernels decomposed from a pooling kernel that performs a pooling operation according to an example.

[0039] Figure 6 is a diagram for describing a method of performing maximum pooling by using sub-pooling kernels decomposed from an original pooling kernel according to an example.

[0040] Figure 7 is a diagram for describing a method of performing average pooling by using sub-pooling kernels decomposed from an original pooling kernel according to an example.

[0041] Figure 8 is a diagram for describing a method of processing a pooling operation using a sub-pooling kernel, performed by a processor and a shared line buffer of a memory according to an example.

[0042] Figure 9 is a diagram for describing a process of updating pooled data and generating an output feature map from the pooled data according to an example.

[0043] Figure 10 is a flowchart for describing a pooling algorithm for a neural network according to an example.

[0044] Figure 11 is a diagram for describing a process of updating pooling data on a shared line buffer by receiving a current input pixel according to an example.

[0045] Figure 12 is a diagram for describing a process of obtaining a final pooling result by post-processing intermediate pooling results stored in a shared line buffer according to an example.

[0046] Figure 13 is a flow chart of a method of processing pooling of a neural network performed by a neural processing device according to an example.

[0047] Throughout the drawings and detailed description, like reference numerals denote like elements. The drawings may not be to scale, and the relative sizes, proportions, and depictions of elements in the drawings may be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0048] The following detailed description is provided to help the reader gain a comprehensive understanding of the methods, devices, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, devices, and / or systems described herein will be clear after understanding the disclosure of the present application. For example, the order of operations described herein is merely an example and is not limited to the order of operations set forth herein, but except for operations that must occur in a specific order, the order of operations may be changed as becomes clear after understanding the disclosure of the present application. In addition, for greater clarity and brevity, descriptions of features known in the art may be omitted.

[0049] The features described herein can be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided merely to illustrate some of the many possible ways to implement the methods, devices, and / or systems described herein, which will become clear after understanding the disclosure of this application.

[0050] As used herein, the term "and / or" includes any one and any combination of any two or more of the associated listed items.

[0051] Although terms such as "first," "second," and "third" may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions should not be limited by these terms. Instead, these terms are used solely to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Thus, what is referred to as a first member, first component, first region, first layer, or first portion in the examples described herein may also be referred to as a second member, second component, second region, second layer, or second portion without departing from the teachings of the examples.

[0052] The terms used herein are intended only to describe various examples and should not be used to limit the disclosure. Unless the context clearly indicates otherwise, the singular is intended to include the plural. The terms "comprise," "include," and "have" indicate the presence of the stated features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0053] As will be apparent after understanding the disclosure of the present application, features of the examples described herein can be combined in various ways. Also, although the examples described herein have various configurations, other configurations are possible as will be apparent after understanding the disclosure of the present application.

[0054] The terms used herein should be interpreted, as having a meaning that is clear to one of ordinary skill in the art after understanding the present disclosure, but can have different meanings according to the intention of one of ordinary skill in the art, precedents, or emergence of new technology after understanding the present disclosure. Also, some terms can be arbitrarily selected by the applicant, and in this case, the meaning of the selected term will be described in detail in the detailed description. Therefore, the terms used herein must be defined based on the meaning of the terms along with the description throughout the specification.

[0055] Here, it is noted that the term "may" used in relation to an example or embodiment (e.g., an example or embodiment can include or implement something) means that there is at least one example or embodiment that includes or implements such a feature, and all examples and embodiments are not limited thereto.

[0056] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, the embodiments can be implemented in many different forms and are not limited to those described herein.

[0057] A device implementing a neural network can use a large amount of calculation with respect to complex input data to the neural network. As the data of the neural network increases and the connectivity of the architecture constituting the neural network becomes complex, an excessive increase in the amount of calculation and memory access frequency of the device occurs, resulting in inefficient performance with respect to miniaturization and commercialization problems.

[0058] Figure 1 is a diagram for describing an architecture of a neural network according to an example.

[0059] Referring to Figure 1 As a non-limiting example, the neural network 1 can be an architecture of a deep neural network (DNN) or an n-layer neural network for image recognition. As a non-limiting example, the DNN or the n-layer neural network can correspond to a convolutional neural network (CNN), a recurrent neural network (RNN), a deep belief network, or a restricted Boltzmann machine, and any combination thereof. For example, the neural network 1 can be implemented as a CNN, but the type of the neural network 1 is not limited to the CNN, but can alternatively also include another type of neural network.

[0060] The neural network 1 can be implemented as a computing architecture having multiple layers, including an input layer, a hidden layer or an intermediate layer, and an output layer, wherein an input image is provided to the input layer, and the hidden layer or the intermediate layer generates a feature map. The input image in the neural network 1 can be convolved with a filter called a weight kernel. As a result of the convolution operation, an output feature map can be output. The output feature map generated at this time can be used as the input feature map of the next layer. In the next layer, the generated output feature map can be again convolved with another kernel, so that a further output feature map is output. As a result of iteratively performing such a convolution operation (wherein the input of the subsequent convolution operation depends on the output of the previous convolution operation), a recognition result regarding the features of the input image can be ultimately output by the neural network 1.

[0061] For example, when a 24×24 pixel image is input to Figure 1 In the example of neural network 1, the input image can be output as a feature map of four channels, each channel having a size of 20×20 pixels (abbreviated as 4@20×20) through the first hidden layer via a convolution operation with the first kernel. Subsequently, the corresponding size of the generated feature map can be gradually reduced by iterative convolution operations for each subsequent hidden layer and corresponding kernel (for example, 4@20×20 can be sequentially reduced to 4@10×10 (i.e., a feature map of four channels, each channel having a size of 10×10 pixels), 8@8×8 (i.e., a feature map of eight channels, each channel having a size of 8×8 pixels), 8@4×4 (i.e., a feature map of eight channels, each channel having a size of 4×4 pixels), and a 20-channel feature map of 1×1 pixel size (i.e., 20@1×1) is output by the hidden layer of the last convolution operation. In such an example, as Figure 1 As shown in , the neural network 1 can filter and output robust features from the input image that can represent the entire input image by iteratively performing convolution operations together with pooling operations or subsampling operations in some layers, and the recognition result of the input image can be obtained through the final output features.

[0062] In addition, a pooling layer that performs a pooling operation may be arranged to appear after the convolution layer. The pooling operation of the pooling layer may be a process for reducing the computational overhead of the next convolution layer by reducing the size (e.g., row to column) of the output feature map of the previous convolution layer input to the next convolution layer.

[0063] In a subsequent example, a method for efficiently performing a pooling operation on an input feature map in a pooling layer will be described in further detail.

[0064] Figure 2 is a diagram used to describe the concept of the pooling operation performed in the pooling layer of a neural network.

[0065] Referring to Figure 2 , the input feature map 21 can have a size of 10x9 (where the size is provided as rows to columns), and the pooling kernel 22 can also be referred to as a pooling window. The pooling kernel 22 that performs the pooling operation on the input feature map 21 can have a size of 3x3. In addition, a stride that indicates a degree to which the pooling kernel 22 slides on the input feature map 21 can be 1. However, hyperparameters of the pooling operation, such as the size of the pooling kernel 22, the stride, or the type of pooling, are not limited to those described with reference to Figure 2 , and can vary in other examples.

[0066] The pooling kernel 22 can perform the pooling operation while sliding in a window (also referred to as a group of pixels, a block, or a scanning window) of 3x3 size in units of 10x9 pixels with respect to the input feature map 21. For example, the sliding order can depend on a raster scan order of the input feature map 21. The raster scan order of the input feature map 21 can denote an order in which pixels of the first row are sequentially processed starting from the pixel of the first column of the first row of the input feature map 21, then pixels from the second row to the tenth row are processed, and finally the pixel of the ninth column of the tenth row is processed.

[0067] When the pixels in the input feature map 21 corresponding to the pooling kernel 22 mapped to the window of the current order are all scanned according to the raster scan order, the pooling kernel 22 can perform the pooling operation on the pixels. For example, when the pixels are all scanned according to the raster scan order, the pooling operation by the pooling kernel 22 mapped to the window 210 of the input feature map 21 can be performed by using the values of the pixels included in the window 210. In addition, when all the pixels included in each of the window 220 and the window 230 are scanned according to the raster scan order, the pooling operation with respect to each of the window 220 and the window 230 can be performed.

[0068] In the example of Figure 2 , only some of the windows 210, 220, and 230 are illustrated for ease of description. The sliding of the pooling kernel 22 with respect to the example windows 210 to 230 and the remaining windows can be performed based on the size and the stride of the pooling kernel 22.

[0069] In addition, when the pooling operation is performed on the input feature map 21 by the pooling kernel 22 in this manner, a pooling result can be obtained, in which the pooling result can correspond to output pixels of an output feature map, respectively.

[0070] Figure 3 is a diagram for describing different types of pooling for describing a pooling operation.

[0071] Referring to Figure 3In the example of , the input feature map 31 may have a size of 4×4, and the pooling kernel 32 may have a size of 2×2. In such an example, for the convenience of describing the pooling type, the stride may be 2. However, these values ​​are merely non-limiting examples.

[0072] The pooling operation for the pixels of the window mapped to the pooling kernel 32 may be an operation based on a pooling type of maximum pooling or a pooling type of average pooling. However, the pooling operation is not limited to these types of pooling, and in other examples, the pooling operation may be based on other types of pooling than the reference pooling operation. Figure 3 Perform pooling operations with a pooling type other than the one described in the example.

[0073] First, maximum pooling will be described in further detail. Maximum pooling indicates that a pooling operation can be performed by using a maximum value of pixels of the input feature map 31 mapped to the pooling kernel 32 as a result of a corresponding pooling operation.

[0074] As a specific example, when the pooling kernel 32 is mapped to four pixels included in the 2×2 upper left window of the input feature map 31, the pooling kernel 32 may obtain a value “20” which is the maximum value among the pixel values ​​“12”, “20”, “8”, and “12” included in the upper left window as a pooling result corresponding to the upper left window. Figure 3 As shown in the first example, the pooling kernel 32 is also used to perform maximum pooling on the 2×2 upper right window, the 2×2 lower left window, and the 2×2 lower right window in the input feature map 31, and the respective pooling results of the 2×2 upper right window, the 2×2 lower left window, and the 2×2 lower right window will be "30", "112", and "37" which are the maximum values ​​in their respective windows. The pooling results "20", "30", "112", and "37" of the maximum pooling of the input feature map 31 may correspond to the pixel values ​​of the output pixels of the output feature map 310, which may then be input to, for example, the next convolutional layer.

[0075] Next, average pooling will be described in further detail. Average pooling indicates that a pooling operation may be performed by using the respective average values ​​of pixels of the input feature map 31 mapped to the pooling kernel 32 .

[0076] As a specific example, when the pooling kernel 32 is mapped to the four pixels included in the 2×2 upper left window of the input feature map 31, the pooling kernel 32 obtains "13" which is the average value of the pixel values ​​"12", "20", "8", and "12" included in the upper left window as the pooling result corresponding to the upper left window. Here, the average value used is the average value. Figure 3As shown in the second example, average pooling is also performed on a 2x2 top-right window, a 2x2 bottom-left window, and a 2x2 bottom-right window in the input feature map 31, and the respective pooling results of the 2x2 top-right window, the 2x2 bottom-left window, and the 2x2 bottom-right window will be “9”, “79”, and “19.5” as the average values of the respective windows using the pooling kernel 32. The pooling results “13”, “9”, “79”, and “19.5” of the average pooling of the input feature map 31 can correspond to pixel values of output pixels of the output feature map 320, which can then be input to, for example, a next convolutional layer.

[0077] In other words, even when the pooling operation is performed by the same pooling kernel 32, the generated output feature map can differ based on the pooling type used in the pooling operation. Non-limiting examples of the pooling types of the max pooling type and the average pooling type are discussed in more detail above, but other pooling types are usable in other examples. The pooling type can be a hyperparameter as described above, and can be predefined with respect to the neural network.

[0078] Figure 4 is a block diagram of a hardware configuration of a neural processing device 100 that processes pooling of a neural network according to an example.

[0079] Referring to Figure 4 In an example of Figure 4 Components of the neural processing device 100 relevant to the current example are shown in the example of Figure 4 In an example of Figure 4 In an example of Figure 4 In an example of

[0080] The neural processing device 100 can correspond to a computing device. For example, as non-limiting examples, the neural processing device 100 can correspond to a personal computer (PC), a server, or a mobile device, or can correspond to an accelerator for performing neural network operations in such devices. Additionally, as non-limiting examples, the neural processing device 100 can represent a device included in an autonomous vehicle, a robot, a smart phone, a tablet device, an augmented reality (AR) device, or an Internet of Things (IoT) device (e.g., the device can perform speech recognition, image recognition, and similar tasks using a neural network). However, the neural processing device 100 is not limited to these non-limiting examples, and can correspond to various types of devices or processing devices that perform neural network operations in such devices.

[0081] The processor 110 may be a hardware component that performs overall control functions for controlling the operation of the neural processing device 100. For example, the processor 110 may generally control the neural processing device 100 by processing or executing instructions and / or data stored in the memory 120 in the neural processing device 100. In an example, the processor 110 may be implemented as a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), a neural processing unit (NPU), or a tensor processing unit (TPU) included in the neural processing device 100, but is not limited to these listed non-limiting examples, and the processor 110 may be implemented as other types of processors, or as a plurality of processors or a combination of processors.

[0082] Memory 120 may be hardware that stores hyperparameters (e.g., including trained parameters) of various received network examples and various types of neural network data processed or to be processed by processor 110. For example, as discussed in further detail above, memory 120 may store input / output feature map data, convolution data, and pooling data processed in a neural network. In addition, memory 120 may store various applications to be driven by processor 110 (e.g., convolution processing applications, pooling processing applications, and other similar applications).

[0083] The memory 120 may correspond to a memory device such as a random access memory (RAM), a read-only memory (ROM), a hard disk drive (HDD), a solid-state drive (SSD), a compact flash (CF) card, a secure digital (SD) card, a micro SD card, a mini SD card, an extreme digital (xD) card, or a memory stick, but the type of the memory 120 is not limited to these non-limiting examples, and in other examples the memory 120 may include other types of memory devices.

[0084] like Figure 4 As shown in the example of , memory 120 may include a shared row buffer 125 as on-chip memory for neural network processing by processor 110. Shared row buffer 125 may be implemented as dynamic RAM (DRAM) or static RAM (SRAM) for high-speed access to processor 110, but shared row buffer 125 is not limited to these non-limiting examples and may include other types of memory technology in other examples. In this example, the term "shared row buffer" may be modified in various ways and represented by another term that also represents a similar portion of memory 120 for buffering. Furthermore, in addition to shared row buffer 125, memory 120 may additionally include other buffers for buffering for other purposes.

[0085] The processor 110 can be implemented to have at least one processor core for processing various operations for performing convolution and pooling of a neural network. Specifically, as described in further detail throughout this disclosure, the processor 110 can generate an output feature map by performing a respective convolution operation between an input feature map and a corresponding kernel weight in a layer of a neural network, each of the layers of the neural network can generate a feature map for a next convolution layer by performing a pooling operation on the generated output feature map. In addition, the processor 110 can perform various operations for implementing or executing a neural network.

[0086] When performing a pooling operation, the processor 110 can read pooling data (such as input pixel values of an input feature map or processed pooling results) from the shared row buffer 125 of the memory 120, can update the pooling data already stored in the shared row buffer 125, or can write the pooling data to the shared row buffer 125. In other words, the shared row buffer 125 can operate as a memory for temporarily storing the pooling data of the processor 110.

[0087] Figure 5 is a diagram for describing a pooling operation performed using sub-pooling kernels decomposed from a pooling kernel according to an example.

[0088] Referring to Figure 5 In an example of FIG. 5, the original pooling kernel 50 can have a 3x3 size, noting that the size of the original pooling kernel 50 is not limited to the 3x3 size and can vary from such a size. Even when using an original pooling kernel of a size different from 3x3, the decomposition method described below can be similarly applied.

[0089] Here, before the kernel is decomposed, the kernel will be denoted by the term "original", for example, the "original" pooling kernel 50.

[0090] The processor 110 can decompose the original pooling kernel 50 into a plurality of sub-pooling kernels 510 to 530 to perform a pooling operation. The sub-pooling kernels 510 to 530 can be one-dimensional (1D) kernels respectively including row elements of the original pooling kernel 50. In an example, the sub-pooling kernels 510 to 530 can be obtained by decomposing the row elements of the original pooling kernel 50 according to the raster scan order described above, but when scanning pixels in the input feature map in a raster scan order different from the raster scan order described above, the sub-pooling kernels can be obtained by decomposing the elements of the original pooling kernel 50 in another direction. For example, the sub-pooling kernels can be obtained by decomposing the elements of the original pooling kernel 50 in a column direction. In other words, in different examples, the decomposition direction can vary based on the raster scan order.

[0091] The number of the sub-pooling kernels 510 to 530 obtained by decomposing the original pooling kernel 50 can correspond to the height of the original pooling kernel 50. For example, because Figure 5 The height of the original pooling kernel 50 is 3, so the number of sub-pooling kernels 510 to 530 is also 3.

[0092] The elements of the sub-pooling core 510 may correspond to the elements of the first row of the original pooling core 50, the elements of the sub-pooling core 520 may correspond to the elements of the second row of the original pooling core 50, and the elements of the sub-pooling core 530 may correspond to the elements of the third row of the original pooling core 50.

[0093] The processor 110 can obtain intermediate pooling results separately by performing pooling operations on the sub-pooling cores 510 to 530 separately. The processor 110 can then merge the intermediate pooling results to output a final pooling result. In other words, according to the example, the pooling operation can be performed in units of sub-pooling cores rather than in units of original pooling cores. In the example, the term "intermediate pooling result" represents the partial pooling data obtained by the sub-pooling core, and the term "final pooling result" represents the final pooling data corresponding to the window obtained from multiple intermediate pooling results.

[0094] Figure 6 6 is a diagram for describing a method of performing maximum pooling (MaxPool) by using sub-pooling kernels 611 to 613 decomposed from an original pooling kernel 610 according to an example.

[0095] Reference Figure 6 For example, the MaxPool operation can be performed on the 4×4 size input feature map 60 by using three 1×3 sub-pooling kernels 611 to 613 decomposed from the 3×3 original pooling kernel 610. Figure 6 In the pooling operation of the example, the stride can be 1.

[0096] The processor 110 can obtain intermediate pooling results 65 corresponding to the sub-pooling kernels 611 to 613 respectively by performing a pooling operation on the input pixels in the current window to be pooled included in the input feature map 60 using the sub-pooling kernels 611 to 613 decomposed from the original pooling kernel 610.

[0097] When all intermediate pooling results 65 are obtained for the current window, the processor 110 may obtain a final pooling result corresponding to the current window by post-processing the intermediate pooling results 65. The processor 110 may then determine an output pixel value of the output feature map 67 based on the final pooling result.

[0098] In such Figure 6In the specific example shown in the example of , when the current window is the first window of the input feature map 60, the subpooling kernel 611 may be mapped to the input pixels of the first row included in the first window. The processor 110 may determine the maximum value "9" selected from the values ​​"9", "5", and "4" of the input pixels of the first row, and may obtain the maximum value "9" as the intermediate pooling result corresponding to the subpooling kernel 611. In addition, the processor 110 may determine the maximum value "8" among the values ​​"1", "6", and "8" of the input pixels of the second row and the maximum value "7" among the values ​​"2", "3", and "7" of the input pixels of the third row, and obtain the maximum values ​​"8" and "7" as the intermediate pooling results corresponding to the subpooling kernels 612 and 613. In other words, the intermediate pooling result 620 obtained by the subpooling kernels 611 to 613 as mapped to the first window may be an element of ("9", "8", "7").

[0099] When all the intermediate pooling results 620 of the current window as the example first window are obtained, the processor 110 can obtain the final pooling result corresponding to the current window as the first window by post-processing the intermediate pooling results 620. Figure 6 The pooling type described in the example of FIG is MaxPool, so the processor 110 can perform post-processing by determining the maximum value "9" from the intermediate pooling result 620 (for example, by implementing "Max(9, 8, 7)"). In this way, the processor 110 can determine the output pixel value "9" of the output feature map 67 corresponding to the current window or the first window based on the final pooling result "9".

[0100] Then, when the current window corresponds to the third window of the input feature map 60 according to the raster scan order, the subpooling kernels 611 to 613 may be mapped to the input pixels of the first to third rows included in the third window, respectively. The processor 110 may determine the maximum values ​​('8', '7', and '1') with respect to the rows, and may obtain the elements ('8', '7', and '1') of the intermediate pooling result 630 corresponding to the subpooling kernels 611 to 613.

[0101] When all the intermediate pooling results 630 of the current window as the third window are obtained, the processor 110 can perform post-processing to determine the maximum value "8" (selected as "max(8, 7, 1)") as the final pooling result from the intermediate pooling results 630, and can determine the output pixel value "8" of the output feature map 67 corresponding to the current window as the third window based on the final pooling result "8".

[0102] exist Figure 6In the example, pooling operations may be performed on some windows (such as the first window and the third window) of the input feature map 60, but the processor 110 may perform pooling operations on the remaining windows in a similar manner, and may ultimately obtain the values ​​of the output pixels of the output feature map 67.

[0103] In other words, according to the example pooling operation, when the original pooling kernel of 3×3 includes, for example, nine elements designated as (a, b, c, d, e, f, g, h, i), the processor 110 may perform the pooling operation by using the following Equation 1.

[0104] Equation 1:

[0105] Pool(a,b,c,d,e,f,g,h,i)=Pool(Pool(a,b,c),Pool(d,e,f),Pool(g,h,i))

[0106] In other words, the processor 110 may perform an intermediate pooling operation on each sub-pooling core (i.e., a sub-pooling core including elements (a, b, c), a sub-pooling core including elements (d, e, f), and a sub-pooling core including elements (g, h, i)), and may perform a post-processing pooling operation on the intermediate pooling results to obtain a final pooling result. Figure 6 In the example of MaxPool described in the example, the pooling operator can be a max operator.

[0107] At the same time, the sub-pooling kernels 612 and 613 when the current window is the first window of the input feature map 60 and the sub-pooling kernels 611 and 612 when the current window is the third window of the input feature map 60 can be based on the same input pixels using the input feature map 60 to perform pooling operations. Therefore, the intermediate pooling results obtained by the sub-pooling kernels 612 and 613 mapped to the first window can be reused as the intermediate pooling results of the sub-pooling kernels 611 and 612 mapped to the third window, because they use some of the same information to perform their calculations in a similar manner. The pooling operations performed on input pixels overlapping in different windows of the input feature map are further described in detail in the corresponding figures below.

[0108] Even if the pooling type is not separately described in the description of the following embodiments, the following examples can be implemented by applying the pooling type of MaxPool, the pooling type of average pooling (AvgPool), and other optional pooling types available in other examples.

[0109] Figure 7 is a diagram for describing a method of performing AvgPool by using sub-pooling cores 711 to 713 decomposed from an original pooling core 710 , according to an example.

[0110] Reference Figure 7 For example, the AvgPool operation can be performed on the 4×4 size input feature map 70 by decomposing three 1×3 sub-pooling kernels 711 to 713 from the 3×3 original pooling kernel 710. Figure 7 In the pooling operation of the example, the stride can be 1.

[0111] like Figure 7 The pooling operation performed on the input feature map 70 by the sub-pooling kernels 711 to 713 decomposed from the original pooling kernel 710 shown in the example of FIG can be compared with the reference Figure 6 The pooling operation described in the example is similar. However, Figure 7 The AvgPool pooling operation of the example is Figure 6 The difference between the MaxPool pooling operation of the example is the method of obtaining the intermediate pooling result and the method of obtaining the final pooling result.

[0112] Specifically, according to Figure 7 For example, when the current window is the first window of the input feature map 70, the processor 110 may add the values ​​(“1”, “2”, and “0”) of the input pixel values ​​of the first row mapped to the subpooling kernel 711 included in the first window, and may obtain their sum “3” or (1+2+0) as the intermediate pooling result corresponding to the subpooling kernel 711. In other words, the method of calculating the intermediate pooling result according to the AvgPool method is different from the method of calculating the intermediate pooling result according to the MaxPool method.

[0113] Similarly, the processor 110 may obtain the sum "6" or (3+2+1) of the values ​​of the second row of input pixels mapped to the subpooling kernel 712, and the sum "0" or (0+0+0) of the values ​​of the third row of input pixels mapped to the subpooling kernel 713, as the intermediate pooling values ​​corresponding to the subpooling kernels 712 and 713. In other words, the intermediate pooling result 720 obtained by mapping the subpooling kernels 711 to 713 to the first window is an element of the group ("3", "6", and "0").

[0114] When all the intermediate pooling results 720 of the current window as the first window are obtained, the processor 110 may obtain the final pooling result corresponding to the current window as the first window by post-processing the intermediate pooling results 720. Figure 7The pooling type described in the example of is AvgPool, so the processor 110 may add the intermediate pooling results 720 ("3", "6", and "0") and may perform post-processing of dividing the sum "9" or (3+6+0) by the size 3×3 or 9 of the original pooling kernel 710. Therefore, the processor 110 may determine an output pixel value "1" of the output feature map 77 corresponding to the current window as the first window based on the final pooling result "1" or (9 / 9).

[0115] Similarly, when the current window is the third window of the input feature map 70, the processor 110 may perform an intermediate pooling operation on the input pixels mapped to each of the sub-pooling kernels 711 to 713 to obtain elements ("6", "0" and "3") of the intermediate pooling result 730.

[0116] When all the intermediate pooling results 730 of the current window, which is the third window, are obtained, the processor 110 may add the intermediate pooling results 730 ("6", "0", and "3"), and may perform post-processing of dividing the sum "9" or (6+0+3) by the size 3×3 or 9 of the original pooling kernel 710. Therefore, the processor 110 may determine an output pixel value "1" of the output feature map 77 corresponding to the current window, which is the third window, based on the final pooling result "1" or (9 / 9).

[0117] exist Figure 7 In the example, pooling operations may be performed on some windows of the input feature map 70 including the first window and the third window, but the processor 110 may perform pooling operations on the remaining windows in a similar manner, and may ultimately obtain the values ​​of the output pixels of the output feature map 77.

[0118] In other words, according to the example pooling operation, when the original pooling kernel of 3×3 includes, for example, elements (a, b, c, d, e, f, g, h, i), the processor 110 may perform the pooling operation by using the above equation 1. However, with Figure 6 Different from the example of , the pooling operator for calculating the intermediate pooling results can be a sum / addition operator, and the pooling operator for calculating the final pooling result can be an average operator.

[0119] As in Figure 6In the example of , the sub-pooling kernels 712 and 713 when the current window is the first window of the input feature map 70 and the sub-pooling kernels 711 and 712 when the current window is the third window of the input feature map 70 can perform pooling operations on the same input pixels of the input feature map 70. Therefore, the intermediate pooling results obtained by mapping the sub-pooling kernels 712 and 713 to the first window can be reused as the intermediate pooling results of the sub-pooling kernels 711 and 712 mapped to the third window. As described above, specific information can be reused in a similar manner, thereby avoiding redundant calculations. The pooling operations performed on overlapping input pixels in different windows of the input feature map are described in more detail below in the corresponding figures.

[0120] exist Figure 6 and Figure 7 In the example of , for ease of description, a 4×4 input feature map, a 3×3 original pooling kernel, and a stride of 1 have been described as a non-limiting example, but in other examples, refer to the example Figure 6 and Figure 7 The described pooling operation can be easily applied in the same way to input feature maps of other sizes, original pooling kernels of other sizes, and strides of other values.

[0121] Figure 8 is a diagram for describing a method of processing a pooling operation using sub-pooling cores 811 to 813 performed by a shared line buffer of a processor and a memory according to an example. As a non-limiting example and for ease of explanation, below, the processor, the shared line buffer, and the memory may correspond to Figure 4 processor 110, shared row buffer 125 and memory 120.

[0122] Reference Figure 8 In the example of , the 3×3 original pooling kernel 810 can be decomposed into 1×3 sub-pooling kernels 811 to 813, and the pooling operation can be performed on the input feature map 820 by using the sub-pooling kernels 811 to 813. However, in the example, the sizes of the original pooling kernel 810 and the sub-pooling kernels 811 to 813 and the size of the input feature map 820 are not limited to Figure 8 The examples shown in FIG. 5 are non-limiting examples and may vary and have other values.

[0123] The example processor 110 may include at least one arithmetic logic unit (ALU). Figure 8 In the non-limiting example of FIG, the processor 110 includes an ALU 110-1 and an ALU 110-2, each of which performs arithmetic operations related to the pooling operation. For ease of description, Figure 8While two ALUs 110 - 1 and 110 - 2 are discussed in the example of FIG, the ALUs in processor 110 may be the same single ALU and / or processor 110 may include three or more ALUs.

[0124] The processor 110 may receive a value of a current input pixel 825 included in a current window of the input feature map 820 according to a raster scan order of the input feature map 820. The processor 110 may update at least one partial pooling result stored in at least one memory unit in the shared line buffer 125 according to the value of the current input pixel 825, wherein the partial pooling result is affected by the value of the current input pixel 825. In such an example, the ALU 110-1 of the processor 110 may perform an arithmetic operation for updating the partial pooling result.

[0125] In such an example, the partial pooling result can be an intermediate value for obtaining the intermediate pooling results of all input pixels mapped to one sub-pooling kernel. For example, the sub-pooling kernel 811 can be mapped to a total of 3 input pixels in one window, and only when all data of the mapped 3 input pixels are received, the intermediate pooling results of the 3 input pixels can be obtained. However, because the input pixels of the input feature map 820 can be sequentially input to the processor 110 according to the raster scan order, it may be difficult to obtain the intermediate pooling results of the 3 input pixels at the same time.

[0126] Therefore, in the case of the MaxPool method, when the value of the first input pixel mapped to the subpooling kernel 811 is received, the processor 110 may store the value of the first input pixel as the maximum value in a memory cell of the shared line buffer 125. When the value of the second input pixel is received, the processor 110 may compare the stored maximum value (i.e., the value of the first input pixel) with the value of the second input pixel, and may update the data of the memory cell to the maximum value among these values ​​as needed. In such an example, the data stored in the memory cell may correspond to a partial pooling result. Finally, when the value of the last third input pixel mapped to the subpooling kernel 811 is received, the processor 110 may compare the stored maximum value (i.e., the maximum value among the values ​​of the first input pixel and the second input pixel) with the value of the third input pixel, and ultimately update the data of the memory cell to the maximum value among all three input pixels. Because the subpooling kernel 811 may have a size of 1×3, the data of the memory cell that is ultimately updated by the value of the third input pixel may correspond to the intermediate pooling result corresponding to the subpooling kernel 811, because each input pixel may have the opportunity to influence the maximum pixel value.

[0127] Similarly, in the case of AvgPool, the processor 110 updates the partial pooling result by adding the value of the received input pixel to the data already stored in the memory cells of the shared line buffer 125.

[0128] In other words, when performing a pooling operation, the processor 110 may read pooled data from the shared line buffer 125 of the memory 120, may update a partial pooling result already stored in the shared line buffer 125 by using the ALU 110-1, and may write the partial pooling result to the shared line buffer 125. By storing the accumulated partial pooling results, information from the input pixels can be temporarily stored, thereby avoiding the need for all pooled data to be available before the pooling process can begin.

[0129] The ALU 110-2 of the processor 110 may read the intermediate pooling result of the current window from the memory unit of the shared line buffer 125, and may perform post-processing on the read intermediate pooling result according to a preset pooling type to obtain a final pooling result corresponding to the input pixel value. Figure 8 In the example, since the 3×3 original pooling core 810 can be decomposed into three 1×3 sub-pooling cores 811 to 813, the ALU 110-2 can read the three intermediate pooling results obtained from the shared line buffer 125 and post-process the three intermediate pooling results.

[0130] When Figure 8 When the 3×3 original pooling kernel 810 is decomposed into three 1×3 sub-pooling kernels 811 to 813 as in the example of , the shared row buffer 125 may store the pooling data in a total of 3 rows of memory cells. In other words, the shared row buffer 125 may store the intermediate pooling results obtained for the window in the input feature map 820 in a circular manner in the memory rows of the number of rows corresponding to the height of the original pooling kernel 810, and the memory rows are shown as row k*i, row k*i+1, and row k*i+2. In such an example, k represents the height of the original pooling kernel 810, i is an arbitrary number used to indicate an arbitrary address in the shared row buffer 125 for the circular manner, and according to Figure 8 For example, k=3. However, Figure 8 is a non-limiting example, and in other examples k may take other values.

[0131] Specifically, the intermediate pooling results corresponding to the same window in the input feature map 820 can be respectively stored in memory cells having memory addresses in the same column and different rows in the shared row buffer 125. For example, when three intermediate pooling results corresponding to the three sub-pooling kernels 811 to 813 are obtained for the window 821 of the input feature map 820, according to this method, the three intermediate pooling results can be respectively stored in the memory cell 1251 of the first row and first column, the memory cell 1252 of the second row and first column, and the memory cell 1253 of the third row and first column of the shared row buffer 125.

[0132] When the stride is 1 and the window 822 to be pooled from the input feature map 820 has the same columns as the previously pooled window 821, the window 821 and the window 822 may share two rows of input pixels. In other words, at least one of the sub-pooling kernels with respect to the current window 821 may be shared as a sub-pooling kernel with respect to at least one other window 822 in the input feature map 820. Therefore, the intermediate pooling result obtained by mapping the sharable sub-pooling kernel among the sub-pooling kernels of the current window 821 may be shared with respect to at least one other window 822.

[0133] When all intermediate pooling results of window 821 are stored in memory units 1251, 1252, and 1253, ALU 110-2 of processor 110 may post-process the intermediate pooling results stored in memory units 1251, 1252, and 1253 to output the final pooling result of window 821. According to the raster scan order, when pooling is subsequently performed on window 822, the data corresponding to the intermediate pooling results of sub-pooling core 811 of window 821 stored in memory unit 1251 may no longer be used for the pooling operation. However, because window 821 shares two rows with window 822, the intermediate pooling results of sub-pooling cores 812 and 813 of window 821 may be reused for window 822. Therefore, the intermediate pooling results stored in memory units 1252 and 1253 may not be deleted, but may be reused for pooling window 822.

[0134] In other words, when the intermediate pooling result stored in one memory cell of the shared row buffer 125 is no longer shared to obtain the final pooling result corresponding to another window, the intermediate pooling result stored in the one memory cell may be deleted or overwritten so that the one memory cell is used to store the intermediate pooling result obtained by another sub-pooling kernel to be stored. In this way, the intermediate pooling results can be stored in the memory cells of the same column of the shared row buffer 125 in a circular manner.

[0135] In various examples, refer to Figure 8The example described in the example can be applied in the same way to pooling operations based on input feature maps of other sizes, original pooling kernels of other sizes, and strides of other values, and Figure 8 The specific details of the examples are to be regarded as non-limiting.

[0136] Figure 9 is a diagram for describing a process of updating pooled data and generating an output feature map 930 from the pooled data according to an example.

[0137] Reference Figure 9 In this example, the input pixel IFM_4-1 of the input feature map 910 may correspond to the current input pixel that overlaps or is shared in different sub-pooling kernels. In such an example, the different sub-pooling kernels may be represented by Figure 9 For example, different sub-pooling kernels may correspond to the third sub-pooling kernel of the 3×3 original pooling kernel mapped to window A, the third sub-pooling kernel of the 3×3 original pooling kernel mapped to window B, and the third sub-pooling kernel of the 3×3 original pooling kernel mapped to window C.

[0138] When receiving a current input pixel, the processor 110 may update at least one partial pooling result affected by the value of the current input pixel. For example, when receiving the input pixel IFM_4-1 according to the raster scan order, the processor 110 may update the partial pooling results IPR_4-1, IPR_4-2, and IPR_4-3 based on how they are affected by the input pixel IFM_4-1. When the update of the partial pooling result IPR_4-1 is completed, the final updated partial pooling result IPR_4-1 may correspond to the intermediate pooling result.

[0139] The processor 110 may obtain a final pooling result by performing post-processing on the intermediate pooling results 925 (ie, IPR_2-1, IPR_3-1, and IPR_4-1). The obtained final pooling result may correspond to the output pixel OFM_2-1 of the output feature map 930.

[0140] In this way, when the processor 110 receives any one input pixel value, at least one partial pooling result stored in the shared line buffer 125 and affected by the input pixel value may also be updated.

[0141] Typically, in the absence of the sub-pooling kernel method discussed herein, when the value of any single input pixel is received, all pooling results of some windows including the input pixel can be repeatedly read / updated / written. For example, when a sufficiently large input feature map is pooled based on a 3×3 pooling kernel with a step size of 1, each time an input pixel is received, the processor and the shared row buffer can read / update / write all pooling data corresponding to the 9 pixel values ​​of each window in a maximum of 9 windows of the shared input pixel. Therefore, the throughput of the processor and the access frequency between the processor and the memory (such as, a shared row buffer) are relatively high. However, when a pooling method utilizing a sub-pooling method according to an embodiment herein (for example, a 1D sub-pooling kernel decomposed from the original pooling kernel) is used, only the pooling data of the 1D sub-pooling kernel of the shared input pixel can be read / updated / written, thereby effectively reducing the overhead of the processor and the memory (such as, a shared row buffer).

[0142] Figure 10 is a flowchart of a neural network method according to an example. Figure 10 For example, a neural network approach can be performed (such as in Figure 4 The neural processing device 100 in the example of FIG. 100 is executed or implemented in a time-based sequence. Figure 10 The hyperparameters of the pooling operation of the neural network method of the example, the size of the original pooling kernel can be k×k, and the step size can be s, where k and s are natural numbers and s <k。

[0143] in addition, Figure 10 The pooling operation shown in the example of may be an operation that receives only one input pixel of the input feature map. Therefore, the pooling operation may be repeated for all pixels of the input feature map. Figure 10 The neural network method of the example is shown in Figure 10 In operation 1001 of the example, current input pixels (X, Y) are sequentially received according to the raster scan order of the input feature map, so that all input pixels are processed.

[0144] In operation 1001, the processor 110 may receive a current input pixel (X, Y) in an input feature map, where the input feature map has a width W and a height H. In such an example, 0≤X <W并且0≤Y<H。

[0145] In operation 1002, the processor 110 may store an address (x p ,y p ) is set to correspond to the update pivot, and the update size of the memory unit of the shared line buffer 125 may be set to correspond to t, where t is a natural number. Then, the processor 110 may initialize i to 0.

[0146] In such an example, before updating the pivot point (x p ,y p ), x p =X%k,y p =Y%k, and when the size t is updated, t is assigned the value of the updated size.

[0147] In operation 1003, the processor 110 may determine whether i <t的条件。当满足i<t的条件时,可执行操作1004。然而,当不满足i<t的条件时,可替代地执行操作1006。

[0148] In operation 1004, the processor 110 may set the target address among the memory cells of the shared line buffer 125 to (x p +i,y p ).

[0149] In operation 1005 , the processor 110 may increase i by 1.

[0150] The processor 110 may perform operations 1003 to 1005 until i is no longer satisfied. <t的条件为止。

[0151] In operation 1006, in order to obtain a final pooling result, the processor 110 may determine an intermediate pooling result (eg, such as Figure 9 The example of the 1×k kernel of reference number 925 is determined to determine whether it exists on the shared line buffer 125. When there is no intermediate pooling result that can be post-processed on the shared line buffer 125, the neural network method for the current input pixel (X, Y) is ended, and operation 1001 is started / restarted for the next input pixel.

[0152] In such an example, the processor 110 can determine whether the update is completed in the update pivot regarding the current input pixel (X, Y) by using the following Formula 2 as a conditional test, and can determine whether a 1×k kernel capable of performing post-processing exists by using the following Formula 3 as a conditional test.

[0153] Formula 2:

[0154] (X-k+1)≥0&&(X-k+1)%s=0

[0155] Formula 3:

[0156] (Y-k+1)≥0&&(Y-k+1)%s=0

[0157] In operation 1007 , the processor 110 may access the x on the shared line buffer 125 . pcolumn as the target column.

[0158] In operation 1008 , the processor 110 may read the x stored on the shared line buffer 125 . p Intermediate pooling results (1×k size) in the columns.

[0159] In operation 1009, the processor 110 may obtain a final pooling result by performing post-processing on the read intermediate pooling result according to a preset pooling type. The processor 110 determines that the obtained final pooling result corresponds to the value of the output pixel at the position ((X-k+1) / 2, (Y-k+1) / 2) of the output feature map.

[0160] The processor 110 may repeatedly execute the neural network method described in further detail above until the output feature map is completed when the final pooling result is obtained for all intermediate pooling results obtained from the input feature map.

[0161] Figure 11 1 is a diagram for describing a process of a processor that updates pooled data 1120 on a shared line buffer by receiving a current input pixel according to an example. Figure 4 The processor 110 and the shared line buffer 125 are described Figure 11 Note that the example is not limited to this.

[0162] Reference Figure 11 In the example of FIG. 1 , when the ALU 110 - 1 of the processor 110 receives a current input pixel in the input feature map 1110, the ALU 110 - 1 reads the pooled data 1120 (i.e., a partial pooling result) affected by the current input pixel stored in the shared row buffer 125. As described in further detail above, the read pooled data 1120 may be data stored in memory cells in the same row but different columns.

[0163] ALU 110-1 may update pooled data 1120 based on pooled data 1120 and the value of the current input pixel. For example, ALU 110-1 may determine a maximum value according to the MaxPool technique as described in further detail above, or perform an addition operation according to AvgPool as described in further detail above.

[0164] In other words, according to the example of employing a 1D sub-pooling kernel, as further described in detail above, only the pooling data 1120 of the 1D sub-pooling kernel sharing the current input pixel is read / updated / written at a given time, and therefore, the overhead of the processor 110 and the shared row buffer 125 can be effectively reduced by managing the data in this manner compared to general processing without sub-pooling.

[0165] Figure 121 is a diagram for describing a process of a processor that obtains a final pooling result by post-processing the intermediate pooling result 1205 stored in the shared line buffer according to an example. Figure 4 The processor 110 and the shared line buffer 125 are described Figure 12 Note that the example is not limited to this.

[0166] Reference Figure 12 In the example of , when the partial pooling result stored in the memory unit 1201 of the shared line buffer 125 is finally updated, the finally updated partial pooling result corresponds to the intermediate pooling result.

[0167] The processor 110 may read the intermediate pooling results 1205 stored in all memory cells in the same column as the memory cell 1201 storing the intermediate pooling results. In such an example, as previously discussed in further detail, it is assumed that the intermediate pooling results 1205 are all final updated results.

[0168] The processor 110 may perform post-processing on the intermediate pooling result 1205 according to a preset pooling type to obtain a final pooling result corresponding to the intermediate pooling result 1205. Various non-limiting examples of pooling have been discussed in more detail above. As a result, the processor 110 may generate an output feature map based on the value of the output pixel 1210 corresponding to the final pooling result.

[0169] As described above, the intermediate pooling results stored in the same column of the shared row buffer 125 may be pre-processed results of the pooling operation performed by each of the sub-pooling cores mapped to the window. Therefore, the processor 110 may complete pooling of the corresponding window by performing post-processing of the pooling operation of merging the intermediate pooling results stored in the same column.

[0170] Figure 13 is a flow chart of a neural network method performed by a device including processing pooling of a neural network according to an example. Figure 13 For example, because Figure 13 The neural network method is related to the example described above with reference to the accompanying drawings, so the above description also applies to Figure 13 operation.

[0171] In operation 1301, the processor 110 may obtain intermediate pooling results respectively corresponding to the sub-pooling kernels by performing a pooling operation on input pixels in a current window to be pooled included in an input feature map using a plurality of sub-pooling kernels decomposed from the original pooling kernel. In such an example, the current window may be determined as the original pooling kernel slides in the input feature map according to a raster scan order.

[0172] In operation 1302 , when all intermediate pooling results for the current window are obtained, the processor 110 may obtain a final pooling result corresponding to the current window by post-processing the intermediate pooling results.

[0173] In operation 1303 , the processor 110 may determine an output pixel value of an output feature map based on the final pooling result.

[0174] The pooling of the neural network according to the example is processed based on hyperparameters, which include information about the size of the original pooling kernel, the step size, and the pooling type. In other words, the number of sub-pooling kernels to be decomposed, the minimum required number of memory rows of the shared row buffer 125, and the reference number can be set based on the hyperparameters. Figure 10 The update pivot and update size of the shared line buffer 125 described in the example of . In addition, the shared line buffer 125 storing the obtained intermediate pooling results can be addressed based on hyper parameters.

[0175] Figures 1 to 13The neural processing device, neural processing device 100, processor, processor 110, ALU, ALU 110-1, ALU 110-2, memory, memory 120, shared row buffer and shared row buffer 125 that perform the operations described in this application are implemented by hardware components, and the hardware components are configured to perform the operations described in this application and performed by the hardware components. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate: controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer can be implemented by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond and execute instructions in a limited manner to achieve desired results). In one example, a processor or computer includes or is connected to one or more memories storing instructions or software executed by a processor or computer. The hardware components implemented by a processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described in this application. The hardware components can also access, manipulate, process, create, and store data in response to the execution of instructions or software. For simplicity, the singular term "processor" or "computer" can be used for the description of the examples described in this application, but in other examples, multiple processors or computers can be used, or a processor or computer can include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component or two or more hardware components. The hardware components may have any one or more of different processing configurations, examples of which include: a single processor, independent processors, parallel processors, single instruction single data (SISD) multiprocessing, single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0176] Figures 1 to 13 The method for performing the operations described in this application shown in the is performed by computing hardware (e.g., by one or more processors or computers), wherein the computing hardware is implemented as described above to execute instructions or software to perform the operations performed by the method described in this application. For example, a single operation or two or more operations may be performed by a single processor or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors, or a processor and a controller, and one or more other operations may be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller may perform a single operation or two or more operations.

[0177] The instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above may be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers to operate as a machine or special-purpose computer to perform the operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by one or more processors or computers. In another example, the instructions or software include high-level code that is executed by one or more processors or computers using an interpreter. Instructions or software may be written in any programming language based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions in the specification, wherein the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions in the specification disclose algorithms for performing the operations performed by the hardware components and methods described above.

[0178] The instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement the hardware components and perform the methods described above, and any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include read-only memory (ROM), random access memory (RAM), flash memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state disk, and any other device configured to store the instructions or software and any associated data, data files, and data structures in a non-transitory manner and provide the instructions or software and any associated data, data files, and data structures to one or more processors or computers so that the one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed over a networked computer system so that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed fashion by one or more processors or computers.

[0179] Although the present disclosure includes specific examples, it will be clear after understanding the disclosure of the present application that various changes in form and detail can be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered merely descriptive and not for purposes of limitation. The description of features or aspects in each example will be considered applicable to similar features or aspects in other examples. Suitable results can be achieved if the described techniques are performed in a different order, and / or if components in the described systems, architectures, devices, or circuits are combined in a different manner, and / or replaced or supplemented by other components or their equivalents. Therefore, the scope of the disclosure is limited not by the specific embodiments, but by the claims and their equivalents, and all changes within the scope of the claims and their equivalents should be interpreted as included in the disclosure.

Claims

1. A processor-implemented method for a neural network for image recognition, the method comprising: Obtain an input feature map from an input image; Performing a pooling operation on the input pixels in the current window included in the input feature map by using a plurality of sub-pooling kernels obtained by decomposing the original pooling kernel to obtain a plurality of intermediate pooling results respectively corresponding to the plurality of sub-pooling kernels; Obtaining a final pooling result corresponding to the current window by post-processing the multiple intermediate pooling results; as well as Based on the final pooling result, the output pixel value of the output feature map is determined for image recognition. Among them, the current window is determined by sliding the original pooling kernel in the input feature map according to the raster scan order. Wherein, the multiple sub-pooling cores are multiple 1-dimensional cores that respectively include row elements of the original pooling core and the total number of the multiple sub-pooling cores obtained by decomposing the original pooling core corresponds to the height of the original pooling core, or the multiple sub-pooling cores are multiple 1-dimensional cores that respectively include column elements of the original pooling core and the total number of the multiple sub-pooling cores obtained by decomposing the original pooling core corresponds to the width of the original pooling core.

2. The method according to claim 1, wherein An intermediate pooling result of the current window obtained by the shared sub-pooling kernel among the multiple sub-pooling kernels is shared with at least one other window in the input feature map.

3. The method according to claim 1, wherein In response to obtaining all of the multiple intermediate pooling results for the current window, a final pooling result is obtained.

4. The method according to claim 1, wherein The intermediate pooling results corresponding to the same window are respectively stored in memory cells including memory addresses of the same column and different rows in the shared row buffer.

5. The method according to claim 4, further comprising: Receive the value of the current input pixel included in the current window according to the raster scan order of the input feature map, The step of obtaining the multiple intermediate pooling results includes: At least one partial pooling result affected by the received current input pixel value and stored in at least one memory unit is updated based on the received current input pixel value.

6. The method according to claim 4, wherein: The steps to obtain the final pooling result corresponding to the current window include: Reading the plurality of intermediate pooling results for the current window from the memory unit of the shared line buffer; and By performing a post-processing operation on the plurality of intermediate pooling results read according to a preset pooling type, a final pooling result corresponding to the output pixel value is obtained.

7. The method according to claim 4, wherein: The shared row buffer stores the intermediate pooling results obtained for other windows in the input feature map in a circular manner in memory rows with a total number of rows corresponding to the height of the original pooling kernel.

8. The method according to claim 7, wherein: In response to the intermediate pooling result stored in a memory unit of the shared row buffer no longer being shared, the intermediate pooling result stored in the one memory unit is deleted so that the one memory unit is used to store a subsequent intermediate pooling result to be stored obtained by another sub-pooling core to obtain a final pooling result corresponding to another window.

9. The method according to claim 1, further comprising: Obtaining hyperparameters of a neural network, the hyperparameters comprising information related to any one or any combination of any two or more of the size of the original pooling kernel, the stride size, and the pooling type, wherein, A shared row buffer storing the obtained multiple intermediate pooling results is addressed based on the obtained hyperparameters.

10. The method according to claim 1, wherein The pooling operation is an operation of the pooling type based on max pooling, in which each intermediate pooling result is the maximum value among the values ​​of the input pixels mapped to the corresponding sub-pooling kernel, and the final pooling result is the maximum value among the multiple intermediate pooling results, or The pooling operation is an operation of the pooling type based on average pooling, in which each intermediate pooling result is the sum of the values ​​of the input pixels mapped to the corresponding sub-pooling kernel, and the final pooling result is the value obtained by dividing the sum of the multiple intermediate pooling results by the size of the original pooling kernel.

11. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method of claim 1.

12. A neural processing device for a neural network for image recognition, comprising: One or more processors configured to: Obtain an input feature map from an input image; By using multiple sub-pooling kernels obtained by decomposing the original pooling kernel, a pooling operation is performed on the input pixels in the current window included in the input feature map to obtain multiple intermediate pooling results respectively corresponding to the multiple sub-pooling kernels, Obtaining a final pooling result corresponding to the current window by post-processing the multiple intermediate pooling results, and Based on the final pooling result, the output pixel value of the output feature map is determined for image recognition. Among them, the current window is determined by sliding the original pooling kernel in the input feature map according to the raster scan order. Wherein, the multiple sub-pooling cores are multiple 1-dimensional cores that respectively include row elements of the original pooling core and the total number of the multiple sub-pooling cores obtained by decomposing the original pooling core corresponds to the height of the original pooling core, or the multiple sub-pooling cores are multiple 1-dimensional cores that respectively include column elements of the original pooling core and the total number of the multiple sub-pooling cores obtained by decomposing the original pooling core corresponds to the width of the original pooling core.

13. The neural processing device of claim 12, further comprising: The memory is configured to store instructions, which, when executed by the one or more processors, configure the one or more processors to perform the steps of obtaining the multiple intermediate pooling results, obtaining the final pooling result, and determining the output pixel value.

14. The neural processing device according to claim 12, wherein: An intermediate pooling result of the current window obtained by the shared sub-pooling kernel among the multiple sub-pooling kernels is shared with at least one other window in the input feature map.

15. The neural processing device of claim 12, wherein: In response to obtaining all of the multiple intermediate pooling results for the current window, a final pooling result is obtained.

16. The neural processing device of claim 12, wherein: The intermediate pooling results corresponding to the same window are respectively stored in memory cells including memory addresses of the same column and different rows in a shared row buffer included in the memory.

17. The neural processing device of claim 16, wherein: The one or more processors are further configured to: receiving a value of a current input pixel included in a current window according to a raster scan order of the input feature map; as well as At least one partial pooling result affected by the received current input pixel value and stored in at least one memory unit is updated based on the received current input pixel value.

18. The neural processing device of claim 16, wherein: The one or more processors are further configured to: The multiple intermediate pooling results for the current window are read from the memory unit of the shared line buffer, and a final pooling result corresponding to the output pixel value is obtained by performing a post-processing operation on the read multiple intermediate pooling results according to a preset pooling type.

19. The neural processing device of claim 16, wherein: The shared row buffer stores the intermediate pooling results obtained for other windows in the input feature map in a circular manner in memory rows with a total number of rows corresponding to the height of the original pooling kernel.

20. The neural processing device of claim 19, wherein: In response to the intermediate pooling result stored in a memory unit of the shared row buffer no longer being shared, the intermediate pooling result stored in the one memory unit is deleted so that the one memory unit is used to store a subsequent intermediate pooling result to be stored obtained by another sub-pooling core to obtain a final pooling result corresponding to another window.

21. The neural processing device of claim 12, wherein: The one or more processors are further configured to: Obtaining hyperparameters of a neural network, the hyperparameters comprising information related to any one or any combination of any two or more of the size of the original pooling kernel, the stride size, and the pooling type, wherein, A shared row buffer included in a memory for storing the obtained plurality of intermediate pooling results is addressed based on the hyperparameter.

22. The neural processing device of claim 12, wherein: The pooling operation is an operation of the pooling type based on max pooling, in which each intermediate pooling result is the maximum value among the values ​​of the input pixels mapped to the corresponding sub-pooling kernel, and the final pooling result is the maximum value among the multiple intermediate pooling results, or The pooling operation is an operation of the pooling type based on average pooling, in which each intermediate pooling result is the sum of the values ​​of the input pixels mapped to the corresponding sub-pooling kernel, and the final pooling result is the value obtained by dividing the sum of the multiple intermediate pooling results by the size of the original pooling kernel.

23. A processor-implemented method for a neural network for image recognition, the method comprising: Obtain an input feature map from an input image; Using multiple sub-pooling kernels obtained by decomposing the original pooling kernel, obtain multiple intermediate pooling results respectively corresponding to the multiple sub-pooling kernels from the input pixels in the current window to be pooled included in the input feature map; In response to the completion of intermediate pooling for the current window, obtaining a final pooling result corresponding to the current window from the multiple intermediate pooling results, and determining the current window as the original pooling kernel slides in the input feature map; as well as Based on the final pooling result, the output pixel value of the output feature map is determined for image recognition. Wherein, the multiple sub-pooling cores are multiple 1-dimensional cores that respectively include row elements of the original pooling core and the total number of the multiple sub-pooling cores obtained by decomposing the original pooling core corresponds to the height of the original pooling core, or the multiple sub-pooling cores are multiple 1-dimensional cores that respectively include column elements of the original pooling core and the total number of the multiple sub-pooling cores obtained by decomposing the original pooling core corresponds to the width of the original pooling core.

24. The method according to claim 23, wherein The original pooling kernel slides according to the raster scan order.

25. The method according to claim 23, wherein At least one sub-pooling kernel among the multiple sub-pooling kernels with respect to the current window is shared as a sub-pooling kernel with respect to at least one other window in the input feature map.

Images