Method and apparatus for processing convolution operations in neural networks

Abstract

A method for performing a convolution operation between a convolution kernel and an input feature map based on reuse of an input feature map, and a neural network device using the method are provided. The neural network device generates an output value of an operation between each weight of the convolution kernel and the input feature map, and generates an output feature map by accumulating the output values ​​at positions in the output feature map that are set based on positions of weights in the convolution kernel.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Korean Patent Application No. 10-2018-0171132 filed on December 27, 2018, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety for all purposes by reference. Technical Field

[0003] Methods and apparatus are provided for processing convolution operations in a neural network. Background Art

[0004] Neural networks are specialized computing architectures that, after extensive training, can provide a computationally intuitive mapping between input patterns and output patterns. According to recent developments in neural network technology, neural network devices are used in various types of electronic systems to analyze input data and extract useful information.

[0005] Neural network devices perform a large number of operations on input data. Research has been conducted on technologies that can efficiently process neural network operations. Summary of the invention

[0006] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0007] In one general aspect, a neural network device is provided, comprising: a processor configured to generate an output value of an operation between each weight of a convolution kernel and an input feature map, and to generate an output feature map by accumulating the output values ​​at positions in the output feature map that are set based on positions of the weights in the convolution kernel.

[0008] The processor can be configured to: generate a first output value by performing an operation between an input feature map and a first weight of a convolution kernel; accumulate the first output value at a first position in the output feature map based on the position of the first weight in the convolution kernel; generate a second output value by performing an operation between the input feature map and a second weight of the convolution kernel; and accumulate the second output value at a second position in the output feature map based on the position of the second weight in the convolution kernel.

[0009] The processor can be configured to: generate a first output value by performing an operation between a first region of the input feature map and each weight of the convolution kernel; generate a first partial output feature map by accumulating the first output value at a position in the first partial output feature map based on the position of the corresponding weight in the convolution kernel; and accumulate the first partial output feature map on the output feature map.

[0010] The processor can be configured to: generate a second output value by performing an operation between a second region in the input feature map and each weight of the convolution kernel, the second region being different from the first region; generate a second partial output feature map by accumulating the second output values ​​at positions in the second partial output feature map based on positions of corresponding weights in the convolution kernel; and accumulate the second partial output feature map on the output feature map.

[0011] The first region may include at least one of n pixels, n×m pixels, or n×m×l pixels in the input feature map, where n, m, and l are natural numbers greater than or equal to 1.

[0012] The processor may be configured to: in response to a first weight in the weights of the convolution kernel being zero, skip an operation between the input feature map and the first weight.

[0013] The processor may be configured to perform an operation between each weight of the convolution kernel and the compressed input feature map by continuously streaming the compressed input feature map from the memory.

[0014] The processor may include: an operation unit configured to generate a partial output feature map by performing operations between different regions among the regions of the input feature map and the convolution kernel; and an output unit configured to generate each region in the region of the output feature map by accumulating one or more partial output feature maps among the partial output feature maps.

[0015] One or more of the operation units can be configured to perform operations between the convolution kernel and different regions independently of each other and in parallel with each other.

[0016] The neural network device may include a bus, wherein the output unit may be configured to receive one or more partial output feature maps from the operation unit via the bus.

[0017] Each of the operation units may include multiple processing units, and the processing units are configured to generate a partial output feature map by performing an operation between a region among the regions of the input feature map and each of the multiple convolution kernels.

[0018] The processing unit includes: a first processing unit, configured to perform operations between the region and the first convolution kernel; and a second processing unit, configured to perform operations between the region and the second convolution kernel, and the first processing unit can be configured to: after completing the operation between the region and the first convolution kernel, perform a part of the operation between the region and the second convolution kernel.

[0019] The neural network device may include a memory storing instructions that, when executed, configure a processor to generate output values ​​and to generate an output feature map.

[0020] In another general aspect, a method of processing a convolution operation in a neural network is provided, the method comprising generating an output value of an operation between each weight of a convolution kernel and an input feature map, and generating an output feature map by accumulating the output values ​​at positions in the output feature map that are set based on positions of weights in the convolution kernel.

[0021] Generating an output value may include: generating a first output value by performing an operation between an input feature map and a first weight of a convolution kernel, and generating a second output value by performing an operation between the input feature map and a second weight of the convolution kernel; and generating an output feature map may include: accumulating the first output value at a first position in the output feature map based on the position of the first weight in the convolution kernel, and accumulating the second output value at a second position in the output feature map based on the position of the second weight in the convolution kernel.

[0022] Generating an output value may include: generating a first output value by performing an operation between a first region of the input feature map and each weight in the convolution kernel, and generating a second output value by performing an operation between a second region of the input feature map and each weight in the convolution kernel, the second region being different from the first region; and generating an output feature map may include: generating a first partial output feature map by accumulating the first output value at a position in the first partial output feature map based on a position of a corresponding weight in the convolution kernel, and accumulating the first partial output feature map on the output feature map, and generating a second partial output feature map by accumulating the second output value at a position in the second partial output feature map based on a position of a corresponding weight in the convolution kernel, and accumulating the second partial output feature map on the output feature map.

[0023] Generating the output value may include: in response to a first weight of the convolution kernel being zero, skipping an operation between the input feature map and the first weight.

[0024] Generating an output value may include: performing an operation between each weight of the convolution kernel and the compressed input feature map by continuously streaming the compressed input feature map.

[0025] The method may include: generating a partial output feature map by performing operations between different regions among the regions of the input feature map and a convolution kernel; and generating each region in the region of the output feature map by accumulating one or more partial output feature maps among the partial output feature maps.

[0026] Generating a partial output feature map may include generating a partial output feature map by performing an operation between a region among the regions of the input feature map and each of a plurality of convolution kernels.

[0027] In another general aspect, a neural network device is provided, comprising: a memory configured to store weights and instructions of a convolution kernel; and a processor configured to execute the instructions to generate a first output value by performing an operation between a first region of an input feature map and the weights, causing the first output value to be located at a position in a first partial output feature map based on a corresponding position of the weights in the convolution kernel, generate a second output value by performing an operation between a second region of the input feature map and the weights, and causing the second output value to be located at a position in a second partial output feature map based on a corresponding position of the weights in the convolution kernel.

[0028] The processor may be configured to generate a second output value by performing an operation between the weight and a portion in a second region different from the first region.

[0029] The processor may be configured to, in response to a weight in the weight being zero, skip an operation between the first region or the second region of the input feature map and the weight.

[0030] Other features and aspects will become apparent from the following detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a diagram showing an example of the architecture of a neural network.

[0032] Figure 2A , Figure 2B and Figure 2C is a diagram showing an example for describing an example of a convolution operation in a neural network.

[0033] Figure 3 is a diagram showing an example of the hardware configuration of a neural network device.

[0034] Figure 4 is a diagram showing an example in which a processor generates an output feature map by reusing an input feature map.

[0035] Figure 5 is a diagram showing an example in which a processor generates a portion of an output feature map by reusing a region of an input feature map.

[0036] Figure 6 is a diagram showing an example of a processor generating a partial output feature map.

[0037] Figure 7is a diagram showing examples of various forms of regions in an input feature map for operating with a convolution kernel.

[0038] Figure 8 An example is shown in which a processor generates a portion of an output feature map by reusing a region of an input feature map.

[0039] Fig. 9 An example is shown in which the processor uses a portion of the convolution kernel to generate a partial output feature map.

[0040] Fig.10 An example is shown in which a processor performs a convolution operation by streaming a compressed input feature map.

[0041] Fig.11 An example of the hardware configuration of the processor is shown.

[0042] Fig.12 An example of the hardware configuration of the processor is shown.

[0043] Fig.13 An example of the hardware configuration of the processor is shown.

[0044] Fig.14 An example is shown in which a processing unit of a processor performs an operation between a convolution kernel and a region of an input feature map.

[0045] Fig.15 An example is shown in which a processing unit of a processor performs an operation between a convolution kernel and a region of an input feature map.

[0046] Fig.16 is a diagram showing an example of an operation method of a neural network.

[0047] Throughout the drawings and detailed description, unless otherwise described or provided, the same reference numerals will be understood to refer to the same elements, features, and structures. The drawings may not necessarily be drawn to scale, and the relative size, proportion, and depiction of the elements in the drawings may be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0048] The following specific embodiments are provided to help the reader gain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be apparent. For example, the order of operations described herein is merely an example and is not limited to those orders set forth herein, and except for operations that must occur in a particular order, the order of operations can be changed as will be apparent after understanding the disclosure of the present application. In addition, for greater clarity and brevity, descriptions of features known in the art may be omitted.

[0049] The features described herein may be implemented in different forms and are not to be construed as being limited to the examples described herein. Rather, the examples described herein are provided merely to illustrate some of the many possible ways to implement the methods, devices, and / or systems described herein, which will be apparent after understanding the disclosure of the present application.

[0050] Although terms such as "first," "second," and "third" may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions are not limited by these terms. Instead, these terms are only used to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Therefore, without departing from the teachings of the examples described herein, a first member, component, region, layer, or portion mentioned in the examples may also be referred to as a second member, component, region, layer, or portion.

[0051] Throughout the specification, when a component is described as being "connected to" or "coupled to" another component, it may be directly "connected to" or "coupled to" the other component, or there may be one or more other components intervening. Conversely, when an element is described as being "directly connected to" or "directly coupled to" another element, there may be no other elements intervening. Similarly, expressions such as "between..." and "immediately between..." and "adjacent to..." and "immediately adjacent to..." may also be interpreted as described above. As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more items in the associated listed items. In addition, unless explicitly described to the contrary, the word "include" and variants such as "comprise" will be understood to imply the inclusion of the stated elements, but not to the exclusion of any other elements.

[0052] The singular forms "a", "an", and "said" are intended to also include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "include / contain", "have / include" when used herein specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof. Throughout the disclosure, the expression "at least one of a, b, or c" indicates only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof. The term "may" used herein with respect to examples or embodiments (e.g., with respect to what an example or embodiment may include or implement) means that there is at least one example or embodiment that includes or implements such a feature, but all examples and embodiments are not limited thereto.

[0053] Figure 1 is a diagram showing an example of the architecture for describing the neural network 1.

[0054] refer to Figure 1 , the neural network 1 can be a deep neural network (DNN) or an n-layer neural network architecture. The DNN or n-layer neural network can correspond to a neural network, such as a convolutional neural network (CNN), a recurrent neural network (RNN), a deep belief network, a restricted Boltzmann machine, a fully connected network (FCN), a deep convolutional network (DCN), a long short-term memory (LSTM) network, and a gated recurrent unit (GRU). For example, the neural network 1 can be embodied as a CNN, but is not limited thereto. Figure 1 In FIG. 1 , some convolutional layers of a CNN corresponding to the example of neural network 1 are shown, but in addition to those convolutional layers shown, the CNN may also include pooling layers or fully connected layers.

[0055] The neural network 1 can be implemented as an architecture having multiple layers including an input image, a feature map, and an output. In the neural network 1, a convolution operation is performed on the input image using a filter called a convolution kernel, and a feature map as a result is output. The output feature map is used as the input feature map and the convolution kernel is used to perform a convolution operation on it again, and a new feature map is output. When the convolution operation is repeatedly performed in this way, a recognition result related to the features of the input image can be finally output through the neural network 1.

[0056] For example, when Figure 1 When the neural network 1 inputs an input image having a size of 24×24 pixels, the input image can be output as a feature map having four channels by performing a convolution operation with a convolution kernel, wherein each feature map has a size of 20×20 pixels. Then, the size of the 20×20 feature map can be reduced by repeatedly performing a convolution operation with the convolution kernel, and finally, features each having a size of 1×1 pixels can be output. In the neural network 1, convolution operations and subsampling (or pooling) operations can be repeatedly performed in several layers to filter and output robust features that can represent the entire input image from the input image, and the recognition result of the input image is derived by the final features output.

[0057] In another example, the neural network 1 may receive an input source sentence (e.g., voice input) instead of an input image. In such an example, a convolution operation with a convolution kernel is performed on the input source sentence, and a feature map as a result is output. The output feature map is used as the input feature map and the convolution operation with the convolution kernel is performed again, and a new feature map is output. When the convolution operation is repeatedly performed in this way, a recognition result related to the feature of the input source sentence can be output by the neural network 1.

[0058] Figure 2A , Figure 2Band Figure 2C is a diagram illustrating an example of a convolution operation in a neural network.

[0059] exist Figure 2A , the input feature map 210 has a size of 6×6 pixels, the convolution kernel 220 has a size of 3×3 pixels, and the output feature map 230 has a size of 4×4 pixels, but the size is not limited thereto, and the neural network may include feature maps and convolution kernels having various sizes. In addition, the values ​​defined in the input feature map 210, the convolution kernel 220, and the output feature map 230 are only examples and are not limited thereto.

[0060] The convolution kernel 220 performs a convolution operation in a region (or patch) unit having a size of 3×3 pixels while sliding on the input feature map 210. The convolution operation means an operation in which each pixel value of the output feature map 230 is obtained by adding all values ​​obtained by multiplying each pixel value of any region of the input feature map 210 by the weight of each element at the corresponding position in the convolution kernel 220. In detail, the convolution kernel 220 may first perform a convolution operation with the first region 211 of the input feature map 210. In other words, the pixel values ​​1, 2, 3, 4, 5, 6, 7, 8, and 9 of the first region 211 are multiplied by the weights -1, -3, +4, +7, -2, -1, -5, +3, and +1 of the elements of the convolution kernel 220, respectively, and the resulting values ​​-1, -6, 12, 28, -10, -6, -35, 24, and 9 are obtained. Then, the values ​​-1, -6, 12, 28, -10, -6, -35, 24, and 9 are added to obtain a value of 15, and the pixel value 231 of the first row and the first column of the corresponding output feature map 230 is determined to be the value 15. The pixel value 231 of the first row and the first column of the output feature map 230 corresponds to the first region 211. Similarly, a convolution operation is performed between the second region 212 of the input feature map 210 and the convolution kernel 220, and thus the pixel value 232 of the first row and the second column of the output feature map 230 is determined to be 4. Finally, a convolution operation is performed between the sixteenth region 213 (i.e., the last window of the input feature map 210) and the convolution kernel 220, and thus the pixel value 233 of the fourth row and the fourth column of the output feature map 230 is determined to be 11.

[0061] In other words, the convolution operation between an input feature map 210 and a convolution kernel 220 can be processed by repeatedly performing multiplication of the values ​​of the elements of the input feature map 210 corresponding to each other and the values ​​of the elements of the convolution kernel 220, and adding the multiplication results, and generating an output feature map 230 as the result of the convolution operation.

[0062] exist Figure 2B, the input feature map 250 has a 1×1 pixel size, the original convolution kernel 260 has a 3×3 pixel size, and the output feature map 270 has a 3×3 pixel size, but the size is not limited thereto, and the neural network may include feature maps and convolution kernels of various sizes with various values.

[0063] The convolution kernel 260 performs a convolution operation in a region (or patch) unit having a size of 3×3 pixels while sliding on the input feature map 250. In detail, the convolution kernel 220 may first perform a convolution operation with the first region 251 of the input feature map 250. That is, the unique pixel value 9 of the first region 251 is multiplied by the weight +1 of the convolution kernel 260, and the pixel value 271 of the first row and the first column of the output feature map 270 is determined as the product 9.

[0064] Similarly, a convolution operation is performed between the second region 252 of the input feature map 250 and the convolution kernel 260, and thus the pixel value 272 of the first row and the second column of the output feature map 270 is determined to be 27. Finally, a convolution operation is performed between the last region (ninth region 253) of the input feature map 250 and the convolution kernel 260, and thus the pixel value 273 of the third row and the third column of the output feature map 270 is determined to be -9.

[0065] At the same time, although reference has been made Figure 2A and Figure 2B A two-dimensional (2D) convolution operation is described, but the convolution operation may correspond to a three-dimensional (3D) convolution operation, where there is an input feature map with multiple channels, a convolution kernel, and an output feature map, as will be referred to in Figure 2C Described.

[0066] refer to Figure 2C , there are X channels in the input feature map 201, and the input feature map 201 of each channel may have a size of H rows and W columns, where X, W, and H are each natural numbers. Each convolution kernel 202 may have a size of R rows and S columns, and has a number of channels corresponding to the number of channels X of the input feature map 201 and the number of channels Y of the output feature map 203, where R, S, and Y are each natural numbers. The output feature map 203 is generated via a 3D convolution operation between the input feature map 201 and the convolution kernel 202, and Y channels may exist in the output feature map 203 via the 3D convolution operation.

[0067] The process of generating an output feature map through a convolution operation between an input feature map and a convolution kernel is the above reference Figure 2A As described above, and repeatedly performed between the input feature maps 201 of all channels and the convolution kernels 202 of all channels Figure 2A The 2D convolution operation is performed to generate output feature maps 203 of all channels.

[0068] Figure 3 is a diagram showing an example of a neural network device 100 .

[0069] Neural network apparatus 100 may be embodied as any of various types of devices that use neural networks to perform speech recognition, image recognition, and image classification, such as a server, a mobile device, a smart phone, an embedded device, a wearable smart device (e.g., a ring, a watch, a pair of glasses, an eyeglass-like device, a wristband, an anklet, a belt, a necklace, an earring, a hairband, a helmet, a device embedded in clothing, or an eyeglass display (EGD)), a computing device (e.g., a server, a laptop, a notebook computer, a subnotebook computer, a netbook, an ultra mobile PC (UMPC), a tablet personal computer (tablet), a tablet phone, a mobile Internet device (MID), a personal digital assistant (PDA), an enterprise digital assistant (EDA), an ultra mobile personal computer (UMPC), a portable lab top PC), an electronic product (e.g., a robot, a digital camera, a digital video camera, a portable The examples described herein may be applied to, but are not limited to, vehicles and vehicle management systems, such as autonomous vehicles, automatic or autonomous driving systems, smart vehicles, advanced driver assistance systems (ADAS), navigation systems that assist vehicles in safely maintaining the lane in which the vehicle is traveling. The examples described herein may be applied to road guidance information in a navigation device of a vehicle, such as an augmented reality head-up display (AR HUD), a smart TV, a smart home appliance, a smart home device, or a security device for door control, a voice authentication system, an augmented reality (AR) device, an Internet of Things (IoT) device, an autonomous vehicle, a robotic device, or a medical device. 3D HUD). In addition, neural network device 100 may be a dedicated hardware accelerator installed in the above-mentioned device, and neural network device 100 may be a hardware accelerator as a dedicated module for driving a neural network, such as a neural processing unit (NPU), a tensor processing unit (TPU), a neural engine, but not limited thereto. The above examples are non-limiting, and other examples (e.g., training, gaming, applications in healthcare, public safety, tourism, and marketing) are considered to be fully within the scope of the present disclosure. These devices perform one or more functions, such as speech recognition, image recognition, and image classification.

[0070] refer to Figure 3, the neural network device 100 includes a processor 110, a memory 120, and a user interface 130. The memory 120, the processor 110, and the user interface 130 may be connected via a system bus or other suitable circuits. Figure 3 In FIG. 1 , only components of the neural network device 100 related to the current embodiment are shown. Therefore, the neural network device 100 may include Figure 3 Common components other than those shown in .

[0071] Processor 110 controls the overall functions of neural network device 100 for executing in a neural network. For example, processor 110 may generally control neural network device 100 by executing a program stored in memory 120 of neural network device 100. Processor 110 is included in reference Figures 4 to 6 and Figures 8 to 15 At least one of the devices described herein may include such a device or perform the steps of Fig.16 At least one of the methods described. Processor 110 refers to a data processing device that is configured as hardware of a circuit having a physical structure for performing a desired operation. For example, the desired operation may include a code or instruction included in a program. For example, processor 110 may be embodied as a microprocessor, a processor core, a multi-core processor, a multiprocessor, a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), an application specific integrated circuit (ASIC), and a field programmable gate array (FPGA) included in neural network device 100, but is not limited thereto. In addition, processor 110 may execute a program stored in memory 120 to control all operations of neural network device 100. For example, processor 110 may be implemented as a microprocessor (MCU) in which a CPU, a memory (read-only memory (ROM) or a random access memory (RAM)), etc. are installed in a single chip, but is not limited thereto. Further details about processor 110 are provided below.

[0072] Memory 120 is hardware that stores various types of data processed in neural network device 100, and memory 120 may store data processed or to be processed by neural network device 100. In addition, memory 120 may store applications or driver programs to be driven by neural network device 100. Memory 120 may include random access memory (RAM) (e.g., dynamic random access memory (DRAM) or static random access memory (SRAM)), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM, Blu-ray disc, optical disc storage device, hard disk drive (HDD), solid state drive (SSD), or flash memory. Further details about memory 120 are provided below.

[0073] User interface 130 is a physical structure including one or more hardware components that provide the ability to present a user interface, present a display, output information, and / or receive user input. User interface 130 outputs the results it receives from neural network device 100. However, user interface 130 is not limited to the above examples, and in the examples, any display (e.g., a computer monitor and an eyeglass display (EGD)) that is effectively connected to neural network device 100 may be used without departing from the spirit and scope of the illustrative examples described.

[0074] The processor 110 may include an operation unit that performs a convolution operation and an on-chip memory that performs a cache function.

[0075] The processor 110 processes the convolution operation between the input feature map and the convolution kernel using the pixel values ​​of the input feature map and the weights of the convolution kernel stored (or buffered) in the on-chip memory from the memory 120. The processor 110 may include at least one operation unit and at least one on-chip memory, and when each of the at least one operation unit and each of the at least one on-chip memory are used to independently and in parallel perform the convolution operation, the convolution operation may be performed efficiently.

[0076] The logic circuit for the convolution operation may be included in the operation unit of the processor 110. That is, the operation unit of the processor 110 may include an operator implemented using a combination of a multiplier, an adder, and an accumulator. The multiplier may include a combination of sub-multipliers, and the adder may also include a combination of sub-adders.

[0077] The operation unit of the processor 110 may include a scheduler that schedules various operands (e.g., pixel values ​​of the input feature map and weights of the convolution kernel) to the convolution operator. The scheduler schedules operands (e.g., pixel values ​​and weights required for the operation unit to perform the convolution operation) to the on-chip memory according to the data of the pixel values ​​of the input feature map and the weights of the convolution kernel, wherein the data is stored in the memory 120. Then, the scheduler schedules the operands scheduled to the on-chip memory to the processing unit in the operation unit again to perform the convolution operation.

[0078] The processor 110 may generate an output feature map by performing a convolution operation between the input feature map and the convolution kernel. For efficient convolution operation, the processor 110 may generate an output value by performing an operation between each weight of the convolution kernel and the input feature map. The processor 110 may perform an operation between the input feature map and each weight of the convolution kernel by reusing the input feature map. In an example, the processor 110 may generate a first output value by multiplying each pixel value of the input feature map by a first weight of the convolution kernel, and may generate a second output value by multiplying each pixel value of the input feature map by a second weight of the convolution kernel.

[0079] The processor 110 can generate an output feature map by accumulating output values ​​at positions set based on the positions of weights in the convolution kernel in the output feature map. The processor 110 can generate an output feature map filled with output values ​​by accumulating output values ​​at positions in the output feature map. The processor 110 can configure the position in the output feature map where the output value is to be accumulated based on the position of the weights in the convolution kernel. For example, the processor 110 can configure the position in the output feature map where the first output value is to be accumulated based on the position of the first weight in the convolution kernel, and configure the position in the output feature map where the second output value is to be accumulated based on the position of the second weight in the convolution kernel. In the example, before performing the operation between the input feature map and the convolution kernel, the processor 110 can pre-configure the position in the output feature map where the output value is to be accumulated. Therefore, the processor 110 can generate an output feature map by accumulating the first output value at a position set based on the first weight in the output feature map and accumulating the second output value at a position set based on the second weight in the output feature map.

[0080] When the processor 110 generates an output feature map by reusing an input feature map read from the memory 120 in each cycle in a convolution operation, the processor 110 can minimize the number of times the input feature map is read from the memory 120 to one regardless of the size of the convolution kernel.

[0081] In the case where the first weight of the convolution kernel is zero, the processor 110 may skip the operation between the input feature map and the first weight. In the example, when the processor 110 sequentially performs the operation between each weight of the convolution kernel and the input feature map in each cycle, the operation between the input feature map and the first weight having a value of zero may be skipped. Therefore, the processor 110 may reduce the time required for the convolution operation between the input feature map and the convolution kernel by as many cycles as the zero weight.

[0082] Figure 4 An example is shown in which a processor generates an output feature map by reusing an input feature map.

[0083] exist Figure 4In the figure, for convenience, the input feature map 410 is shown as a 1×1 pixel area, and the convolution kernel 420 is shown as a 3×3 pixel area, but the input feature map and the convolution kernel can be areas of different sizes without being limited to the illustration.

[0084] In the first cycle, the processor 110 may generate a first output value by performing an operation between the input feature map 410 and the first weight 422 of the convolution kernel 420. In an example, the processor 110 may generate a first output value by performing a multiplication operation between the pixel value of the input feature map 410 and the first weight 422 of the convolution kernel 420. The processor 110 may accumulate the first output value at a position in the output feature map 430 that is set based on the position of the first weight 422 in the convolution kernel 420. The position in the output feature map 430 (which corresponds to the position of the first weight 422 in the convolution kernel 420) may be set to the third row and the third column of the output feature map 430. Therefore, the processor 110 may accumulate the first output value to the third row and the third column of the output feature map 430.

[0085] In the second cycle, the processor 110 may generate a second output value by performing an operation between the input feature map 410 and the second weight 424 of the convolution kernel 420. The processor 110 may accumulate the second output value at a position in the output feature map 430 that is set based on the position of the second weight 424 in the convolution kernel 420. The position in the output feature map 430 (which corresponds to the position of the second weight 424 in the convolution kernel 420) may be set to the third row and second column of the output feature map 430. That is, when the position of the weight as an operand changes from the first weight 422 to the right by one block to become the second weight 424, the position in the output feature map 430 for accumulating the output value may change from the third row and the third column to the left by one block to become the third row and the second column. Therefore, the processor 110 may accumulate the second output value to the third row and the second column of the output feature map 430.

[0086] In the 3rd cycle, the processor 110 may generate a third output value by performing an operation between the input feature map 410 and the third weight 426 of the convolution kernel 420, and accumulate the third output value in the third row and the first column of the output feature map 430. In the 4th cycle, the processor 110 may generate a fourth output value by performing an operation between the input feature map 410 and the fourth weight 428 of the convolution kernel 420, and accumulate the fourth output value in the second row and the third column of the output feature map 430. Similarly, in the 5th to 9th cycles, the processor 110 may generate an output value by performing an operation between each weight of the convolution kernel 420 and the input feature map 410. The processor 110 may generate an output feature map 430 filled with the output value by accumulating the output value in a position corresponding to the position of the weight of the convolution kernel 420 in the output feature map 430.

[0087] For convenience, Figure 4 4 shows that the operation between each weight of the convolution kernel 420 and the input feature map 410 is performed in a total of 9 cycles, but the operation between the zero weight and the input feature map 410 can be skipped. That is, the processor 110 can perform the operation between each weight of the convolution kernel 420 and the input feature map 410 as many times as the number of non-zero weights of the convolution kernel 420.

[0088] Therefore, if Figure 2A and Figure 2B As shown, the processor 110 can perform a convolution operation by reusing the input feature map in each cycle while setting the position for accumulating output values ​​on the output feature map based on the position of the weight of the convolution kernel in advance, rather than performing a convolution operation while reading the input feature map including overlapping areas several times, thereby efficiently performing the convolution operation.

[0089] refer to Figure 3 , the processor 110 can generate a first output value by performing an operation between each weight in the convolution kernel and the first region based on the reuse of the first region in the input feature map. The processor 110 can generate a first partial output feature map by accumulating the first output value at a position in the first partial output feature map that is set based on the position of the weight in the convolution kernel. Then, the processor 110 can accumulate the first partial output feature map on the output feature map. The processor 110 can set a position in the output feature map for accumulating the first partial output feature map based on the position of the first region in the input feature map, and accumulate the first partial output feature map at the set position.

[0090] The processor 110 may generate a second output value by performing an operation between each weight in the convolution kernel and the second region based on the reuse of a second region different from the first region in the input feature map. The processor 110 may generate a second partial output feature map by accumulating the second output value at a position set based on the position of the weight in the convolution kernel in the second partial output feature map. Then, the processor 110 may accumulate the first partial output feature map on the output feature map. The processor 110 may set a position in the output feature map for accumulating the second partial output feature map based on the position of the second region in the input feature map, and accumulate the second partial output feature map at the set position.

[0091] Similarly, the processor 110 may generate an Nth partial output feature map by performing an operation between each weight in the convolution kernel and the Nth region based on reusing the Nth region different from the first region and the second region in the input feature map, where N is a natural number greater than or equal to 3. Therefore, the processor 110 may generate an output feature map by accumulating the first partial output feature map to the Nth partial output feature map on the output feature map.

[0092] The processor 110 may include a multiplier MUL corresponding to each pixel in the area of ​​the input feature map, and a multiplexer MUX, an adder, and an accumulator & register ("Acc.Register"), which respectively correspond to the pixels of the partial output feature map, for generating the partial output feature map.

[0093] The processor 110 may configure various forms of regions in the input feature map, and generate a portion of the output feature map by performing an operation between the configured region and the convolution kernel. The various forms of regions may include n pixels, n×m pixels, or n×m×l pixels, where n, m, and l may be natural numbers greater than or equal to 1. The input feature map may be a two-dimensional (2D) input feature map or a three-dimensional (3D) input feature map, and the region of the input feature map may be a 2D region or a 3D region.

[0094] The processor 110 may generate a partial output feature map by performing an operation between an area of ​​the input feature map and a partial area of ​​the convolution kernel. By limiting the operation to a partial area of ​​the convolution kernel, the processor 110 may reduce the size of the partial output feature map, thereby reducing the size of the buffer for the partial output feature map. For example, when the area of ​​the input feature map has a size of 1×10 pixels and the convolution kernel has a size of 3×3 pixels, the partial output feature map as a result of the operation may have an area of ​​3×12 pixels. In this case, by limiting the size of the convolution kernel to 1×3 pixels, the processor 110 may perform a convolution operation so that the partial output feature map has an area of ​​1×12 pixels, and thus the size of the buffer for the partial output feature map may be reduced.

[0095] Figure 5 An example is shown in which a processor generates a portion of an output feature map by reusing a region of an input feature map.

[0096] exist Figure 5 In the figure, for convenience, the first area 510 of the input feature map 501 is shown as a 4×4 pixel area, and the convolution kernel 520 is shown as a 3×3 pixel area, but the first area of ​​the input feature map and the convolution kernel can be areas of different sizes without being limited to the illustration.

[0097] In the first cycle, the processor 110 may generate a first output value by performing an operation between the first region 510 and the first weight 522 of the convolution kernel 520. In an example, the processor 110 may generate a first output value by performing a multiplication operation between each pixel value of the first region 510 and the first weight 522. The processor 110 may generate sixteen first output values ​​by performing a multiplication operation between each of the sixteen pixel values ​​of the first region 510 and the first weight 522. The processor 110 may accumulate the first output value at a position in the first partial output feature map 530 that is set based on the position of the first weight 522 in the convolution kernel 520. The position in the first partial output feature map 530 that corresponds to the position of the first weight 522 in the convolution kernel 520 may form a region 532 of the first partial output feature map 530. Therefore, the processor 110 may accumulate the first output value in the region 532 of the first partial output feature map 530. That is, the processor 110 may accumulate the product of the pixel values ​​of n rows and m columns of the first region 510 and the first weight 522 in the region 532 of the first partial output feature map 530 , where n and m are natural numbers.

[0098] In the second cycle, the processor 110 may generate a second output value by performing an operation between the first region 510 and the second weight 524 of the convolution kernel 520. The processor 110 may accumulate the second output value at a position in the first partial output feature map 530 that is set based on the position of the second weight 524 in the convolution kernel 520. The position in the first partial output feature map 530, which corresponds to the position of the second weight 524 in the convolution kernel 520, may form an area 534 of the first partial output feature map 530. That is, when the position of the weight as an operand changes from the first weight 522 to the right by one block to become the second weight 524, the area in the first partial output feature map 530 for accumulating output values ​​may change from the area 532 to the left by one block to become the area 534. Therefore, the processor 110 may accumulate the second output value in the area 534 of the first partial output feature map 530.

[0099] Similarly, in the 3rd to 9th cycles, the processor 110 may generate an output value by performing an operation between each weight of the convolution kernel 520 and the first region 510. The processor 110 may generate the first partial output feature map 530 by accumulating the output values ​​in the region corresponding to the position of the weight of the convolution kernel 520 in the first partial output feature map 530.

[0100] The processor 110 may accumulate the generated first partial output feature map 530 on the output feature map 531. The processor 110 may accumulate the first partial output feature map in a position in the output feature map 531 that is set based on the position of the first region 510 in the input feature map 501.

[0101] For the Nth region in the input feature map 501 that is different from the first region 510, the processor 110 may generate an output value by performing an operation between each weight in the convolution kernel 520 and the Nth region based on the reuse of the Nth region, and generate the Nth partial output feature map by accumulating the output values ​​at a position in the Nth partial output feature map that is set based on the position of the weights in the convolution kernel 520. The processor 110 may accumulate the generated Nth partial output feature map on the output feature map 531. The processor 110 may generate the output feature map 531 by accumulating the first partial output feature map to the Nth partial output feature map on the output feature map 531. In other words, the processor 110 may generate the output feature map 531 filled with the output values ​​of the first partial output feature map to the Nth partial output feature map. For convenience, Figure 5, the operation between each weight of the convolution kernel 520 and the first region 510 is performed in a total of 9 cycles, but the operation between the zero weight and the first region 510 can be skipped. That is, the processor 110 can perform the operation between each weight of the convolution kernel 520 and the first region 510 as many times as the number of non-zero weights of the convolution kernel 520.

[0102] Figure 6 An example of a processor generating a partial output feature map is shown.

[0103] exist Figure 6 In the embodiment, the processor 110 may include sixteen multipliers (MUL), thirty-six multiplexers (MUX), thirty-six adders (Adder), and thirty-six accumulators & registers (Acc.Register) to generate Figure 5 The first part outputs feature map 530.

[0104] Each of the sixteen multipliers can be used with Figure 5 Corresponding to each pixel of the first area 510. For each of the sixteen multipliers, the weight of the convolution kernel 520 and each pixel of the first area 510 may be input. For example, the first weight of the convolution kernel 520 and the first pixel of the first area 510 may be input to the first multiplier, the first weight of the convolution kernel 520 and the second pixel of the first area 510 may be input to the second multiplier, and the first weight of the convolution kernel 520 and the sixteenth pixel of the first area 510 may be input to the sixteenth multiplier. In each of the nine cycles, the first weight to the ninth weight of the convolution kernel 520 may be sequentially input to the sixteen multipliers, respectively, and each pixel of the first area 510 may be repeatedly input. Therefore, in each of the nine cycles, the sixteen multipliers may perform a multiplication operation between each weight in the convolution kernel 520 and the first area 510, and thereby output an output value.

[0105] In the example, each of the thirty-six multiplexers, adders, and accumulators & registers may correspond to each of the thirty-six pixels of the first partial output feature map 530. That is, a group of multiplexers, adders, and accumulators & registers may correspond to any one of the thirty-six pixels. Each of the thirty-six multiplexers may receive a preset number of output values ​​from among the output values ​​of the sixteen multipliers.

[0106] exist Figure 6, 610 indicates the number of output values ​​accumulated for each of the thirty-six pixels of the first partial output feature map 530. For example, one output value may be accumulated in the pixel values ​​of the first row and the first column of the first partial output feature map 530, but nine output values ​​may be accumulated in the pixel values ​​of the third row and the third column of the first partial output feature map 530. The number of output values ​​accumulated for each of the thirty-six pixels of the first partial output feature map 530 may represent the number of inputs to the multiplexer. For example, the multiplexer corresponding to the pixel of the third row and the third column of the first partial output feature map 530 may receive the output values ​​output from the nine multipliers as inputs.

[0107] Each of the thirty-six multiplexers can receive a plurality of output values ​​from among the output values ​​of the sixteen multipliers, and select an output value from among the plurality of output values. Each of the thirty-six multiplexers can receive a number of output values ​​corresponding to each pixel of the first partial output feature map 530 (as shown in 610), and select an output value based on the position of the weight in the convolution kernel 520. For example, a multiplexer corresponding to a pixel of the third row and the third column of the first partial output feature map 530 can receive nine output values ​​from the first row and the first column to the third row and the third column in the area output as the result of the operation between the first area 510 and the weight in the convolution kernel 520. In the operation between the first area 510 and the first weight 522 in the convolution kernel 520, the multiplexer can select the output value of the first row and the first column from among the nine output values ​​in the area 532 based on the position of the first weight 522 in the convolution kernel 520. In the operation between the first region 510 and the second weight 524 in the convolution kernel 520 , the multiplexer may select the output values ​​of the first row and second column from among the nine output values ​​in the region 534 based on the position of the second weight 524 in the convolution kernel 520 .

[0108] Each of the thirty-six adders and accumulators & registers can accumulate the output value selected from each of the thirty-six multiplexers. Therefore, as a result of accumulating the output values ​​in a total of nine cycles, each of the thirty-six accumulators & registers can generate a first partial output feature map 530 configured with thirty-six pixel values.

[0109] Figure 7 Examples of various forms of regions in the input feature map for operating with convolution kernels are shown.

[0110] The processor 110 may configure various forms of regions in the input feature map 710, and generate a partial output feature map by performing an operation between the configured regions and the convolution kernel.

[0111] In an example, the processor 110 may generate a partial output feature map for the region 720 by configuring the region 720 consisting of n×n pixels in the input feature map 710 through an operation with a convolution kernel, and accumulate the partial output feature map on the output feature map.

[0112] In the example, the processor 110 may generate a partial output feature map for the region 730 by configuring the region 730 consisting of 1×n pixels in the input feature map 710 through an operation with the convolution kernel, and accumulate the partial output feature map on the output feature map. In other words, the processor 110 may configure a region that is input only in one direction, such as the region 730, in addition to the square region 720 in the input feature map 710.

[0113] According to an example, the processor 110 may generate a partial output feature map for the region 740 by configuring the region 740 consisting of 1×1×n pixels in the input feature map 710 through an operation with a convolution kernel, and accumulate the partial output feature map on the output feature map.

[0114] Figure 8 An example is shown in which a processor generates a portion of an output feature map by reusing a region of an input feature map.

[0115] exist Figure 8 In the figure, for convenience, the first area 810 of the input feature map is shown as a 1×10 pixel area, and the convolution kernel 820 is shown as a 3×3 pixel area, but the first area of ​​the input feature map and the convolution kernel can be areas of different sizes without being limited to the illustration.

[0116] In the first cycle, the processor 110 may generate a first output value by performing an operation between the first region 810 and the first weight 822 of the convolution kernel 820, and accumulate the first output value at a position in the first partial output feature map 830 that is set based on the position of the first weight 822 in the convolution kernel 820. In other words, the processor 110 may accumulate the first output value in the region 832 of the first partial output feature map 830.

[0117] In the 2nd to 9th cycles, the processor 110 may generate a first partial output feature map 830 by performing an operation between each weight of the convolution kernel 820 and the first region 810 based on the reuse of the first region 810 .

[0118] Fig. 9 An example is shown in which the processor uses a portion of the convolution kernel to generate a partial output feature map.

[0119] By restricting the operation to Figure 8The processor 110 may perform a partial area 920 of the convolution kernel 820. Figure 8 An operation is performed between the first area 810 of the input feature map and the partial area 920 of the convolution kernel to generate a partial output feature map 930.

[0120] The processor 110 may generate a first output value by performing an operation between the first weight 922 of the partial region 920 and the first region 810, and accumulate the first output value in region 932 of the partial output feature map 930. The processor 110 may generate a second output value by performing an operation between the second weight 924 of the partial region 920 and the first region 810, and accumulate the second output value in region 934 of the partial output feature map 930. The processor 110 may generate a third output value by performing an operation between the third weight of the partial region 920 and the first region 810, and generate the partial output feature map 930 by accumulating the third output value in region 936 of the partial output feature map 930.

[0121] The processor 110 may generate a partial output feature map by performing an operation between another region of the convolution kernel 820 and the first region 810 of the input feature map.

[0122] Therefore, in Figure 8 In comparison, Fig. 9 In the embodiment of the present invention, by limiting the operation to a partial area of ​​the convolution kernel, the processor 110 can reduce the size of the partial output feature map, thereby reducing the size of the buffer used for the partial output feature map.

[0123] Return to reference Figure 3 , the processor 110 can continuously read the input feature map or its region in the form of a stream, and perform a convolution operation with the convolution kernel based on the read input feature map or its region. The processor 110 performs the convolution operation with the convolution kernel by reusing the input feature map or its region, and therefore does not need to read the input feature map or its region again after reading it for the first time, thereby continuously reading the input feature map or its region like a continuous stream.

[0124] The processor 110 may read the compressed input feature map and perform a convolution operation between the compressed input feature map and the convolution kernel. The input feature map and the compressed input feature map may be stored in the memory 120, and the processor 110 may access the memory 120 to read the compressed input feature map and perform the convolution operation. For example, the processor 110 may store the output feature map as a result of the convolution operation in the memory 120 as the input feature map of the next layer. The processor 110 may compress the input feature map or store the compressed input feature map in the memory 120. The processor 110 may read the compressed input feature map from the memory 120 or perform a convolution operation based on the compressed input feature map.

[0125] Therefore, the processor 110 can perform the convolution operation by reading the compressed input feature map or a region thereof and the input feature map like a continuous stream, thereby improving the speed of the convolution operation.

[0126] Fig.10 An example is shown in which a processor performs a convolution operation by reading a compressed input feature map like a stream.

[0127] The memory 120 can store the compressed input feature map 1010 and the input feature map 1010. Figure 1 The compressed input feature map 1010 may be compressed in units of regions of the input feature map. For example, the compressed input feature map 1010 may be compressed in units of 4×4 regions. The processor 110 may perform a convolution operation by reading the compressed input feature map 1010 like a continuous stream.

[0128] The compressed input feature map 1010 may include non-zero pixels, so that the processor 110 can implement zero skipping by performing a convolution operation between the compressed input feature map 1010 and a convolution kernel, and thereby reduce memory bandwidth.

[0129] Fig.11 An example of the hardware configuration of the processor is shown.

[0130] The processor 110 may include operation units 1112 , 1114 , and 1116 and output units 1122 , 1124 , and 1126 .

[0131] Each of the operation units 1112, 1114, and 1116 can generate a partial output feature map by performing an operation between different regions among the regions IFM_1, IFM_2 to IFM_N and the convolution kernel. For example, the first operation unit 1112 can generate a first partial output feature map by performing an operation between the first region IFM_1 and the convolution kernel based on the reuse of the first region IFM_1 of the input feature map. In addition, the Nth operation unit 1116 can generate an Nth partial output feature map by performing an operation between the Nth region IFM_N and the convolution kernel based on the reuse of the Nth region IFM_N of the input feature map.

[0132] Each of the operation units 1112, 1114, and 1116 may include a scheduler, a processing unit, and a first buffer located at the front end (FrontEnd). The scheduler of the first operation unit 1112 may read the first region IFM_1 of the input feature map from the memory 120, and schedule the read first region to the processing unit. The processing unit may generate an output value by performing an operation between the first region IFM_1 and the convolution kernel. In the example, the processing unit includes various operators, such as a multiplier, an adder, and an accumulator. The processing unit may generate a first output value by performing an operation between the first region IFM_1 and the first convolution kernel, a second output value by performing an operation between the first region IFM_1 and the second convolution kernel, and an Nth output value by performing an operation between the first region IFM_1 and the Nth convolution kernel. Then, the first buffer 1113 may generate a first partial output feature map by accumulating the output values. For example, buffer 1 in the first buffer 1113 can generate a first “first partial output feature map” by accumulating the first output value generated by the processing unit, buffer 2 in the first buffer 1113 can generate a first “second partial output feature map” by accumulating the second output value generated by the processing unit, and buffer N in the first buffer 1113 can generate a first “Nth partial output feature map” by accumulating the Nth output value generated by the processing unit.

[0133] Similarly, other operating units 1114 and 1116 can generate the second partial output feature map to the Nth partial output feature map by performing operations between the convolution kernel and other areas IFM_2 to IFM_N of the input feature map based on the reuse of other areas IFM_2 to IFM_N of the input feature map through the scheduler, the processing unit and the first buffer.

[0134] The processing units included in each of the operation units 1112, 1114 and 1116 may include parallelized processing units. For example, the processing units of the first operation unit 1112 may include: a first processing unit that performs an operation between the first area IFM_1 of the input feature map and the first convolution kernel; and a second processing unit that performs an operation between the first area IFM_1 and the second convolution kernel. In this case, after completing the operation between the first area IFM_1 of the input feature map and the first convolution kernel, the first processing unit may perform a part of the operation between the first area IFM_1 of the input feature map and the second convolution kernel instead of the second processing unit. As a result, load balancing can be achieved and the total processing time can be reduced. Reference will now be made to Fig.15 To describe a detailed example.

[0135] The output units 1122, 1124, and 1126 may generate regions OFM_0, OFM_1 to OFM_N of the output feature maps by accumulating the required partial output feature maps among the partial output feature maps generated by the operation units 1112, 1114, and 1116. The output units 1122, 1124, and 1126 may generate regions OFM_0, OFM_1 to OFM_N of the output feature maps and output them to the memory 120.

[0136] Each of the output units 1122, 1124, and 1126 may include a second buffer and an output processor at the back end.

[0137] The second buffer of the first output unit 1122 may receive a partial output feature map from each of the operation units 1112, 1114, and 1116, and generate a first region OFM_1 of the output feature map by accumulating the received partial output features. For example, the second buffer of the first output unit 1122 may receive a first "first partial output feature map" from buffer 1 of the first operation unit 1112, a second "first partial output feature map" from buffer 1 of the second operation unit 1114, and an Nth "first partial output feature map" from buffer 1 of the Nth operation unit 1116. The second buffer of the first output unit 1122 may generate a first region OFM_1 of the output feature map by accumulating the received first "first partial output feature map" to the Nth "first partial output feature map". The output processor of the first output unit 1122 may perform pixel processing on the first region OFM_1 of the output feature map, and output the first region OFM_1 of the pixel-processed output feature map.

[0138] Similarly, other output units 1124 and 1126 can receive the required partial output feature map from each of the operating units 1112, 1114 and 1116 through the second buffer and the output processor, and generate the second region OFM_2 to the Nth region OFM_N of the output feature map by accumulating the received partial output feature maps.

[0139] Each of the operation units 1112, 1114, and 1116 can perform operations with the convolution kernel by reusing different regions of the input feature map, and thus the operation units 1112, 1114, and 1116 can respectively perform independent and parallel operations. In each of the operation units 1112, 1114, and 1116, the scheduler can only schedule a region of the input feature map to a processing unit on the same operation unit, and not schedule the region of the input feature map to a processing unit on another operation unit, thereby reducing the complexity at the front end of the FrontEnd of the processor 110.

[0140] refer to Fig.11 , in the example, the operation units 1112, 1114, and 1116 may be fully connected to the output units 1122, 1124, and 1126. Therefore, it may be erroneously presented that the complexity of the processor 110 at the backend (BackEnd) increases, and the complexity of the processor 110 at the frontend FrontEnd decreases. The output units 1122, 1124, and 1126 perform an operation of selectively accumulating a portion of the feature maps from each of the operation units 1112, 1114, and 1116. Therefore, the output units 1122, 1124, and 1126 may perform the operation more sparsely in time than the operation units 1112, 1114, and 1116 without significantly increasing the complexity.

[0141] Fig.12 An example of the hardware configuration of the processor is shown.

[0142] The processor 110 may include operation units 1212, 1214, and 1216 and output units 1222, 1224, and 1226. Fig.12 Description, Fig.12 The operation units 1212, 1214 and 1216 and the output units 1222, 1224 and 1226 can be connected with Fig.11 The operation units 1112, 1114 and 1116 and the output units 1122, 1124 and 1126 correspond to each other, so that the repeated contents are incorporated into this document by reference. Therefore, the above description may not be repeated here.

[0143] refer to Fig.12, the operation units 1212 , 1214 and 1216 can be connected to the output units 1222 , 1224 and 1226 via the bus 1210 .

[0144] The output units 1222 , 1224 , and 1226 may perform an operation of selectively accumulating partial output feature maps from each of the operation units 1212 , 1214 , and 1216 , and may thus receive partial output feature maps from the operation units 1212 , 1214 , and 1216 via the bus 1210 .

[0145] Therefore, the processor 110 can implement the sending / receiving path of part of the output feature map between the operation units 1212, 1214 and 1216 and the output units 1222, 1224 and 1226 through the bus 1210, instead of implementing a full connection between the two, thereby reducing hardware overhead.

[0146] Fig.13 An example of the hardware configuration of the processor is shown.

[0147] The processor 110 may include operation units 1312, 1314, and 1316. Each of the operation units 1312, 1314, and 1316 may generate a partial output feature map by performing an operation between different regions among the regions of the input feature map and the convolution kernel. Each of the operation units 1312, 1314, and 1316 may include a scheduler, a processing unit, and a buffer. For example, the scheduler of the first operation unit 1312 may read the first region of the input feature map from the memory 120, and schedule the read first region to the processing unit. The processing unit may generate an output value by performing an operation between the first region and the convolution kernel, and the buffer may generate a first partial output feature map by accumulating the output value.

[0148] Each of the operation units 1312, 1314, and 1316 can generate each of the multiple regions of the output feature map by accumulating the required partial output feature map from another operation unit. In an example, the buffers of adjacent operation units can be interconnected, and the buffer of each of the operation units 1312, 1314, and 1316 can receive the required partial output feature map from the buffer of a different operation unit. For example, when the buffer of the first operation unit 1312 needs the partial output feature map output from the Nth operation unit 1316, the first operation unit 1312 can receive the partial output feature map output from the Nth operation unit 1316 through the buffer of the second operation unit 1314.

[0149] Fig.14 An example is shown in which a processing unit of a processor performs an operation between a convolution kernel and a region of an input feature map.

[0150] The processor 110 may include a first operation unit 1412, a second operation unit 1414, a third operation unit 1416, and a fourth operation unit 1418. The processor 110 may include a first output unit 1422, a second output unit 1424, a third output unit 1426, and a fourth output unit 1428. The processor 110 may also include a bus 1430.

[0151] The first operation unit 1412 can generate a first partial output feature map by performing an operation between the first area IFMO and the convolution kernel (Kernel) of the input feature map. In the example, the first operation unit 1412 can generate a first "first partial output feature map" by performing an operation between the first area IFMO and the first convolution kernel via the first processing unit, generate a first "second partial output feature map" by performing an operation between the first area IFMO and the second convolution kernel via the second processing unit, generate a first "third partial output feature map" by performing an operation between the first area IFMO and the third convolution kernel via the third processing unit, and generate a first "fourth partial output feature map" by performing an operation between the first area IFMO and the fourth convolution kernel via the fourth processing unit.

[0152] Similarly, the second operation unit 1414, the third operation unit 1416 and the fourth operation unit 1418 can generate the second "first partial output feature map" to the second "fourth partial output feature map", the third "first partial output feature map" to the third "fourth partial output feature map", and the fourth "first partial output feature map" to the fourth "fourth partial output feature map" by respectively performing operations between the second area IFM1, the third area IFM2 and the fourth area IFM3 of the input feature map and the convolution kernel through four processing units.

[0153] The first output unit 1422 can receive the required partial output feature map from the operation units 1412, 1414, 1416 and 1418 through the bus 1430. For example, the first output unit 1422 can receive the first "first partial output feature map", the second "first partial output feature map", the third "first partial output feature map" and the fourth "first partial output feature map" through the bus 1430, and generate the first region OFMO of the output feature map by accumulating the first "first partial output feature map", the second "first partial output feature map", the third "first partial output feature map" and the fourth "first partial output feature map".

[0154] Similarly, the second output unit 1424, the third output unit 1426 and the fourth output unit 1428 may receive partial output feature maps through the bus 1430, and generate a second region OFM1, a third region OFM2 and a fourth region OFM3, respectively.

[0155] Fig.15 An example is shown in which a processing unit of a processor performs an operation between a convolution kernel and a region of an input feature map.

[0156] When each of the operation units 1412 to 1418 performs an operation between a region of the input feature map and the convolution kernel "Kernel", the operation time of each of the processing units in the operation units 1412 to 1418 may be different. In the example, as can be seen in 1510, the operation times of the first processing unit to the fourth processing unit of the first operation unit 1412 may be different from each other. The time required for the first processing unit to perform the operation between the first region IFMO of the input feature map and the first convolution kernel may be shorter than the time required for the second processing unit to perform the operation between the first region IFMO and the second convolution kernel, and the time required for the fourth processing unit to perform the operation between the first region IFMO and the fourth convolution kernel may be the longest. As a result, the total processing time may increase.

[0157] Therefore, when performing the operation between the region of the input feature map and the convolution kernel, in the example, each of the operation units 1412 to 1418 can control the processing unit that completes the operation earlier to perform the operation on behalf of another processing unit to achieve load balancing. In the example, the third processing unit of the first operation unit 1412 can generate the first "third partial output feature map" through the operation between the first region IFMO and the third convolution kernel, and then perform a part of the operation between the first region IFM0 and the fourth convolution kernel on behalf of the fourth processing unit. As a result, as shown in 1530, the total processing time can be reduced.

[0158] In addition, even when a processing unit in the operation unit performs an operation on behalf of another processing unit, the output operation unit can selectively obtain a required portion of the output feature map, so that the output operation unit can generate an area of ​​the output feature map regardless of the load balance in the operation unit.

[0159] Fig.16 is a diagram showing an example of an operation method of a neural network according to an embodiment. Fig.16 The operations in the diagram may be performed in the order and manner shown; however, the order of some operations may be changed or some operations may be omitted without departing from the spirit and scope of the illustrative examples described. Fig.16 Many of the operations shown can be performed in parallel or concurrently. Fig.16One or more blocks and combinations of these blocks may be implemented by a computer (eg, a processor) based on special-purpose hardware that performs the specified functions, or a combination of special-purpose hardware and computer instructions.

[0160] Fig.16 The operation method shown in Figures 3 to 15 The present invention is performed by the elements of the neural network device 100 shown in , and therefore repeated descriptions will be omitted. In the example, Fig.16 The operation can be performed by a neural network device ( Figure 3 100) in the execution. Except for the following Fig.16 In addition to the description of Figures 1 to 15 The description also applies to Fig.16 , and is incorporated herein by reference. Therefore, the above description may not be repeated here.

[0161] In operation 1610, neural network device 100 may generate an output value by performing an operation between the weight of the convolution kernel and the input feature map. Neural network device 100 may generate a first output value by performing an operation between the input feature map and a first weight of the convolution kernel. Neural network device 100 may generate a second output value by performing an operation between the input feature map and a second weight of the convolution kernel.

[0162] Neural network device 100 may generate a first output value by performing an operation between a first region of the input feature map and each weight of the convolution kernel. Neural network device 100 may generate a second output value by performing an operation between each weight in the convolution kernel and a second region of the input feature map that is different from the first region.

[0163] When the first weight of the convolution kernel is zero, the neural network device 100 can skip the operation between the input feature map and the first weight.

[0164] The neural network device 100 can perform operations between each weight of the convolution kernel and the compressed input feature map by continuously streaming the compressed input feature map.

[0165] In operation 1620, neural network device 100 may generate an output feature map by accumulating output values ​​at positions in the output feature map that are set based on the positions of weights in the convolution kernel. In an example, neural network device 100 may accumulate a first output value at a first position in the output feature map that is configured based on the position of a first weight in the convolution kernel. Neural network device 100 may accumulate a second output value at a second position in the output feature map that is configured based on the position of a second weight in the convolution kernel.

[0166] Neural network device 100 may generate a first partial output feature map by accumulating first output values ​​at positions in the first partial output feature map that are set based on positions of weights in the convolution kernel, and accumulate the first partial output feature map on the output feature map. Neural network device 100 may generate a second partial output feature map by accumulating second output values ​​at positions in the second partial output feature map that are set based on positions of weights in the convolution kernel, and accumulate the second partial output feature map on the output feature map.

[0167] Neural network device 100 may generate a partial output feature map by performing an operation between each region in the region of the input feature map and a convolution kernel. Neural network device 100 may generate each region in the region of the output feature map by accumulating required partial output feature maps among the partial output feature maps. Neural network device 100 may generate a partial output feature map by performing an operation between a region in the region and each convolution kernel among a plurality of convolution kernels.

[0168] Neural network device 100, processor 110, multiplier (MUL), multiplexer (MUX), adder (Adder), accumulator & register (Acc.Register), operation units 1112, 1114 and 1116, output units 1122, 1124 and 1126, operation units 1212, 1214 and 1216, output units 1222, 1224 and 1226, operation units 1312, 1314 and 1316, first operation unit 1412, second operation unit 1414, third operation unit 1416, fourth operation unit 1418, first output unit 1422, second output unit 1424, third output unit 1426, fourth output unit 1428, scheduler, processing unit, scheduler (FrontEnd), output processor (BackEnd), and other devices, units, modules, devices and other components described herein are implemented by hardware components. Examples of hardware components that can be used to perform the operations described in this application include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application, where appropriate. In other examples, one or more hardware components that perform the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer may be implemented by one or more processing elements (e.g., logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond and execute instructions in a defined manner to achieve desired results. In one example, a processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. Hardware components implemented by a processor or computer may execute instructions or software, such as an operating system (OS) and one or more software applications running on the OS, to perform the operations described in this application. The hardware components may also access, manipulate, process, create and store data in response to the execution of instructions or software. For simplicity, the singular term "processor" or "computer" may be used in the description of the examples described in this application, but in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller.One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors or another processor and another controller. One or more processors or a processor and a controller may implement a single hardware component, or two or more hardware components. A hardware component may have any one or more of different processing configurations, examples of which include a single processor, an independent processor, a parallel processor, a single instruction single data (SISD) multiprocessing, a single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0169] Perform the operations described in this application Figures 1 to 11 The method shown is performed by computing hardware, for example, by one or more processors or computers implemented as described above, executing instructions or software to perform operations described in the present application (operations implemented by the method). For example, a single operation or two or more operations can be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations can be performed by one or more processors or a processor and a controller, and one or more other operations can be performed by one or more other processors or another processor and another controller. One or more processors or a processor and a controller can perform a single operation or two or more operations.

[0170] The instructions or software for controlling the processor or computer to implement the hardware components and perform the method as described above are written as computer programs, code segments, instructions or any combination thereof, for individually or collectively instructing or configuring the processor or computer to operate as a machine or a special-purpose computer to perform the operations performed by the hardware components and the above-mentioned methods. In an example, the instructions or software include at least one of the following: an applet, a dynamic link library (DLL), middleware, firmware, a device driver, an application program for storing the method of output status information. In one example, the instructions or software include machine code directly executed by the processor or computer, such as machine code generated by a compiler. In another example, the instructions or software include a higher-level code executed by the processor or computer using an interpreter. An ordinary programmer in the art can easily write instructions or software based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions in the specification, wherein an algorithm for performing the operations performed by the hardware components and methods as described above is disclosed.

[0171] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform methods as described above, and any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), flash memory, card type storage (e.g., multimedia card, secure digital (SD) card, or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, and any other device configured to store instructions or software in a non-temporary manner and as described below, any other device configured to store instructions or software in a non-temporary manner, and any associated data, data files and data structures, and provide instructions or software and associated data, data files and data structures to a processor or computer so that the processor or computer can execute the instructions. In one example, the instructions or software and any associated data, data files and data structures are distributed on a networked computer system so that one or more processors or computers store, access and execute the instructions and software and any associated data, data files and data structures in a distributed manner.

[0172] Although the present disclosure includes specific examples, it will be apparent after understanding the disclosure of the present application that various changes in form and detail may be made to these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are considered to be illustrative only and not for limiting purposes. The description of features or aspects in each example is considered to be applicable to similar features or aspects in other examples. If the described techniques are performed in a different order and / or if the components in the described systems, architectures, devices, or circuits are combined in different ways and / or replaced or supplemented by other components or their equivalents, suitable results may be achieved. Therefore, the scope of the present disclosure is not limited by specific embodiments, but by the claims and their equivalents, and all changes within the scope of the claims and their equivalents are interpreted as being included in the present disclosure.

Claims

1. A neural network device for speech recognition, image recognition or image classification, comprising: The processor is configured as: By each operation between different weights of the convolution kernel and the same pixel of the input feature map, for each of the plurality of weights of the convolution kernel, generating an output value of the corresponding operation of the processor for the input feature map; as well as For each of the one or more corresponding pixels of the input feature map, generating the output feature map by accumulating the output values ​​at positions in the output feature map that are set based on the positions of the plurality of weights in the convolution kernel, Wherein, for generating the output value, the processor is further configured to: generating a first output value by performing a first operation between the input feature map and a first weight of the convolution kernel; and generating a second output value by performing a second operation between the input feature map and a second weight of the convolution kernel, and Wherein, for generating the output feature map, the processor is further configured to: Accumulating the first output value at a first position in the output feature map based on the position of the first weight in the convolution kernel; and Accumulating the second output value at a second position in the output feature map based on the position of the second weight in the convolution kernel, Wherein, in the convolution kernel, the first weight is adjacent to the second weight.

2. The neural network device according to claim 1, wherein: The processor is further configured to: generating a first output value by performing a corresponding first operation between a first region of the input feature map and each of the plurality of weights of the convolution kernel; generating the first partial output feature map by accumulating the first output values ​​at positions in the first partial output feature map based on positions of corresponding weights in the convolution kernel; as well as The first part of the output feature map is accumulated on the output feature map.

3. The neural network device according to claim 2, wherein: The processor is further configured to: generating a second output value by performing a corresponding second operation between a second region in the input feature map and each of the plurality of weights of the convolution kernel, the second region being different from the first region; generating the second partial output feature map by accumulating the second output values ​​at positions in the second partial output feature map based on positions of corresponding weights in the convolution kernel; as well as The second part of the output feature map is accumulated on the output feature map.

4. The neural network device according to claim 2, wherein: The first area includes at least one of n pixels, n×m pixels, or n×m×l pixels in the input feature map, where n, m, and l are natural numbers greater than or equal to 1.

5. The neural network device according to claim 1, wherein: The processor is also configured to: in response to the first weight among the multiple weights of the convolution kernel being zero, skip an operation between the input feature map and the first weight among the multiple weights.

6. The neural network device according to claim 1, wherein: The processor is also configured to perform an operation between each of the plurality of weights of the convolution kernel and the compressed input feature map by continuously streaming the compressed input feature map from a memory.

7. The neural network device according to claim 1, wherein: The processor further comprises: an operation unit configured to generate a partial output feature map by respectively performing an operation between each of the plurality of regions of the input feature map and the convolution kernel; and An output unit is configured to generate each of the multiple regions of the output feature map by accumulating at least one partial output feature map among the partial output feature maps.

8. The neural network device according to claim 7, wherein: One or more of the computing units of the processor are further configured to respectively perform operations between the convolution kernel and each of the multiple regions of the input feature map independently and in parallel with each other.

9. The neural network device according to claim 7, further comprising a bus, in, The output unit is further configured to receive the one or more partial output feature maps from the operation unit of the processor through the bus.

10. The neural network device according to claim 7, wherein: Each of the computing units of the processor includes multiple processing units, and the processing units are configured to generate the partial output feature map by respectively performing operations between the regions of the multiple regions of the input feature map and each of the multiple convolution kernels.

11. The neural network device according to claim 10, wherein: The processing unit includes: a first processing unit configured to perform an operation between the region and a first convolution kernel weight among a plurality of weights of the first convolution kernel; and a second processing unit configured to perform an operation between the region and a second convolution kernel weight among a plurality of weights of the second convolution kernel, and The first processing unit is also configured to: after completing the operation between the region and the first convolution kernel weight of the first convolution kernel, perform a part of the operation between the region and the second convolution kernel weight of the second convolution kernel.

12. The neural network device of claim 1, further comprising a memory storing instructions that, when executed, configure the processor to generate the output value and to generate the output feature map.

13. A method for processing convolution operations in a neural network for speech recognition, image recognition or image classification, the method comprising: By each operation between different weights of the convolution kernel and the same pixel of the input feature map, for the input feature map, for each of the multiple weights of the convolution kernel, an output value of the corresponding operation is generated; as well as For each of the one or more corresponding pixels of the input feature map, generating the output feature map by accumulating the output values ​​at positions in the output feature map that are set based on the positions of the plurality of weights in the convolution kernel, Wherein, generating the output value comprises: generating a first output value by performing a first operation between the input feature map and a first weight of the convolution kernel; and generating a second output value by performing a second operation between the input feature map and a second weight of the convolution kernel, and Wherein, generating the output feature map comprises: Accumulating the first output value at a first position in the output feature map based on the position of the first weight in the convolution kernel; and Accumulating the second output value at a second position in the output feature map based on the position of the second weight in the convolution kernel, Wherein, in the convolution kernel, the first weight is adjacent to the second weight.

14. The method according to claim 13, wherein: Generated output values ​​also include: generating a first output value by performing a corresponding first operation between a first region in the input feature map and each of the plurality of weights of the convolution kernel; and generating a second output value by performing a corresponding second operation between a second region in the input feature map and each of the plurality of weights of the convolution kernel, the second region being different from the first region; and Generating the output feature map further includes: generating the first partial output feature map by accumulating the first output values ​​at positions in the first partial output feature map based on positions of corresponding weights in the convolution kernel, and accumulating the first partial output feature map on the output feature map; and The second partial output feature map is generated by accumulating the second output values ​​at positions in the second partial output feature map based on positions of corresponding weights in the convolution kernel, and accumulating the second partial output feature map on the output feature map.

15. The method according to claim 13, wherein: Generating an output value also includes: in response to the first weight among the multiple weights of the convolution kernel being zero, skipping the operation between the input feature map and the first weight among the multiple weights.

16. The method according to claim 13, wherein: Generating an output value also includes: performing an operation between each of the multiple weights of the convolution kernel and the compressed input feature map by continuously streaming the compressed input feature map.

17. The method according to claim 13, wherein: Generated output values ​​also include: Generate a partial output feature map by respectively performing an operation between each of the plurality of regions of the input feature map and the convolution kernel; and Each of the multiple regions of the output feature map is generated by accumulating at least one partial output feature map among the partial output feature maps.

18. The method according to claim 17, wherein: Generating a partial output feature map includes: generating the partial output feature map by respectively performing operations between an area among the multiple areas of the input feature map and each of the multiple convolution kernels.

19. A non-transitory computer-readable storage medium storing instructions which, when executed by a processor, cause the processor to perform the method according to claim 13.

20. A neural network device for speech recognition, image recognition or image classification, comprising: A memory configured to store a plurality of weights and instructions of a convolution kernel; as well as A processor configured to execute the instructions so that the processor: generating a first output value for the first region of the input feature map by performing a corresponding operation of the processor for each of the plurality of weights through each operation between a different weight and the first region of the input feature map; Cause the first output value to be located at a position in the first partial output feature map based on corresponding positions of the plurality of weights in the convolution kernel; By each operation between a different weight and the second region of the input feature map, for the second region of the input feature map, for each weight in the plurality of weights, generating a second output value by performing a corresponding other operation of the processor; as well as The second output value is located at a position in the second partial output feature map based on corresponding positions of multiple weights in the convolution kernel.

21. The neural network device of claim 20, wherein: The processor is further configured to generate the second output value by performing an operation between the plurality of weights and a portion of the second region different from the first region.

22. The neural network device of claim 20, wherein: The processor is also configured to: in response to a first weight among the multiple weights being zero, skip an operation between the first region or the second region of the input feature map and the first weight among the multiple weights.

Images