Method and apparatus for processing convolutional neural networks

Abstract

A neural network device includes: one or more processors, including: a controller configured to determine, based on either or both of features of an input feature map and features of a convolution kernel, a shared operand to be shared in a parallel operation as any one of pixel values ​​among pixel values ​​of the input feature map and weight values ​​among weight values ​​of the convolution kernel; and one or more processing units configured to perform the parallel operation based on the determined shared operand.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 799,190, filed in the U.S. Patent and Trademark Office on January 31, 2019, and Korean Patent Application No. 10-2019-0038606, filed in the Korean Intellectual Property Office on April 2, 2019, the disclosures of which are incorporated herein by reference in their entireties. Technical Field

[0003] The present disclosure relates to methods and apparatus having convolutional neural network processing. Background Art

[0004] Neural networks refer to a computing architecture that some electronic systems can use to analyze input data and extract useful information.

[0005] Neural network devices can perform a large number of operations on input data. Summary of the Invention

[0006] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0007] In one general embodiment, a neural network device includes: one or more processors, including: a controller configured to determine, based on either or both of features of an input feature map and features of a convolution kernel, a shared operand to be shared in a parallel operation as any one of pixel values ​​among pixel values ​​of the input feature map and weight values ​​among weight values ​​of the convolution kernel; and one or more processing units configured to perform the parallel operation based on the determined shared operand.

[0008] The one or more processing units may be configured to, in response to the controller determining the shared operand to be the pixel value of the input feature map, perform parallel operations in parallel between the pixel values ​​of the input feature map and the weight values ​​of the convolution kernel.

[0009] The controller can be configured to: determine a shared operand as a first pixel value of an input feature map, and determine another shared operand as a second pixel value of the input feature map, and the one or more processing units may include: one or more first processing units configured to perform parallel operations in parallel between the first pixel value of the input feature map and the weight value of the convolution kernel; and one or more second processing units configured to perform parallel operations in parallel between the second pixel value of the input feature map and the weight value of the convolution kernel.

[0010] The one or more processing units may be configured to, in response to the controller determining the shared operand as the weight value of the convolution kernel, perform parallel operations in parallel between the weight value of the convolution kernel and the pixel value of the input feature map.

[0011] The controller can be configured to: determine a shared operand as a first weight value of the convolution kernel, and determine another shared operand as a second weight value of the convolution kernel, and the one or more processing units may include: one or more first processing units, configured to perform parallel operations in parallel between the first weight value of the convolution kernel and the pixel value of the input feature map; and one or more second processing units, configured to perform parallel operations in parallel between the second weight value of the convolution kernel and the pixel value of the input feature map.

[0012] The shared operand, the pixel value of the input feature map and the weight value of the convolution kernel all involve the first layer of the neural network, and the controller can also be configured to: for the second layer of the neural network, based on any one or both of the features of the input feature map of the second layer and the features of the convolution kernel of the second layer, determine the shared operand of the second layer as any one of the pixel values ​​of the input feature map of the second layer and the weight value of the convolution kernel of the second layer.

[0013] The characteristics of the input feature map may include either or both of the percentage of pixels having zero values ​​within the input feature map and the shape of the input feature map, and the characteristics of the convolution kernel may include the percentage of weights having zero values ​​within the convolution kernel.

[0014] The controller may be configured to determine the shared operand as the weight value of the convolution kernel in response to a percentage of weights having zero values ​​in the convolution kernel being greater than a set value.

[0015] The number of the one or more processing units that jointly share the shared operand may be equal to any one of the number of pixels of the input channels of the input feature map and the number of output channels of the convolution kernel.

[0016] The one or more processors may be configured to perform a 3D convolution operation between an input feature map having a three-dimensional (3D) size and a convolution kernel having a four-dimensional (4D) size.

[0017] One or more processing units may be configured to skip parallel operations having a shared operand of 0 among the parallel operations.

[0018] In response to the controller determining the shared operand as a pixel value of the input feature map, one or more processing units can be configured to: in response to a first pixel value among the pixel values ​​of the input feature map being 0, skip the parallel operation between the first pixel value and the first weight value among the weight values ​​of the convolution kernel among the parallel operations, and perform the parallel operation between the second pixel value among the pixel values ​​of the input feature map and the second weight value among the weight values ​​of the convolution kernel among the parallel operations.

[0019] In response to the controller determining that the shared operand is the weight value of the convolution kernel, one or more processing units can be configured to: in response to a first weight value among the weight values ​​of the convolution kernel being 0, skip the parallel operation between the first weight value and the first pixel value among the pixel values ​​of the input feature map among the parallel operations, and perform the parallel operation between the second weight value among the weight values ​​of the convolution kernel and the second pixel value among the pixel values ​​of the input feature map among the parallel operations.

[0020] The apparatus may further include a memory storing instructions that, when executed by one or more processors, configure the one or more processors to perform the following operations: obtain a convolution kernel bit-serial block; generate a feature map bit-serial block; and generate at least a portion of an output feature map.

[0021] In another general aspect, a processor-implemented neural network method includes: determining a shared operand to be shared in parallel operations as any one of pixel values ​​among pixel values ​​of the input feature map and weight values ​​among weight values ​​of the convolution kernel based on any one or both of features of the input feature map and features of the convolution kernel; and performing the parallel operations based on the determined shared operand.

[0022] The determining may include determining the shared operand as a pixel value of the input feature map, and the performing may include performing a parallel operation in parallel between the pixel value of the input feature map and a weight value of the convolution kernel.

[0023] The determining may include determining the shared operand as a weight value of a convolution kernel, and the performing may include performing a parallel operation in parallel between the weight value of the convolution kernel and the pixel value of the input feature map.

[0024] The executing may include skipping a parallel operation having a shared operand of 0 among the parallel operations.

[0025] The determining may include determining the shared operand as a pixel value of the input feature map, and the executing includes: in response to a first pixel value among the pixel values ​​of the input feature map being 0, skipping a parallel operation between the first pixel value and a first weight value among the weight values ​​of the convolution kernel among the parallel operations, and executing a parallel operation between a second pixel value among the pixel values ​​of the input feature map and a second weight value among the weight values ​​of the convolution kernel among the parallel operations.

[0026] The determining may include determining the shared operand as a weight value of the convolution kernel, and the executing may include: when a first weight value among the weight values ​​of the convolution kernel is 0, skipping a parallel operation between the first weight value and a first pixel value among the pixel values ​​of the input feature map among the parallel operations, and executing a parallel operation between a second weight value among the weight values ​​of the convolution kernel and a second pixel value among the pixel values ​​of the input feature map among the parallel operations.

[0027] A non-transitory computer-readable storage medium may store instructions that, when executed by one or more processors, cause the one or more processors to perform the method.

[0028] In another general aspect, a processor-implemented neural network method includes: obtaining an input feature map and a convolution kernel; determining a shared operand as any one of a pixel value among pixel values ​​of the input feature map and a weight value among weight values ​​of the convolution kernel; and determining a pixel value of an output feature map by performing an operation between any one of each weight value and each pixel value of the convolution kernel and the shared operand.

[0029] The pixel value of the input feature map may correspond to the first channel among the channels of the input feature map, the weight value may correspond to the first channel among the channels of the convolution kernel, and each pixel value of the output feature map may correspond to the same position in the corresponding channel among the channels of the output feature map.

[0030] Determining the shared operand may include: in response to the two-dimensional area size of the input feature map being less than or equal to a set value, determining the shared operand as a pixel value of the input feature map; and in response to the two-dimensional area size of the input feature map being greater than or equal to another set value, determining the shared operand as a weight value of the convolution kernel. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] These and / or other aspects will become more apparent and easier to understand through the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0032] Figure 1 The architecture of a neural network in accordance with one or more embodiments is shown.

[0033] Figure 2Aand Figure 2B Figure 2 shows the convolution operation in a neural network.

[0034] Figure 3 FIG. 4 shows a hardware structure of a neural network device according to one or more embodiments.

[0035] Figure 4 1 shows a controller and hardware processing units included in a neural network device according to one or more embodiments.

[0036] Figure 5A and Figure 5B A hardware processing unit configured to perform parallel operations based on shared operands is shown in accordance with one or more embodiments.

[0037] Figure 6 A process for a controller to determine a shared operand based on the shape of an input feature map is shown in accordance with one or more embodiments.

[0038] Figure 7 A hardware processing unit configured to perform parallel operations based on pixel values ​​of an input feature map as a shared operand of the processing unit is shown in accordance with one or more embodiments.

[0039] Figure 8 A hardware processing unit configured to perform parallel operations based on pixel values ​​of an input feature map as a shared operand of the processing unit is shown in accordance with one or more embodiments.

[0040] Figure 9 A hardware processing unit configured to perform parallel operations based on weight values ​​of convolution kernels as shared operands is shown in accordance with one or more embodiments.

[0041] Figure 10 A hardware processing unit configured to perform parallel operations based on weight values ​​of convolution kernels as shared operands is shown in accordance with one or more embodiments.

[0042] Figure 11 A hardware processing unit is shown performing zero skipping based on a shared operand being zero, according to one or more embodiments.

[0043] Figure 12 A process for a hardware processing unit to perform zero skipping based on a shared operand being zero is shown in accordance with one or more embodiments.

[0044] Figure 13 Methods of operating a neural network device according to one or more embodiments are shown. DETAILED DESCRIPTION

[0045] The following detailed description is provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be apparent. For example, the order of operations described herein is merely an example and is not limited to those orders set forth herein, but can be changed to an order that will be apparent after understanding the disclosure of the present application, except that the operations must occur in a specific order. In addition, for greater clarity and brevity, descriptions of features known in the art may be omitted.

[0046] In the description of the embodiments, when a component is connected to another component, it includes not only the case where the component is directly connected, but also the case where the component is electrically connected through another component therebetween. In addition, when a component includes a constituent element, unless otherwise specifically stated, it means that the embodiment may include other elements rather than excluding other elements.

[0047] The terms used herein are only used to describe various examples and are not intended to limit the present disclosure. Unless the context clearly indicates otherwise, the articles "a", "an" and "the" are also intended to include plural forms. The terms "include", "comprising" and "having" indicate the presence of the recited features, numbers, operations, components, elements and / or combinations thereof, but do not preclude the presence or addition of one or more other features, numbers, operations, components, elements and / or combinations thereof.

[0048] Although terms such as "first," "second," and "third" may be used herein to describe various components, assemblies, regions, layers, or sections, these components, assemblies, regions, layers, or sections are not limited by these terms. Instead, these terms are used only to distinguish one component, component, region, layer, or section from another component, component, region, layer, or section. Thus, a first component, component, region, layer, or section mentioned in the examples described herein may also be referred to as a second component, component, region, layer, or section without departing from the teachings of the examples.

[0049] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art after understanding the disclosure of this application. Terms such as those defined in commonly used dictionaries should be interpreted as having the same meaning as in the context of the relevant technology and / or the disclosure of this application, and should not be interpreted as ideal or overly formal meanings unless explicitly defined as such herein.

[0050] With reference now to embodiment in detail, the example of embodiment is shown in the accompanying drawings, wherein similar reference numerals throughout the drawings refer to similar elements. In this regard, one or more embodiments may have different forms and should not be construed as being limited to the description set forth herein. Therefore, the following embodiments are described only with reference to the accompanying drawings to illustrate the various aspects of this description. As used herein, the term "and / or" includes any and all combinations of one or more items in the associated listed items. Statements such as "at least one of ... " modify the entire element list when preceding an element list, rather than modifying the individual elements in the list.

[0051] The following embodiments relate to methods and apparatus having convolutional neural network processing.

[0052] Figure 1 The architecture of a neural network in accordance with one or more embodiments is shown.

[0053] refer to Figure 1 , the neural network 1 can be a deep neural network (DNN) or an n-layer neural network architecture. DNN or n-layer neural network can correspond to convolutional neural network (CNN), recurrent neural network (RNN), deep belief network, restricted Boltzmann machine. For example, the neural network 1 can be embodied as CNN, but is not limited thereto. Figure 1 In FIG, some convolutional layers of a CNN corresponding to the example of neural network 1 are shown, but in addition to the convolutional layers shown, the CNN may also include pooling layers or fully connected layers.

[0054] Therefore, as shown in the figure, the neural network 1 can be embodied as an architecture having multiple layers including an input image layer, a feature map generation layer, and an output layer. In the neural network 1, a convolution operation is performed on the input image using a filter called a convolution kernel, and as a result, a feature map (or activation map or convolution feature) can be output. The output feature map is used as the input feature map and the convolution kernel is again convolved to it, and a new feature map is output. When the convolution operation is repeated in this way, the neural network 1 can ultimately output a recognition result related to the features of the input image.

[0055] For example, when Figure 1When the neural network 1 inputs an input image having a size of 24×24 pixels, the input image can be output as a feature map having four channels by performing a convolution operation with a convolution kernel, and the feature map of each channel has a size of 20×20 pixels. Then, the size of the 20×20 feature map can be reduced by repeatedly performing a convolution operation with the convolution kernel, and finally, features each having a size of 1×1 pixels can be output. In the neural network 1, convolution operations and subsampling (or pooling) operations can be repeatedly performed in several layers to filter and output robust features that can represent the entire input image from the input image, and the recognition result of the input image is derived by the final features output. Based on the above, it should be understood that the example embodiments discussed herein include embodiments having one or more such layers, levels, edges, etc. or all such layers, levels, edges, etc. in various combinations in the operations of the neural network 1.

[0056] Figure 2A and Figure 2B Figure 2 shows the convolution operation in a neural network.

[0057] In one or more embodiments, the input feature map 210 may have a size of 6×6 pixels, the convolution kernel 220 may have a size of 3×3 pixels, and the output feature map 230 may have a size of 4×4 pixels (e.g., Figure 2A ), but the size is not limited thereto, and the neural network may include feature maps and convolution kernels of various sizes. The values ​​defined in the input feature map 210, the convolution kernel 220, and the output feature map 230 are all example values, and the embodiment is not limited thereto. Each pixel of the input feature map may include a pixel value.

[0058] The convolution kernel 220 can perform a convolution operation in units of an area (or patch) having a size of 3×3 pixels while sliding on the input feature map 210. The convolution operation may include the following operation: each pixel value of the output feature map 230 is obtained by adding the values ​​obtained by multiplying each pixel value of any area of ​​the input feature map 210 by the weight of each element at the corresponding position in the convolution kernel 220. For example, the convolution kernel 220 first performs a convolution operation on the first area 211 of the input feature map 210. For example, the pixel values ​​1, 2, 3, 4, 5, 6, 7, 8, and 9 of the first area 211 can be multiplied by the weights of the elements of the convolution kernel 220, namely, -1, -3, +4, +7, -2, -1, -5, +3, and +1, respectively, and the resulting values ​​can be -1, -6, 12, 28, -10, -6, -35, 24, and 9. Next, the obtained values ​​-1, -6, 12, 28, -10, -6, -35, 24, and 9 may be added together to obtain a value of 15, and a pixel value 231 on the first row and first column of the output feature map 230 may be determined to be 15. The pixel value 231 on the first row and first column of the output feature map 230 corresponds to the first region 211. Similarly, a convolution operation may be performed between the second region 212 of the input feature map 210 and the convolution kernel 220, and thus a pixel value 232 of the first row and second column of the output feature map 230 may be determined to be 4. Convolution operations may be performed between the first to sixteenth regions of the input feature map and the convolution kernel 220, respectively, so that a pixel value for each position of the output feature map 230 is determined. For example, a convolution operation may be performed between the sixteenth region 213 (e.g., the last window of the input feature map 210) and the convolution kernel 220, and thus a pixel value 233 of the fourth row and fourth column of the output feature map 230 may be determined to be 11. That is, the convolution operation between the input feature map 210 and the convolution kernel 220 can be performed by repeatedly processing the multiplication of the values ​​of the corresponding elements in the input feature map 210 and the convolution kernel 220 and the sum of the multiplication results, and the output feature map 230 can be generated as the result of the convolution operation.

[0059] Already referenced Figure 2A A two-dimensional (2D) convolution operation is described. However, in other examples, a three-dimensional (3D) convolution operation can be performed, where there are input feature maps, convolution kernels, and output feature maps with multiple channels, as will be referred to in the example of FIG. Figure 2B As described in the example.

[0060] refer to Figure 2B, the input feature map 201 may have a 3D size, there may be X input channels in the input feature map 201, and the 2D input feature map for each input channel may have a size of H rows and W columns, where X, W, and H are each natural numbers. The convolution kernel 202 may have a four-dimensional (4D) size, and each 2D convolution kernel having a size of R rows and S columns may have X input channels and Y output channels, where R, S, and Y are each natural numbers. For example, the number of channels of the convolution kernel 202 may correspond to (e.g., may be equal to) the number X of input channels of the input feature map 201 and the number Y of output channels of the output feature map 203, and the 2D convolution kernel for each channel may have a size of R rows and S columns. The output feature map 203 may be generated via a 3D convolution operation between the input feature map 201 having a 3D size and the convolution kernel 202 having a 4D size, and as a result of the 3D convolution operation, the output feature map 203 may have Y channels.

[0061] The process of generating an output feature map by a convolution operation between a 2D input feature map and a 2D convolution kernel can be as described above. Figure 2A As described above, the above reference can be repeatedly performed between the X input channels of the input feature map 201 and the Y output channels of the convolution kernel 202. Figure 2A The 2D convolution operation is performed to generate an output feature map 203 having Y output channels. Based on the foregoing, it should be understood that the example embodiments discussed herein may include embodiments having one or more channels in various combinations of the X channels in the input feature map and / or the Y channels in the output feature map, or all channels.

[0062] Figure 3 The hardware architecture of a neural network device according to one or more embodiments is shown.

[0063] In one or more embodiments, neural network device 100 may be and / or may be implemented as any of various types of devices, such as a personal computer (PC), a server, a mobile device, and an embedded device, and may be, for example, or may include, a smartphone, a tablet device, an augmented reality (AR) device, an Internet of Things (IoT) device, an autonomous vehicle, a robot, or a medical device, which performs speech recognition, image recognition, and image classification using any one or any combination of neural network layers and / or neural networks composed of one or more of the layers of node convolutional interactions discussed herein, but is not limited thereto. In another example, neural network device 100 may correspond to a dedicated hardware (HW) accelerator installed on such a device, and may be a HW accelerator such as a neural processing unit (NPU), a tensor processing unit (TPU), or a neural engine, which is a dedicated module for driving a neural network, as non-limiting examples, but is not limited thereto.

[0064] refer to Figure 3 , the neural network device 100 may include a processor 110 and a memory 120. Among the components of the neural network device 100, the components related to one or more embodiments are Figure 3 However, after understanding the present disclosure, those skilled in the art will understand that: Figure 3 Neural network device 100 may also include general components in addition to those shown in FIG.

[0065] Processor 110 may control all functions for executing a neural network in neural network device 100. For example, processor 110 may control all functions of neural network device 100 by executing instructions stored in memory 120 in neural network device 100. Processor 110 may be implemented as a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), etc. included in neural network device 100, but is not limited thereto.

[0066] Memory 120 is hardware for storing various pieces of data processed in neural network device 100. For example, memory 120 may store data that has been processed or is to be processed in neural network device 100. In addition, memory 120 may store applications, drivers, and the like to be driven by neural network device 100. Memory 120 may include, but is not limited to, random access memory (RAM) (e.g., dynamic random access memory (DRAM) or static random access memory (SRAM)), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM, Blu-ray disc, other optical disc storage devices, hard disk drive (HDD), solid-state drive (SSD), or flash memory.

[0067] The processor 110 reads and / or writes neural network data (e.g., image data, feature map data, or convolution kernel data) from the memory 120 and executes the neural network by using the read / written neural network data. When executing the neural network, the processor 110 may drive a processing unit in which a convolution operation is provided to repeatedly perform a convolution operation between the input feature map and the convolution kernel, thereby generating data related to the output feature map. Here, the operation count of the convolution operation may be determined based on various factors, such as the number of channels of the input feature map, the number of channels of the convolution kernel, the size of the input feature map, the size of the convolution kernel, and the accuracy of the value.

[0068] The processor 110 may obtain an input feature map and a convolution kernel, and may perform a convolution operation between the input feature map and the convolution kernel. The processor 110 may include a hardware processing unit 112 to perform the convolution operation. The processing unit 112 may perform the convolution operation in parallel, and may each include a logic circuit for the convolution operation. In detail, each of the processing units 112 may include an operator, including a combination of a multiplier, an adder, and an accumulator. The multiplier may be embodied as a combination of multiple sub-multipliers, and the adder may be embodied as a combination of multiple sub-adders.

[0069] The processor 110 may further include: an on-chip memory that performs a cache function to perform convolution operations; and a scheduler that schedules various operands (e.g., pixel values ​​of the input feature map and weights of the convolution kernel). The scheduler may schedule operands (e.g., pixel values ​​and weights required for the processing unit 112 to perform the operation) to the on-chip memory based on the pixel values ​​of the input feature map and the weights of the convolution kernel, wherein the data is stored in the memory 120. The scheduler may also schedule the operands scheduled to the on-chip memory to the processing unit 112 for convolution operations.

[0070] The processor 110 may include a controller 114 that determines an operand to be processed by the processing unit 112. During a convolution operation between an input feature map and a convolution kernel, the controller 114 may determine a shared operand to be shared in the parallel operations performed by the processing unit 112 as one of the following: a pixel value of the input feature map or a weight value of the convolution kernel. According to an example, during a convolution operation between a first input feature map and a first convolution kernel, the controller 114 may determine the shared operand as the pixel value of the first input feature map. According to another example, during a convolution operation between a second input feature map and a second convolution kernel, the controller 114 may determine the shared operand as the weight value of the second convolution kernel.

[0071] The controller 114 may determine the shared operand as one of the following: pixel values ​​of the input feature map or weight values ​​of the convolution kernel based on at least one of the features of the input feature map or the features of the convolution kernel. Specifically, the controller 114 may determine the shared operand as one of the following: pixel values ​​of the input feature map or weight values ​​of the convolution kernel based on at least one of the percentage of pixels with zero values ​​in the input feature map, the percentage of weights with zero values ​​in the convolution kernel, or the shape of the input feature map. In one example, when the percentage of pixels with zero values ​​in the input feature map is equal to or greater than a preset critical value, the controller 114 may determine the shared operand as the pixel values ​​of the input feature map. The controller 114 may determine the percentage of pixels with zero values ​​in the input feature map of the current layer based on the percentage of pixels with zero values ​​in the input feature map of the previous layer. In another example, when the percentage of weights with zero values ​​in the convolution kernel is equal to or greater than a preset critical value, the controller 114 may determine the shared operand as the weight value of the convolution kernel. According to another example, when the number of input channels of the input feature map is large or the size of a two-dimensional (2D) input feature map is small, the controller 114 may determine the shared operand as a pixel value of the input feature map.

[0072] For each layer of the neural network, the controller 114 may determine the shared operand as one of the following: the pixel value of the input feature map or the weight value of the convolution kernel. For example, during a convolution operation between the input feature map of the first layer and the convolution kernel of the first layer, the controller 114 may determine the shared operand of the first layer as the pixel value of the input feature map of the first layer. During a convolution operation between the input feature map of the second layer and the convolution kernel of the second layer, the controller 114 may determine the shared operand of the second layer as the weight value of the convolution kernel of the second layer. In other words, each time the layer is changed, the controller 114 may change the shared operand from the pixel value of the input feature map to the weight of the convolution kernel, or may change the shared operand from the weight of the convolution kernel to the pixel value of the input feature map.

[0073] The processing unit 112 may perform parallel operations based on the shared operands determined by the controller 114. For example, when the pixel value of the first input feature map is determined as a shared operand, the processing unit 112 may jointly receive the pixel value of the first input feature map as one operand in the operand pair, and may receive different weight values ​​of the first convolution kernel as the other operand. As another example, when the weight value of the first convolution kernel is determined as a shared operand, the processing unit 112 may jointly receive the weight value of the first convolution kernel as one operand in the operand pair, and may receive different pixel values ​​of the first input feature map as the other operand. Because the processing unit 112 jointly receives the shared operands and performs parallel operations, the processing unit 112 may implement a scalar vector multiplication operation. The processing unit 112 may output an output value by performing parallel operations, and the output value may be accumulated in the output feature map.

[0074] Because the neural network device 100 can control the processing units 112 to share the shared operands, the processing units 112 can be controlled equally, thereby simplifying the hardware structure and reducing hardware costs. In addition, because the neural network device 100 can determine the shared operands of the processing units 112 as one of the following: the pixel values ​​of the input feature map or the weight values ​​of the convolution kernel, the shared operands can be selected to effectively use the parallelism of the convolution operation. According to an example, in order to more effectively implement zero skipping, the neural network device 100 can determine the shared operands as one of the following: the pixel values ​​of the input feature map or the weight values ​​of the convolution kernel. Figure 11 and Figure 12 According to another example, the neural network device 100 can select shared operands according to the shape of the input feature map of a layer to more efficiently implement the parallel operation of the processing unit 112. Figure 6 A detailed embodiment will be described.

[0075] When neural network device 100 performs a three-dimensional (3D) convolution operation between an input feature map having a 3D size and a convolution kernel having a 4D size, although the dimensions of the input feature map and the convolution kernel are different from each other, neural network device 100 can determine one of the pixel value of the input feature map or the weight value of the convolution kernel as a shared operand, and thereby can implement a hardware structure capable of performing parallel operations.

[0076] Figure 4 FIG1 shows a controller and a hardware processing unit included in a neural network device according to one or more embodiments. In the example, the controller and the processing unit are respectively Figure 3 The controller 114 corresponds to the processing unit 112. For ease of explanation, this example will be used to discuss examples below, but it is noted that the embodiments are not limited thereto.

[0077] As a non-limiting example, the controller 114 may obtain the input feature map and the convolution kernel from the memory 120. That is, although Figure 4 The operation of the neural network device 100 will be described with reference to the neural network device 100, but the embodiment is not limited thereto. The controller 114 may determine one of the pixel values ​​of the input feature map or one of the weight values ​​of the convolution kernel as a shared operand to be shared by the processing unit 112 based on at least one feature of the input feature map and / or at least one feature of the convolution kernel.

[0078] The controller 114 may control the pixel values ​​of the input feature map and the weight values ​​of the convolution kernel to be input to the processing units 112, and may specifically control the predetermined shared operands to be commonly input to the processing units 112. According to an example, when the pixel values ​​of the input feature map are determined as shared operands, the controller 114 may control the pixel values ​​of the input feature map to be input to each processing unit 112, and may control different weight values ​​of the convolution kernel to be input to different processing units 112, respectively. In other words, the controller 114 may control different weight values ​​of the convolution kernel to be input as operands 1 to n, respectively, to different processing units 112. According to another example, when the weight value of the convolution kernel is determined as a shared operand, the controller 114 may control the weight value of the convolution kernel to be input to each processing unit 112, and may control different pixel values ​​of the input feature map to be input to different processing units 112, respectively. In other words, the controller 114 may control different pixel values ​​of the input feature map to be input as operands 1 to n, respectively, to different processing units 112. Under the control of the controller 114, the scheduler of the processor 110 can schedule operand pairs including shared operands to each processing unit 112. Figure 4 As shown, the hardware processing units 1 to n of the processing unit 112 may each perform one or more operations on the shared operands and the corresponding operands 1 to n to generate corresponding outputs 1 to n, but the embodiment is not limited thereto.

[0079] Because each processing unit 112 can be any one or any combination of any two or more of a hardware multiplier, adder, and accumulator, each processing unit 112 can output an output value (output 1 to n) via a multiplication and / or addition operation on an input operand pair, and the output value (output 1 to n) can be accumulated into an output feature map. In addition, because each processing unit 112 can perform operations between shared operands and different operands (e.g., corresponding operands), the processing unit 112 can implement a 1×n hardware architecture that implements a scalar (shared operand)-vector (operand 1, operand 2, up to operand n) multiplication operation.

[0080] Figure 5A and Figure 5B The hardware processing unit is shown, which is configured to perform parallel operations based on the shared operands of the processing unit. In the example, the controller and the processing unit are respectively connected to Figure 3 The controller 114 corresponds to the processing unit 112. For ease of explanation, this example will be used to discuss examples below, but it is noted that the embodiments are not limited thereto.

[0081] In one or more embodiments, Figure 5A As shown, the controller 114 may determine the first weight value w_1 of the convolution kernel as a shared operand of the processing unit 112. Then, the controller 114 may control the first weight value w_1 to be input to each of the 16 processing units 112, and may control the 16 pixel values ​​IFM1 to IFM16 of the input feature map to be input to the 16 processing units 112, respectively. For example, Figure 5A The 16 pixel values ​​IFM1 to IFM16 may be pixel values ​​corresponding to the first input channel of the input feature map IFM. Each of the 16 processing units 112 may output 16 output values ​​by performing one or more operations on the input operand pairs, and the 16 output values ​​may be accumulated in the output feature map OFM.

[0082] In one or more embodiments, Figure 5B As shown, the controller 114 can determine the first pixel value IFM1 of the input feature map as a shared operand of the processing unit 112. Then, the controller 114 can control the first pixel value IFM1 to be input to each of the 16 processing units 112, and can control the 16 weight values ​​w_1 to w_16 of the convolution kernel to be input to the 16 processing units 112 respectively. For example, the 16 weight values ​​w_1 to w_16 of the convolution kernel can be the corresponding weight values ​​of the 16 output channels of the convolution kernel. Each of the 16 processing units 112 can output 16 output values ​​by performing one or more operations on the input operands, and the 16 output values ​​can be accumulated in the output feature map OFM.

[0083] exist Figure 5A and Figure 5B , the number of pixels of one input channel of the input feature map, the number of output channels of the convolution kernel, and the number of processing units 112 are all 16. However, this is merely an example, and embodiments are not limited thereto.

[0084] Figure 6 FIGURE 1 shows a process for a controller to determine a shared operand based on the shape of an input feature map. In the example, the controller and the processing unit are respectively Figure 3The controller 114 and the processing unit 112 correspond to each other, and for ease of explanation, this example will be used to discuss an example below, without limiting the scope of the present invention.

[0085] In one or more embodiments, the controller 114 may determine the number of shared operations of the processing unit 112 based on the shape of the input feature map. For example, the controller 114 may determine the number of shared operations considering the number of input channels of the input feature map or the size of the 2D input feature map of each input channel.

[0086] According to an example, when it is determined (e.g., in response to determining) that the size of the 2D input feature map of one channel of the input feature map 610 is large and the number of channels of the input feature map 610 is small, the controller 114 may determine the shared operand of the processing unit 112 as the weight value(s) of the convolution kernel. For example, when it is determined that the area value (W×H) of the input feature map 610 is large and its depth value C is small (e.g., shallow depth), the controller 114 may determine the shared operand of the processing unit 112 as the weight value(s) of the convolution kernel. As the area value (W×H) of the input feature map 610 increases and its depth value C decreases, compared to performing the convolution operation by reusing the pixel values ​​of the input feature map, performing the convolution operation by reusing the weight values ​​of the convolution kernel can more efficiently implement parallel operations of the processing unit 112, and therefore the controller 114 may determine the weight values ​​of the convolution kernel as the shared operand. For example, when it is determined that the area value (W×H) of the input feature map 610 is greater than or equal to a preset critical value and / or its depth value C is less than or equal to a preset critical value, the controller 114 may determine the shared operand of the processing unit 112 as the weight value of the convolution kernel. As another example, the controller 114 may determine the shared operand of the processing unit 112 based on a comparison between the area value (W×H) and the depth value C. For example, when it is determined that the area value (W×H) is sufficiently large compared to the depth value C, the controller 114 may determine the shared operand of the processing unit 112 as the weight value of the convolution kernel. For example, when it is determined that the area value (W×H) is greater than or equal to a preset critical value determined as a multiple or a portion of the depth value C, the controller 114 may determine the shared operand of the processing unit 112 as the weight value of the convolution kernel.

[0087] According to another example, when it is determined (e.g., in response to determining) that the size of the 2D input feature map of one channel of the input feature map 620 is small and the number of input channels of the input feature map 620 is large, the controller 114 may determine the shared operand of the processing unit 112 as the pixel value of the input feature map. For example, when it is determined that the area value (W×H) of the input feature map 620 is small and its depth value C is large, the controller 114 may determine the shared operand of the processing unit 112 as the pixel value of the input feature map. As the area value (W×H) of the input feature map 610 decreases and its depth value C increases, compared to performing the convolution operation by reusing the weight value of the convolution kernel, since performing the convolution operation by reusing the pixel value of the input feature map can more efficiently implement parallel operation of the processing unit 112, the controller 114 may determine the pixel value of the input feature map as the shared operand. For example, when it is determined that the area value (W×H) of the input feature map 620 is less than or equal to a preset critical value and / or its depth value C is greater than or equal to a preset critical value, the controller 114 may determine the shared operand of the processing unit 112 as the pixel value of the input feature map. As another example, the controller 114 may determine the shared operand of the processing unit 112 based on a comparison between the area value (W×H) and the depth value C. For example, when it is determined that the depth value C is sufficiently large compared to the area value (W×H), the controller 114 may determine the shared operand of the processing unit 112 as the pixel value of the input feature map. For example, when it is determined that the depth value C is greater than or equal to a preset critical value determined as a multiple or a portion of the area value (W×H), the controller 114 may determine the shared operand of the processing unit 112 as the pixel value of the input feature map.

[0088] As the convolution operations for each layer of the neural network are performed sequentially, the output feature map of the layer generated based on the input feature map of the layer can have a smaller "area value (W×H) to depth value C ratio" than the input feature map of the layer, wherein the input feature map of the subsequent layer can be (or can be based on) the output feature map of the layer, so that the input feature map of the early layer of the neural network can have the same "area value (W×H) to depth value C ratio" as the input feature map of the layer. Figure 6 The input feature map 610 of the neural network may have a shape similar to that of the input feature map 610 of the neural network, and the input feature map of the late layer of the neural network may have a shape similar to that of the input feature map 610 of the neural network. Figure 6 The shape of the input feature map 620 is similar to that of the input feature map 620. Therefore, the controller 114 can determine the shared operands of the processing units 112 as the weight values ​​of the convolution kernels in the early layers of the neural network, and can determine the shared operands of the processing units 112 as the pixel values ​​of the input feature map in the late layers of the neural network.

[0089] Figure 7A hardware processing unit according to one or more embodiments is shown, which is configured to perform parallel operations based on pixel values ​​of an input feature map IFM as a shared operand of the processing unit. In the example, the controller and the processing unit are respectively connected to Figure 3 The controller 114 and the processing unit 112 correspond to each other, and for ease of explanation, this example will be used to discuss an example below, without limiting the scope of the present invention.

[0090] The controller 114 may determine the shared operand as the pixel value of the input feature map IFM. Therefore, the controller 114 may control the pixel values ​​of the input feature map IFM to be commonly input to the processing unit 112.

[0091] For example, the controller 114 may control the first pixel value IFM1 of the input feature map IFM to be input to each processing unit 112 as a shared operand, and may also control the weight values ​​w_1, w_2, through w_m of the convolution kernel to be input to the processing unit 112, respectively. The first pixel value IFM1 may be a pixel value corresponding to the position (0, 0) of the 0th input channel of the input feature map IFM, and each of the weight values ​​w_1, w_2, through w_m of the convolution kernel may be a weight value corresponding to the position (0, 0) of the 0th input channel in each of the m output channels of the convolution kernel.

[0092] The first processing unit among the processing units 112 may output an output value O0(0, 0) by performing a multiplication and / or addition operation between the first pixel value IFM1 and the weight value w_1, and the mth processing unit among the processing units 112 may output an output value O0(0, 0) by performing a multiplication and addition operation between the first pixel value IFM1 and the weight value w_m. m-1 In other words, the processing unit 112 can output the output values ​​O0(0,0), O1(0,0) to O1(0,0) by performing multiplication and / or addition operations between the first pixel value IFM1 and each of the weight values ​​w_1, w_2 to w_m. m-1 (0, 0), wherein the multiplication and / or addition operation may include a scalar vector multiplication operation. In addition, the output values ​​O0(0, 0), O1(0, 0) up to O m-1 (0, 0) may be accumulated to the 0th to mth output channels of the output feature map OFM, respectively. For example, each output value may relate to a corresponding channel of the output feature map and may relate to the same 2D position of the output feature map.

[0093] Similarly, the controller 114 may control another pixel value of the input feature map IFM to be input to each processing unit 112 as a shared operand, and may also control different weight values ​​of the convolution kernel to be input to the processing units 112, respectively. The processing unit 112 may then output an output value by performing a multiplication and / or addition operation between another pixel value of the input feature map IFM and each of the different weight values ​​of the convolution kernel, and the output value may be accumulated to the output feature map OFM.

[0094] Figure 8 A hardware processing unit according to one or more embodiments is shown, which is configured to perform parallel operations based on pixel values ​​of an input feature map IFM as a shared operand of the processing unit. In the example, the controller and the processing unit are respectively connected to Figure 3 The controller 114 and the processing unit 112 correspond to each other, and for ease of explanation, this example will be used to discuss an example below, without limiting the scope of the present invention.

[0095] The controller 114 may determine the shared operands as the first pixel value IFM1, the second pixel value IFM2, the third pixel value IFM3, and the fourth pixel value IFM4 of the input feature map IFM. The first pixel value IFM1, the second pixel value IFM2, the third pixel value IFM3, and the fourth pixel value IFM4 may be pixel values ​​corresponding to positions (0, 0), (0, 1), (1, 0), and (1, 1), respectively, of the 0th input channel of the input feature map IFM. The controller 114 may control the first pixel value IFM1 to be input as the shared operand to each first processing unit 171 among the processing units 112, and may also control the weight values ​​w_1, w_2, through w_m of the convolution kernel to be input to the first processing unit 171, respectively. Similarly, the controller 114 may control the second pixel value IFM2 to be input as the shared operand to each second processing unit 172 among the processing units 112, and may also control the weight values ​​w_1, w_2, through w_m of the convolution kernel to be input to the second processing unit 172, respectively. Similarly, the controller 114 may control the third pixel value IFM3 to be input as a shared operand to each third processing unit 173 among the processing units 112, and may also control the weight values ​​w_1, w_2, through w_m of the convolution kernel to be input to the third processing unit 173. Similarly, the controller 114 may control the fourth pixel value IFM4 to be input as a shared operand to each fourth processing unit 174 among the processing units 112, and may also control the weight values ​​w_1, w_2, through w_m of the convolution kernel to be input to the fourth processing unit 174. For example, the controller 114 may control the first pixel value IFM1 to the fourth pixel value IFM4 from the IFM buffer (0, 0) to the IFM buffer (1, 1) to be input to the first processing unit 171 to the fourth processing unit 174, respectively, and may control the weight values ​​w_1, w_2, through w_m from the weight buffer to be input to the first processing unit 171 to the fourth processing unit 174, respectively.

[0096] The first processing unit 171 may output output values ​​O0(0,0), O1(0,0) to O1(0,0) by performing a multiplication operation between the first pixel value IFM1 and each of the weight values ​​w_1, w_2 to w_m. m-1 (0,0) (e.g., as referenced above Figure 7 The second processing unit 172 may output output values ​​O0(0,1), O1(0,1) to O1(0,1) by performing a multiplication operation between the second pixel value IFM2 and each of the weight values ​​w_1, w_2 to w_m. m-1(0, 1). The third processing unit 173 may output output values ​​O0(1, 0), O1(1, 0) to O1(1, 0) by performing a multiplication operation between the third pixel value IFM3 and each of the weight values ​​w_1, w_2 to w_m. m-1 The fourth processing unit 174 may output output values ​​O0(1,1), O1(1,1) to O1(1,1) by performing a multiplication operation between the fourth pixel value IFM4 and each of the weight values ​​w_1, w_2 to w_m. m-1 (1, 1). Therefore, the output value O output by the first processing unit 171 to the fourth processing unit 174 is 0～m-1 (0, 0), O 0～m-1 (0, 1), O 0～m-1 (1,0),O 0～m-1 (1, 1) can be accumulated to the output feature map OFM.

[0097] Figure 7 and Figure 8 The input feature map IFM shown in FIG is of size H×W and includes n input channels, and the size is Ti r And includes a convolution kernel with n input channels and m output channels, and a size of T r ×T c And each map in the output feature map OFM including m output channels is just an example, and the embodiment is not limited thereto. Figure 7 and Figure 8 The positions of the first to fourth pixel values ​​of the input feature map IFM and the positions of the weight values ​​w_1, w_2 to w_m of the convolution kernel shown in are also merely examples, and the embodiment is not limited thereto.

[0098] Figure 9 A hardware processing unit is shown according to one or more embodiments, which is configured to perform parallel operations based on weight values ​​of convolution kernels as shared operands of the processing unit. In an example, the processing unit and Figure 3 Corresponding to the processing unit 112, and for ease of explanation, this example will be used to discuss the example below, and it is noted that the embodiment is not limited thereto.

[0099] In one or more embodiments, the controller 114 may determine the shared operand as the weight value of the convolution kernel. Therefore, the controller 114 may control the weight value of the convolution kernel to be commonly input to the processing unit 112.

[0100] For example, the controller 114 may control the first weight value w_1 of the convolution kernel to be input to each processing unit 112 as a shared operand, and may also control the first pixel value IFM1, the second pixel value IFM2, the third pixel value IFM3, and the fourth pixel value IFM4 of the input feature map IFM to be output to the processing unit 112, respectively. The first weight value w_1 of the convolution kernel may be a weight value corresponding to the position (0, 0) of the 0th output channel and the 0th input channel of the convolution kernel. The first pixel value IFM1, the second pixel value IFM2, the third pixel value IFM3, and the fourth pixel value IFM4 may be pixel values ​​corresponding to the positions (0, 0), (0, 1), (1, 0), and (1, 1) in the 0th input channel of the input feature map IFM, respectively.

[0101] The first processing unit among the processing units 112 may output an output value O0(0,0) by performing a multiplication and / or addition operation between the first weight value w_1 and the first pixel value IFM1. Similarly, the second processing unit, the third processing unit, and the fourth processing unit among the processing units 112 may output an output value O0(0,1), an output value O0(1,0), and an output value O0(1,1), respectively, by performing a multiplication and / or addition operation between the first weight value w_1 and the second pixel value IFM2, a multiplication and / or addition operation between the first weight value w_1 and the third pixel value IFM3, and a multiplication and / or addition operation between the first weight value w_1 and the fourth pixel value IFM4, respectively. In other words, the processing unit 112 can output output values ​​O0(0,0), O0(0,1), O0(1,0), and O0(1,1) respectively by performing multiplication and / or addition operations between the first weight value w_1 and each of the first pixel value IFM1, the second pixel value IFM2, the third pixel value IFM3, and the fourth pixel value IFM4, wherein the multiplication and / or addition operations may include scalar vector multiplication operations. The output values ​​O0(0,0), O0(0,1), O0(1,0), and O0(1,1) can be accumulated to the output feature map OFM. For example, each output value can relate to a corresponding 2D position of the output feature map and can relate to the same channel of the output feature map.

[0102] Similarly, the controller 114 may control another weight value of the convolution kernel to be input to each processing unit 112 as a shared operand, and may also control different pixel values ​​of the input feature map IFM to be respectively input to the processing unit 112. The processing unit 112 may then output an output value by performing a multiplication and / or addition operation between another weight value of the convolution kernel and each of the different pixel values ​​of the input feature map IFM, and the output value may be accumulated to the output feature map OFM.

[0103] Figure 10A hardware processing unit according to one or more embodiments is shown, which is configured to perform parallel operations based on weight values ​​of convolution kernels as shared operands of the processing unit. In the example, the controller and the processing unit are respectively connected to Figure 3 The controller 114 and the processing unit 112 correspond to each other, and for ease of explanation, this example will be used to discuss an example below, without limiting the scope of the present invention.

[0104] The controller 114 may determine the shared operands as the first weight value w_1, the second weight value w_2, the third weight value w_3, and the fourth weight value w_4 of the convolution kernel. The first weight value w_1, the second weight value w_2, the third weight value w_3, and the fourth weight value w_4 may be weight values ​​corresponding to the position (0, 0) of the 0th input channel corresponding to the 0th output channel, the first output channel, the second output channel, and the third output channel, respectively. The controller 114 may control the first weight value w_1 to be input to each first processing unit 191 among the processing units 112 as the shared operand, and may also control the first pixel value IFM1, the second pixel value IFM2, the third pixel value IFM3, and the fourth pixel value IFM4 of the input feature map IFM to be input to the first processing unit 191, respectively. Similarly, the controller 114 may control the second weight value w_2 to be input as a shared operand to each second processing unit 192 among the processing units 112, and may also control the first pixel value IFM1, the second pixel value IFM2, the third pixel value IFM3, and the fourth pixel value IFM4 of the input feature map IFM to be input to the second processing unit 192, respectively. Similarly, the controller 114 may control the third weight value w_3 to be input as a shared operand to each third processing unit 193 among the processing units 112, and may also control the first pixel value IFM1, the second pixel value IFM2, the third pixel value IFM3, and the fourth pixel value IFM4 of the input feature map IFM to be input to the third processing unit 193, respectively. Similarly, the controller 114 may control the fourth weight value w_4 to be input as a shared operand to each fourth processing unit 194 among the processing units 112, and may also control the first pixel value IFM1, the second pixel value IFM2, the third pixel value IFM3, and the fourth pixel value IFM4 of the input feature map IFM to be input to the fourth processing unit 194, respectively.

[0105] The first processing unit 191 may output output values ​​O0(0,0), O0(0,1), O0(1,0), and O0(1,1) (for example, as described above with reference to FIG) by performing a multiplication operation between the first weight value w_1 and each of the first pixel value IFM1, the second pixel value IFM2, the third pixel value IFM3, and the fourth pixel value IFM4. Figure 9The second processing unit 192 may output output values ​​O1(0,0), O1(0,1), O1(1,0), and O1(1,1) by performing a multiplication operation between the second weight value w_2 and each of the first pixel value IFM1, the second pixel value IFM2, the third pixel value IFM3, and the fourth pixel value IFM4. The third processing unit 193 may output output values ​​O2(0,0), O2(0,1), O2(1,0), and O2(1,1) by performing a multiplication operation between the third weight value w_3 and each of the first pixel value IFM1, the second pixel value IFM2, the third pixel value IFM3, and the fourth pixel value IFM4. The fourth processing unit 194 can output output values ​​O3(0,0), O3(0,1), O3(1,0), and O3(1,1) by performing a multiplication operation between the fourth weight value w_4 and each of the first pixel value IFM1, the second pixel value IFM2, the third pixel value IFM3, and the fourth pixel value IFM4. Therefore, the output values ​​O0(0-1,0-1), O1(0-1,0-1), O2(0-1,0-1), and O3(0-1,0-1) output by the first processing unit 191, the second processing unit 192, the third processing unit 193, and the fourth processing unit 194 can be accumulated in the output feature map OFM.

[0106] Figure 9 and Figure 10 The input feature map IFM shown in FIG is of size H×W and includes n input channels, and the size is Ti r And includes a convolution kernel with n input channels and m output channels, and a size of T r ×T c And each map in the output feature map OFM including m output channels is just an example, and the embodiment is not limited thereto. Figure 9 and Figure 10 The positions of the first to fourth weight values ​​w_1 to w_4 and the positions of the first, second, third, or fourth pixel values ​​IFM1, IFM2, IFM3, and IFM4 shown in FIG. 1 are merely examples, and the embodiment is not limited thereto.

[0107] Return Reference Figure 3 , the processing unit 112 may perform the parallel operation based on the shared operand determined by the controller 114. At this time, when the shared operand is 0, the processing unit 112 may skip the parallel operation. In other words, the processing unit 112 may implement zero skipping based on the shared operand.

[0108] For example, the controller 114 may determine the pixel values ​​of the input feature map as shared operands, for example, as described above with reference to Figure 5B 、 Figure 7 and / or Figure 8 According to the example, when the first pixel value of the input feature map is 0, the processing unit 112 may skip the parallel operation between the first pixel value of the input feature map and the first weight value of the convolution kernel. Therefore, the processing unit 112 may perform a parallel operation between a non-zero second pixel value (instead of the first pixel value of 0) in the input feature map and the second weight value of the convolution kernel. According to another example, when all the first weight values ​​of the convolution kernel are 0, the processing unit 112 may skip the parallel operation between the first pixel value of the input feature map and the first weight value of the convolution kernel, and may perform a parallel operation between the second pixel value of the input feature map and the second weight value of the convolution kernel.

[0109] As another example, the controller 114 may determine the weight values ​​of the convolution kernel as shared operands, for example, as described above with reference to Figure 5A 、 Figure 9 and / or Figure 10 According to the example, when the first weight value of the convolution kernel is 0, the processing unit 112 may skip the parallel operation between the first weight value of the convolution kernel and the first pixel value of the input feature map. Therefore, the processing unit 112 may perform a parallel operation between the non-zero second weight value of the convolution kernel (instead of the first weight value of 0) and the second pixel value of the input feature map. According to another example, when all the first pixel values ​​of the input feature map are 0, the processing unit 112 may skip the parallel operation between the first weight value of the convolution kernel and the first pixel value, and may perform a parallel operation between the second weight value of the convolution kernel and the second pixel value.

[0110] Since the neural network device 100 can implement zero skipping by determining whether the shared operand of the processing unit 112 is 0 or whether all the remaining operands of the processing unit 112 are 0 as described above, a simplified hardware structure for zero skipping can be implemented.

[0111] Figure 11 FIGURE 1 illustrates a process for a hardware processing unit to perform zero skipping based on a shared operand being zero, according to one or more embodiments. Figure 3 Corresponding to the processing unit 112, and for ease of explanation, this example will be used to discuss the example below, and it is noted that the embodiment is not limited thereto.

[0112] In one or more embodiments, the processing unit 112 may output an output value by performing a parallel operation based on the first pixel value IFM1 of the input feature map IFM as a shared operand in the (N-1)th cycle. For example, the processing unit 112 may output an output value by performing a parallel operation between the first pixel value IFM1 at (0, 0) of the 0th input channel of the input feature map IFM and the weight value at (0, 0) of the 0th input channel corresponding to the 0th output channel to the (m-1)th output channel of the convolution kernel, respectively. The output value may be accumulated to the O at the position (0, 0) of the 0th output channel to the (m-1)th output channel of the output feature map OFM. 0～m-1 (0, 0).

[0113] Then, the processing unit 112 may perform a parallel operation based on the second pixel value IFM2 of the input feature map IFM in the Nth cycle. However, when it is determined that the second pixel value IFM2 is 0, the processing unit 112 may skip the parallel operation between the second pixel value IFM2 and the weight value. For example, the processing unit 112 may skip the parallel operation between the second pixel value IFM2 at (0, 0) of the first input channel of the input feature map IFM and the weight value at (0, 0) of the first input channel corresponding to the 0th output channel to the (m-1)th output channel of the convolution kernel, respectively.

[0114] Then, the processing unit 112 may output the output value by performing a parallel operation based on the third pixel value IFM3 of the input feature map IFM in the Nth cycle. In detail, the processing unit 112 may output the output value by performing a parallel operation between the third pixel value IFM3 at (0, 0) of the second input channel of the input feature map IFM and the weight value at (0, 0) of the second input channel corresponding to the 0th output channel to the (m-1)th output channel of the convolution kernel, respectively. The output value may be accumulated to the O at the position (0, 0) of the 0th output channel to the (m-1)th output channel of the output feature map OFM. 0～m-1 (0, 0).

[0115] Therefore, when the second pixel value IFM2 is 0, the processing unit 112 can perform a parallel operation based on the third pixel value IFM3 after the first pixel value IFM1, and the position on the output feature map OFM where the output value based on the first pixel value IFM1 is accumulated can be the same as the position on the output feature map OFM where the output value based on the third pixel value IFM3 is accumulated. Therefore, the neural network device 100 can reduce the operation count and operation time period of the convolution operation through zero skipping while maintaining the same position of the accumulated output value, thereby efficiently processing the convolution operation of the neural network.

[0116] Figure 12 FIGURE 1 illustrates a process for a hardware processing unit to perform zero skipping based on a shared operand being zero according to one or more embodiments. Figure 3 Corresponding to the processing unit 112, and for ease of explanation, this example will be used to discuss the example below, and it is noted that the embodiment is not limited thereto.

[0117] In one or more embodiments, the processing unit 112 may output an output value by performing a parallel operation based on the first weight value w_1 of the convolution kernel as a shared operand in the (N-1)th cycle. For example, the processing unit 112 may output an output value by performing a parallel operation on the first weight value w_1 at (0, 0) of the 0th output channel and the 0th input channel of the convolution kernel and the pixel value at (0-1, 0-1) of the 0th input channel of the input feature map IFM. The output value may be accumulated to O0 (0-1, 0-1) of (0-1, 0-1) of the 0th output channel of the output feature map OFM.

[0118] Then, the processing unit 112 may perform a parallel operation based on the second weight value w_2 of the convolution kernel in the Nth cycle. However, when it is determined that the second weight value w_2 is 0, the processing unit 112 may skip the parallel operation between the second weight value w_2 and the pixel value. For example, the processing unit 112 may skip the parallel operation between the second weight value w_2 located at (0, 0) of the 0th output channel and the first input channel of the convolution kernel and the pixel value located at (0-1, 0-1) of the first input channel of the input feature map IFM.

[0119] Then, the processing unit 112 may output an output value by performing a parallel operation based on the third weight value w_3 of the convolution kernel in the Nth cycle. In detail, the processing unit 112 may output an output value by performing a parallel operation between the third weight value w_3 located at (0, 0) of the 0th output channel and the second input channel of the convolution kernel and the pixel value located at (0-1, 0-1) of the second input channel of the input feature map IFM. The output value may be accumulated to O0 (0-1, 0-1) of (0-1, 0-1) of the 0th output channel of the output feature map OFM.

[0120] Therefore, when the second weight value w_2 is 0, the processing unit 112 can perform a parallel operation based on the third weight value w_3 after the first weight value w_1, and the position on the output feature map OFM where the output value based on the first weight value w_1 is accumulated can be the same as the position on the output feature map OFM where the output value based on the third weight value w_3 is accumulated. Therefore, the neural network device 100 can reduce the operation count and operation time period of the convolution operation through zero skipping, while maintaining the position of the accumulated output value to be the same, thereby efficiently processing the convolution operation of the neural network.

[0121] Figure 13 Methods of operating a neural network device according to one or more embodiments are shown.

[0122] Figure 13 The operation method can be Figures 1 to 12 The above operations are performed by components of the neural network device 100, and redundant descriptions thereof will be omitted here.

[0123] In operation 1310, neural network device 100 may determine, based on at least one of the features of the input feature map or the features of the convolution kernel, a shared operand to be shared in the parallel operation as one of the following: pixel values ​​of the input feature map and weight values ​​of the convolution kernel. According to an example, during a convolution operation between a first input feature map and a first convolution kernel, neural network device 100 may determine the shared operand as the weight value of the first convolution kernel. According to another example, during a convolution operation between a second input feature map and a second convolution kernel, neural network device 100 may determine the shared operand as the pixel value of the second input feature map.

[0124] Neural network device 100 may determine the shared operand as one of the following: pixel values ​​of the input feature map or weight values ​​of the convolution kernel based on at least one of a feature of the input feature map or a feature of the convolution kernel. For example, neural network device 100 may determine the shared operand as one of the following: pixel values ​​of the input feature map or weight values ​​of the convolution kernel based on at least one of a percentage of zero-valued pixels in the input feature map, a percentage of zero-valued weights in the convolution kernel, or a shape of the input feature map.

[0125] For each layer of the neural network, neural network device 100 may determine the shared operand as one of the following: pixel values ​​of the input feature map or weight values ​​of the convolution kernel. For example, during a convolution operation between the input feature map of the first layer and the convolution kernel of the first layer, neural network device 100 may determine the shared operand of the first layer as the weight values ​​of the convolution kernel of the first layer. During a convolution operation between the input feature map of the second layer and the convolution kernel of the second layer, neural network device 100 may determine the shared operand of the second layer as the pixel values ​​of the input feature map of the second layer.

[0126] In operation 1320, neural network device 100 may perform parallel operations based on shared operands. For example, when the pixel values ​​of the first input feature map are determined as shared operands, neural network device 100 may perform parallel operations between the pixel values ​​of the first input feature map and the weight values ​​of the first convolution kernel. As another example, when the weight values ​​of the first convolution kernel are determined as shared operands, neural network device 100 may perform parallel operations between the weight values ​​of the first convolution kernel and the pixel values ​​of the first input feature map. Neural network device 100 may accumulate output values ​​obtained through the parallel operations to the output feature map.

[0127] When the determined shared operand is 0, neural network device 100 may skip the parallel operation. According to an example, when the first pixel value is 0, neural network device 100 may skip the parallel operation between the first pixel value and the first weight value of the convolution kernel. Therefore, neural network device 100 may perform a parallel operation between a non-zero second pixel value (instead of the first pixel value being 0) and the second weight value of the convolution kernel. According to another example, when the first weight value is 0, neural network device 100 may skip the parallel operation between the first weight value and the first pixel value of the input feature map. Therefore, neural network device 100 may perform a parallel operation between the non-zero second weight value (instead of the first weight value being 0) and the second pixel value of the input feature map.

[0128] According to one or more embodiments, because the shared operand to be shared in the parallel operation of the convolution operation can be determined as one of the pixel value of the input feature map or the weight value of the convolution kernel, the parallelism of the convolution operation can be achieved more efficiently.

[0129] According to one or more embodiments, when a 3D convolution operation is performed between an input feature map having a 3D size and a convolution kernel having a 4D size, although the dimensions of the input feature map and the convolution kernel are different from each other, one of the pixel values ​​of the input feature map or the weight values ​​of the convolution kernel can be determined as a shared operand, and thus a hardware structure capable of performing parallel operations can be implemented.

[0130] According to one or more embodiments, since zero skipping is implemented by whether a shared operand of a processing unit is 0 or whether all remaining operands of a processing unit are 0, a simplified hardware structure for zero skipping can be implemented.

[0131] In this article Figures 1 to 13The neural network device 100, controller 114, processing unit 112, memory 120, buffers (IFM buffers, weight buffers), and other devices, units, modules, equipment, and other components described herein are implemented by or represent hardware components. Examples of hardware components that can be used to perform the operations described herein include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described herein, where appropriate. In other examples, one or more hardware components that perform the operations described herein are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer may be implemented by one or more processing elements (e.g., logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond to and execute instructions in a defined manner to achieve desired results. In one example, the processor or computer includes or is connected to one or more memories that store instructions or software to be executed by the processor or computer. Hardware components implemented by the processor or computer may execute instructions or software (e.g., an operating system (OS) and one or more software applications running on the OS) to perform the operations described herein. The hardware components may also access, manipulate, process, create, and store data in response to the execution of the instructions or software. For simplicity, the singular term "processor" or "computer" may be used in the description of the examples described herein, but may be used in other examples. A plurality of processors or computers, or processors or computers may include a plurality of processing elements or various types of processing elements or both have both. For example, a single hardware component or two or more hardware components may be realized by a single processor or two or more processors or a processor and a controller. One or more hardware components may be realized by one or more processors or a processor and a controller, and one or more other hardware components may be realized by one or more other processors or another processor and another controller. One or more processors or processors and a controller may realize a single hardware component or two or more hardware components. A hardware component may have any one or more of the different processing configurations, and the example of the processing configuration may comprise a single processor, an independent processor, a parallel processor, a single instruction single data (SISD) multiprocessing, a single instruction multiple data (SIMD) multiprocessing, a multiple instruction single data (MISD) multiprocessing, and a multiple instruction multiple data (MIMD) multiprocessing.

[0132] Perform the operations described in this application Figures 1 to 13The method shown is performed by computing hardware, for example, by one or more processors or computers that execute an execution instruction or software implemented as described above to perform the operation described herein (the operation implemented by the method). For example, a single operation or two or more operations can be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations can be performed by one or more processors or a processor and a controller, and one or more other operations can be performed by one or more other processors or another processor and another controller. One or more processors or a processor and a controller can perform a single operation or two or more operations.

[0133] The instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above can be written as computer programs, code segments, instructions, or any combination thereof, for individually or collectively indicating or configuring one or more processors or computers to operate as machines or special-purpose computers to perform the operations performed by the above-mentioned hardware components and methods. In one example, the instructions or software include machine code directly executed by one or more processors or computers, such as machine code generated by a compiler. In another example, the instructions or software include higher-level codes executed by one or more processors or computers using an interpreter. Any programming language can be used to write instructions or software based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding description used herein (which discloses an algorithm for performing the operations performed by the hardware components and methods described above).

[0134] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, as well as any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random-access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), flash memory, card type memory (such as, multimedia card or micro card (for example, Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device configured to store instructions or software and any associated data, data files and data structures in a non-transitory manner, and provide instructions or software and any associated data, data files and data structures to one or more processors or computers so that one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files and data structures are distributed on a networked computer system so that one or more processors or computers store, access and execute the instructions and software and any associated data, data files and data structures in a distributed manner.

[0135] Although this disclosure includes specific examples, it will be apparent after an understanding of the disclosure of this application that various changes in form and detail may be made to these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered merely descriptive and not for purposes of limitation. The description of features or aspects in each example is considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order and / or if components in the described systems, architectures, devices, or circuits are combined in different ways and / or replaced or supplemented by other components or their equivalents. Therefore, the scope of the disclosure is defined not by the detailed description, but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents are interpreted as being included in this disclosure.

Claims

1. A neural network device for performing image recognition, image classification, or speech recognition, the neural network device comprising: One or more processors, including: a controller configured to: determine, based on either or both of a shape of an input feature map and a weight value of a convolution kernel, a shared operand to be shared in parallel operations as either a pixel value among pixel values ​​of the input feature map or a weight value among weight values ​​of the convolution kernel, and determine an operand pair including the shared operand; and A processing unit is configured to perform the parallel operation based on the operand pair including the determined shared operand and a corresponding operand from the input feature map or the convolution kernel.

2. The device according to claim 1, wherein The plurality of processing units are configured to, in response to the controller determining the shared operand as a pixel value of the input feature map, perform the parallel operation in parallel between the pixel value of the input feature map and the corresponding weight value of the convolution kernel as the corresponding operand.

3. The device according to claim 1, wherein The plurality of processing units are configured to, in response to the controller determining the shared operand as the weight value of the convolution kernel, perform the parallel operation in parallel between the weight value of the convolution kernel and the corresponding pixel value of the input feature map as the corresponding operand.

4. The device according to claim 1, wherein When the controller determines the shared operand as a pixel value, the controller is further configured to determine another shared operand as another pixel value of the input feature map, and in, The plurality of processing units include: a first processing unit configured to perform the parallel operation in parallel between the pixel values ​​of the input feature map and the weight values ​​of the convolution kernel; and The second processing unit is configured to perform the parallel operation in parallel between the other pixel value of the input feature map and the weight value of the convolution kernel.

5. The device according to claim 1, wherein When the controller determines the shared operand as a weight value, the controller is further configured to determine another shared operand as another weight value of the convolution kernel, and in, The plurality of processing units include: a first processing unit configured to perform the parallel operation in parallel between the weight value of the convolution kernel and the pixel value of the input feature map; and The second processing unit is configured to perform the parallel operation in parallel between the other weight value of the convolution kernel and the pixel value of the input feature map.

6. The device according to claim 1, wherein The shared operand, the pixel values ​​of the input feature map, and the weight values ​​of the convolution kernel relate to a first layer of a neural network, and The controller is further configured to, for a second layer of the neural network, determine the shared operand of the second layer as either a pixel value of the input feature map of the second layer or a weight value of the convolution kernel of the second layer based on either or both of the features of the input feature map of the second layer and the features of the convolution kernel of the second layer.

7. The device according to claim 1, wherein The characteristics of the input feature map include either or both of a percentage of pixels having a zero value within the input feature map and a shape of the input feature map, and The characteristic of the convolution kernel includes a percentage of weights within the convolution kernel having a value of zero.

8. The device according to claim 1, wherein The controller is configured to determine the shared operand as a weight value of the convolution kernel in response to a percentage of weights having a zero value in the convolution kernel being greater than a set value.

9. The device according to claim 1, wherein The number of the processing units that jointly share the shared operand is equal to any one of the number of pixels of the input channels of the input feature map and the number of output channels of the convolution kernel.

10. The device according to claim 1, wherein The one or more processors are configured to perform a 3D convolution operation between the input feature map having a three-dimensional 3D size and the convolution kernel having a four-dimensional 4D size.

11. The device according to claim 1, wherein The processing unit is configured to skip a parallel operation among the parallel operations in response to the shared operand being zero.

12. The device according to claim 1, wherein When the controller determines the shared operand as a pixel value, the controller is further configured to determine another shared operand as another pixel value of the input feature map, and Wherein, the plurality of processing units are configured as follows: In response to the pixel value being 0, skipping the parallel operation between the pixel value and a first weight value among the weight values ​​of the convolution kernel among the parallel operations, and A parallel operation between the other pixel value and a second weight value among the weight values ​​of the convolution kernel is performed among the parallel operations.

13. The device according to claim 1, wherein When the controller determines the shared operand as a weight value, the controller is further configured to determine another shared operand as another weight value of the convolution kernel, and Wherein, the plurality of processing units are configured as follows: In response to the weight value being 0, skipping a parallel operation between the weight value and a first pixel value among pixel values ​​of the input feature map among the parallel operations, and A parallel operation between the other weight value and a second pixel value among the pixel values ​​of the input feature map is performed among the parallel operations.

14. The apparatus of claim 1 , further comprising a memory storing instructions that, when executed by the one or more processors, configure the one or more processors to perform the following operations: determining the shared operands, performing the parallel operations, and generating at least a portion of an output feature map based on the performed parallel operations.

15. The device according to claim 1, wherein Determining the shared operand based on either or both of the shape of the input feature map and the weight value of the convolution kernel includes: In response to determining that a ratio of the area value of the input feature map to the depth value of the input feature map is greater than a specified value, specifying a weight value of the convolution kernel as the shared operand; and In response to determining that a ratio of the area value of the input feature map to the depth value of the input feature map is less than the designated value, designating a pixel value as the shared operand.

16. A processor-implemented neural network method for performing image recognition, image classification, or speech recognition, the method comprising: Based on either or both of a shape of an input feature map and a weight value of a convolution kernel, determining a shared operand to be shared in parallel operation as either a pixel value among pixel values ​​of the input feature map or a weight value among weight values ​​of the convolution kernel, and selecting an operand pair including the shared operand; and The parallel operation is performed based on the operand pairs including the determined shared operand and a corresponding operand from the input feature map or the convolution kernel.

17. The method according to claim 16, wherein Executing the parallel operation includes: In response to the pixel value being the determined shared operand, the parallel operation is performed in parallel between the pixel value of the input feature map and the corresponding weight value of the convolution kernel as the corresponding operand.

18. The method according to claim 16, wherein Executing the parallel operation includes: In response to the weight value being the determined shared operand, the parallel operation is performed in parallel between the weight value of the convolution kernel and the corresponding pixel value of the input feature map as the corresponding operand.

19. The method according to claim 16, wherein The executing includes skipping a parallel operation among the parallel operations in response to the shared operand being zero.

20. The method of claim 16, wherein In response to the pixel value being the determined shared operand, performing the parallel operation includes: In response to the pixel value being 0, skipping the parallel operation between the pixel value and a first weight value among the weight values ​​of the convolution kernel among the parallel operations, and A parallel operation between another pixel value among the pixel values ​​of the input feature map and a second weight value among the weight values ​​of the convolution kernel is performed among the parallel operations.

21. The method according to claim 16, wherein In response to the weight value being the determined shared operand, performing the parallel operation includes: When the weight value is 0, skipping the parallel operation between the weight value and the first pixel value among the pixel values ​​of the input feature map among the parallel operations, and A parallel operation between another weight value among the weight values ​​of the convolution kernel and a second pixel value among the pixel values ​​of the input feature map is performed.

22. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method of claim 16.

23. A processor-implemented neural network method for performing image recognition, image classification, or speech recognition, the method comprising: Get the input feature map and convolution kernel; Determining a shared operand as either a pixel value among pixel values ​​of the input feature map or a weight value among weight values ​​of the convolution kernel based on either or both of a shape of the input feature map and a weight value of the convolution kernel; and A pixel value of an output feature map is determined by performing an operation between any one of each weight value and each pixel value of the convolution kernel and the shared operand.

24. The method according to claim 23, wherein The pixel value of the input feature map corresponds to the first channel among the channels of the input feature map, the weight value corresponds to the first channel among the channels of the convolution kernel, and each pixel value of the output feature map corresponds to the same position in the corresponding channel among the channels of the output feature map.

25. The method according to claim 23, wherein Determining the shared operand based on either or both of the shape of the input feature map and the weight value of the convolution kernel includes: In response to a two-dimensional area size of the input feature map being less than or equal to a set value, determining the shared operand to be a pixel value of the input feature map; and In response to a two-dimensional area size of the input feature map being greater than or equal to another set value, the shared operand is determined as a weight value of the convolution kernel.

Images