Npu device performing convolution operation based on number of channels and operating method thereof

Abstract

A method of generating an output feature map based on an input feature map, a neural processing unit device, and an operating method thereof are provided. The method of generating the output feature map based on the input feature map includes generating an input feature map vector for a plurality of input feature map blocks when a number of channels of the input feature map is less than a certain number of reference channels, performing a convolution operation on the input feature map based on a target weight map and an additional weight map having the same weight as a weight of the target weight map when a number of the target weight maps is less than a reference number, and generating the output feature map based on the performed convolution operation.

Description

[0001] Cross-references to related applications

[0002] This application is based on and claims priority to Korean Patent Application No. 10-2020-0174731, filed on December 14, 2020, with the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure relates to a neural processing unit (NPU) device and its operation method, and more specifically, to an NPU device and its operation method that perform convolution operations based on the number of channels of an input feature map and an output feature map. Background Technology

[0004] A neural network is a computational architecture used to model the biological brain. Recently, with the development of neural network technology, various electronic systems have been actively researched for use with neural network devices to analyze input data and extract useful information using multiple neural network models.

[0005] Neural network devices are required to perform a large number of calculations on complex input data. Therefore, in order for neural network devices to analyze high-quality inputs and extract information in real time, techniques that can efficiently process neural network calculations are needed.

[0006] That is, since neural network devices need to perform operations on complex input data, there is a need for a method and apparatus that can efficiently extract the data required for the operation from complex and large input data with less resources and minimal power consumption. Summary of the Invention

[0007] The present invention provides a neural processing unit (NPU) device for performing effective convolution operations when the number of channels in the input feature map and the output feature map is small.

[0008] According to one aspect of the inventive concept of this disclosure, a method for generating an output feature map based on an input feature map is provided. The method includes: generating an input feature map vector for multiple input feature map blocks based on the fact that the number of channels in the input feature map is less than the number of reference channels; performing a convolution operation between the input feature map vector and a weight map based on the fact that the number of the one or more target weight maps is less than the reference number, the weight map including the one or more target weight maps and an additional weight map having the same weight as one of the one or more target weight maps; and generating an output feature map based on the convolution operation.

[0009] According to another aspect of the inventive concept of this disclosure, a neural processing unit (NPU) device is provided. The NPU device may include: a vector generator configured to generate input feature map vectors for multiple input feature map blocks based on the number of channels in the input feature map being less than the number of reference channels; and computational circuitry configured to: perform a convolution operation between the input feature map vectors and weight maps based on the number of one or more target weight maps being less than the reference number, the weight maps including the one or more target weight maps and an additional weight map having the same weights as one of the one or more target weight maps; and generate an output feature map based on the result of the convolution operation.

[0010] According to another aspect of the inventive concept of this disclosure, an operating method is provided for an NPU device that performs convolution operations based on a convolution operation schedule. The operating method includes: adjusting the convolution operation schedule based on at least one of the number of channels in an input feature map and the number of channels in an output feature map being less than the number of reference channels; performing a weighted convolution operation on the input feature map based on the adjusted convolution operation schedule; and generating the output feature map based on the convolution operation.

[0011] According to another aspect of the inventive concept of this disclosure, a neural processing unit (NPU) device is provided, the NPU device comprising: a memory storing one or more instructions; and a processor configured to execute the one or more instructions to: determine whether the number of channels of an input feature map is less than the number of reference channels; generate an input feature map vector based on the fact that the number of channels of the input feature map is less than the number of reference channels; determine whether the number of target weight maps is less than the number of available channels of an output feature map; generate an additional weight map having the same weights as one of the target weight maps based on the fact that the number of target weight maps is less than the number of available channels of the output feature map; and perform a convolution operation on the input feature map vector using the target weight maps and the additional weight map to generate the output feature map. Attached Figure Description

[0012] Embodiments of the inventive concept will become clearer from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0013] Figure 1 This is a block diagram of the components of an NPU device according to an example embodiment;

[0014] Figure 2 and Figure 3 This is a view of the structure of a convolutional neural network according to an example embodiment;

[0015] Figure 4 This is a view used to describe the convolution operation according to the example embodiment;

[0016] Figure 5 This is a flowchart illustrating an operation method of an NPU device according to an example embodiment;

[0017] Figure 6 This is a view of the channels of an input feature map for multiple available channels, according to an example embodiment;

[0018] Figure 7 This is a block diagram of the configuration for generating an output feature map by generating an input feature map vector, according to an example embodiment;

[0019] Figure 8 It is a view of multiple input feature map patches corresponding to the weight map of the 3D structure according to the example embodiment;

[0020] Figure 9 This is a view of the input feature map vector generated based on multiple input feature map patches according to an example embodiment;

[0021] Figure 10 and Figure 11 This is a view of the weight graph and weight graph vectors based on the example embodiment;

[0022] Figure 12 This is a block diagram illustrating an example where two of multiple vector generators generate input feature map vectors.

[0023] Figure 13 This is a view illustrating an input feature map including multiple input feature map blocks according to another example embodiment;

[0024] Figure 14 It is based on Figure 13 A view of an input feature map vector generated based on multiple input feature map patches in an embodiment;

[0025] Figure 15 This is a view of the output feature map generated according to an example embodiment by performing convolution operations using multiple target weight maps;

[0026] Figure 16 It is a block diagram of the configuration for generating an output feature map based on an additional weight map according to an example embodiment;

[0027] Figure 17 It is a view that includes a collection of weight graphs, including additional weight graphs generated according to the example embodiment;

[0028] Figure 18 It is a view of the output feature map generated from a set of weight maps, including an additional weight map;

[0029] Figure 19 It is a view of the input feature map that includes multiple input feature map patches when performing a depthwise convolution operation;

[0030] Figure 20 This is a view of the configuration of the computational circuitry used to perform a comparative example of depthwise convolution operations;

[0031] Figure 21 It is a block diagram of the configuration for generating an output feature map based on an additional weight map according to an example embodiment;

[0032] Figure 22 It is a view of the input feature map vectors generated based on the same channel regions among multiple input feature map patches when performing a depthwise convolution operation; and

[0033] Figure 23 According to Figure 21 A schematic diagram of multiple computing circuits performing depthwise convolution operations in an embodiment. Detailed Implementation

[0034] In the following, embodiments of the inventive concept will be described in detail with reference to the accompanying drawings.

[0035] Figure 1 This is a block diagram of the components of a neural processing unit (NPU) device according to an example embodiment.

[0036] Reference Figure 1 The NPU device 10 can analyze input data in real time based on a neural network to extract valid information, determine the status based on the extracted information, or control the configuration of an electronic device in which the NPU device 10 is installed. According to an example embodiment, the NPU device 10 can identify the status based on the extracted information. For example, the NPU device 10 can be applied to drones, advanced driver assistance systems (ADAS), smart TVs, smartphones, medical devices, mobile devices, video display devices, measuring devices, Internet of Things (IoT) devices, etc., and can be installed on one of various types of electronic devices. However, this disclosure is not limited thereto, and similarly, the NPU device 10 can be combined with any type of electronic device. According to another example embodiment, the NPU device 10 can be implemented as a standalone device.

[0037] NPU device 10 may include at least one intellectual property (IP) block and neural network processor 300. NPU device 10 may include various types of IP blocks. For example, such as... Figure 1As shown, the IP block may include a main processor 100, random access memory (RAM) 200, input / output (I / O) devices 400, and memory 500. Furthermore, the NPU device 10 may also include other general-purpose components such as multi-format codecs (MFC), video modules (e.g., camera interfaces, Joint Image Experts Group (JPEG) processors, video processors, or mixers), 3D graphics cores, audio systems, display drivers, graphics processing units (GPUs), digital signal processors (DSPs), etc.

[0038] Various components of the NPU device 10 (e.g., main processor 100, RAM 200, neural network processor 300, input / output device 400, and memory 500) can send and receive data via the system bus 600. For example, the Advanced Microcontroller Bus Architecture (AMBA) protocol of an Advanced RISC machine (ARM) can be applied as a standard bus specification to the system bus 600. However, the inventive concept is not limited to this, and various types of protocols can be applied.

[0039] According to an example embodiment, the components of the NPU device 10 (including a main processor 100, RAM 200, a neural network processor 300, an input / output device 400, and a memory 500) are implemented as a single semiconductor chip. For example, the NPU device 10 can be implemented as a system-on-a-chip (SoC). However, the inventive concept is not limited thereto, and the NPU device 10 can be implemented using multiple semiconductor chips. In an embodiment, the NPU device 10 can be implemented as an application processor mounted on a mobile device.

[0040] The main processor 100 controls all operations of the NPU device 10, and by way of example, the main processor 100 may be a central processing unit (CPU). The main processor 100 may include a single core or may include multiple cores. The main processor 100 can process or execute programs and / or data stored in RAM 200 and memory 500. For example, the main processor 100 can control various functions of the NPU device 10 by executing programs stored in memory 500.

[0041] RAM 200 can temporarily store programs, data, or instructions. For example, under the control of the main processor 100 or the boot code, programs and / or data stored in memory 500 can be temporarily loaded into RAM 200. RAM 200 can be implemented using memory such as dynamic RAM (DRAM) or static RAM (SRAM).

[0042] The input / output device 400 can receive input data from a user or external device and can output the data processing results of the NPU device 10. The input / output device 400 can be implemented using at least one of a touchscreen panel, a keyboard, and various types of sensors. According to an embodiment, the input / output device 400 can collect information from the vicinity of the NPU device 10. For example, the input / output device 400 can include at least one of various types of sensing devices such as an imaging device, an image sensor, a LiDAR sensor, an ultrasonic sensor, and an infrared sensor, or can receive sensing signals from such a device. In an embodiment, the input / output device 400 can sense or receive image signals from outside the NPU device 10 and can convert the sensed or received image signals into image data, i.e., image frames. The input / output device 400 can store the image frames in the memory 500 or provide the image frames to the neural network processor 300.

[0043] Memory 500 is a storage area for storing data and can store, for example, an operating system (OS), various programs, and various types of data. Memory 500 may be DRAM, but is not limited to it. Memory 500 may include at least one of volatile memory and non-volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, phase-change RAM (PRAM), magnetic RAM (MRAM), resistive RAM (RRAM), or ferroelectric RAM (FRAM). Volatile memory may include DRAM, SRAM, synchronous DRAM (SDRAM), or PRAM. Furthermore, in embodiments, memory 500 may be implemented as a storage device such as a hard disk drive (HDD), solid-state drive (SSD), compact flash memory (CF), secure digital card (SD), micro-secure digital card (Micro-SD), mini-secure digital card (Mini-SD), extreme digital card (xD), or memory stick.

[0044] The neural network processor 300 can generate neural networks, train or learn neural networks, perform operations based on received input data, generate information signals based on the results of the operations, and retrain the neural network. Neural networks can include various types of neural network models, such as, but are not limited to, convolutional neural networks (CNNs), region-based CNNs (R-CNNs), region proposal networks (RPNs), recurrent neural networks (RNNs), stacked deep neural networks (S-DNNs), state-space dynamic neural networks (S-SDNs), deconvolutional networks, deep belief networks (DBNs), restricted Boltzmann machines (RBMs), fully convolutional networks, long short-term memory (LSTM) networks, and classification networks. (See reference...) Figure 2 An exemplary neural network structure is described.

[0045] Figure 2 and Figure 3 This is a view of the structure of a convolutional neural network according to an example embodiment.

[0046] Reference Figure 2 A neural network (NN) can include multiple layers L1 to Ln. The NN can be a deep neural network (DNN) or an n-layer neural network architecture. The multiple layers L1 to Ln can be implemented as convolutional layers, pooling layers, activation layers, and fully connected layers.

[0047] For example, the first layer L1 can be a convolutional layer, the second layer L2 can be a pooling layer, and the nth layer Ln is the output layer and can be a fully connected layer. The neural network NN can also include activation layers and layers that perform other types of operations.

[0048] Each of the multiple layers L1 to Ln can receive input data (e.g., image frames) or a feature map generated in the previous layer as an input feature map, and an output feature map or recognition signal REC can be generated by computing on the input feature maps. In this case, a feature map refers to data in which various features of the input data are expressed. Feature maps FM1, FM2, ..., and FMn can have, for example, a 2D matrix or a 3D matrix (or tensor) structure. Feature maps FM1, FM2, ..., and FMn can include at least one channel CH in which feature values ​​are arranged in a matrix. When feature maps FM1, FM2, ..., and FMn include multiple channels CH, the number of rows H and the number of columns W of the multiple channels CH are the same. In this case, the rows, columns, and channels CH can correspond to the x-axis, y-axis, and z-axis of coordinates, respectively. Feature values ​​arranged in specific rows and columns of a 2D matrix in the x-axis and y-axis directions (hereinafter, a matrix in this disclosure refers to a 2D matrix in the x-axis and y-axis directions) can be referred to as elements of the matrix. For example, a 4×5 matrix structure can include 20 elements.

[0049] The first layer L1 generates a second feature map FM2 by convolving a first feature map FM1 with a weighted kernel WK. The weighted kernel WK can be referred to as a filter, weight map, etc. The weighted kernel WK filters the first feature map FM1. The structure of the weighted kernel WK is similar to that of the feature map. The weighted kernel WK includes at least one channel CH in which weights are arranged in a matrix. Furthermore, the number of channels CH of the weighted kernel WK can be the same as the number of channels CH of the corresponding feature map (e.g., the first feature map FM1). The same channels CH of the weighted kernel WK and the first feature map FM1 can be convolved. For example, the first channel CH of the weighted kernel WK can be convolved with the corresponding first channel CH of the first feature map FM1. Hereinafter, the weighted kernel WK can be referred to as a weight map. When the second feature map FM2 is generated by convolving the first feature map FM1 with the weight map, the first feature map FM1 can be referred to as the input feature map, and the second feature map FM2 can be referred to as the output feature map.

[0050] As the weighted kernel WK is shifted across the first feature map FM1 using a sliding window, it can be convolved with a window (or slice) of the first feature map FM1. During each shift, each weight included in the weighted kernel WK is multiplied by the corresponding eigenvalue among all eigenvalues ​​in the region of the first feature map FM1 that overlaps with the weighted kernel WK, and all multiplication results are summed. With the convolution of the first feature map FM1 and the weighted kernel WK, one channel of the second feature map FM2 can be generated. Although... Figure 2 The diagram shows a weighted kernel WK, but multiple weighted kernels WK can be convolved with the first feature map FM1 to generate a second feature map FM2 that includes multiple channels.

[0051] The neural network according to the example embodiment can be a segmentation network such as DeepLabV3, and the NPU device 10 can perform a decoding operation after the encoding operation to recreate the image. In this case, when performing the decoding operation, the NPU device 10 can receive input feature maps of some available channels, or it can generate output feature maps of some channels. For example, the NPU device 10 can perform convolution operations using only 4 of the 32 available channels.

[0052] Reference Figure 3The input feature map (IFM) 301 can include D channels, and each channel of the input feature map can have a size of H rows and W columns (D, H, and W are natural numbers). Each of the kernels 302 has a size of R rows and S columns, and kernel 302 can include a number of channels corresponding to the number of channels (or depth) D of the input feature map 301 (R and S are natural numbers). The output feature map (OFM) 303 can be generated by a 3D convolution operation between the input feature map 301 and kernel 302, and can include Y channels depending on the convolution operation. Y can correspond to the number of kernels performing the convolution operation. The output feature map (OFM) 303 can include multiple output feature elements 304.

[0053] For reference Figure 4 This describes the process of generating an output feature map through a convolution operation between an input feature map and a kernel. The operation is performed between the input feature map 301 (all channels) and the kernel 302 (all channels). Figure 4 The 2D convolution operation described in the text enables the generation of output feature maps 303 for all channels.

[0054] Figure 4 This is a view used to describe the convolution operation according to the example embodiment.

[0055] Reference Figure 4 For ease of explanation, it is assumed that the input feature map 301 has a size of 6×6, the kernel 302 has a size of 3×3, and the output feature map 303 has a size of 4×4; however, the inventive concept is not limited to this. Neural networks can be implemented using feature maps and kernels of various sizes. Furthermore, the values ​​defined in the input feature map 301, kernel 302, and output feature map 303 are exemplary values, and the embodiments of this disclosure are not limited thereto.

[0056] Kernel 302 can perform convolution operations while sliding within a 3×3 window unit in the input feature map 301. The convolution operation can be represented as an operation used to obtain each feature data of the output feature map 303, wherein each feature data of the output feature map 303 is obtained by summing all values ​​obtained by multiplying each feature data of the window of the input feature map 301 by each weight corresponding to the position of kernel 302. The data included in the window of the input feature map 301 and multiplied by the weights can be referred to as extracted data extracted from the input feature map 301. More specifically, kernel 302 can first perform convolution operations using the first extracted data 301a of the input feature map 301. That is, the feature data 1, 2, 3, 4, 5, 6, 7, 8, and 9 of the first extracted data 301a are multiplied by -1, -3, 4, 7, -2, -1, -5, 3, and 1, respectively, the corresponding weights of kernel 302. As a result, we obtain -1, -6, 12, 28, -10, -6, -35, 24, and 9. Next, we add -1, -6, 12, 28, -10, -6, -35, 24, and 9 to generate 15, which is the result of adding all the obtained values ​​-1, -6, 12, 28, -10, -6, -35, 24, and 9. Thus, we can determine that the feature element 304a in the first row and first column of the output feature map 303 is 15. Here, the feature element 304a in the first row and first column of the output feature map 303 corresponds to the first extracted data 301a. Similarly, by performing a convolution operation between the second extracted data 301b of the input feature map 301 and the original kernel 302, we can determine that the feature element 304b in the first row and second column of the output feature map 303 is 4. Finally, by performing a convolution operation between the 16th extracted data 301c, which is the last extracted data of the input feature map 301, the feature element 304c in the fourth row and fourth column of the output feature map 303 can be determined to be 11.

[0057] In other words, the convolution operation between an input feature map 301 and a kernel 302 can be processed by repeatedly performing the multiplication of the extracted data of the input feature map 301 with the corresponding weights of the original kernel 302 and summing the multiplication results, and the output feature map 303 can be generated through the convolution operation.

[0058] Figure 4The convolution operation of the input feature map 301 with a 2D structure is shown. However, the input feature map 301 according to the example embodiment has a 3D structure, and the NPU device 10 performs a convolution operation on the input feature map 301 and kernel 302 corresponding to the same channel, thereby providing an output feature map 303 for the input feature map 301 with a 3D structure including multiple channels. Furthermore, the NPU device 10 can output an output feature map 303 by performing a convolution operation on one kernel 302 and the input feature map 301. However, the NPU device 10 can output an output feature map 303 by performing convolution operations on multiple kernels 302 and the input feature map 301. Here, when multiple kernels 302 are present, the number of channels in the output feature map 303 can correspond to the number of kernels.

[0059] Figure 5 This is a flowchart illustrating an operation method of the NPU device 10 according to an example embodiment.

[0060] Reference Figure 5 When the number of channels in the input feature map is less than a specific number of reference channels, when the NPU device 10 performs a depthwise convolution operation, and when the number of channels in the output feature map is less than the reference number due to the number of target weight maps being less than the reference number, the NPU device 10 can perform convolution operations using as many available channels as possible by generating an input feature map vector or an additional weight map. The number of reference channels and the reference number can be preset numbers.

[0061] In operation S10, the NPU device 10 can compare the number of channels in the input feature map with the number of reference channels. In operation S20, when the number of channels in the input feature map is less than or equal to the number of reference channels, an input feature map vector can be generated. The NPU device 10 according to the example embodiment can determine whether to generate an input feature map vector in the corresponding layer based on the result of comparing the number of reference channels with the number of channels in the input feature map, but this disclosure is not limited thereto. Thus, according to another example embodiment, the layer performing the convolution operation can be set by generating the input feature map vector.

[0062] In operation S30, the NPU device 10 can determine whether to perform a depthwise convolution operation, and can generate an input feature map vector in operation S20 based on the determination to perform a depthwise convolution operation. The input feature map vector can be a vector generated by concatenating at least some of a plurality of input feature map patches, and the input feature map patches can include elements corresponding to at least one input value. For example, the input feature map vector can be a vector generated by concatenating all of a plurality of input feature map patches, or it can be a vector generated by concatenating some input feature map patches located in the same channel region among a plurality of input feature map patches. See below for further details. Figures 6 to 18A detailed description of an embodiment for generating input feature map vectors.

[0063] In operation S40, NPU device 10 may determine whether to generate an additional weight map. For example, NPU device 10 may determine whether the number of weight maps is less than a reference number. In operation S50, when the number of weight maps is less than the reference number, NPU device 10 may generate at least one additional weight map having the same weights as the target weight map. (Refer to...) Figure 4 The number of weight maps can be the number of kernels performing convolution operations on the input feature maps, and the number of weight maps can correspond to the number of channels in the output feature maps. According to an example embodiment, the NPU device 10 can determine whether to generate additional weight maps in the corresponding layer based on a comparison of the number of weight maps and the number of reference maps, but this disclosure is not limited thereto. Thus, according to another example embodiment, the layer performing the convolution operation can be set by generating additional weight maps.

[0064] In operation S60, when generating the input feature map vector, the NPU device 10 can perform convolution operations using multiple weight maps. More specifically, the NPU device 10 can generate a weight map vector based on the weight maps using the method for generating the input feature map vector from the input feature map, and can perform a dot product operation on the input feature map vector and the weight map vector.

[0065] When the NPU device 10 generates an additional weight map, it can perform a convolution operation on the input feature map or input feature map vector using the target weight map and the additional weight map. For example, when the NPU device 10 generates an input feature map vector, it can perform a convolution operation on the input feature map vector using a weight map vector based on the target weight map and the additional weight map. However, when the NPU device 10 does not generate an input feature map vector, it can still perform a convolution operation on the input feature map using the target weight map and the additional weight map. (See below for further details.) Figures 19 to 22 An embodiment is described in which the NPU device 10 generates an additional weight map to perform convolution operations.

[0066] In operation S70, the NPU device 10 can generate the result of performing convolution operations as elements of the output feature map, and can generate an output feature map with multiple output feature map elements. The number of channels in the output feature map can be configured to be the same as the number of weight maps, and when the NPU device 10 generates an additional weight map, the NPU device 10 can output an output feature map that includes more channels than the target weight map.

[0067] Figure 6 This is a view of the channels of an input feature map for multiple available channels, according to an example embodiment.

[0068] Reference Figure 6The NPU device 10 of this invention can generate an input feature map 301 including multiple channels to perform convolution operations. The input feature map 301 can be an output feature map from another layer, and the NPU device 10 can use the output feature map from another layer as the input feature map 301 to perform convolution operations. However, this disclosure is not limited to this, and therefore, the input feature map 301 may not originate from a previous layer. The NPU device 10 can ensure that the hardware space or hardware resources used to perform convolution operations are available channels C, and can most efficiently perform neural network operations when using all available channels C to perform convolution operations on the input feature map 301. For example, the NPU device 10 can allocate hardware resources for performing convolution operations as available channels C, and can most efficiently perform neural network operations when using all available channels C to perform convolution operations on the input feature map 301. Figure 6 In this embodiment, although the NPU device 10 ensures 16 channels as available channels C, the NPU device 10 performs operations on an input feature map that includes 4 channels, and thus can perform convolution operations at 25% of the maximum performance.

[0069] The NPU device 10 can load a weight map 302 with a 3D structure to perform convolution operations on an input feature map 301, which includes a finite number of channels among the available channels C, wherein the 3D structure has a number of channels corresponding to the input feature map 301. The NPU device 10 can perform convolution operations on some elements of the weight map 302 and the input feature map 301 to generate an output value corresponding to an element in the output feature map. (Refer to...) Figure 6 The input feature map, including four channels, can contain 256 (8*8*4) elements. The NPU device 10 can perform convolution operations on 36 (3*3*4) elements from these 256 (8*8*4) elements that correspond to the weight map 302, to generate an output feature map element. In this case, the NPU device 10 can perform convolution operations on one input feature map block in one loop. The input feature map block can be an element line formed in the channel direction, and the number of elements included in the input feature map block can correspond to the number of channels in the input feature map. (See reference...) Figure 6 In one embodiment, the element lines formed in the channel direction in each row and each column can be an input feature map block. The NPU device 10 can perform vector dot product operations for nine consecutive loops based on a weight map 302 comprising three rows and three columns to generate an output feature map element.

[0070] When the input feature map 301 is configured with a limited number of channels, the NPU device 10 according to the example embodiment can generate an input feature map vector based on the input feature map blocks, and can generate an output feature map using as many channels as possible by performing a convolution operation on the input feature map vector. Therefore, the NPU device 10 according to the example embodiment can generate an output feature map by performing a convolution operation with fewer cycles compared to performing a convolution operation on an input feature map 301 configured with a limited number of channels. Hereinafter, reference will be made to... Figures 7 to 14 An embodiment of the NPU device 10 is described, which generates an output feature map for an input feature map configured with a limited number of channels.

[0071] Figure 7 This is a block diagram of a configuration for generating an output feature map by generating an input feature map vector, according to an embodiment.

[0072] Reference Figure 7 The NPU device 10 may include a buffer, and the buffer may include multiple vector generators 11 that generate an input feature map vector IFMV for the generated input feature map. The NPU device 10 may determine whether to activate the multiple vector generators 11 based on the number of channels in the input feature map. For example, the NPU device 10 may determine which vector generator 11 to activate based on the ratio of the number of channels in the input feature map to the number of available channels. (See also...) Figure 7 When the number of available channels is 16 and the number of channels in the input feature map is 4, the NPU device 10 can activate the first vector generator 11a among the four vector generators 11. The first vector generator 11a can generate an input feature map vector IFMV based on the input feature map blocks corresponding to the first to fourth channels among multiple input feature map blocks.

[0073] According to an example embodiment, multiple computing circuits 12 can receive input feature map vectors IFMV from a vector generator 11, and can perform convolution operations on the weight map corresponding to each computing circuit 12 and the broadcast input feature map vectors IFMV. The computing circuits 12 may include arithmetic circuits or accumulator circuits. For example, a first computing circuit 12a can receive a first input feature map vector IFMV1 generated from a first vector generator 11a, and can generate an output feature map by performing a convolution operation on the first input feature map vector IFMV1 and the weight map. The number of channels in the generated output feature map can be determined based on the first input feature map vector IFMV1 and the number of weight maps on which the convolution operation is performed.

[0074] The NPU device 10 may include a plurality of computing circuits 12, and each of the computing circuits 12 may generate a plurality of output feature maps by performing convolution operations in parallel. (Refer to...) Figure 7The NPU device 10 may include four computing circuits 12, and each of the computing circuits 12 can generate four output feature maps by performing convolution operations based on different weight maps. Furthermore, each of the computing circuits 12 can generate multiple output feature maps in parallel based on multiple weight maps. For example, the first computing circuit 12a can generate first to fourth output feature maps based on first to fourth weight maps, and in this way, the four computing circuits 12 can generate 16 output feature maps.

[0075] Figure 8 It is a view of multiple input feature map patches BL corresponding to the weight map of the 3D structure according to the example embodiment, and Figure 9 This is a view of the input feature map vector IFMV generated based on multiple input feature map patches BL according to the example embodiment.

[0076] Figure 8 Only a portion of the input feature map is shown, representing the part in which the convolution operation is performed to generate one output feature map element. The input feature map may include multiple input feature map patches BL, and each input feature map patch BL may be a line of elements in the channel direction that includes at least one input feature map element. The number of elements included in an input feature map patch BL may correspond to the number of channels in the input feature map. Figure 6 The NPU device 10 of the comparative embodiment can perform a convolution operation on an input feature map block BL in one cycle, and generate an output feature map element by performing nine cycles of convolution operation.

[0077] Reference Figure 9 According to an embodiment, the NPU device 10 can generate a single input feature map vector IFMV from multiple input feature map blocks BL. For example, when nine input feature map blocks BL1 to BL9 are needed to generate one output feature map element, the NPU device 10 can generate an input feature map vector IFMV by combining the nine input feature map blocks BL1 to BL9 together. The NPU device 10 can perform a convolution operation on the elements of the generated input feature map vector IFMV corresponding to the number of available channels within a single loop. Figure 9 In one embodiment, the NPU device 10 can perform convolution operations on four input feature maps BL1 to BL4 in one loop, and can perform convolution operations on nine input feature maps BL1 to BL9 in three loops.

[0078] According to the comparative example, the hardware of NPU device 10 has the capability to perform convolution operations corresponding to the number of available channels within a single loop. However, when the number of channels in the input feature map is limited, NPU device 10 may only perform convolution operations on a limited number of input feature map elements. Therefore, multiple loops of convolution operations are required to generate a single output feature map element. According to an embodiment, when the number of channels in the input feature map is limited, NPU device 10 can generate an input feature map vector IFMV to perform convolution operations on multiple input feature map blocks BL within a single loop, thereby efficiently performing convolution operations on the available channels. Therefore, NPU device 10 according to the embodiment can perform fewer convolution operations to generate a single output feature map element.

[0079] Figure 10 and Figure 11 This is a view of the weight graph and weight graph vectors according to an embodiment.

[0080] Reference Figure 10 The weight map may include multiple weight map blocks (WBLs), and the size of the weight map may correspond to the size of the input feature map. The NPU device according to the embodiment may further include a weight vector generator that performs the same operations as the vector generator 11, and the weight vector generator may be configured with the same hardware as the vector generator 11 that generates the input feature map vector to generate the weight map vector, but is not limited thereto, and may be configured with different hardware. The NPU device 10 performs a convolution operation by multiplying the input feature map having a 3D structure and the corresponding elements of the input feature map and weight map in the weight map, and summing the results of the multiplication. As described above. Figure 8 As described, the NPU device 10 can perform a convolution operation on an input feature map patch BL and a weight map patch WBL within one loop, and according to... Figure 10 In one embodiment, an output feature map element can be generated by performing nine cycles of convolution operations.

[0081] Reference Figure 11 The NPU device 10 can generate a weight map vector based on the weight map in the same way as generating the input feature map vector IFMV, so as to perform convolution operations using the input feature map vector IFMV. For example, when the NPU device 10 combines nine input feature map blocks BL1 to BL9 together to generate an input feature map vector IFMV, the NPU device 10 can generate a weight map vector by concatenating the nine weight map blocks WBL1 to WBL9 together in the order in which the input feature map blocks BL1 are concatenated. The NPU device 10 can generate an output feature map element by performing convolution operations on the nine input feature map blocks BL1 to BL9 and the nine weight map blocks WBL1 to WBL9 for up to three loops.

[0082] Figure 12 This is a block diagram illustrating an example of generating an input feature map vector IFMV from two of a plurality of vector generators 11.

[0083] Reference Figure 12 The NPU device 10 can activate two or more of the multiple vector generators 11 according to the number of channels in the input feature map. Figures 7 to 11 This illustrates an example embodiment of generating the input feature map vector IFMV by activating only one of the multiple vector generators 11. However, according to Figure 12 The example embodiment shown illustrates that two or more of a plurality of vector generators 11 can be activated to generate an input feature map vector IFMV. Each of the plurality of vector generators 11 may correspond to a channel region including some channels of the input feature map, and the NPU device 10 determines whether to activate the corresponding vector generator 11 based on the presence or absence of input feature map elements in the corresponding channel region. That is, the NPU device 10 may determine the vector generator 11 to be activated based on the ratio of the number of channels of the input feature map to the number of available channels. For example, in Figure 12 In this process, vector generators 11a and 11b can be activated to generate the input feature map vector IFMV. For example, vector generator 11a can generate the input feature map vector IFMV1, and vector generator 11b can generate the input feature map vector IFMV2. Subsequently, the input feature map vectors IFMV1 and IFMV2 can be combined to generate the input feature map vector IFMV. (The above has already been referred to...) Figure 7 The vector generator 11, which generates the input feature map vector IFMV and outputs the generated input feature map vector IFMV to the computing circuit 12, is described, so its detailed description will not be provided here.

[0084] Figure 13 It is based on and Figure 8 The example embodiments of different embodiments include views of input feature maps comprising multiple input feature map blocks BL, and Figure 14 It is based on Figure 13 The embodiment is a view of the input feature map vector IFMV generated based on multiple input feature map patches BL.

[0085] Reference Figure 12 and Figure 13 When the number of available channels is 16 and the number of channels in the input feature map is 5, the NPU device 10 can activate the first vector generator 11a and the second vector generator 11b among the four vector generators 11a, 11b, 11c, and 11d. For example, according to... Figure 13In the input feature map, the first vector generator 11a can generate a first input feature map vector IFMV1 based on the input feature map elements of the first channel CH1 to the fourth channel CH4, and the second vector generator 11b can generate a second input feature map vector IFMV2 based on the input feature map elements of the fifth channel CH5 to the eighth channel CH8. The first vector generator 11a and the second vector generator 11b can broadcast the generated first input feature map vector IFMV1 and second input feature map vector IFMV2 to multiple computing circuits 12a, 12b, 12c, and 12d.

[0086] Multiple computational circuits 12 can receive multiple input feature map vectors IFMV generated from multiple vector generators 11, and can generate an input feature map vector IFMV for performing convolution operations by combining the multiple input feature map vectors IFMV. (Refer to...) Figure 14 When multiple input feature map vectors IFMV are received, each of the multiple computing circuits 12 can combine multiple input feature map vectors IFMV in units of input feature map blocks BL. For example, the input feature map vectors IFMV may include partial input feature map vectors IFMV corresponding to input feature map blocks BL, and partial input feature map vectors IFMV generated by different vector generators 11 can be cross-linked.

[0087] according to Figure 13 and Figure 14 In one embodiment, the first vector generator 11a can generate a first input feature map vector IFMV1 based on the input feature map elements corresponding to the first channel CH1 to the fourth channel CH4 in the first input feature map blocks BL1 to the ninth input feature map blocks BL9. In this case, the first vector generator 11a can generate a first partial input feature map vector based on the input feature map elements corresponding to the first channel CH1 to the fourth channel CH4 in the first input feature map block BL1, and can generate the second to ninth partial input feature map vectors in this way.

[0088] According to an embodiment, when the computing circuit 12 receives an input feature map vector IFMV, which includes partial input feature map vectors, from multiple vector generators 11, the computing circuit 12 can combine the partial input feature map vectors on a per-input feature map block BL basis. For example, the computing circuit 12 can perform a convolution operation by combining partial input feature map vectors corresponding to the first channel CH1 to the fourth channel CH4 in the first input feature map block BL1 received from the first vector generator 11a and partial input feature map vectors corresponding to the fifth channel CH5 to the eighth channel CH8 in the first input feature map block BL1 received from the second vector generator 11b, and then combining the partial input feature map vectors corresponding to the second input feature block BL2. Therefore, the computing circuit 12 can perform a convolution operation based on the input feature map vector IFMV generated by the multiple vector generators 11.

[0089] However, the NPU device 10 according to the embodiment is not limited to the one according to the embodiment. Figure 14 In this embodiment, the input feature map block BL is a unit combination of the input feature map vector IFMV received from the vector generator 11, or the input feature map vector IFMV can be combined on a unit basis with the vector generator 11. For example, the NPU device 10 can perform a convolution operation by connecting a second input feature map vector IFMV2 received from the second vector generator 11b to a first input feature map vector IFMV1 received from the first vector generator 11a. According to the example embodiment, since the number of channels of the weight map to which the convolution operation is performed corresponds to the number of channels of the input feature map, the NPU device 10 can also generate the weight map vector in the same manner as the method for generating the input feature map vector IFMV. Furthermore, since the above has already referred to Figure 10 and Figure 11 The method for generating weighted graph vectors has been described, and will not be described in detail here.

[0090] Figure 15 This is a view of the output feature map generated according to an example embodiment by performing a convolution operation using multiple target weight maps.

[0091] Reference Figure 15 The NPU device 10 can generate an output feature map with a number of channels corresponding to the number of weight maps by performing convolution operations on the input feature map and multiple weight maps WM1 to WM4. The NPU device 10 can also generate the output feature map by performing convolution operations between the input feature map and weight maps having the same number of channels as the input feature map. For example, the NPU device 10 can generate an output feature map with four channels by performing convolution operations using four weight maps WM1 to WM4.

[0092] The hardware of the NPU device 10 according to the embodiment can perform sufficient computation to generate an output feature map as many as the number of available channels. However, when the number of weight maps is limited, the NPU device 10 may generate an output feature map with fewer channels than the number of available channels. That is, in accordance with... Figure 15 In one embodiment, the hardware of the NPU device 10 can generate an output feature map with 16 channels based on 16 weight maps, while the NPU device 10 can generate an output feature map with 4 channels within the same time period by performing convolution operations based on 4 weight maps. When the NPU device 10... Figure 15 When performing convolution operations based on four weight graphs as in the embodiment, the NPU device 10 is inefficient in performing convolution operations because it only handles 25% of the computational load compared to its maximum performance.

[0093] According to an embodiment, the NPU device 10 can generate an additional weight map with the same weights as the target weight map (which is an existing weight map), and the hardware of the NPU device 10 can be effectively utilized by performing convolution operations on different input weight map blocks using the target weight map and the additional weight map.

[0094] Figure 16 It is a block diagram of the configuration for generating an output feature map based on an additional weight map according to an embodiment.

[0095] Reference Figure 16 When the number of target weight maps is less than the reference number, multiple vector generators 11a, 11b, 11c, and 11d included in the buffer of the NPU device 10 can provide different input feature map blocks BL to the computation circuits 12a, 12b, 12c, and 12d one-to-one. When vector generator 11 determines that the number of channels in the input feature map is greater than the number of reference channels, or determines that no depthwise convolution operation will be performed, vector generator 11 can generate an input feature map vector IFMV by merging at least some of the multiple input feature map blocks BL. In other words, vector generator 11 can provide different input feature map blocks BL to the computation circuit 12 corresponding to each vector generator 11. When vector generator 11 determines that the number of channels in the input feature map is less than or equal to the number of reference channels, or when vector generator 11 determines that a depthwise convolution operation will be performed, vector generator 11 can generate an input feature map vector IFMV based on at least some of the multiple input feature map blocks BL, and can provide the input feature map vector IFMV to the computation circuit 12.

[0096] According to the comparative example, the NPU device 10 can determine which computing circuits to activate from among a plurality of computing circuits 12 based on the number of target weight maps. For example, each computing circuit 12 can perform convolution operations on multiple weight maps in parallel. Each computing circuit 12 can perform convolution operations on four weight maps, and when the number of target weight maps to be convolved in parallel is four or less, the NPU device 10 can activate any one of the four computing circuits 12 to perform the convolution operation. That is, the NPU device 10 according to the comparative example disables the other three computing circuits 12 and generates the output feature map by one computing circuit 12, which may take four times longer than the case where all computing circuits 12 are activated.

[0097] According to an embodiment, the NPU device 10 can generate an output feature map using the deactivated computing circuitry 12 in the comparative example by generating at least one additional weight map having the same weights as the target weight map. The generated additional weight map can be assigned such that the convolution operation is performed in a different computing circuitry 12 than the one that performs the convolution operation on the target weight map, and the input feature map blocks BL or input feature map vectors IFMV sent to the computing circuitry 12 from the plurality of vector generators 11 can include different input feature map elements.

[0098] according to Figure 15 and Figure 16 When the number of target weight maps is 4 and the number of available channels in the output feature map is 16, the hardware of the NPU device 10 can be in a state capable of performing convolution operations on 16 weight maps. The NPU device 10 can generate 12 additional weight maps by generating three additional weight maps, each with the same weights as the target weight map, for each of the four target weight maps. Therefore, the 16 weight maps, including the additional weight maps and the target weight maps, can be assigned to each of the four computing circuits 12, and based on the assigned weight maps, while generating 4 output feature map elements in the comparative embodiment, the multiple computing circuits 12 can generate 16 output feature map elements. At this time, because the input feature map blocks BL or input feature map vectors IFMV received by each computing circuit 12 are different from each other, the four computing circuits 12 can generate 16 different output feature map elements.

[0099] Figure 17 It is a view that includes a set of weight graphs, including additional weight graphs generated according to the embodiments, and Figure 18 It is a view of the output feature map generated from a set of weight maps, including an additional weight map.

[0100] Reference Figure 17An additional weight map corresponding to the target weight map can be generated based on the number of available channels in the output feature map. The NPU device 10 can generate the additional weight map by determining whether to generate it during the inference process of generating inference data based on the input data. However, according to an embodiment, the NPU device 10 can determine whether to generate the additional weight map based on the number of weight maps generated during the training process of generating the weight map.

[0101] The NPU device 10 can generate additional weight maps such that the number of target weight maps and additional weight maps is maximized, which is less than or equal to the number of available channels in the output feature map. For example, when the number of available channels in the output feature map is 16 and the number of target weight maps is 4, since a maximum of 12 additional weight maps can be generated, the NPU device 10 can generate three additional weight maps for each of the four target weight maps. Target weight maps and additional weight maps with different weights can be assigned as a set of weight maps to each computing circuit 12. Therefore, the set of weight maps assigned to each computing circuit 12 can be a set of weight maps with the same weight maps as the set of weight maps assigned to other computing circuits 12.

[0102] Reference Figure 17 and Figure 18 The NPU device 10 can generate an output feature map based on a target weight map and an additional weight map. For example, the NPU device 10 can generate a first output feature map block O1 by performing a convolution operation on a first input feature map block I1 and a first weight map set SET1 in the input feature map. For example, the first input feature map block I1 can be an input feature map block corresponding to the first row and first column, the first row and second column, the second row and first column, and the second row and second column in a 3*3 input feature map, and the first computing circuit 12a can receive the first input feature map block I1 from the first vector generator 11a. The first computing circuit 12a that receives the first input feature map block I1 can generate the first output feature map block O1 based on the first weight map set SET1. In this way, the second computing circuit 12b to the fourth computing circuit 12d can generate the second output feature blocks O2 to the fourth output feature blocks O4 by performing convolution operations in parallel based on the second input feature map blocks I2 to the fourth input feature blocks I4.

[0103] Figure 18 This demonstrates generating output feature map patches without generating an input feature map vector IFMV for the input feature map patches. However, when the number of channels in the input feature map is as... Figures 7 to 14 When restricted, the NPU device 10 can perform convolution operations based on a weight map including an additional weight map by generating an input feature map vector IFMV. In other words, it describes... Figures 7 to 14 Processing when the input feature map channels are limited Figures 15 to 18The processing is performed when the number of channels in the input feature map is limited. However, when the number of channels in the input feature map and the number of channels in the output feature map are limited, the NPU device 10 according to the embodiment can generate the output feature map by performing both processes.

[0104] Figure 19 It is a view of the input feature map that includes multiple input feature map patches (BL) when performing a depthwise convolution operation, and Figure 20 This is a view of the configuration of computational circuit 12 for performing a comparative example of depthwise convolution operations.

[0105] Reference Figure 19 Even if the number of channels in the input feature map equals the number of available channels in the NPU device 10, the NPU device 10 of this invention can still generate the input feature map vector IFMV when a depthwise convolution operation is requested. The depthwise convolution operation can be a method for computing neural networks that reduces computational cost and enables real-time operation. The depthwise convolution operation can refer to performing a convolution operation after generating a 2D weight map by separating each channel from the weight map of the 3D structure. In other words, when the NPU device 10 performs a depthwise convolution operation, the NPU device 10 may not perform convolution in the channel direction, but may only perform convolution operations in the spatial direction.

[0106] Reference Figure 20 When the NPU device 10 according to the comparative embodiment performs a depthwise convolution operation, each computing circuit 12 can generate an output feature map for an input feature map block BL by performing convolution operations at different timings based on weight maps with different weights. For example, when the first input feature map block BL1 is provided to the four computing circuits 12, the NPU device 10 can perform a convolution operation on the first channel region and the first weight map set of the first input feature map block BL1 by activating the first computing circuit 12a at the first timing. At the second timing after the first timing, the NPU device 10 can perform a convolution operation on the second channel region and the second weight map set of the first input feature map block BL1 by activating the second computing circuit 12b. Similarly, the NPU device 10 can output multiple output feature map block elements for the first input feature map block BL by performing convolution operations on the third channel region and the fourth channel region in the third computing circuit 12c and the fourth computing circuit 12d at the third and fourth timings, respectively. For example, the first channel region can be the first channel CH1 to the fourth channel CH4, and the fourth channel region can be the thirteenth channel CH13 to the sixteenth channel CH16.

[0107] In this case, the number of elements in the output feature map can correspond to the number of weight maps included in the multiple computational circuits 12, and when performing depthwise convolution, it can correspond to the number of channels in the input feature map. That is, the number of channels in the input feature map can be the same as the number of channels in the output feature map.

[0108] According to the comparative example, the NPU device 10 generates output feature map elements for an input feature map block BL by activating only one computing circuit and deactivating the remaining computing circuits so that they do not perform operations. On the other hand, the NPU device 10 of the embodiment can generate multiple output feature maps simultaneously by performing a convolution operation on a second input feature map block BL2 at the same timing point where a convolution operation is performed on a first input feature map block BL1.

[0109] Figure 21 This is a block diagram illustrating the configuration for generating an output feature map by generating an input feature map vector IFMV when performing a depthwise convolution operation. Figure 22 It is a view of the input feature map vectors IFMV generated on the same channel regions in multiple input feature map blocks BL when performing depthwise convolution operations.

[0110] Reference Figure 21 Multiple vector generators 11 can generate an input feature map vector IFMV based on input feature map elements corresponding to partial channel regions in multiple input feature map blocks BL1 to BL9. More specifically, each vector generator 11 can generate the input feature map vector IFMV using input feature map elements corresponding to preset channel regions. (See also...) Figure 22 The first vector generator 11a can generate the first input feature map vector IFMV1 by concatenating the input feature map elements corresponding to the first channel CH1 to the fourth channel CH4 in the first input feature map blocks BL1 to the ninth input feature map block BL9. Similarly, as... Figure 19 In one embodiment, when all channels of the input feature map are filled with up to the available channels, four input feature map vectors IFMV1 to IFMV4 can be generated from the four vector generators 11 respectively.

[0111] According to the comparison example, because the same input feature map block BL is provided to each of the computation circuits 12, it is necessary to wait until some of the input feature map blocks BL are convolved by each of the computation circuits 12. Conversely, each of the vector generators 11 according to the embodiment can provide the corresponding computation circuit 12 with input feature map vectors IFMV1 to IFMV4, respectively corresponding to different channel regions.

[0112] Figure 23 It shows the basis Figure 21A diagram of multiple computing circuits 12 performing depthwise convolution operations in an embodiment.

[0113] Reference Figure 23 ,and Figure 20 Unlike the previous comparison example, NPU device 10 can perform convolution operations on multiple input feature map blocks BL without interrupting the computation circuit 12. The computation circuit 12 can receive input feature map vectors IFMV from respective vector generators 11. The input feature map vectors IFMV can each include input feature map elements from the same channel region in multiple input feature map blocks BL, as shown above. Figure 22 As stated above.

[0114] The computing circuit 12 of the NPU device 10 can perform convolution operations on the input feature map vector IFMV at all timings to generate output feature map elements for multiple input feature map blocks BL respectively. For example, the computing circuit 12 can receive the first input feature map vectors IFMV1 to IFMV4 generated for different channel regions based on the first input feature map blocks BL1 to BL4 respectively. The first computing circuit 12a that receives the first input feature map vector IFMV1 can perform convolution operations on the input feature map elements in the first input feature map block BL1 corresponding to the first channel CH1 to the fourth channel CH4 at a first timing. In the same manner, the second computing circuits 12b to 12d can perform convolution operations on the input feature map elements in the first input feature map block BL1 corresponding to the fifth channel CH5 to the eighth channel CH8, the ninth channel CH9 to the twelfth channel CH12, and the thirteenth channel CH13 to the sixteenth channel CH16 at a first timing. That is, the convolution operation performed by the NPU device 10 according to the comparative example at the second to fourth timing can be performed by the NPU device 10 according to the embodiment of the present invention at the first timing.

[0115] When the NPU device 10 according to the comparative embodiment is based on 16 weight maps, according to Figure 19 When performing depthwise convolution operations on an input feature map including 16 channels in the embodiment, the NPU device 10 can generate 16 output feature map elements for one input feature map block BL in four timing periods. On the other hand, the NPU device 10 of the present invention only needs to perform convolution operations in one timing period to generate 16 output feature map elements identical to the output feature map elements of the comparative example by generating an input feature map vector IFMV, and can generate 64 output feature map elements for four input feature map blocks BL in four timing periods.

[0116] According to one or more exemplary embodiments of the present disclosure, one or more components or elements of an NPU device may be implemented as hardware. However, the present disclosure is not limited thereto, and therefore, according to exemplary embodiments, one or more components or elements of an NPU device may be implemented as software or a combination of hardware and software. For example, according to exemplary embodiments, each of a vector generator, a weight graph generator, etc., may be implemented by hardware, a software module, or a combination of hardware and software.

[0117] Although the inventive concept has been specifically shown and described with reference to exemplary embodiments thereof, it will be understood that various changes in form and detail may be made therein without departing from the spirit and scope of the appended claims.

Claims

1. A method for generating an output feature map based on an input feature map, the method comprising: The neural processing unit device generates an input feature map vector for multiple input feature map blocks based on the fact that the number of channels in the input feature map is less than the number of reference channels. The neural processing unit device generates one or more additional weight maps with the same weights as the one or more target weight maps, based on the fact that the number of one or more target weight maps is less than the number of references. The neural processing unit device performs a convolution operation between the input feature map and the weight map, the weight map including the one or more target weight maps and the one or more additional weight maps; and The neural processing unit device generates an output feature map based on the convolution operation.

2. The method according to claim 1, wherein, The input feature map vector is a vector information generated based on the multiple input feature map blocks in the three-dimensional input feature map that correspond to the size of the weight map.

3. The method according to claim 2, wherein, Each of the plurality of input feature map blocks includes: A data block corresponds to one or more channels among multiple available channels that contain input values, and Generating the input feature map vector includes: Each of the plurality of input feature map blocks is generated as a partial input feature map vector.

4. The method according to claim 3, wherein, Generating the input feature map vector includes: The input feature map vector is generated by combining multiple partial input feature map vectors corresponding to each of the multiple input feature map blocks in the order of convolution operations.

5. The method according to claim 4, wherein, The length of the input feature map vector is determined based on the ratio of the number of channels containing the input value to the number of available channels.

6. The method according to claim 2, wherein, Generating the input feature map vector includes: Based on the determination of performing depthwise convolution operations, the input feature map vector is generated using the input values ​​corresponding to the same channel in the plurality of input feature map blocks.

7. The method according to claim 2, wherein, Performing the convolution operation includes: Generate a weight vector with a size corresponding to the input feature map vector based on the weight map; and Perform a dot product operation on the weight vector and the input feature map vector.

8. The method according to claim 1, wherein, Generating the one or more additional weight graphs includes: The number of additional weight maps to be generated is determined based on the ratio of the number of the one or more target weight maps to the number of available channels.

9. The method according to claim 1, wherein, Performing the convolution operation includes: Convolution operations are performed on different input feature map blocks in the input feature map based on the one or more target weight maps and the one or more additional weight maps.

10. A neural processing unit device, comprising: A vector generator configured to generate input feature map vectors for multiple input feature map patches based on the fact that the number of channels in the input feature map is less than the number of reference channels. A weight map generator configured to generate one or more additional weight maps having the same weights as the one or more target weight maps, based on the fact that the number of one or more target weight maps is less than the number of references. as well as The computing circuit is configured as follows: Perform a convolution operation between one of the plurality of input feature map blocks and the input feature map vector and a weight map, wherein the weight map includes the one or more target weight maps and the one or more additional weight maps, and An output feature map is generated based on the result of the convolution operation.

11. The neural processing unit device according to claim 10, wherein, The input feature map vector is a vector information generated based on the multiple input feature map blocks in the three-dimensional input feature map that correspond to the size of the weight map.

12. The neural processing unit device according to claim 11, wherein, The vector generator performs a depthwise convolution operation based on a predetermined condition, generating the input feature map vector using the input values ​​corresponding to the same channel in the plurality of input feature map blocks.

13. The neural processing unit device according to claim 10, wherein, The computing circuit performs convolution operations on different input feature map blocks in the input feature map based on the one or more target weight maps and the one or more additional weight maps.

14. A method of operating a neural processing unit device, wherein the neural processing unit device schedules and executes convolution operations based on convolution operations, the method comprising: The convolution operation schedule is adjusted based on the fact that at least one of the number of channels in the input feature map and the number of channels in the output feature map is less than the number of reference channels. The weight map convolution operation is performed on the input feature map based on the adjusted convolution operation schedule; as well as The output feature map is generated based on the convolution operation. The adjustment of the convolution operation schedule includes: Given that the number of one or more target weight maps is less than the number of reference maps, generate one or more additional weight maps with the same weights as the target weight maps; and The convolution operation schedule is adjusted to perform convolution operations on different input feature map blocks using the target weight map and the one or more additional weight maps.

15. The operating method according to claim 14, wherein, Adjusting the convolution operation schedule also includes: Based on the fact that the number of channels in the input feature map is less than the number of channels in the first reference map, an input feature map vector is generated for multiple input feature map blocks; and The convolution operation schedule is adjusted based on the length of the input feature map vector relative to the number of available channels.

16. The operating method according to claim 14, wherein, Adjusting the convolution operation schedule includes: Based on the determination of the depthwise convolution operation, the input feature map vector is generated using the input values ​​corresponding to the same channel in multiple input feature map blocks.

17. The operating method according to claim 14, wherein, When the number of target weight maps is less than the reference number, the number of channels in the output feature map is increased by generating one or more additional weight maps.

Images