Core of neural processing units and method to configure the same
By optimizing NPU data paths with a 2-to-1 multiplexer and adder tree structure, the NPU array achieves substantial utilization improvements for depth-wise convolution, addressing inefficiencies in conventional NPUs.
Patent Information
- Application Number
- KR1020220068209
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2021-08-12
- Filing Date
- 2022-06-03
- Publication Date
- 2026-11-11
- Estimated Expiration
- 2042-06-03
AI Technical Summary
Conventional NPUs exhibit low utilization for depth-wise convolution operations, particularly due to the lack of spatial feature map locality, leading to inefficient utilization and idle resources.
The proposed solution involves configuring an array of NPUs with a 4x4 processing element arrangement and utilizing a 2-to-1 multiplexer to optimize data paths for depth-wise convolution, enabling improved spatial feature map locality through an adder tree structure.
This configuration significantly enhances NPU utilization for depth-wise convolution, achieving up to 9x improvement for 3x3 filters and 25x improvement for 5x5 filters, while reducing activation data bandwidth by 60-80%.
Smart Images

Figure R1020220068209_ABST
Abstract
Description
Technology Field
[0001] The subject matter disclosed herein relates to neural processing units (NPUs). More specifically, the subject matter disclosed herein relates to a system and method for configuring an array of NPUs to perform depth-wise convolution, wherein the NPUs are configured to maximize spatial feature map locality using an adder tree. Background Technology
[0002] Conventional Neural Processing Units (NPUs) are optimally designed for normal convolution operations. However, Depth-wise (DW) convolution operations exhibit low NPU utilization when executed on an NPU configured for general convolution operations.
[0003] Conventional NPU cores are optimally designed for normal convolution operations so that input channels (C) and output channels (K) are spatially spread out. Parallelized input channels are reduced using a combination of adders and accumulators. Depth-wise convolution can omit many sub-operations that may be associated with general convolution operations. While general convolution operations perform an inner product of the activation tensor and weight tensor along the input channel direction (C-axis), depth-wise convolution does not reduce the product along the input channel direction. Instead, each input channel of the activation tensor has a corresponding weight filter with a depth of 1, which is convolved with the pixels of the weight filter and then reduced. Since the introduction of MobileNet-type architectures (which focus on depth-wise convolution), numerous deep learning architectures have included depth-wise convolution operations to reflect the increasing demand for depth-wise convolution using NPU cores.
[0004] A conventional approach to addressing the low utilization of existing NPUs for depth-wise convolution is to use a dedicated vector processor attached to the NPU core. However, this approach may not function properly with 5x5 filters due to a load imbalance between the NPU core and the vector processor. Furthermore, during typical convolution operations, the vector processor remains idle, thereby making the NPU / vector processor chip inefficient. The problem to be solved
[0005] The problem that the present invention aims to solve is to provide a core of a neural processing unit with improved performance and a method for configuring the core of the neural processing unit.
[0006] The technical problems of the present invention are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by those skilled in the art from the description below. means of solving the problem
[0007] An exemplary embodiment provides a core of an NPU that may include a first memory; a second memory; and an array of NPUs. The first memory may store a first value. The second memory may store a second value. The array of NPUs may be arranged in C rows and K columns, where C and K both include integers divisible by 4, and each NPU may include a 4x4 processing element (PE) arranged in 4 rows and 4 columns, and each PE may include a first input and a second input. In a first operating mode of the core of the NPU, the core of the NPU may be composed of Int(C / 3) row groups (RG), and each row group may include three consecutively adjacent rows RG1 to RG3 of the PE, and the first row RG1 is a PE in each k-th NPU for k = 1 to K. 11It may include a 2-to-1 multiplexer having an output coupled to the first input, each 2-to-1 multiplexer including a first input and a second input, and when k = 1, the first input for the 2-to-1 multiplexer is coupled to the first memory, and for k = 1 to K-1, each PE of each k-th NPU 12 inner PE 14 The above first input is the PE of the k-th NPU 11 At the second input of the above 2-to-1 multiplexer coupled to, the PE of the (k+1)th NPU 11 Combined with the first input of the 2-to-1 multiplexer coupled to, and coupled to the first memory, and the second row RG2 is PE in each k-th NPU for k = 1 to K 23 It may include a 2-to-1 multiplexer having an output coupled to the first input of, and when k = 1, the PE of the k-th NPU 21 and PE 22 The first input of is the PE of the k-th NPU mentioned above. 23 and the first input of the 2-to-1 multiplexer coupled to the first memory and can be coupled to the first memory, and for k = 2 to K-1, the PE of each k-th NPU 21 and PE 22 The above first input is the PE of the (k-1)th NPU. 23 The second input of the above 2-to-1 multiplexer coupled to, the PE of the (k-1)th NPU 24 The above first input of, the above PE of the k-th NPU 23 The first input of the 2-to-1 multiplexer coupled to, and can be coupled to the first memory, and the third row RG3 is the PE of the k-th NPU 31 PE to PE34 coupled to each other's first input and first memory 31 inner PE 34Each of the above-mentioned first inputs may be included, and the second input of each PE in the column of the PE may be coupled to the second memory. In one embodiment, the PE of each NPU 14 , PE 24 , PE 34 and PE 44 is disabled in the first operating mode above. In another embodiment, when k = odd, the PE of each row of the NPU 12 , PE 22 , PE 32 and PE 42 is disabled, and if k = even, the PE of each row of the NPU 11 , PE 12 , PE 13 , PE 14 , PE 13 , PE 23 , PE 33 and PE 43 ... is disabled in the first operating mode. In another embodiment, the second value stored in the second memory includes a value for a 5x5 filter. In another example, in the second operating mode, the core of the NPU is: for k = 1 to K, the PE of the k-th NPU 31 inner PE 34 PE coupled to each other's first input and first memory 31 inner PE 34It may be composed of each row of a PE containing each of the above-mentioned first inputs; and the second input of each PE in a column of a PE coupled to the second memory. In one embodiment, the core of the NPU may generate a convolution result from the first value and the second value in a second operating mode. In another embodiment, the core of the NPU may generate a depth-wise convolution result from the first value and the second value in the first operating mode. In yet another embodiment, the first memory may store an activation value and the second memory may store a weight value. In yet another embodiment, the second value stored in the second memory may include a value for a 3x3 filter.
[0008] An exemplary embodiment provides a method for configuring a core of an NPU, wherein the core of the NPU comprises an array of NPUs arranged in C rows and K columns, where C and K are both integers divisible by 4, and each NPU may comprise a 4x4 processing element (PE) arranged in 4 rows and 4 columns, and each PE comprises a first input and a second input, and the method comprises, for a first operating mode, configuring the core of the NPU into Int(C / 3) row groups (RG) to generate depth-wise convolution results from a first value and a second value, wherein each row group comprises three consecutively adjacent rows RG1 to RG3 of the PE, and for k = 1 to K, a 2-to-1 multiplexer configuring the PE in each k-th NPU 11 A step of configuring a first row RG1 to have an output coupled to the first input of the above - each 2-to-1 multiplexer includes a first input and a second input -, when k = 1, a step of configuring the first input for the 2-to-1 multiplier to be coupled to a first memory storing the first value, for k = 1 to K-1, each PE of each k-th NPU12 inner PE 14 The above first input of the PE of the k-th NPU 11 Coupled to the second input of the above 2-to-1 multiplexer coupled to, and the PE of the (k+1)th NPU 11 A step of configuring to be coupled to the first input of the 2-to-1 multiplexer coupled to and coupled to the first memory, wherein the 2-to-1 multiplexer is PE at each k-th NPU for k = 1 to K 23 A step of configuring to have an output coupled to the first input of, where k = 1, the PE of the k-th NPU 21 and PE 22 The above first input of the PE of the k-th NPU 23 and a step of configuring to be coupled to the first input of the 2-to-1 multiplexer coupled to the first memory and coupled to the first memory, wherein for k = 2 to K-1, the PE of each k-th NPU 21 and PE 22 The above first input of the above PE of the (k-1)th NPU 23 Coupled to the second input of the above 2-to-1 multiplexer coupled to, and the PE of the (k-1)th NPU 24 Combined to the first input of the above, and the PE of the k-th NPU 23 A step of configuring to be coupled to the first input of the 2-to-1 multiplexer coupled to and coupled to the first memory, PE 31 inner PE 34 Each of the above first inputs is the PE of the k-th NPU 31 The method may include the step of configuring PE34 to be coupled to each other's first inputs and coupled to the first memory, and the step of configuring the second input of each PE in the column of PE to be coupled to the second memory storing the second value. In one embodiment, the method comprises the PE of each NPU 14, PE 24 , PE 34 and PE 44 The method may further include the step of configuring to disable in the first operating mode. In another embodiment, the method, when k = odd, the PE of each row of the NPU 12 , PE 22 , PE 32 and PE 42 The step of configuring to disable, and if k = even, the PE of each row of the NPU 11 , PE 12 , PE 13 , PE 14 , PE 13 , PE 23 , PE 33 and PE 43 The method may further include the step of configuring to be disabled in the first operating mode. In another embodiment, the second value stored in the second memory may include a value for a 5x5 filter. In another embodiment, the method further includes the step of configuring the core of the NPU in the second operating mode, and for k = 1 to K, PE 31 inner PE 34 Each of the above first inputs is the PE of the k-th NPU 31 inner PE 34The method may further include the step of configuring each row of PE to be combined with each other's first inputs and the first memory; and the step of configuring the second input of each PE in the column of PE to be combined with the second memory. In one embodiment, the method may further include the step of generating a convolution result from the first value and the second value by the core of the NPU in the second operating mode. In another embodiment, the method may further include the step of generating a depth-wise convolution result from the first value and the second value by the core of the NPU in the first operating mode. In another embodiment, the first memory may store an activation value and the second memory may store a weight value. In one embodiment, the second value stored in the second memory may include a value for a 3x3 filter. Brief explanation of the drawing
[0009] In the following section, aspects of the subject matter disclosed in this specification will be described with reference to exemplary embodiments illustrated in the drawings: FIG. 1 illustrates an exemplary array of multipliers for an NPU core conventionally configured for a typical convolution operation. FIG. 2a shows a block diagram of an NPU configured for conventional convolution operations. FIG. 2b illustrates a modified NPU that improves the activation bandwidth for depth-wise convolution operations compared to the activation bandwidth provided by the NPU of FIG. 2a; FIG. 3a illustrates the characteristics of a depth-wise convolution operation performed by a conventional NPU having a data path switched for activation and weights; FIG. 3b illustrates the characteristics of a depth-wise convolution operation performed by an NPU having a data path that is switched and modified for activation and weights, and uses an RSxHW-based data flow in accordance with the subject matter disclosed herein: FIG. 4a illustrates a logical mapping of a method in which weights are broadcast to a multiplier for a single exemplary cycle to maximize spatial feature map locality in accordance with the subject matter disclosed herein; FIG. 4b illustrates how an exemplary 3x3 filter is applied to a map of activation values arranged in rows and columns:. FIGS. 5a and 5b respectively illustrate the logical mapping of activations and weights assigned to the first cycle of a multicycle according to the subject matter disclosed herein; FIGS. 5C and FIGS. 5D respectively illustrate the logical mapping of weights and activations assigned to the next cycle of a multicycle according to the subject matter disclosed herein; FIG. 6a illustrates an exemplary rotation sequence for six exemplary cycles of a 3x3 DW filter according to the subject matter disclosed herein; FIG. 6b illustrates an exemplary 3x3 filter that is downshifted for activation values for four cycles according to the subject matter disclosed herein; FIG. 6c illustrates a method in which an activation value is transmitted to a multiplier using a 4-row activation buffer as a line buffer in accordance with the subject matter disclosed herein; FIGS. 7a and 7b conceptually illustrate that diagonal mapping can be implemented as horizontal broadcasting by rotating weight values in a weight vector according to the subject matter disclosed herein; FIG. 8a illustrates four filter groups in four columns configured as part of an NPU core for conventional convolution operations; FIG. 8b illustrates the NPU core of FIG. 8a modified into an NPU core that maximizes spatial feature map locality in accordance with the subject matter disclosed herein; FIG. 8c illustrates an example of a method in which an activation value is input to an NPU core through an activation buffer in accordance with the subject matter disclosed herein; FIG. 9a illustrates a logical mapping of a method in which a 3x3 filter with weights having a stride of 2 for a single exemplary cycle is broadcast to a multiplier to maximize spatial feature map locality in accordance with the subject matter disclosed herein: FIG. 9b corresponds to FIG. 9a and shows a column of multipliers (adders not shown) that can be disabled for a 3x3 filter having a stride of 2: FIG. 10a illustrates a logical mapping of a method in which a 5x5 filter of weights having a stride of 1 during a single exemplary cycle is broadcast to a multiplier to maximize spatial feature map locality in accordance with the subject matter disclosed herein: FIG. 10b conceptually illustrates a method in which the three adder-tree diagonal mappings of the activation values of FIG. 7a can be extended to five adder-tree diagonal mappings for a 5x5 filter according to the subject matter disclosed herein: FIG. 10c conceptually illustrates that the diagonal mapping of activation values in FIG. 10a and 10b can be implemented as horizontal sharing of activation values by rotating the weight values of a 5x5 filter so that weight broadcasting becomes horizontal: FIG. 11 illustrates a logical mapping of a method in which a 5x5 filter with weights having a stride of 2 is broadcast to a multiplier during a single exemplary cycle to maximize spatial feature map locality according to the subject matter disclosed herein; and FIG. 12 illustrates an electronic device comprising a neural processing unit configured to maximize spatial feature map locality using an adder tree in accordance with the subject matter disclosed in this specification. Specific details for implementing the invention
[0010] In the following detailed description, many specific details are described to provide a complete understanding of the present disclosure. However, it will be understood by those skilled in the art that the disclosed aspects may be practiced without these specific details. In other examples, well-known methods, procedures, components, and circuits are not described in detail so as not to obscure the invention disclosed herein.
[0011] Throughout this specification, references to “one embodiment” or “an embodiment” mean that specific features, structures, or characteristics described in relation to the embodiment may be included in at least one embodiment disclosed herein. Accordingly, references to “in one embodiment,” “in an embodiment,” or “according to one embodiment” (or other phrases having a similar meaning) in various places throughout this specification may not all refer to the same embodiment. Additionally, specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In this regard, as used herein, the word “exemplary” means “serves as an example, a case, or an example.” Any embodiment described as “exemplary” in this specification should not be interpreted as necessarily being preferred or advantageous over other embodiments. Furthermore, specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. Also, as discussed herein, singular terms may include corresponding plural terms, and plural terms may include corresponding singular terms. Similarly, hyphenated terms (e.g., "2-dimensional," "predetermined," "pixel-specific," etc.) may sometimes be used interchangeably with their corresponding unhyphenated versions (e.g., "2-dimensional," "predetermined," "pixel-specific," etc.), and uppercase items (e.g., "Counter Clock," "Row Select," "PIXOUT," etc.) may be used interchangeably with their corresponding non-uppercase versions (e.g., "counter clock," "row select," "pixout," etc.). Such interchangeable use should not be considered inconsistent.
[0012] Additionally, depending on the context discussed herein, singular terms may include corresponding plural forms, and plural terms may include corresponding singular forms. It should be noted that the various drawings (including component diagrams) shown and discussed herein are for illustrative purposes only and are not drawn in proportion. For example, the dimensions of some elements may be exaggerated relative to others for clarity. Additionally, where deemed appropriate, reference numbers are repeated between drawings to indicate corresponding and / or similar elements.
[0013] The terms used herein are intended to describe some exemplary embodiments and are not intended to limit the essence of the claimed invention. As used herein, the singular form includes the plural form unless the context clearly indicates otherwise. It will be understood that when used herein, the terms “comprising” and / or “comprising” specify the presence of the mentioned features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.
[0014] When one element or layer is referred to as being "connected" or "combined" to another element or layer, it may be located directly above the other element or layer, be connected to or combined therewith, or an intermediate element or layer may exist. In contrast, when one element is referred to as being "directly above," "directly connected," or "directly combined" to another element or layer, no intermediate element or layer exists. The same number indicates the same element throughout. As used herein, the term "and / or" includes any and all combinations of one or more associated enumerated items.
[0015] Terms used herein such as “first,” “second,” etc. are used as labels for the preceding nouns and, unless explicitly defined, do not imply any type of order (e.g., spatial, temporal, logical, etc.). Additionally, the same reference number may be used across two or more drawings to refer to parts, components, blocks, circuits, units, or modules having the same or similar functions. However, such usage is intended for simplicity of description and ease of discussion; it does not imply that the structure or structural details of such components or units are the same across all embodiments or that a commonly referenced part / module is the only way to implement a part of the exemplary embodiments disclosed herein.
[0016] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which this subject belongs. It will be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with that meaning in the context of the relevant technology, and should not be interpreted in an idealized or overly formal sense unless explicitly defined in this specification.
[0017] As used herein, the term “module” refers to any combination of software, firmware, and / or hardware configured to provide the functions described herein in relation to the module. For example, software may be implemented as a software package, code and / or instruction set or instruction, and the term “hardware” as used in any implementation described herein may include, for example, assemblies, hardwired circuits, programmable circuits, state machine circuits, and / or firmware storing instructions executed by programmable circuits, in a single or any combination. Modules may be implemented collectively or individually as circuits forming part of a larger system, such as an integrated circuit (IC), a system-on-a-chip (SoC), an assembly, etc.
[0018] The subject matter disclosed herein relates to an NPU comprising its own SRAM, which connects the SRAM of different NPUs to other NPUs in a vertical / horizontal / diagonal direction and uses this connection to support different NPU configurations having different input-channels and output channels and spatial reuse of SRAM data paths.
[0019] The subject disclosed herein inverts (switches) the data paths of activations and weights, adds a 2-to-1 multiplexer (MUX) every 2 / 9 multipliers, and executes NPU cores using row x column and cell (RSxHW) for depth-wise convolution. For normal convolution, the data paths of activations and weights can be configured for a normal convolution configuration where the 2-to-1 MUX is idle. By changing the NPU data flow to become an RSxHW data flow, NPU utilization for depth-wise convolution for 3x3 filters and 5x5 filters can be improved by ~9x and ~25x, respectively.
[0020] In one embodiment, the subject disclosed herein provides high data bandwidth for depth-wise convolution operations by modifying an existing NPU core by inverting (or switching) the data paths of activations and weights so that an activation buffer stores and transmits weights, while a weight buffer inside the NPU core stores and transmits (or broadcasts) activations. To match weight and activation pairs, a 2-to-1 multiplexer is added every 2 / 9 multipliers, and the core operates using RSxHW data streams to maximize locality of feature maps, thereby providing high data bandwidth for depth-wise convolution.
[0021] The utilization of an NPU configured for general convolution operations is approximately 3.3% for depth-wise convolution operations with a stride of 1. By using the RSxHW data stream, NPU utilization for 3x3 depth-wise convolution with a stride of 1 can be improved to approximately 28%, which is an increase in utilization of about 9 times. For a stride of 2, the utilization improvement can be about 2 times. For a 5x5 DW with a stride of 1, utilization can be improved to approximately 78%, which is about 25 times the utilization improvement. For a 5x5 DW with a stride of 2, the utilization improvement can be about 6 times higher than the existing NPU configuration.
[0022] In another embodiment, the subject matter disclosed herein provides an implementation of depth-by-depth convolution operations on an NPU core that improves NPU utilization compared to the NPU utilization of an NPU core configured for general convolution operations. Activation data bandwidth can be improved by 60% to 80% by adding a 2-to-1 MUX to the activation data paths for approximately 22% of the NPUs in a 32x32 NPU core and using an adder tree to provide spatial locality for the activation data, which increases NPU utilization for depth-by-depth convolution by approximately 9 to 25 times depending on the weight kernel size. The added MUX can be used to maximize spatial feature map locality by specifying activation paths loaded into the WBUF. The added MUX also reduces the activation data bandwidth requirements for depth-by-depth convolution operations compared to the bandwidth required for depth-by-depth convolution operations performed on an NPU core configured for general convolution operations.
[0023] The subject disclosed in this specification provides a logical mapping of depth-wise convolutions that maximizes spatial feature map (activation) locality using an adder tree that reduces the total activation data bandwidth; and a physical mapping of depth-wise convolutions using two (2-to-1) multiplexers per nine multiplication units added to a standard NPU core architecture of 32 rows (C) * 32 columns (K) * 4 cells (HW).
[0024] Depth-wise convolution operations performed using an NPU configured to utilize spatial parallelism for the C and K axes conventionally have a utilization rate of about 3% in, for example, a 32x32 multiplier (MULT) array, because depth-wise convolution does not reduce the multiplication in the direction of the input channels. As illustrated in FIG. 1, the input channels of the convolution operation are mapped to the rows of the NPU core and the output channels are mapped to the columns of the NPU core, whereas depth-wise convolution uses Multiplication Accumulator (MAC) units along the main diagonal within the NPU core.
[0025] FIG. 1 illustrates an array of multipliers of an exemplary NPU core (100) typically configured for a typical convolution operation.
[0026] The NPU core (100) may be referred to as a Conv Core. In FIG. 1, each square (101) represents a multiplier (MULT) unit within the NPU core. The gray square (102) is the only MULT unit used for depth-wise convolution, which results in a MULT utilization ratio of 1 / row = 1 / column (where row = column). In a specific NPU core configuration consisting of 32 rows and 32 columns, the utilization is only ~3%.
[0027] FIG. 2a shows a block diagram of an NPU (200) configured for conventional convolution operations.
[0028] Activation data (Input Feature Map (IFM) data) and weight data may initially be stored in memory (201), which may be static random access memory (SRAM). During a convolution operation, the activation data may be transferred to an activation buffer (ABUF) (203), and the weight data may be transferred to a weight buffer WBUF (202). The activation data and weight data are transferred to a processing entity (PE) (204) that includes a multiplier for processing. Various functional units of the NPU (200) may be implemented as circuits and / or modules.
[0029] A conventional NPU (200) modification to improve activation bandwidth for depth-wise convolution operations is to switch data paths for activations and weights. That is, the weight buffer stores and transmits activations, and the activation buffer stores and transmits (or broadcasts) weights.
[0030] FIG. 2b illustrates a modified NPU (200') that improves the activation bandwidth for depth-wise convolution operations compared to the activation bandwidth provided by the NPU (200) of FIG. 2a.
[0031] Activation data and weight data can initially be stored in memory (201'). During depth-wise convolution operations, activation data can be transferred to an activation buffer (ABUF) (202'), and weight data can be transferred to a weight buffer WBUF (203'). Then, activation data can be transferred, and weight data can be broadcast to a processing entity (PE) (201) containing a multiplier for processing. Various functional units of the NPU (200') can be implemented as circuits and / or modules.
[0032] FIG. 3a illustrates the characteristics of a depth-wise convolution operation performed by a conventional NPU having a data path switched for activation and weighting (like the NPU (200) in FIG. 2a).
[0033] Since a weight kernel (filter) is applied to the activation data, the weight kernel moves over the activation data along a zigzag path to provide spatial feature map locality. Spatial feature map locality is the proportion of feature map pixels that are reused in both time and space. The zigzag path can be considered to provide temporal locality in that a portion of the activation data is reused for each stride.
[0034] During the first cycle 0 of FIG. 3a, a 3x3 exemplary weight kernel (301) is applied to nine exemplary activation pixels (values) (302) that are surrounded (or overlapped) by the 3x3 weight kernel. For the next cycle 1, the 3x3 kernel moves horizontally to the right by one stride. (Other horizontal strides are possible.) Then, the 3x3 weight kernel is applied to the nine activation values containing it. For cycle 2, the 3x3 kernel moves horizontally to the right by one step again and is applied to the nine activation values containing it, and so on. When the kernel (301) reaches the end of the IFM (302) in the horizontal direction, the kernel moves downward by one stride, as illustrated by the zigzag path of the arrow in FIG. 3a, and processing continues again in the other direction. (Other strides are possible.)
[0035] In contrast to FIG. 3a, FIG. 3b illustrates the characteristics of depth-wise convolution operations performed by a modified NPU as disclosed herein, with the data path for activation and weights switched (similar to the NPU (200') in FIG. 2b), using an RSxHW-based data flow.
[0036] The conventional approach illustrated in FIG. 1, FIG. 3a, provides a spatial feature map and temporal locality based on the overlap of input (activated) pixels in the current and next cycles, whereas the subject disclosed herein maximizes feature map locality by utilizing adder tree (AT) locality in conjunction with temporal locality. Adder tree locality enables additional overlap of input pixels (activations) by sharing input pixels between different adder trees (i.e., spatial reuse of input pixels).
[0037] In FIG. 3b, four adder trees AT0 through AT3 partially share activations for a 3x3 depth-wise convolution. During cycle 0, nine activation pixels (values) displayed under the AT0 identifier, which is included by the 3x3 weight kernel (303a), are reduced by the first adder tree (AT0). Nine activation values displayed under the AT1 identifier, which is included by the 3x3 weight kernel (303b), are reduced by the second adder tree (AT1). Nine activation values displayed under the AT2 identifier, which is included by the 3x3 weight kernel (303c), are reduced by the third adder tree (AT2). Finally, nine activation values displayed under the AT3 identifier, which is included by the 3x3 weight kernel (303d), are reduced by the fourth adder tree (AT4).
[0038] For the next cycle 1, the weight kernel (303a, 303d) is shifted one row (1 stride) down of the activation value, and the weight kernel is applied to the activation value in a manner similar to how the weight kernel was applied to the activation value in cycle 0. For cycle 2, the weight kernel (303a, 303d) is again moved one row (1 stride) down of the activation value, and the weight kernel is applied to the activation value in a manner similar to how the weight kernel was applied to the activation value in cycle 0. Other stride shifts are also possible. For example, at the bottom of the activation value map, the kernel is moved horizontally by the width of the kernel, and the process is repeated one stride up.
[0039] The reuse (partial sharing) of activation data can be illustrated by considering the dotted lines between Cycle 0 and Cycle 1, and between Cycle 1 and Cycle 2. For a 3x3 filter, the bottom two rows of activation data processed in Cycle 0 are reused as the top two rows of activation data in Cycle 1. For Cycle 2, the bottom two rows of activation data processed in Cycle 1 are reused as the top two rows of activation data in Cycle 2. The partial sharing of activation values provided by the process illustrated in Fig. 3b can provide a 50% activation bandwidth saving. For a larger number of adder trees (e.g., 32 adder trees of an NPU core with 32 columns), activation bandwidth can be saved by approximately 65% for a 3x3 weight kernel and approximately 80% for a 5x5 weight kernel.
[0040] FIG. 4a illustrates a logical mapping of a method in which a 3x3 filter with a weight of 1 stride per single exemplary cycle is broadcast to a multiplier to maximize spatial feature map locality according to the subject matter disclosed herein.
[0041] In FIG. 4a, a 3x3 DW filter is illustrated in 401. In FIG. 4a (and throughout this disclosure), weight values are indicated by the row and column positions of the weight values in the filter. For example, a weight value at row 0, column 0 is indicated as 0,0. The DW filter (401) is expanded from a 3x3 configuration to become a 9x1 weight vector (402) in a WBUF (such as the WBUF (203') in FIG. 2b). An array of multipliers is arranged in rows and columns and is illustrated in FIG. 4a as an array of squares (403), with only one of the multipliers shown. Although only a 6-column multiplier (403) is shown, it should be understood that more or fewer columns of multipliers may be used. Each column of the multiplier has a corresponding adder tree (AT) (404), with only one of the adder trees shown.
[0042] Each weight is broadcast along the corresponding row, as indicated by 405. The activation value is assigned to each multiplier (403). Neighboring multipliers (403) located along the diagonal (406) (only one diagonal is shown) use the same activation value. That is, each activation value can be reused by different multipliers (403) and adder trees (404). The diagonal mapping of activation values to weight values maximizes spatial feature map locality according to the subject matter disclosed herein. The maximum AT locality is 2 / 3 for a 3x3 filter.
[0043] Figure 4b illustrates how an exemplary 3x3 filter is applied to activation map (IFM) values arranged in rows and columns.
[0044] The activation values are indicated by row and column positions in the IFM map. The blackening to gray shades from left to right corresponds to the blackening to gray shades of the multiplier (403) and adder tree (404) in FIG. 4a. Although only 6 rows and 10 columns of the activation value map are shown in FIG. 4b, it should be understood that the activation value map may contain more or fewer rows and / or columns.
[0045] FIGS. 5a and 5b respectively illustrate the logical mapping of activations and weights assigned to multiple cycles according to the subject matter disclosed herein.
[0046] Fig. 5a illustrates the logical mapping of the weights of a 3x3 filter for cycle 0 (similar to Fig. 4a). Fig. 5b is identical to Fig. 4b. For the next cycle 1 illustrated in Fig. 5, the weight kernel is shifted down by one stride. Fig. 5d illustrates an exemplary 3x3 filter that is shifted downward for the activation values. The weight values are rotated so that the top three rows now become the bottom three rows of a 9x1 vector, which causes the other six weight values to rotate upward. As the weight kernel moves downward, the new activation values replace the top three rows of activation values as indicated in box 501. Due to temporal feature map locality, only the first three rows of activation values are taken. The logical mapping of the weights for the multiplier (403) is a diagonal mapping (diagonal not illustrated), identical to the diagonal logical mapping illustrated in Fig. 4a, but differs only in the activation values updated for cycle 1.
[0047] In the next cycle 2 (not shown), the activation values of row 3 are added to the top 3 rows of the multiplier, while the weight values continue to rotate. This sequence of adding activation values to the top 3 rows of the multiplier and adding rotations to the weight values continues as the weight kernel moves through the activation values by the path indicated by the bold line in Fig. 3b.
[0048] In the next cycle 2 (not shown), the activation value of row 3 is added to the top 3 rows of the multiplier, and the weight value continues to rotate. As the weight kernel moves through the activation value by the path shown by the bold line in FIG. 3b, the sequence of adding the activation value to the top 3 rows of the multiplier and rotating the weight value continues.
[0049] FIG. 6a illustrates an exemplary rotation sequence for six exemplary cycles of a 3x3 DW filter according to the subject matter disclosed herein.
[0050] The rows of the 3x3 DW filter are shifted down by three rows in one cycle to match the row indices of the activations and weights. To match the position (i.e., row) of the weight values, shuffling logic can be used to reroute the weights to the corresponding rows. Since rotating groups of three rows generates all possible weight vectors for the filter, there is no need to fetch new weights until the DW convolution for the filter is complete.
[0051] FIG. 6b illustrates an exemplary 3x3 filter that is downshifted for activation values for four cycles according to the subject matter disclosed herein. FIG. 6c illustrates a method in which activation values are transmitted to a multiplier using a 4-row activation buffer (ABUF) as a line buffer according to the subject matter disclosed herein.
[0052] Three rows are used to send activation values to the core, and one row is refreshed by SRAM. For example, during cycle 0, rows y = 0 through y = 2 are sent to the core, and row y = 3 is updated in memory (SRAM). During cycle 1, rows y = 1 through y = 3 are sent to the core, and row y = 4 (located in the top row) is updated in memory. During cycle 2, rows y = 2, 3, and 4 are sent to the core, and row y = 5 is updated in memory. Activations can be routed to their respective targets using, for example, a 4x4 crossbar circuit.
[0053] FIGS. 7A and 7B conceptually illustrate that a diagonal mapping (Fig. 7A) can be implemented as a horizontal mapping (Fig. 7B) by rotating weight values in a weight vector according to the subject matter disclosed herein.
[0054] The weight values of a 9x1 vector can be rotated as illustrated in 701 to implement horizontal broadcasting. The "%" in "AT%X" is the modulus operator. For example, "AT%3=1" means that the remainder when the addition tree index (AT) is divided by 3 is 1. AT(1)%3 = AT(4)%3 = 1. The remainder is the number of sliding steps in which the weight values are rotated. That is, the diagonal mapping of activation values can be implemented as horizontal sharing of activation values so that weight broadcasting is horizontal. To provide horizontal weight broadcasting and horizontal activation sharing, three of the four multipliers can be used in the baseline NPU core (32x32x4) described in FIGS. 8a and 8b.
[0055] FIG. 8a illustrates four filter groups (8010, 8013) of four columns (columns 0 to 3) configured as part of an NPU core (800) for conventional convolution operations.
[0056] The NPU core (800) may include additional columns of filter groups, and each column not shown may include additional filter groups. Each filter group (801) includes a 3x4 multiplier (802xy). In each filter (801), a row of four OFM pixels (802) shares a single weight value (wxy). Each OFM pixel is connected to an independent activation input (not shown) and an adder tree (not shown). The activation input is shared for a given column of OFM pixels.
[0057] FIG. 8b illustrates the NPU core of FIG. 8a modified into an NPU core (850) for depth-wise convolution that maximizes spatial feature map locality for a 3x3 filter and 1 stride, in accordance with the subject matter disclosed herein.
[0058] In the first row (861) of each pixel (851), as illustrated, a 2-to-1 multiplexer (853) is the first multiplier (852 0,0It is inserted into the input of ) and a connection (854) is added along the first row (861) between each pixel (851). In the second row (862) of each pixel (851), as illustrated, a 2-to-1 multiplexer (853) is added to the third multiplier (852 1,2 It is inserted into the input of ) and the connection (854) is added along the second row (862) between each pixel (851). The third row (863) of each pixel (851) is not modified. The fourth multiplier (852) of each pixel (851) in each row x3 ) is disabled as indicated by the white circle. The modification of the 3-row group is repeated for all 3-row groups to complete the modification in which the NPU core of FIG. 8a becomes the NPU core (850).
[0059] FIG. 8c shows an example where an activation value is input to the NPU core (850) through ABUF.
[0060] An exemplary activation value corresponding to the activation value of FIG. 7b is shown as being input to the multiplier (851). The bold line in FIG. 8c indicates how the activation value is shared between the multipliers (851). For example, the activation value 0,0 is input to the multiplier (not shared based on mapping). The activation value 0,3 is input to the multiplier (852) of the pixel (8510). 01 and 852 02 ) and the multiplier (852) of the pixel (8511) 00 It is shared between ). The activation value 0,6 is the multiplier (852) of the pixel (8511). 01 and 852 02 ) and the multiplier (852) of the pixel (8512) 00 It is shared between ). The activation value 0,9 is the multiplier (852) of the pixel (8512) 01 and 852 02 ) and the multiplier (852) of the pixel (8513) 00 It is shared among, etc.
[0061] FIG. 9a illustrates a logical mapping of a method in which a 3x3 filter with a weight of 2 strides is broadcast to a multiplier during a single exemplary cycle to maximize spatial feature map locality according to the subject matter disclosed herein.
[0062] FIG. 9a is identical to FIG. 4a, except that 2 strides are used for a 3x3 DW filter for every 2nd column of the multiplier and the corresponding adder tree does not contribute to OFM data and can be disabled. The columns of the multiplier and the corresponding adder tree that can be disabled for 2 strides are indicated by 901.
[0063] FIG. 9b shows a column of multipliers (adder not shown) that corresponds to FIG. 9b and can be disabled for a 3x3 filter with a stride of 2. FIG. 9b is identical to FIG. 8c except that the column of multipliers that can be disabled for a 3x3 filter with a stride of 2 is labeled 901.
[0064] FIG. 10a illustrates a logical mapping of a method in which a 5x5 filter with a weight of 1 stride is broadcast to a multiplier during a single exemplary cycle to maximize spatial feature map locality according to the subject matter disclosed herein.
[0065] When using a 5x5 filter, five rows work together to represent horizontal inputs. A single IFM value is shared across five adder trees. In FIG. 10a, the 5x5 DW filter is illustrated as 1001. The DW filter (1001) is expanded in a 5x5 configuration to become a 25x1 weight vector (1002) in WBUF. An array of multipliers is arranged in rows and columns and is illustrated in FIG. 10a as a square array (1003), with only one of the multipliers shown. Although only six columns of multipliers (1003) are shown, it should be understood that more or fewer columns of multipliers may be used. Each column of a multiplier has a corresponding adder tree (AT) (1004), with only one of the adder trees shown.
[0066] Each weight is broadcast along the corresponding row, as indicated by 1005. The activation value is assigned to each multiplier (1003). Neighboring multipliers (1003) located along the diagonal (1006) (only one diagonal is shown) use the same activation value. The diagonal mapping of the activation value to the weight value maximizes the spatial feature map locality according to the subject matter disclosed herein. The maximum AT locality is 4 / 5 for a 5x5 filter.
[0067] FIG. 10b conceptually shows that the three adder tree diagonal mappings of the activation values of FIG. 7a can be extended to five adder tree diagonal mappings for a 5x5 filter according to the subject matter disclosed herein.
[0068] To do this, each row of the 5x5 can be conceptually transformed into a 2x3 filter where the weight values at 2,3 are not used (indicated by x). The extended mapping location is indicated by the bold dashed line 1007.
[0069] In Fig. 10c, the diagonal mapping of the activation values in Figs. 10a and 10b can be implemented as horizontal sharing of activation values by rotating the weight values of a 5x5 filter so that weight broadcasting becomes horizontal.
[0070] The rotation of weight values spans three rows of the 15x1 weight vector (1002). The thick dashed line (1008) provides sharing of weight values, so that activation values are provided to only three of the six rows due to space reuse of activation values (one row is idle). To provide horizontal weight broadcasting, it is disabled every 6th multiplier on the baseline NPU core (32x32x4) (no activity).
[0071] FIG. 11 illustrates a logical mapping of a method in which a 5x5 filter of 2-strine weights is broadcast to a multiplier during a single exemplary cycle to maximize spatial feature map locality according to the subject matter disclosed herein.
[0072] Similar to a 3x3 filter with 2 strides, as shown in 1101, it is disabled for every other adder tree.
[0073] FIG. 12 illustrates an electronic device (1200) comprising a neural processing unit that can be configured for depth-wise convolution that maximizes spatial feature map locality using an adder tree, in accordance with the subject matter disclosed in this specification.
[0074] The electronic device (1200) includes a controller (or CPU) (1210), a keypad, a keyboard, a display, a touchscreen display, a camera and / or an image sensor, memory (1230), an interface (1240), a GPU (1250), an image processing unit (1260), and a neural processing unit (1270) connected to each other via a bus (1280). The controller (1210) may include, for example, at least one microprocessor, at least one digital signal processor, at least one microcontroller, etc. The memory (1230) may be configured to store command codes or user data used by the controller (1210). The electronic device (1200) and various system components of the electronic device (1200) may be formed by one or more modules.
[0075] In one embodiment, the neural processing unit (1270) may be configured to maximize spatial feature map locality using an adder tree, in accordance with the subject matter disclosed herein. The interface (1240) may be configured to include a wireless interface configured to transmit and receive data with a wireless communication network using RF signals. The wireless interface (1240) may include, for example, an antenna. The electronic system (1200) is also used in communication interface protocols of communication systems such as Code Division Multiple Access (CDMA), Global Mobile Communication System (GSM), North American Digital Communication (NADC), Extended Time Division Multiple Access (ETDMA), Broadband CDMA (WCDMA), CDMA2000, Wi-Fi, Municipal Wi-Fi (Muni Wi-Fi), Bluetooth, Digital Enhanced Radio Communication, Wireless Universal Serial Bus (DECT), Wireless Universal Serial Bus (Wireless USB), Seamless Handoff Orthogonal Frequency Division Multiplexed High Speed Low Latency Access (Flash OFDM), IEEE 802.20, General Packet Radio Service (GPRS), iBurst, Wireless Broadband (WiBro), WiMAX, WiMAX Advanced, Universal Mobile Communication Service - Time Division Duplex (UMTS TDD), High Speed Packet Access (HSPA), Evolution Data Optimization (EVDO), Long Term Evolution Advanced (LTE-Advanced), Multi-Channel Multi-Point Distribution Service (MMDS), 5th Generation Wireless (5G), 6th Generation Wireless (6G), etc.
[0076] Embodiments of the subject matter and operation described herein may be implemented in digital electronic circuits, or in computer software, firmware, or hardware comprising structures and structural equivalents disclosed herein, or in a combination of one or more of these. Embodiments of the subject matter described herein may be implemented in one or more computer programs, i.e., one or more modules of computer program instructions, encoded in a computer storage medium to control execution by a data processing device or operation of a data processing device. Alternatively or additionally, program instructions may be encoded in artificially generated radio signals, for example, mechanically generated electrical, optical, or electromagnetic signals generated to encode information to be transmitted to a suitable receiver device for execution by a data processing device. The computer storage medium may be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination thereof, or may be included therein. Furthermore, although the computer storage medium is not a radio signal, the computer storage medium may be a source or destination of computer program instructions encoded in an artificially generated radio signal. A computer storage medium may also be one or more individual physical components or media (e.g., multiple CDs, disks, or other storage devices) or may be included therein. Additionally, the operations described herein may be implemented as operations performed by a data processing device on data stored in one or more computer-readable storage devices or received from other sources.
[0077] While this specification may contain many specific implementation details, these implementation details should not be interpreted as a limitation on the scope of the claimed subject matter, but rather as a description of features specific to a particular embodiment. Specific features described herein in relation to separate embodiments may be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. Furthermore, although features were described and initially claimed to function in a specific combination, one or more features from the claimed combination may be removed from this combination in some cases, and the claimed combination may relate to a sub-combination or a variation of a sub-combination.
[0078] Similarly, although operations are depicted in a specific order in the drawings, this should not be understood as requiring that the operations be performed in the specific order shown or in a sequential order, or that all illustrated operations be performed, in order to achieve the desired results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products.
[0079] Accordingly, specific embodiments of the subject matter have been described herein. Other embodiments are within the scope of the following claims. In some cases, the measures specified in the claims may be performed in a different order and still obtain a desirable result. Furthermore, the process illustrated in the accompanying drawings does not necessarily require the specific order or sequential order illustrated to achieve a desirable result. In certain implementations, multitasking and parallel processing may be advantageous.
[0080] As recognized by those skilled in the art, the innovative concepts described herein may be modified and changed across a wide range of applications. Accordingly, the scope of the claimed subject matter should not be limited to the specific exemplary teachings set forth above, but is instead defined by the following claims.
Claims
Claim 1 As a core of a neural processing unit (NPU), a first memory for storing a first value; a second memory for storing a second value; and an array of NPUs arranged in C rows and K columns, wherein C and K both contain integers divisible by 4, and each NPU includes a 4x4 processing element (PE) arranged in 4 rows and 4 columns, and each PE includes an array of NPUs including a first input and a second input, and in a first operating mode of the core of the NPU, the core of the NPU is composed of Int(C / 3) row groups (RG), and each row group consists of three consecutively adjacent first to third rows (RG1 to RG3) of the PE. 3) ...including, wherein the first row RG1 is a PE placed in the first row and first column of each k-th NPU for k = 1 to K. 11 It includes a 2-to-1 multiplexer having an output coupled to the first input, wherein the 2-to-1 multiplexer includes a first input and a second input, and when k = 1, the first input for the 2-to-1 multiplexer is coupled to the first memory, and for k = 1 to K-1, PEs disposed in each first row and second to fourth columns of each k-th NPU 12 inner PE 14 The above first input is the PE of the k-th NPU 11 The second input of the above 2-to-1 multiplexer coupled to, and the PE of the (k+1)th NPU 11 The first input of the 2-to-1 multiplexer coupled to, and coupled to the first memory, and the second row RG2 is a PE placed in the second row and third column of the respective k-th NPU for k = 1 to K. 23 It includes a 2-to-1 multiplexer having an output coupled to the first input of, and when k = 1, a PE placed in the second row and first column of the k-th NPU 21 and PE placed in the 2nd row and 2nd column 22 The first input of is the PE of the k-th NPU mentioned above. 23 and the first input of the 2-to-1 multiplexer coupled to the first memory and coupled to the first memory, and for the case k = 2 to K, the PE of each k-th NPU 21 and PE 22 The above first input is the PE of the (k-1)th NPU. 23 The second input of the above 2-to-1 multiplexer coupled to, and the PE placed in the second row and fourth column of the (k-1)th NPU 24 The above first input of and the above PE of the k-th NPU 23 The first input of the 2-to-1 multiplexer coupled to, and coupled to the first memory, and the third row RG3 is a PE placed in the third row and columns 1 to 4 of the k-th NPU 31 inner PE 34 Includes the above-mentioned first input, wherein the PE 31 inner PE 34 A core of an NPU, wherein the first inputs are combined with each other and combined to the first memory, and the third row RG3 includes the second inputs of each PE in a column of PE that is combined to the second memory. Claim 2 In claim 1, the PE of each NPU 14 , PE 24 , PE 34 and PE 44 is a core of the NPU that is disabled in the first operating mode above. Claim 3 In Clause 2, when k = odd, the PE of each row of the NPU 12 , PE 22 , PE 32 and PE 42 is disabled, and if k = even, the PE of each row of the NPU 11 , PE 12 , PE 13 , PE 14 , PE 13 , PE 23 , PE 33 and PE 43 The core of the NPU that is disabled in the first operating mode above. Claim 4 In claim 3, the second value stored in the second memory includes a value for a 5x5 filter, the core of the NPU. Claim 5 In claim 1, in the second operating mode, the core of the NPU, for k = 1 to K, the PE of the k-th NPU 31 inner PE 34 PE that is coupled to each other's first input and first memory 31 inner PE 34 A core of an NPU comprising each row of a PE including each of the first inputs; and the second inputs of each PE in a column of a PE coupled to the second memory. Claim 6 In claim 5, the core of the NPU generates a convolution result from the first value and the second value in a second operating mode. Claim 7 In claim 1, the core of the NPU generates depth-wise convolution results at the first value and the second value in the first operating mode. Claim 8 A core of an NPU according to claim 1, wherein the first memory stores an activation value and the second memory stores a weight value. Claim 9 In claim 1, the second value stored in the second memory includes a value for a 3x3 filter, the core of the NPU. Claim 10 A method for configuring a core of a neural processing unit (NPU), wherein the core of the NPU comprises an array of NPUs arranged in C rows and K columns, where C and K are both integers divisible by 4, and each NPU comprises a 4x4 processing element (PE) arranged in 4 rows and 4 columns, and each PE comprises a first input and a second input, and the method comprises: for a first operating mode, configuring the core of the NPU into Int(C / 3) row groups (RG) to generate depth-wise convolution results from a first value and a second value, wherein each row group comprises three consecutively adjacent rows RG1 to RG3 of the PE; for k = 1 to K, a 2-to-1 multiplexer is placed in the first row and first column of each k-th NPU and PE 11 A step of configuring a first row RG1 to have an output coupled to the first input, wherein the 2-to-1 multiplexer includes a first input and a second input, and when k = 1, the first input for the 2-to-1 multiplexer is configured to be coupled to a first memory storing the first value, and for k = 1 to K-1, PEs placed in each first row and second to fourth columns of each k-th NPU 12 inner PE 14 The above first input of the PE of the k-th NPU 11 The second input of the above 2-to-1 multiplexer coupled to, and the PE of the (k+1)th NPU 11 A step of configuring the first input of the 2-to-1 multiplexer coupled to the first memory; wherein the 2-to-1 multiplexer is a PE placed in the second row and third column of each k-th NPU for k = 1 to K. 23 A step of configuring a second row RG2 to have an output coupled to the first input of, wherein when k = 1, a PE placed in the second row and first column of the k-th NPU 21 and PE placed in the 2nd row and 2nd column 22 The above first input of the PE of the k-th NPU 23 and the first input of the 2-to-1 multiplexer coupled to the first memory, configured to be coupled to the first memory, and for the case k = 2 to K, the PE of each k-th NPU 21 and PE 22 The above first input of the above PE of the (k-1)th NPU 23 The second input of the above 2-to-1 multiplexer coupled to, and the PE placed in the second row and fourth column of the (k-1)th NPU 24 The above first input of and the above PE of the k-th NPU 23 A step of configuring the first input of the 2-to-1 multiplexer coupled to the first memory to be coupled to the PE disposed in the third row and the first to fourth columns 31 inner PE 34 Each of the above first inputs is the PE of the k-th NPU 31 A method comprising: configuring a third row RG3 such that the first inputs of PE34 are combined with each other and the first memory; and configuring the second inputs of each PE in the column of PE to be combined with the second memory that stores the second value. Claim 11 In Clause 10, the PE of each NPU 14 , PE 24 , PE 34 and PE 44 A method further comprising the step of configuring to disable in the first operating mode. Claim 12 In Clause 11, when k = odd, the PE of each row of the NPU 12 , PE 22 , PE 32 and PE 42 The step of configuring to disable, and if k = even, the PE of each row of the NPU 11 , PE 12 , PE 13 , PE 14 , PE 13 , PE 23 , PE 33 and PE 43 A method comprising further including the step of configuring to be disabled in the first operating mode. Claim 13 In claim 12, the second value stored in the second memory includes a value for a 5x5 filter. Claim 14 In claim 10, the step of configuring the core of the NPU in a second operating mode is further included, and for k = 1 to K, PE 31 inner PE 34 Each of the above first inputs is the PE of the k-th NPU 31 inner PE 34 A method comprising: configuring each row of PE to be combined with each other's first inputs and the first memory; and further comprising the step of configuring the second inputs of each PE in the column of PE to be combined with the second memory. Claim 15 A method according to claim 14, further comprising the step of generating a convolution result from the first value and the second value in a second operating mode by the core of the NPU. Claim 16 A method according to claim 10, further comprising the step of generating depth-wise convolution results at the first value and the second value in the first operating mode by the core of the NPU. Claim 17 A method according to claim 10, wherein the first memory stores an activation value and the second memory stores a weight value. Claim 18 In claim 10, the method wherein the second value stored in the second memory includes a value for a 3x3 filter.
Citation Information
Patent Citations
An accelerator-aware pruning method for convolution neural networks and a recording medium thereof
KR101989793B1
Storage device and method for performing convolution operations
KR1020170099848A
Convolutional network hardware accelerator device, system and method
US20200310758A1
