Neural processing unit

Abstract

Neural processing units are provided. The neural processing units can be reconfigured to process fine-grained structured weight sparsity arrangements selected from N:M = 1:4, 2:4, 2:8, and 4:8 fine-grained structured weight sparsity arrangements. A weight buffer stores weight values, and a weight multiplexer array outputs one or more weight values stored in the weight buffer as first operand values based on the selected fine-grained structured weight sparsity arrangement. An activation buffer stores activation values, and an activation multiplexer array outputs one or more activation values stored in the activation buffer as second operand values based on the selected fine-grained structured weight sparsity arrangement, where each respective second operand value and respective first operand value form an operand value pair. A multiplier array outputs a product value for each operand value pair.

Description

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 408,827, filed September 21, 2022; U.S. Provisional Patent Application No. 63 / 408,828, filed September 21, 2022; and U.S. Patent Application No. 17 / 980,544, filed November 3, 2022, the entire disclosure of which is incorporated herein by reference. Technical Field

[0002] The subject matter disclosed herein relates to neural network processing apparatuses. More specifically, the subject matter disclosed herein relates to neural processing units that can be reconfigured to process fine-grained structured sparse weight arrangements selected from N:M = 1:4, 2:4, 2:8, and 4:8 fine-grained structured weight sparsity arrangements. Background Technology

[0003] Processing in deep neural networks (DNNs) can be accelerated by neural processing units (NPUs). That is, the sparsity of operands associated with the generalized matrix multiplication (GEMM) operation in DNNs can be used to accelerate operations performed by the NPU. Fine-grained structured sparsity (especially N:M sparsity (N non-zero elements out of M weight values)) can help maintain DNN accuracy and save hardware overhead compared to random sparsity. However, while additional fine-grained structured sparsity configurations exist (e.g., N:M = {1:4, 4:8, 2:8}), current techniques only support one N:M configuration (i.e., N:M = 2:4). Summary of the Invention

[0004] An example embodiment provides a neural processing unit that may include a weight buffer, a weight reusing unit, an activation buffer, an activation multiplexer, and a multiplier array. The weight buffer may be configured to store weight values ​​in a fine-grained structured sparse weight arrangement, the fine-grained structured sparse weight arrangement being selected from a group including at least two arrangements selected from 1:4, 2:4, 4:8, and 2:8 fine-grained structured sparse weight arrangements. The weight reusing array may be configured to output one or more weight values ​​stored in the weight buffer as a first operand value based on the selected fine-grained structured sparse weight arrangement. The activation buffer may be configured to store activation values. An activation multiplexer array may include inputs to the activation multiplexer array, the activation multiplexer array may be incorporated into an activation buffer, and the activation multiplexer array may be configured to output one or more activation values ​​stored in the activation buffer as second operand values, wherein each corresponding second operand value and a corresponding first operand value form an operand value pair. A multiplier array may be configured to output the product of each operand value pair. In one embodiment, the activation buffer may include eight activation registers for storing eight activation values, the weight reusable array may include a first weight reusable, the first weight reusable being configured to select a weight register based on a selected fine-grained structured sparse weight arrangement and output the weight value stored in the selected weight register as a first operand value, the activation multiplexer array may include a first activation multiplexer, the first activation multiplexer may include seven inputs, wherein each corresponding input of the first activation multiplexer may be connected to a corresponding activation register within a first set of activation registers, the first activation multiplexer may be configured to select an activation register in the first set of activation registers based on a selected fine-grained structured sparse weight arrangement and output the activation value stored in the selected activation register as a second operand value, wherein the second operand value corresponds to the first operand value and forms a first operand value pair, and the multiplier array may include a first multiplier unit, the first multiplier unit being configured to output the product value of the first operand value pair. In another embodiment, the weight values ​​can be stored in the weight buffer with a 1:4 fine-grained structured sparse weight arrangement or a 2:8 fine-grained structured sparse weight arrangement, and the first set of activation registers can include 7 activation registers. In yet another embodiment, the weight values ​​can be stored in the weight buffer with a 2:4 fine-grained structured sparse weight arrangement, and the first set of activation registers can include 4 activation registers. In yet another embodiment, the weight values ​​can be stored in the weight buffer with a 4:8 fine-grained structured sparse weight arrangement, and the first set of activation registers can include 6 activation registers.In one embodiment, the weight values ​​can be arranged in a 2:8 fine-grained structured sparsity configuration, and the activation registers can include four activation registers in two rows, wherein two output multiplexers can be configured to select one activation register from each row. In another embodiment, the weight values ​​can be arranged in a 2:4 fine-grained structured sparsity configuration, and the activation registers can include two activation registers in two rows, wherein two output multiplexers are configured to select one activation register from each row.

[0005] An example embodiment provides a neural processing unit that may include a first weight buffer, a first weight reusing unit, a first activation buffer, a first activation multiplexer, and a first multiplier unit. The first weight buffer may include an array of first weight registers, wherein each first weight register is configured to store weight values ​​in a fine-grained structured sparse weight arrangement, the fine-grained structured sparse weight arrangement being selected from a group including at least two arrangements selected from 1:4, 2:4, 4:8, and 2:8 fine-grained structured sparse weight arrangements. The first weight reusing unit is configured to select a first weight register based on the selected fine-grained structured sparse weight arrangement and output the weight value stored in the selected first weight register as a first operand value. The first activation buffer may be a first predetermined number of first activation registers, wherein each first activation register is configured to store an activation value. The first activation multiplexer may include a second predetermined number of first activation multiplexer inputs, wherein each corresponding input of the first activation multiplexer may be connected to a corresponding first activation register within a first set of first activation registers, and wherein the first activation multiplexer may be configured to select a first activation register based on a selected fine-grained structured sparsity weight arrangement, and output an activation value stored in the selected first activation register as a second operand value, the activation value output as the second operand value corresponding to a weight value output as the first operand value. A first multiplier unit may be configured to output a first product of the first operand value and the second operand value. In one embodiment, the first predetermined number of first activation registers may be 8, and the second predetermined number of activation inputs may be 7. In another embodiment, the weight values ​​may be arranged in a 1:4 fine-grained structured sparsity configuration. In yet another embodiment, the weight values ​​may be arranged in a 2:4 fine-grained structured sparsity configuration. In yet another embodiment, the weight values ​​may be arranged in a 4:8 fine-grained structured sparsity configuration. In one embodiment, the weight values ​​may be arranged in a 2:8 fine-grained structured sparsity configuration.In another embodiment, the neural processing unit may further include a second weighting multiplexer, a second activation multiplexer, and a second multiplier unit. The second weighting multiplexer is configured to select a first weight register based on a selected fine-grained structured sparse weight arrangement and output a weight value stored in the selected first weight register as a third operand value. The second activation multiplexer may include a second predetermined number of second activation multiplexer inputs, wherein each corresponding input of the second activation multiplexer is connected to a corresponding first activation register within a second set of activation registers different from the first set of first activation registers. The second activation multiplexer is configured to select a first activation register based on a selected fine-grained structured sparse weight arrangement and output an activation value stored in the selected first weight register as a fourth operand value. The activation value output as the fourth operand value corresponds to a weight value output as the third operand value. The second multiplier unit is configured to output a second product of the third operand value and the fourth operand value.In another embodiment, the neural processing unit may further include a second weight buffer, a third weight resizer, a second activation buffer, a third activation multiplexer, a third multiplier unit, a fourth weight resizer, a fourth activation multiplexer, and a fourth multiplier unit. The second weight buffer may be configured to store weight values ​​of fine-grained structured sparse weights based on a selected fine-grained structured sparse weight arrangement. The third weight resizer may be configured to select a second weight register based on the selected fine-grained structured sparse weight arrangement and output the weight value stored in the selected second weight register as a fifth operand value. The second activation buffer may include a first predetermined number of second activation registers, each second activation register being configured to store an activation value. The third activation multiplexer includes a second predetermined number of third activation multiplexer inputs, each corresponding input of the third activation multiplexer being connected to a corresponding second activation register within a first group of second activation registers. The third activation multiplexer is configured to select a second activation register based on the selected fine-grained structured sparse weight arrangement and... The activation value stored in the selected second activation register is output as the sixth operand value. The activation value output as the sixth operand value corresponds to the weight value output as the fifth operand value. The third multiplier unit can be configured to output the third product of the fifth operand value and the sixth operand value. The fourth weight multiplexer can be configured to select the second weight register based on the selected fine-grained structured sparse weight arrangement and output the weight value stored in the selected second weight register as the seventh operand value. The fourth activation multiplexer can include a second predetermined number of fourth activation multiplexer inputs, wherein each corresponding input of the fourth activation multiplexer can be connected to a corresponding second activation register in a fourth group of second activation registers different from the third group of activation registers. The fourth activation multiplexer can be configured to select the second activation register based on the selected fine-grained structured sparse weight arrangement and output the activation value stored in the selected second activation register as the eighth operand value. The fourth multiplier unit can be configured to output the fourth product of the seventh operand value and the eighth operand value. In one embodiment, the first predetermined number of first activation registers can be 8 first activation registers, the second predetermined number of first activation multiplexer inputs can be 7 first activation multiplexer inputs, the second predetermined number of second activation multiplexer inputs can be 7 second activation multiplexer inputs, the first predetermined number of second activation registers can be 8 second activation registers, the second predetermined number of third activation multiplexer inputs can be 7 third activation multiplexer inputs, and the second predetermined number of fourth activation multiplexer inputs can be 7 fourth activation multiplexer inputs.In another embodiment, the weight values ​​can be arranged in a 1:4 fine-grained structured sparsity configuration. The first group of first activation registers may include four first activation registers, the second group of first activation registers may include four first activation registers and is different from the first group of first activation registers, the third group of second activation registers may include four second activation registers, and the fourth group of second activation registers may include four second activation registers and is different from the third group of second activation registers. In yet another embodiment, the weight values ​​can be arranged in a 2:8 fine-grained structured sparsity configuration. The first group of first activation registers may include seven first activation registers, the second group of first activation registers may include seven first activation registers and is different from the first group of first activation registers, the third group of second activation registers may include seven second activation registers, and the fourth group of second activation registers may include seven second activation registers and is different from the third group of second activation registers. In another embodiment, the weight values ​​can be arranged in a 2:4 fine-grained structured sparsity configuration. Activation values ​​can be stored in four first activation registers of a first activation buffer and in four second activation registers of a second activation buffer. A first set of first activation registers may include four first activation registers storing activation values. A second set of first activation registers may include the same four first activation registers as the first set of second activation registers. A third set of second activation registers may include four second activation registers storing activation values. A fourth set of second activation registers may include the same four second activation registers as the third set of activation registers. In one embodiment, the weight values ​​can be arranged in a 4:8 fine-grained structured sparsity configuration. Activation values ​​can be stored in six first activation registers of a first activation buffer and six second activation registers of a second activation buffer. A first set of first activation registers may include six first activation registers storing activation values. A second set of first activation registers may include the same six activation registers as the first set of second activation registers. A third set of second activation registers may include six second activation registers storing activation values. A fourth set of second activation registers may include the same six second activation registers as the third set of second activation registers. In another embodiment, the weight values ​​can be arranged in a 2:8 fine-grained structured sparsity configuration, and the first activation register can include four activation registers in two rows, wherein two output multiplexers can be configured to select one activation register from each row. In yet another embodiment, the weight values ​​can be arranged in a 2:4 fine-grained structured sparsity configuration, and the first activation register can include two activation registers in two rows, wherein two output multiplexers can be configured to select one activation register from each row. Attached Figure Description

[0006] In the following sections, aspects of the subject matter disclosed herein will be described with reference to exemplary embodiments shown in the accompanying drawings, wherein:

[0007] Figure 1A Describe an example dot product operation that is typically performed in a neural network;

[0008] Figure 1B Describe an example dot product operation performed by a single multiplication and accumulation (MAC) unit;

[0009] Figure 1C Describe a sparse example dot product operation where one of the multiple sets of operands is used.

[0010] Figure 2 An example depicting a set of dense weight values ​​being formed into a set of N:M fine-grained structured sparse weight values;

[0011] Figure 3A Describe four possible sparse mask cases or patterns for cross-channel C0 for a 1:4 fine-grained structured weighted sparsity arrangement.

[0012] Figure 3B An example configuration is depicted for routing logic that selects appropriate activation values ​​from the activation buffer based on a weight sparse mask for a 1:4 fine-grained structured weight sparsity arrangement, according to the subject matter disclosed herein.

[0013] Figure 4A Describe six possible sparse mask cases or patterns for cross-channel C0 for a 2:4 fine-grained structured weighted sparsity arrangement.

[0014] Figure 4B An example configuration is depicted for routing logic for selecting appropriate activation values ​​from an activation buffer based on a weight zero-bit mask for, for example, a 2:4 fine-grained structured weight sparsity arrangement, according to the subject matter disclosed herein.

[0015] Figure 5A Describe 13 of the 28 possible sparse mask cases or patterns for cross-channel C0 for a 2:8 fine-grained structured weighted sparsity arrangement.

[0016] Figure 5B An example configuration is depicted for routing logic for selecting appropriate activation values ​​from an activation buffer based on a weight zero-bit mask, for example, for a 2:8 fine-grained structured weight sparsity arrangement, according to the subject matter disclosed herein.

[0017] Figure 6A Describe two of the 70 possible sparse mask cases or patterns for cross-channel C0 for a 4:8 fine-grained structured weighted sparsity arrangement.

[0018] Figure 6BAn example configuration of routing logic for selecting appropriate activation values ​​from the activation buffer based on a weight zero-bit mask for a 4:8 fine-grained structured weight sparsity arrangement, as described in the subject matter disclosed herein.

[0019] Figure 7 Example embodiments of neural processing units that are reconfigurable for fine-grained structured sparse arrangements of 1:4, 2:4, 2:8 and 4:8, based on the subject matter disclosed herein;

[0020] Figure 8A and Figure 8B The active buffers, which can be used in 2:8 and 2:4 weighted fine-grained structured sparse NPU architectures with greater area efficiency, are described respectively, based on the subject matter disclosed herein.

[0021] Figure 9A and Figure 9B This demonstrates, respectively, how to make 2:8 and 2:4 weighted fine-grained structured sparse NPU architectures more area-efficient according to the subject matter disclosed herein; and

[0022] Figure 10 An electronic device 1000, which may include at least one NPU configured for one or more N:M fine-grained structured sparsity arrangements, can be described according to the subject matter disclosed herein. Detailed Implementation

[0023] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the disclosure. However, those skilled in the art will understand that the aspects disclosed may be practiced without these specific details. In other instances, well-known methods, processes, components, and circuits have not been described in detail so as not to obscure the subject matter disclosed herein.

[0024] Throughout this specification, references to "an embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment disclosed herein. Therefore, the phrases "in one embodiment," "in an embodiment," or "according to an embodiment" (or other phrases with similar meanings) appearing in various places throughout this specification do not necessarily all refer to the same embodiment. Furthermore, in one or more embodiments, a particular feature, structure, or characteristic may be combined in any suitable manner. In this regard, as used herein, the word "exemplary" means "serving as an example, instance, or illustration." Any embodiment described herein as "exemplary" is not to be construed as necessarily preferred or superior to other embodiments. Additionally, in one or more embodiments, a particular feature, structure, or characteristic may be combined in any suitable manner. Furthermore, depending on the context discussed herein, singular terms may include corresponding plural forms, and plural terms may include corresponding singular forms. Similarly, hyphenated terms (e.g., "two-dimensional", "pre-determined", "specific pixel", etc.) may occasionally be used interchangeably with their corresponding non-hyphenated versions (e.g., "two-dimensional", "pre-determined", "specific pixel", etc.), and uppercase entries (e.g., "Counter Clock", "Row Select", "PIXOUT", etc.) may be used interchangeably with their corresponding non-uppercase versions (e.g., "counter clock", "row select", "pixout", etc.). Such occasional interchangeability should not be considered inconsistent with each other.

[0025] Furthermore, depending on the context of this discussion, singular terms may include corresponding plural forms, and plural terms may include corresponding singular forms. It should also be noted that the various figures shown and discussed herein (including component diagrams) are for illustrative purposes only and are not drawn to scale. For example, the size of some elements may be exaggerated relative to others for clarity. Additionally, reference numerals have been repeated in the figures where appropriate to indicate corresponding and / or similar elements.

[0026] The terminology used herein is for the purpose of describing some exemplary embodiments only and is not intended to limit the claimed subject matter. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “including” are used in this specification, they specify the presence of stated features, integrals, steps, operations, elements, components, and / or groups thereof, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. As used herein, the terms “first,” “second,” etc., serve as labels for nouns following them and do not indicate any type of order (e.g., spatial, temporal, logical, etc.) unless so explicitly defined. Furthermore, the same reference numerals may be used between two or more figures to denote parts, components, blocks, circuits, units, or modules having the same or similar functions. However, such use is merely for simplification and ease of discussion and does not imply that the construction or structural details of such components or units are identical across all embodiments, or that such commonly referenced parts / modules are the only way to implement some of the exemplary embodiments disclosed herein.

[0027] It will be understood that when an element or layer is referred to as being "on," "connected to," or "bonded to" another element or layer, it may be directly on, directly connected to, or directly bonded to the other element or layer, or there may be intermediate elements or layers present. Conversely, when an element or layer is referred to as being "directly on," "directly connected to," or "directly bonded to" another element or layer, there are no intermediate elements or layers present. The same reference numerals always denote the same element. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0028] As used herein, the terms “first,” “second,” etc., serve as labels for nouns that follow them and do not indicate any type of order (e.g., spatial, temporal, logical, etc.) unless explicitly defined as such. Furthermore, the same reference numerals may be used between two or more figures to denote parts, components, blocks, circuits, units, or modules having the same or similar functions. However, such use is merely for simplification and ease of discussion and does not imply that the construction or structural details of such components or units are identical across all embodiments, or that such commonly referenced parts / modules are the only way to implement some of the exemplary embodiments disclosed herein.

[0029] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this subject pertains. It will also be understood that, unless clearly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and shall not be interpreted in an idealized or overly formalized manner.

[0030] As used herein, the term "module" means any combination of software, firmware, and / or hardware configured to provide the functionality described herein in conjunction with modules. For example, software may be implemented as a software package, code, and / or instruction set or instructions, and the term "hardware" as used in any implementation described herein may individually or in any combination include, for example, assemblies, hardwired circuit systems, programmable circuit systems, state machine circuit systems, and / or firmware storing instructions executed by the programmable circuit system. Modules may be implemented collectively or individually as circuit systems forming part of a larger system (e.g., but not limited to, integrated circuits (ICs), system-on-a-chip (SoCs), assemblies, etc.).

[0031] The subject matter disclosed herein provides efficient hardware logic architectures in NPUs supporting N:M fine-grained structured sparsity for N:M = 1:4, 4:8, and 2:8. Additionally, the subject matter disclosed herein provides reconfigurable sparse logic architectures supporting N:M = {1:4, 2:4, 2:8, 4:8} fine-grained structured sparsity. That is, a single sparse NPU logic architecture can be reconfigured to efficiently support four different N:M sparsity modes. For example, the NPU architecture disclosed herein provides a 4x speedup for N:M = 2:8 or for N:M = 1:4. Furthermore, in the N:M sparse logic architecture disclosed herein, the activation buffer size and multiplexer complexity can be reduced at the cost of a demultiplexer per multiplier unit and an additional adder tree, but the overall power reduction and area efficiency can be improved by more than 3.5x.

[0032] While conventional N:M sparsity clusters M weights in a reducible dimension (the input channels used for convolution operations, or the column vectors of the weight matrix in a GEMM operation), the subject matter disclosed herein can also cluster M weights from both a reducible dimension and an irreducible dimension (the output channels or output pixels used for convolution operations, or the row vectors of the weight matrix in a GEMM operation).

[0033] Figure 1AThe diagram depicts an example dot product operation 100 typically performed in a neural network. In 103, the dot product of a first set of dense operands 101 (which can be considered activation values) and a second set of dense operands 102 (which can be considered weight values) is formed. The input dimension C0 is reduced after the dot product operation. Figure 1B An example dot product operation 100' is depicted, performed by a single multiplication and accumulation (MAC) unit 110. A first set of dense operands 101' and a second set of dense operands 102' are sequentially input into multiplier (X) 111 to form a series of product values, which are then summed together by accumulator 112. As previously described, operand 101' can be considered as activation values, and operand 102' can be considered as weight values. Figure 1C One of the multiple sets of operands is depicted as a sparse example dot product operation 100". The dot product operation is depicted as being performed by a single MAC unit 110 on a first set of dense operands 101" and a set of sparse operands 102". Figure 1C In the example depicted, only one operand 102” has a non-zero value, while the other operands 102” have zero values. Similarly, operand 101” can be considered an activation value, and operand 102” can be considered a weight value. Activation multiplexer (AMUX) 113 can be used to select the appropriate operand 101” to correspond to the non-zero value operand 102”. Controller (not shown) can control the multiplexer to select the appropriate activation value based on, for example, metadata or a weight zero-bit mask associated with operand 102”.

[0034] The weights of the trained neural network are fixed, known values, while the activation values ​​depend on and vary depending on the input to the network. The weights of the trained neural network can be dense, or optionally, they can be pruned and then compressed to form dense weights. The weights can be arranged in an N:M fine-grained structured sparse arrangement. In one embodiment, the weights can be stored in a weight buffer (e.g., a weight register of the weight buffer) in a selected N:M fine-grained structured sparse arrangement. In one embodiment, a weight reusable array can output one or more weight values ​​stored in the weight buffer as operand 102 based on the selected N:M fine-grained structured sparse arrangement (e.g., a weight reusable in the weight reusable array can select a corresponding weight register based on the selected N:M fine-grained structured sparse arrangement and output the weight values ​​stored in the selected weight register as operand 102).

[0035] Figure 2An example of a dense set of weight values ​​being formed as a set of N:M fine-grained structured sparse weight values ​​is depicted. In 201, the dense weight values ​​W are depicted in the example matrix, where R is the number of output channels and C is the number of channels in the linear layer of the neural network. Relative values ​​are depicted as light matrix elements (blocks) and dark matrix elements (blocks), where relatively low value elements are depicted as lighter gray and relatively high value elements are depicted as darker gray. In 202, before pruning, the weight values ​​W are grouped into 4×4 groups. In 203, sparse subnet masks for the two weight groups are indicated. After pruning, in 205, the pruned weights are deployed in an N:M fine-grained structured sparse arrangement, in which at most N weights in each group of M consecutive weights have non-zero values. Indicated in 205... This means that since only N elements of the M weights in C channels are retained, the channel size of the weight tensor shrinks from C to [a smaller value].

[0036] Figure 3A Four possible sparse masking cases or patterns for cross-channel C0 are depicted for a 1:4 fine-grained structured weighted sparse arrangement. As indicated by the gray shading blocks, Figure 3A The sparse mask cases 1 to 4 indicated in the text can be considered as depicting four different register positions in which non-zero weight values ​​can be located in a four-register weight buffer or a four-position weight buffer. Figure 3B This document describes an example configuration of routing logic for selecting appropriate activation values ​​from the Activation Buffer (ABUF) based on a weight sparse mask or weight metadata for a 1:4 fine-grained structured weight sparsity arrangement, according to the subject matter disclosed herein.

[0037] The corresponding inputs to the 4-to-1 Activation Multiplexer (AMUX) are each connected to the corresponding ABUF register (REG) of the four-register ABUF. The output of AMUX is input to the multiplier MULT. AMUX can be controlled by a control unit (not shown) to select a specific ABUF register based on, for example, a weight zero-bit mask or weight metadata. Figure 3B The routing logic described in [the document] is not configured to "borrow" activation values ​​from future periods (i.e., advance "borrowing"), but rather to "borrow" activation values ​​from neighboring channels (i.e., backup "borrowing"). w Indicates the maximum channel range (backup distance) to which the active value can be routed. That is, if the active value is in its original position (0), the maximum (farthest) backup position to which it can be routed is position 3. For Figure 3B The routing logic described in C w The value is 3. If bidirectional routing is allowed, then C equals 3. wThis will be indicated as ±1.5. That is, the activation value can be routed evenly from the left middle position (-1.5) to the right middle position (+1.5). 1+C w Instructions for AMUX to allow the activation value to be routed from its original location to C w Backup position input fan-in.

[0038] Figure 3B The routing logic described in the text for structured sparsity of N:M=1:4 can also be used in (T w C w ,K w The operation is performed on a random (irregular) sparse pattern of T = (3,0,0), where T w It is the precedence in time, C w It is the backup in the input channel, and K w It is the backup in the output channel, and the w subscript indicates the weight. By providing a 4x speedup, operating under structured sparsity of N:M=1:4 is more efficient than operating under random sparsity in (3,0,0), and since the speedup is based on a non-zero value distribution pattern, the ideal speedup for random sparsity in (3,0,0) is not always feasible.

[0039] Figure 4A Six possible sparse masking cases or patterns for cross-channel C0 are depicted for a 2:4 fine-grained structured weighted sparse arrangement. As indicated by the gray shading blocks, Figure 4A The sparse mask cases 1 to 6 indicated in the text can be considered as depicting six different register locations in which non-zero weight values ​​can reside in the four-register weight buffer. Figure 4B This document describes an example configuration of routing logic for selecting appropriate activation values ​​from the Activation Buffer (ABUF) based on a weight zero-bit mask or weight metadata for, for example, a 2:4 fine-grained structured weight sparsity arrangement, according to the subject matter disclosed herein.

[0040] exist Figure 4BIn the example configuration of the routing logic depicted, the AMUX array is an array of two multiplexers (MUX), where each multiplexer is a 3-to-1 multiplexer. As shown, the corresponding inputs to the multiplexers in the AMUX array are connected to the register (REG) locations of the four-register ABUF. More specifically, the three inputs to the leftmost multiplexer are connected to the leftmost three ABUF registers. In one embodiment, the ABUF array may include two active buffers, each with a register width of 2. The three inputs to the rightmost multiplexer are connected to the rightmost three ABUF registers. The output of each corresponding multiplexer is input to the corresponding multiplier in the multiplier MULT array. Each multiplier in the MULT array is indicated by a block containing X. The multiplexers in the AMUX array can each be controlled by a control unit (not shown), which selects a specific ABUF register based on, for example, a weight zero-bit mask or weight metadata. Figure 4B The example configuration of the routing logic depicted is not configured to "borrow" activation values ​​from future periods, but rather to "borrow" activation values ​​from neighboring channels.

[0041] against Figure 4B The maximum channel range C of the routing logic w It is 2, or ±1 for bidirectional routes. Figure 4B The routing logic can also be based on (T) w C w ,K w The operation is performed on the random sparse pattern of (1,1,0).

[0042] Figure 5A Depicts 13 of the 28 possible sparse mask cases or patterns for cross-channel C0 used in a 2:8 fine-grained structured weighted sparse arrangement. As indicated by the gray shading blocks, Figure 5A The 13 sparse mask cases indicated in the text can be considered as depicting different register positions of the eight-register weight buffer where non-zero weight values ​​can reside. Two different gray shadings depict two different non-zero weight values. Figure 5B This document describes an example configuration of routing logic for selecting appropriate activation values ​​from an activation buffer (ABUF) based on, for example, a weight zero-bit mask or weight metadata for a 2:8 fine-grained structured weight sparsity arrangement, according to the subject matter disclosed herein.

[0043] exist Figure 5BIn the example configuration of the routing logic depicted, the AMUX array is an array of two multiplexers (MUX), each of which is a 7-to-1 multiplexer. As shown, the corresponding inputs to the multiplexers in the AMUX array are connected to the register (REG) locations of the eight-register ABUF. That is, the seven inputs to the leftmost multiplexer are connected to the leftmost seven ABUF registers. The seven inputs to the rightmost multiplexer are connected to the rightmost seven ABUF registers. In one embodiment, the ABUF array may include two active buffers, each with a register width of 4. The output of each corresponding multiplexer is input to the corresponding multiplier in the multiplier MULT array. The multipliers in the MULT array are indicated by blocks containing X. The multiplexers in the AMUX array can each be controlled by a control unit (not shown), which selects a specific ABUF register based on, for example, a weight zero-bit mask or weight metadata. Figure 5B The example configuration of the routing logic depicted is not configured to "borrow" activation values ​​from future periods, but rather to "borrow" activation values ​​from neighboring channels.

[0044] against Figure 5B The maximum channel range C of the routing logic w It is 6, or ±3 for bidirectional routes. Figure 5B The routing logic can also be based on (T) w C w ,K w The operation is performed on the random sparse pattern of (3,1,0).

[0045] Figure 6A Depicts two of the 70 possible sparse mask cases or patterns for cross-channel C0 used in a 4:8 fine-grained structured weighted sparse arrangement. As indicated by four distinct gray shading blocks, Figure 6A The two sparse mask cases indicated in the text describe the cases when all non-zero weight values ​​are on the left and when all non-zero weight values ​​are on the right. Figure 6B This document describes an example configuration of routing logic for selecting appropriate activation values ​​from the Activation Buffer (ABUF) based on a weight zero-bit mask or weight metadata for a 4:8 fine-grained structured weight sparsity arrangement, according to the subject matter disclosed herein.

[0046] exist Figure 6BIn the example configuration of the routing logic depicted, the AMUX array is an array of four multiplexers (MUX), where each multiplexer is a 5-to-1 multiplexer. As shown, the corresponding inputs to the multiplexers in the AMUX array are connected to the register (REG) locations of the eight-register ABUF. For example, the five inputs to the leftmost multiplexer are connected to the leftmost five ABUF registers. Similarly, the five inputs to the rightmost multiplexer are connected to the rightmost five ABUF registers. In one embodiment, the ABUF array may include four active buffers, each with a register width of 2. The output of each corresponding multiplexer is input to the corresponding multiplier in the multiplier MULT array. The multipliers in the MULT array are indicated by blocks containing X. The multiplexers in the AMUX array can each be controlled by a control unit (not shown), which selects a specific ABUF register based on, for example, a weight zero-bit mask or weight metadata. Figure 6B The routing logic described is not configured to "borrow" activation values ​​from future periods, but rather to "borrow" activation values ​​from neighboring channels.

[0047] against Figure 6B The maximum channel range C of the routing logic w It is 4, or ±2 for bidirectional routes. Figure 6B The routing logic can also be based on (T) w C w ,K w The operation is performed on the random sparse pattern of (1,2,0).

[0048] Figure 7 Example embodiments of a neural processing unit (NPU) 700, reconfigurable for fine-grained structured sparse arrangements of 1:4, 2:4, 2:8, and 4:8, are depicted according to the subject matter disclosed herein. The NPU 700 may include four multipliers configured in a MULT array. Multipliers in the MULT array are represented by blocks containing X. The NPU 700 may also include four activation multiplexers configured in an AMUX array. Multiplexers in the AMUX array are represented by a trapezoidal shape. Activation buffers may be configured as four four-register buffers and arranged in an ABUF array. Each multiplexer in the AMUX array may be a 7-to-1 multiplexer. Inputs to two of the multiplexers in the AMUX array can be combined with those described herein. Figure 5B The same method described is used to connect to the two four-register buffers. The connection between the multiplexer of the AMUX array and the registers of the ABUF array can be as follows: Figure 7 As shown in the image.

[0049] By selectively placing activation values ​​in the registers of the ABUF array, the NPU 700 architecture can be used for fine-grained structured weight sparsity arrangements of 1:4, 2:4, 2:8, and 4:8. (See reference...) Figure 7 As indicated on the far left of each NPU 700 configuration, the corresponding active channels can be indexed. The channel index changes based on which of the four fine-grained structured sparsity arrangements the NPU 700 has been configured for.

[0050] When the NPU 700 is configured for 2:8 fine-grained structured weight sparsity, the connections between the ABUF array, AMUX array, and MULT array are depicted as an N:M = 2:8 configuration. Each of the sixteen active channels is input to a register in the corresponding ABUF array. The multiplexer in the AMUX array is controlled by a controller (not shown) to select the appropriate ABUF register based on, for example, a weight zero-bit mask or weight metadata associated with the 2:8 fine-grained structured weight sparsity value.

[0051] When the NPU 700 is configured for 1:4 fine-grained structured weight sparsity, the connection between the ABUF array, AMUX array, and MULT array is depicted as an N:M = 1:4 configuration. The N:M = 1:4 configuration is the same as the N:M = 2:8 configuration. For the N:M = 1:4 configuration, each of the 16 active channels is input to a corresponding register in the ABUF array. The multiplexer in the AMUX array is controlled by a controller (not shown) to select the appropriate ABUF register based on, for example, a weight zero-bit mask or weight metadata associated with the 1:4 fine-grained structured weight sparsity value.

[0052] When the NPU 700 is configured for 2:4 fine-grained structured weight sparsity, the connections between the ABUF array, AMUX array, and MULT array are depicted as an N:M = 2:4 configuration. For the N:M = 2:4 configuration, each of the eight active channels is input to a corresponding register in the ABUF array as indicated. The multiplexer in the AMUX array is controlled by a controller (not shown) to select the appropriate ABUF register based on, for example, a weight zero-bit mask or weight metadata associated with the 2:4 fine-grained structured weight sparsity value.

[0053] When the NPU 700 is configured for 4:8 fine-grained structured weight sparsity, the connections between the ABUF array, AMUX array, and MULT array are depicted as an N:M = 4:8 configuration. For the N:M = 4:8 configuration, each of the eight active channels is input to the corresponding register in the ABUF array as indicated. More specifically, the two topmost multipliers have access to channels 1 through 6. The topmost multiplier has access to channels 1 through 5, and the next multiplier has access to channels 2 through 6, which... Figure 6B This corresponds to the NPU configuration depicted. Additionally, the two bottom multipliers have access to channels 3 through 8, the third multiplier from the top has access to channels 3 through 7, and the bottom multipliers have access to channels 4 through 8, which also corresponds to... Figure 6B The NPU configuration depicted corresponds to this. The multiplexer in the AMUX array is controlled by a controller (not shown) to select the appropriate ABUF register based on, for example, a weight zero-bit mask or weight metadata associated with the 4:8 fine-grained structured weight sparsity values.

[0054] Table 1 illustrates the hardware costs and acceleration benefits of the four different NPU-only (W-Only) structured sparse core architectures disclosed herein. Figure 3A and Figure 3B The W1:4 sparse NPU logic design described herein can only be configured to operate in both 1:4 and 2:4 sparse modes, while Figure 4A and Figure 4B The W 2:4 sparse NPU logic design described herein can be configured to operate only in 2:4 sparse mode. For example... Figure 7 The description in Figure 5A and Figure 5B The W 2:8 sparse NPU logic design described herein can only be configured to operate in all four sparse modes, and Figure 6A and Figure 6B The W-only 4:8 sparse NPU logic design described herein can be configured to operate in both 2:4 and 4:8 sparse modes. The W-only 2:8 NPU logic design provides programmers with the freedom to choose any of four (1:4, 2:4, 2:8, or 4:8) N:M fine-grained structured sparsity modes. Table 1 also includes information on AMUX fan-in, ABUF size (width), and computational acceleration associated with each of the four different NPU logic designs disclosed herein.

[0055] Table 1 also shows the approximate computational speedup provided by each random weight sparsity pattern for approximately 80% sparsity in the different NPU logic designs disclosed herein.

[0056] Table 1

[0057]

[0058]

[0059] Figure 8A and Figure 8BThe paper describes how the area efficiency of ABUF can be improved for 2:8 and 2:4 weighted fine-grained structured sparse NPU architectures, based on the topics disclosed herein. The size of the ABUF for both 2:8 and 2:4 NPU architectures can be reduced, which allows for a reduction in the complexity of the corresponding AMUX. This reduction in size and complexity can be achieved through the hardware cost of one demultiplexer and one additional adder tree per multiplier.

[0060] Figure 8A The left side depicts 13 of the 28 possible sparse mask cases or patterns for cross-channel C0 for a 2:8 fine-grained structured weighted sparsity arrangement. ABUF configurations for a 2:8 fine-grained structured weighted sparsity NPU (such as...) Figure 5B (As shown) is an 8-channel ABUF with a register depth dimension of 1. This configuration of the ABUF corresponds to a weighted buffer as an 8-channel WBUF with a depth of 1. Figure 8A As shown on the right, the ABUF size can be reconfigured to a 4-channel ABUF with a register depth dimension of 2, where some example 8-to-2 sparsity may be depicted. To further reduce the size of the ABUF used in this NPU architecture, the WBUF can also be reconfigured to a 4-channel buffer with a register depth dimension of 2. The AMUX corresponding to the reduced-size ABUF becomes instead... Figure 5B The diagram depicts two 4-to-1 multiplexers for each of the 7-to-1 multiplexers. The outputs of the two smaller multiplexers are fed into two multipliers. The output of each multiplier is combined into a 1-to-2 demultiplexer. The corresponding output of each 1-to-2 demultiplexer is combined into an adder tree, in which a second adder tree results in an increase in the size of the ABUF and WBUF.

[0061] Figure 8B The left side depicts six possible sparse masking cases or patterns for cross-channel C0 for a 2:4 fine-grained structured weighted sparsity arrangement. ABUF configurations for a 2:4 fine-grained structured weighted sparsity NPU (such as...) Figure 4B (As shown) is a 4-channel ABUF with a register depth of 1. This configuration of the ABUF corresponds to a weight buffer as a 4-channel ABUF with a register depth of 1. Figure 8B As shown on the right, the ABUF size can be reconfigured to a 2-channel ABUF with a depth dimension of 2, where some example 4-to-2 sparsity may be depicted. To further reduce the size of the ABUF used in this NPU architecture, the WBUF can also be reconfigured to a 2-channel buffer with a register depth dimension of 2. The AMUX corresponding to the reduced-size ABUF becomes instead... Figure 5BThe diagram depicts two 2-to-1 multiplexers for each of the 3-to-1 multiplexers. The outputs of the two size-reduced multiplexers are fed into two multipliers, respectively. The output of each multiplier is combined into a 1-to-2 demultiplexer. The corresponding output of each 1-to-2 demultiplexer is combined into an adder tree, in which a second adder tree results in an increase in the size of the ABUF and WBUF.

[0062] Figure 9A and Figure 9B The paper demonstrates how, based on the subject matter disclosed herein, 2:8 and 2:4 weighted fine-grained structured sparse NPU architectures can be made more area-efficient. Figure 9A The example dense weighted data path layout is depicted, where the WBUF includes two output weight channels for four weight input channels, and the ABUF includes one active output channel for four active input channels. Each weight output channel is input to a corresponding multiplier (indicated by a block containing X). The active output channel is broadcast to each of the two multipliers. The respective outputs of the multipliers are each combined into a single adder tree.

[0063] Figure 9B The example sparse weighted data path arrangement is depicted, where the WBUF includes two output weight channels for four weight input channels, and the ABUF includes one activation output channel for four activation input channels. Each weight output channel is input to a corresponding multiplier (indicated by a block containing X). An activation output channel is combined to each of the two multipliers via two 4-to-1 activation multiplexers. Each activation multiplexer is controlled to select the activation value corresponding to the non-zero weight input to the multiplier. The corresponding outputs of the multipliers are each combined to a 1-to-2 demultiplexer. Each output of the demultiplexer is combined to a first adder tree and a second adder tree, respectively.

[0064] The reduction in ABUF size described herein applies to all four NPU weight-only (W-only) structured sparse core architectures disclosed herein. Table 2 illustrates the hardware costs and acceleration benefits of the four different NPU weight-only (W-only) structured sparse core architectures disclosed herein. As indicated in the "→2D" column immediately to the right of the different N:M sparsity modes, the hardware costs and benefits for the reduced ABUF size (i.e., for ABUFs that have been reconfigured from a depth dimension of 1 to a depth dimension of 2) are also shown.

[0065] Table 2

[0066]

[0067]

[0068] Figure 10The electronic device 1000, depicting the subject matter disclosed herein, may include at least one NPU configured for one or more N:M fine-grained structured sparsity arrangements. The electronic device 1000 and its various system components may be formed from one or more modules. The electronic device 1000 may include a controller (or CPU) 1010, input / output (I / O) devices 1020 (such as, but not limited to, a keypad, keyboard, display, touchscreen display, 2D image sensor, 3D image sensor), memory (e.g., a memory device) 1030, an interface 1040, a graphics processing unit (GPU) 1050, an imaging processing unit (e.g., an image processing unit) 1060, a neural processing unit 1070, and a time-of-flight (TOF) processing unit 1080, all coupled to each other via a bus 1090. In one embodiment, the 2D image sensor and / or the 3D image sensor may be part of the imaging processing unit 1060. In another embodiment, the 3D image sensor may be part of the TOF processing unit 1080. The controller 1010 may include, for example, at least one microprocessor, at least one digital signal processor, at least one microcontroller, etc. The memory 1030 may be configured to store command codes to be used by the controller 1010 and / or to store user data. The neural processing unit 1070 may include at least one NPU configured for one or more N:M fine-grained structured sparse arrangements according to the subject matter disclosed herein.

[0069] Interface 1040 may be configured to include a wireless interface, which is configured to transmit data to or receive data from, for example, a wireless communication network using RF signals. Wireless interface 1040 may include, for example, an antenna. Electronic device 1000 can also be used in communication interface protocols of communication systems, such as, but not limited to, Code Division Multiple Access (CDMA), Global System for Mobile Communications (GSM), North American Digital Communications (NADC), Extended Time Division Multiple Access (E-TDMA), Wideband CDMA (WCDMA), CDMA 2000, WiFi, Muni WiFi, Bluetooth, Digital Enhanced Cordless Telecommunications (DECT), Wireless Universal Serial Bus (Wireless USB), Fast Low Latency Access with Seamless Switching (Flash-OFDM), IEEE 802.20, General Packet Radio Service (GPRS), iBurst, Wireless Broadband (WiBro), WiMAX, Advanced WiMAX, Universal Mobile Telecommunications Service - Time Division Duplex (UMTS-TDD), High Speed ​​Packet Access (HSPA), Evolved Data Optimized (EVDO), Advanced Long Term Evolution (LTE-Advanced), Multichannel Multipoint Distribution Service (MMDS), 5G, 6G, etc.

[0070] Embodiments of the subject matter and operations described in this specification may be implemented in digital electronic circuit systems, or in computer software, firmware, or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of these. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs (i.e., one or more modules of computer program instructions) encoded on a computer storage medium to be executed by or to control the operation of a data processing device. Optionally or additionally, the program instructions may be encoded on artificially generated propagating signals (e.g., electrical, optical, or electromagnetic signals generated by a machine to be generated as encoded information for transmission to a receiver device suitable for execution by the data processing device). The computer storage medium may be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or apparatus, or a combination thereof, or may be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or apparatus, or a combination thereof. Furthermore, although the computer storage medium is not a propagating signal, it may be a source or destination of computer program instructions encoded in artificially generated propagating signals. Computer storage media may also be one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices), or may be included within one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices). Furthermore, the operations described herein can be implemented as operations performed by a data processing device on data stored on one or more computer-readable storage devices or received from other sources.

[0071] While this specification may contain numerous specific implementation details, these details should not be construed as limiting the scope of any claimed subject matter, but rather as descriptions of features specific to particular embodiments. Specific features described in the context of individual embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in a particular combination or even initially claimed in this way, in some cases one or more features from a claimed combination may be removed from the combination, and the claimed combination may be for sub-combinations or variations thereof.

[0072] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order shown or in a sequential order, or to perform all the shown operations to achieve the desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the embodiments described above should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0073] Therefore, specific embodiments of the subject matter have been described herein. Other embodiments are within the scope of the appended claims. In some cases, the actions set forth in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing may be advantageous.

[0074] As those skilled in the art will recognize, the innovative concepts described herein can be modified and varied across a wide range of applications. Therefore, the scope of the claimed subject matter should not be limited to any particular exemplary teachings discussed above, but is defined by the appended claims.

Claims

1. A neural processing unit, comprising: A weight buffer is configured to store weight values ​​in a fine-grained structured sparse weight arrangement, wherein the fine-grained structured sparse weight arrangement is selected from a group of at least two arrangements selected from 1:4 fine-grained structured sparse weight arrangement, 2:4 fine-grained structured sparse weight arrangement, 4:8 fine-grained structured sparse weight arrangement and 2:8 fine-grained structured sparse weight arrangement, wherein the N:M fine-grained structured sparse weight arrangement is a weight arrangement in which at most N weights in each group of M consecutive weights have non-zero values. The weight reusable is configured to output one of the weight values ​​stored in the weight buffer as the first operand value based on a selected fine-grained structured sparse weight arrangement. The activation buffer is configured to store the activation value. An activation multiplexer, combined with an activation buffer, is configured to output one of the activation values ​​stored in the activation buffer as a second operand value, the second operand value and the first operand value forming an operand value pair; and The multiplier unit is configured to output the product of operand pairs, wherein... The weight buffer is a first weight buffer that includes an array of first weight registers, each of which is configured to store weight values ​​in a fine-grained, structured, sparse weight arrangement. The weight reusable is a first weight reusable, which is configured to select a first weight register based on a selected fine-grained structured sparse weight arrangement and output the weight value stored in the selected first weight register as the first operand value. The activation buffer is a first activation buffer comprising a first predetermined number of first activation registers, each first activation register being configured to store an activation value; and The activation multiplexer is a first activation multiplexer input including a second predetermined number of first activation multiplexer inputs. Each corresponding input of the first activation multiplexer is connected to a corresponding first activation register within a first group of first activation registers. The first activation multiplexer is configured to select a first activation register based on a selected fine-grained structured sparse weight arrangement and output the activation value stored in the selected first activation register as a second operand value. The activation value output as the second operand value corresponds to the weight value output as the first operand value. The multiplier unit is a first multiplier unit, which is configured to output a first product of a first operand and a second operand as the product value.

2. The neural processing unit according to claim 1, comprising: A weight reusable array, including weight reusables, is configured to output one or more weight values ​​stored in a weight buffer as first operand values ​​based on a selected fine-grained structured sparse weight arrangement. An activation multiplexer array, including an activation multiplexer and an input to an activation multiplexer array coupled to an activation buffer, wherein the activation multiplexer array is configured to output one or more activation values ​​stored in the activation buffer as second operation values, and each corresponding second operation value and a corresponding first operation value form an operation value pair. as well as A multiplier array, comprising multiplier units, is configured to output the product of each operand pair.

3. The neural processing unit according to claim 2, wherein, The activation buffer includes eight activation registers, which store eight activation values. The weight reusable array includes: a first weight reusable, configured to select a weight register based on a selected fine-grained structured sparse weight arrangement, and output the weight value stored in the selected weight register as a first operation value. The activation multiplexer array includes: a first activation multiplexer with seven inputs, each corresponding input of which is connected to a corresponding activation register within a first set of activation registers. The first activation multiplexer is configured to select an activation register in the first set of activation registers based on a selected fine-grained structured sparse weight arrangement, and outputs an activation value stored in the selected activation register as a second operand value. The second operand value corresponds to the first operand value and forms a first operand value pair. The multiplier array includes a first multiplier unit configured to output the product of a first operand pair.

4. The neural processing unit according to claim 3, wherein, The weight values ​​are stored in the weight buffer with either a 1:4 fine-grained structured sparse weight arrangement or a 2:8 fine-grained structured sparse weight arrangement, and The first group of activation registers includes 7 activation registers.

5. The neural processing unit according to claim 3, wherein, The weight values ​​are stored in a weight buffer with a 2:4 fine-grained structured sparse weight arrangement, and The first group of activation registers includes four activation registers.

6. The neural processing unit according to claim 3, wherein, The weight values ​​are stored in a weight buffer with a fine-grained, structured, sparse weight arrangement of 4:8, and The first group of activation registers includes six activation registers.

7. The neural processing unit according to claim 3, wherein, The weights are arranged in a fine-grained, structured, sparse configuration of 2:8, and The activation registers consist of four activation registers in two rows, with two output multiplexers configured to select one activation register from each row.

8. The neural processing unit according to claim 3, wherein, The weights are arranged in a 2:4 fine-grained structured sparsity configuration, and The activation registers consist of two activation registers in two rows, with two output multiplexers configured to select one activation register from each row.

9. The neural processing unit according to any one of claims 1 to 8, wherein, The first predetermined number of the first activation registers includes 8, and the second predetermined number of activation multiplexer inputs includes 7.

10. The neural processing unit according to claim 9, wherein, The weight values ​​are arranged in a 1:4 fine-grained structured sparsity configuration or a 2:4 fine-grained structured sparsity configuration.

11. The neural processing unit according to claim 9, wherein, The weight values ​​are arranged in a fine-grained, structured, sparse configuration of 4:

8.

12. The neural processing unit according to claim 9, wherein, The weight values ​​are arranged in a 2:8 fine-grained structured sparsity configuration.

13. The neural processing unit according to any one of claims 1 to 8, further comprising: The second weight reusable unit is configured to select the first weight register based on the selected fine-grained structured sparse weight arrangement, and output the weight value stored in the selected first weight register as the third operation value. The second activation multiplexer includes a second predetermined number of second activation multiplexer inputs, each corresponding input of the second activation multiplexer being connected to a corresponding first activation register in a second set of activation registers that is different from the first set of first activation registers. The second activation multiplexer is configured to select the first activation register based on a selected fine-grained structured sparse weight arrangement and output the activation value stored in the selected first activation register as a fourth operand value. The activation value output as the fourth operand value corresponds to the weight value output as the third operand value. as well as The second multiplier unit is configured to output the second product of the third operand and the fourth operand.

14. The neural processing unit according to claim 13, further comprising: The second weight buffer is configured to store the weight values ​​of the fine-grained structured sparse weights based on the selected fine-grained structured sparse weight arrangement. The third weight reusable unit is configured to select the second weight register based on the selected fine-grained structured sparse weight arrangement, and output the weight value stored in the selected second weight register as the fifth operand value. The second activation buffer includes a first predetermined number of second activation registers, each of which is configured to store an activation value. The third activation multiplexer includes a second predetermined number of third activation multiplexer inputs, each corresponding input of the third activation multiplexer is connected to a corresponding second activation register in the first group of second activation registers, the third activation multiplexer is configured to select the second activation register based on a selected fine-grained structured sparse weight arrangement, and output the activation value stored in the selected second activation register as a sixth operation value, the activation value output as the sixth operation value corresponds to the weight value output as the fifth operation value; The third multiplier unit is configured to output the third product of the fifth operand and the sixth operand; The fourth weight reusable unit is configured to select the second weight register based on the selected fine-grained structured sparse weight arrangement, and output the weight value stored in the selected second weight register as the seventh operation value. The fourth activation multiplexer includes a second predetermined number of fourth activation multiplexer inputs, each corresponding input of the fourth activation multiplexer being connected to a corresponding second activation register within a fourth set of second activation registers that is different from the third set of activation registers. The fourth activation multiplexer is configured to select the second activation register based on a selected fine-grained structured sparsity weight arrangement and output the activation value stored in the selected second activation register as the eighth operation value. as well as The fourth multiplier unit is configured to output the fourth product of the seventh and eighth operands.

Images