Hardware unit for performing matrix multiplication with clock gating

Abstract

This disclosure relates to a hardware unit with clock gating for performing matrix multiplication. The disclosure provides hardware units and methods for performing matrix multiplication via a multi-stage pipeline, wherein memory elements associated with one or more stages of the pipeline are clock-gated based on data elements and / or portions thereof known to have zero values ​​or which can be considered to have zero values. In some cases, memory elements may be clock-gated based on each data element, based on whether the data element has a zero value or can be considered to have a zero value. In other cases, memory elements may be clock-gated based on a subset of elements, based on the bit width of the data element. For example, if the bit width of a data element is less than the maximum bit width of the data element, a subset of bits associated with that data element may be considered to have zero values ​​and a subset of memory elements associated with that data element may not be clock-controlled.

Description

[0001] Divisional

[0002] This application is a divisional application of the invention patent application with the application date of November 2, 2018, the application number of 201811299933.X, and the title of “Hardware unit for performing matrix multiplication with clock gating”. TECHNICAL FIELD

[0003] The present disclosure relates to matrix multiplication, and more specifically, to a hardware unit for performing matrix multiplication with clock gating. BACKGROUND

[0004] Matrix multiplication is multiplying a first matrix A and a second matrix B to produce a third matrix C. If A is an a x b matrix and B is a b x c matrix, as follows:

[0005]

[0006]

[0007] The result of the matrix multiplication is then an a x c matrix C, as follows:

[0008]

[0009] where each element C ij The calculation is done by multiplying the elements in the i-th row of matrix A with the elements in the j-th column of matrix B and summing the results, as shown in equation (1):

[0010]

[0011] Matrix multiplication is a key operation in many applications. For example, matrix multiplication can be used to compute the convolution of input data and weights in a deep neural network (DNN). However, matrix multiplication can be time-consuming, especially for large matrices. Therefore, there is a need for a hardware configured to perform matrix multiplication in an efficient manner, i.e., a manner that requires less silicon area or less processing power when operating. Moreover, different matrix multiplications (e.g., in terms of matrix size, etc.) can be performed for different applications. Therefore, there is also a need for a hardware configured to perform matrix multiplication that can flexibly support various matrix multiplications.

[0012] The embodiments described below are provided by way of example only, and do not limit the implementations to solve any or all of the disadvantages of known hardware units for performing matrix multiplication. SUMMARY

[0013] This summary is provided to introduce a selection of concepts, which are further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0014] Described herein are hardware units and methods for performing matrix multiplication via multiple pipeline stages, where each stage is preceded by a plurality of storage elements for storing data elements input to the stage, the storage elements associated with one or more stages of the pipeline being clocked based on whether the data elements and / or portions thereof have a zero value (or can be treated as having a zero value). In some cases, the storage elements can be clocked for each data element based on whether the data element has a known zero value (or can be treated as having a zero value). In other cases, the initial data elements can be represented in a fixed point number format including an exponent and a mantissa bit width, and the mantissa bit width can vary between matrix multiplications or even between hardware lanes of the same matrix multiplication. In these cases, the storage elements can be clocked on a partial data element basis based on the mantissa bit width of the initial data elements input to the hardware. For example, if the mantissa bit width of an initial data element is less than a maximum mantissa bit width, a portion of bits associated with that data element can be treated as having a zero value and a portion of the storage elements associated with that data element can not be clocked. In other cases, the storage elements can be clocked based on both the full and partial elements.

[0015] A first aspect provides a hardware unit for performing matrix multiplication, the hardware unit comprising: a multiplier stage comprising a plurality of multipliers, each multiplier configured to multiply a first data element and a second data element to produce a multiplication data element; one or more adder stages following the multiplier stage, the one or more adder stages forming an adder tree to produce a sum of the multiplication data elements, each adder stage comprising one or more adders configured to add at least two data elements output by a preceding stage to produce an addition data element; wherein at least one of the multipliers and / or at least one of the adders is preceded by a storage element corresponding to each bit of a data element input to the at least one adder or the at least one multiplier; and control logic configured to clock all or part of the storage elements corresponding to a data element in response to determining that all or part of the data element can be treated as having a zero value.

[0016] The second aspect provides a method of clock gating storage elements in a hardware unit for performing matrix multiplication, the hardware unit comprising: a plurality of multipliers each configured to multiply data elements to output a product data element; and a plurality of adders forming an adder tree to produce a sum of the product data elements by adding data elements from the multipliers or the adders, wherein at least one of the multipliers and / or at least one of the adders is preceded by a storage element for each bit of each input data element. The method comprises: receiving information indicating whether each data element input to at least one of the multipliers and / or at least one of the adders can be considered to have a zero value, and / or information indicating whether a portion of each data element input to at least one of the multipliers and / or at least one of the adders can be considered to have a zero value; and in response to receiving information indicating that a data element or a portion of a data element can be considered to have a zero value, clock gating all or a portion of the storage elements corresponding to that data element.

[0017] The third aspect provides a hardware implementation of a deep neural network (DNN) comprising the hardware unit of the first aspect, the hardware implementation of the DNN being configured to compute a convolution associated with one or more convolutional layers and / or one or more fully connected layers of the DNN.

[0018] The hardware unit for performing matrix multiplication can be implemented in hardware on an integrated circuit. A method of manufacturing the hardware unit in an integrated circuit manufacturing system can be provided. A set of integrated circuit definition data can be provided, which when processed in the integrated circuit manufacturing system, configures the system to manufacture the hardware unit for performing matrix multiplication. A non-transitory computer readable storage medium can be provided having stored thereon a computer readable description of the hardware unit for performing matrix multiplication, which when processed in the integrated circuit manufacturing system, causes the integrated circuit manufacturing system to manufacture an integrated circuit embodying the hardware unit for performing matrix multiplication.

[0019] An integrated circuit manufacturing system can be provided comprising: a non-transitory computer readable storage medium having stored thereon a computer readable description of the hardware unit for performing matrix multiplication; a layout processing system configured to process the computer readable description to generate a circuit layout description of an integrated circuit embodying the hardware unit for performing matrix multiplication; and an integrated circuit generation system configured to manufacture the hardware unit for performing matrix multiplication from the circuit layout description.

[0020] Computer program code for performing the methods described herein can be provided. A non-transitory computer readable storage medium having stored thereon computer readable instructions, which when executed at a computer system, cause the computer system to perform the methods described herein can be provided.

[0021] As the skilled person will readily appreciate, the above features can be combined as appropriate, and with any of the aspects of the examples described herein. BRIEF DESCRIPTION OF DRAWINGS

[0022] Examples will be described in detail with reference to the attached drawings, in which:

[0023] Figure 1 is a block diagram of an example hardware unit for performing matrix multiplication;

[0024] Figure 2 is a block diagram of a set of storage elements preceding an arithmetic unit;

[0025] Figure 3 is a block diagram of the storage elements and arithmetic unit of Figure 2 with clock gating control logic;

[0026] Figure 4 is a block diagram of a first example implementation of the clock gating control logic of Figure 3

[0027] Figure 5 is a table illustrating the calculation of the output element strobe signal of Figure 4

[0028] Figure 6 is a block diagram of a second example implementation of the clock gating control logic of Figure 3

[0029] Figure 7 is a schematic diagram illustrating the most significant bit-aligned two’s complement format;

[0030] Figure 8 is a table illustrating an example of the storage element blocks preceding the multiplier that are not clocked for various bit widths;

[0031] Figure 9 is a table illustrating an example of the storage element blocks preceding the adder that are not clocked for various bit widths;

[0032] Figure 10 is a block diagram of a third example implementation of the clock gating control logic of Figure 3

[0033] is a flowchart of an example method for clock gating the storage elements of the hardware unit of Figure 11 Figure 1 is a flowchart of a first example implementation of the method of

[0034] Figure 12 Figure 11

[0035] Figure 13 is a flowchart of a second example implementation of the method of Figure 11 ​​​​​​a flowchart of a second example implementation of the method of

[0036] Figure 14 is a block diagram of an example computer system in which a hardware unit for performing matrix multiplication as described herein is implemented; and

[0037] Figure 15 is a block diagram of an example integrated circuit manufacturing system for generating an integrated circuit embodying a hardware unit for performing matrix multiplication as described herein.

[0038] The drawings illustrate various examples. A skilled artisan understands that the boundaries shown in the drawings (e.g., boxes, groups of boxes, or other shapes) represent one example of a boundary. In some examples, one element can be designed as multiple elements, or multiple elements can be designed as one element. Common reference numbers can be used throughout the drawings as appropriate to indicate similar features. DETAILED DESCRIPTION

[0039] The following description is presented to enable any person skilled in the art to make and use the application. The present application is not limited to the embodiments described herein and various modifications to the disclosed embodiments will be readily apparent to those skilled in the art. The embodiments are described by way of example only.

[0040] Hardware that performs matrix multiplication can be implemented as a pipeline having multiple pipeline stages. For example, Figure 1 An example hardware unit 100 for performing matrix multiplication is shown, in which the matrix multiplication is performed via multiple pipeline stages 102, 104, 106, 108. The first stage 102 (which can be referred to as a multiplier stage) includes multiple parallel multipliers 110 that each multiply a first data element (D1) in a first set of data elements with a second data element (D2) in a second set of data elements to produce a multiplication data element. The first set of data elements (D1) can represent elements of one matrix (e.g., matrix A), and the second set of data elements (D2) can represent elements of another matrix (e.g., matrix B). The first and second data elements (D1, D2) can be stored in external memory such as a buffer and provided to the first stage 102 via a communication channel established between the hardware unit 100 and the external memory. The multiplication data elements are provided to the second stage 104 (which can be referred to as an accumulator stage) that includes multiple parallel accumulators 112 that each accumulate the multiplication data elements to produce an accumulated data element. The accumulated data elements are provided to the third stage 106 (which can be referred to as an exponentiation stage) that includes multiple parallel exponentiation units 114 that each exponentiate the accumulated data elements to produce an exponentiated data element. The exponentiated data elements are provided to the fourth stage 108 (which can be referred to as a reduction stage) that includes multiple parallel reduction units 116 that each reduce the exponentiated data elements to produce a reduced data element. The reduced data elements are provided to an output 118 of the hardware unit 100. Figure 1 In the example, the multiplier stage 102 includes eight multipliers 110, however, it will be readily understood by those skilled in the art that this is merely an example and that the multiplier stage 102 can have any number of multipliers greater than or equal to two. In some cases, the multiplier stage 102 can have 128 multipliers.

[0041] The remaining stages 104, 106, 108 form an adder tree to produce a sum of the multiplied data elements. Specifically, each of the remaining stages 104, 106, 108, which can be referred to as adder stages, includes one or more adders 112 configured to add at least two data elements generated by a preceding stage to produce an added data element. For example, each adder 112 of the second stage 104 is configured to add two multiplied data elements from the first stage 102 to produce an added data element; each adder 112 of the third stage 106 is configured to add two added data elements from the second stage 104 to produce an added data element; and each adder 112 of the fourth stage 108 is configured to add two added data elements from the third stage 106 to produce the final output. In Figure 1 In the example of FIG. 1, there are three adder stages 104, 106, 108, however, it will be readily understood by those skilled in the art that this is merely an example, and other examples can have more or fewer adder stages. In general, if the multiplier stage 102 includes N multipliers 110, then the adder tree includes N-1 adders 112 over (N / 2)-1 stages.

[0042] Figure 1 The hardware unit 100 of FIG. 1 calculates a sum of products of a set of first data elements (D1) and a set of second data elements (D2). As such, Figure 1 The hardware unit 100 of FIG. 1 can perform matrix multiplication over one or more hardware lanes, where a hardware lane includes inputting a set of first data elements (D1) and a set of second data elements (D2) and outputting a sum of products thereof. For example, depending on the number of multipliers 110 and the number of elements in the matrix, there can be one hardware lane for each element of the final matrix. However, in other cases, there can be more multiplications to generate an element of the final matrix that can be done in a single hardware lane, and thus multiple hardware lanes can be needed to generate an element of the final matrix. In these cases, the outputs from multiple hardware lanes can be added together to generate an element of the final matrix.

[0043] For data flowing through the pipeline 102, 104, 106, and 108 in a predictable manner, each arithmetic unit (e.g., multiplier 110 or adder 112) can be preceded by a set of storage elements for each data element input into the arithmetic unit 110 or 112. For example, in Figure 1In the example of FIG. 1, each arithmetic unit (e.g., multiplier 110 or adder 112) receives two data elements (a first data element (d1) and a second data element (d2)) to be processed by that arithmetic unit 110 or 112. The data elements received by the multiplier 110 are the original data elements input to the hardware unit 100 in the hardware pipeline, and the data elements received by the adder 112 are data elements generated by an earlier stage of the pipeline (e.g., output data elements of a multiplier 110 or an adder 112 of an earlier stage of the pipeline).

[0044] Referring now to FIG. 2, Figure 2 which shows an arithmetic unit (e.g., multiplier 110 or adder 112) before which is a storage element 202 for each bit of a first data element and a second data element (D1, D2) input to the arithmetic unit (e.g., multiplier 110 or adder 112). The notation Dx(i) is used herein to refer to the i-th bit of the x-th data element. For example, D1(5) denotes the 5th bit of the first (1) data element. The storage element receives a bit of a data element (either from an external source in the case of a multiplier, or from an earlier stage of the pipeline in the case of an adder), stores the received bit, and outputs the stored bit to the corresponding arithmetic unit (e.g., multiplier 110 or adder 112). The storage element 202 can be implemented as a flip-flop or any other form of register or memory unit.

[0045] As known to those skilled in the art, the storage elements 202 are “clocked,” which means that the storage elements are configured to update the stored value based on the input value at the time of a clock signal (CLK) transition. In other words, the storage elements 202 ignore their inputs except at the time of a clock signal (CLK) transition. When the storage elements 202 are clocked, the storage elements change or maintain their stored value based on the input value. Some storage elements are triggered on the rising edge of the clock signal (CLK), while other storage elements are triggered on the falling edge of the clock signal (CLK). As known to those skilled in the art, each storage element consumes static power and dynamic power. Static power is the power consumed by a storage element when the storage element is in a stable state (i.e., the storage element is not toggling or changing state). In contrast, dynamic power is the power consumed to invoke a state transition (i.e., the power consumed when the storage element toggles or changes state). Generally, static power is small, and the power consumed by a storage element is based on the dynamic power consumed by the storage element. Thus, the power consumed by a storage element can be reduced by reducing the number of state transitions.

[0046] In many matrix multiplications, one or more data elements (or a portion thereof) can be known a priori to have a zero value (or can be treated as having a zero value). Rather than passing these data elements (or a portion thereof) to the corresponding arithmetic unit via a set of storage elements, these data elements (or a portion thereof) can not be used with storage elements (i.e., not clocked to storage elements) and a zero value can be provided directly to the arithmetic unit. This can be accomplished through a technique known as “clock gating.” As known to those skilled in the art, “clock gating” is a technique in which a circuit (e.g., a latch) is used to disable the clock to the circuit under certain conditions. By not “clocking” a storage element when a bit is known to be zero (or can be treated as zero), the number of state transitions of that storage element is reduced, which reduces the power consumption of the storage element and the overall hardware unit.

[0047] Accordingly, described herein are hardware units and methods for performing matrix multiplications via a plurality of pipeline stages, where each stage is preceded by a plurality of storage elements for storing data elements input to the stage, and where the storage elements associated with one or more stages of the pipeline are clock-gated based on whether the data elements input to the stage and / or a portion thereof have a zero value (or can be treated as having a zero value). In some cases, the storage elements can be clock-gated on a per-data-element basis based on whether the data element has a zero value (or can be treated as having a zero value). In other cases, the initial data elements can be represented in a fixed-point number format that includes an exponent and a mantissa bit width, and the mantissa bit width can vary between matrix multiplications or even between hardware lanes of the same matrix multiplication. In these cases, the storage elements can be clock-gated on a partial-data-element basis based on the mantissa bit width of the initial data elements input to the hardware unit. For example, if the mantissa bit width of an initial data element is less than a maximum mantissa bit width, a portion of bits associated with that data element can be treated as having a zero value and a portion of the storage elements associated with that data element can not be clocked. In other cases, the storage elements can be clock-gated based on both per-element and partial-element. This can reduce the power consumption of the hardware unit used to perform matrix multiplications without significantly increasing the complexity of the hardware unit.

[0048] Reference is now made to Figure 3 which illustrates an example arithmetic unit (e.g., multiplier 110 or adder 112) performing an arithmetic operation (e.g., multiplication or addition) on a first data element (D1) and a second data element (D2), where the first and second data elements (D1 and D2) are provided to the arithmetic unit via a set of storage elements 202. Specifically, there is one storage element for each bit of the first and second data elements (D1 and D2). However, unlike Figure 2The storage elements 202 and the arithmetic units 110 / 112 differ in that the clocking of the storage elements 202 is controlled by clock gating control logic 302. Specifically, the clock gating control logic 302 includes hardware logic configured to clock gate (i.e., not clock control) all or a portion of the storage elements corresponding to a data element in response to determining that all or a portion of the data element has a zero value (or can be considered to have a zero value).

[0049] In some cases, the clock gating control logic 302 can be configured to perform clock gating on a per data element basis. Clock gating on a per element basis means that all of the storage elements corresponding to a particular data element are either clocked together or not clocked together. In these cases, the clock gating control logic can be configured to clock gate the storage elements associated with a data element in response to determining that the data element can be considered to have a zero value. There can be a number of ways to determine that a value can be considered to have a zero value. For example, a data element can be determined to be considered to have a zero value if it is determined to have a zero value, and / or if the data element is an input to a multiplier and other data elements input to the multiplier have zero values. This is because multiplying anything by a zero value will result in a result of zero, and thus the data element can be considered to have a zero value.

[0050] In other cases, the clock gating control logic 302 can be configured to alternatively or additionally perform clock gating on a partial data element basis. Clock gating on a partial data element basis means that not all of the storage elements corresponding to a particular data element must be clocked / clocked out at the same time. Specifically, clock gating on a partial data element basis means that one or more of the storage elements associated with a data element can be clocked and one or more of the storage elements associated with the same data element can not be clocked. For example, in cases where the bit width of an initial data element can be less than the bit width supported by the hardware unit (e.g., the hardware unit supports 16-bit input data elements, but the data element received has a bit width of only 12), not all of the storage elements will receive values that are of value or use, and thus some of the storage elements can be considered to have a zero value. Accordingly, the clock gating control logic 302 can be configured to clock gate one or more of the storage elements associated with a data element based on the bit width of the initial data element.

[0051] Example implementations of the clock gating control logic 302 of the Figure 4 , Figure 6 and Figure 10 are described below with reference to Figure 3 . Although Figure 3Clock gating control logic is shown for controlling clocking of storage elements before a single arithmetic unit (e.g., multiplier 110 or adder 112), but the clock gating control logic can be replicated for each arithmetic unit of the same tier and / or for all or a portion of the arithmetic units of other tiers.

[0052] Reference is now made to Figure 4 which shows a first example implementation of clock gating control logic 400, which can be used as Figure 3 the clock gating control logic 302 of FIG. 1. In this example, the clock gating control logic 400 is configured to clock gate the storage elements 202 before a particular arithmetic unit (e.g., multiplier 110 or adder 112) based on each data element based on the sparsity of the data elements. While Figure 4 clock gating control logic is shown for controlling clocking of storage elements 202 before a single arithmetic unit (e.g., multiplier 110 or adder 112), the clock gating control logic can be replicated for each arithmetic unit of the same tier and / or for all or a portion of the arithmetic units of other tiers.

[0053] Figure 4 The clock gating control logic 400 includes gating logic 402, an AND gate 404, and a gating signal generation logic 406. The clock gating control logic 400 receives an element gating signal for each data element to be processed by the arithmetic unit 110 or 112, which indicates whether the data element can be considered to have a zero value and thus whether the data element should be clock gated (i.e., not clocked). For example, as shown in Figure 4 the clock gating control logic 400 receives a first element gating signal for a first data element Dl, which indicates whether the first data element Dl can be considered to have a zero value and thus whether it should be gated; and the clock gating control logic 400 receives a second element gating signal for a second data element D2, which indicates whether the second data element D2 can be considered to have a zero value and thus whether it should be gated. In some cases, if a data element is to be clock gated (i.e., not clocked), the corresponding element gating signal can have a value of "1", and if a data element is not to be clock gated (i.e., clocked), the corresponding element gating signal can have a value of "0".

[0054] In the case where clock gating control logic 400 is configured to control storage elements preceding multiplier 110, the element gating signal can be generated by logic external to the hardware unit (not shown) or by logic internal to the hardware unit. The external or internal logic can be configured to set the element gating signal for both data elements to indicate that the respective data elements can be considered to have a zero value and should be gated if either data element has a zero value (or if either data element can be considered to have a zero value). This is because if either input data element to the multiplication is zero (or can be considered to be zero), the output of the multiplication will be zero, so the multiplication need not be performed. This means that both data elements can be gated (i.e., not clocked). The external or internal logic can determine that a data element can be considered to have a zero value based on one or more criteria. For example, in the case where the hardware unit is used to perform a convolution operation on a layer of a deep neural network, the internal or external logic can be configured to determine that a data element can be considered to have a zero value if the data element is an input data value and the input data value is in a plane outside of the layer; or if the data element is a weight and the weight is outside of the current window. As will be readily understood by one of skill in the art, this is merely an example, and the external or internal logic can be configured to determine that a data element can be considered to have a zero value based on one or more additional or alternative criteria.

[0055] In contrast, in the case where clock gating control logic 400 is configured to control storage elements preceding adder 112, the element gating signal indicating whether the respective data element can be considered to have a zero value and thus should be clock gated (i.e., not clocked) can be generated by the clock gating control logic 400 of the preceding stage. Specifically, as described in greater detail below, gating signal generation logic 406 can be configured to generate an output element gating signal indicating whether the output element generated by arithmetic unit 110 or 112 should be gated in the subsequent pipeline stage.

[0056] For each set of storage elements 202 controlled by the same clock signal, there is a gating logic unit 402. For example, in Figure 4 In the case of FIG. 4, all storage elements associated with each data element are controlled by a single clock signal, so there are only two gating logic units 402, one for the storage elements associated with the first data element (D1) and one for the storage elements associated with the second data element (D2). As will be readily understood by one of skill in the art, this is merely an example, and the storage elements associated with a data element can be grouped in different ways. For example, in other cases, the storage elements for each data element can be grouped into sets of four storage elements. In this case, there would be four gating logic units, one for each set of four storage elements.

[0057] Each strobe logic unit 402 is configured to receive an input clock signal (CLK) and a corresponding element strobe signal (if the element strobe signal is set to "1" to indicate that the corresponding data element is to be clocked, after a NOT gate) and provide an output clock signal to the clock input of each corresponding storage element, where the output clock signal causes the storage element to be clocked when the element strobe signal indicates that the corresponding data element is to be clocked, and the output clock signal causes the storage element to not be clocked when the element strobe signal indicates that the corresponding data element is not to be clocked. In other words, the strobe logic unit 402 effectively generates the output clock signal by performing an AND operation on the input clock signal (CLK) and the corresponding element strobe signal. For example, as shown in Figure 4 the strobe logic unit 402 controlling the storage element of the first data element is configured to generate an output clock signal that causes the storage element of the first data element to be clocked when the element strobe signal for the first data element NOT'd Dl is "1" and that causes the storage element of the first data element to not be clocked when the element strobe signal for the first data element NOT'd Dl is "0"; and the strobe logic unit 402 controlling the storage element of the second data element is configured to generate an output clock signal that causes the storage element of the second data element to be clocked when the element strobe signal for the second data element NOT'd D2 is "1" and that causes the storage element of the second data element to not be clocked when the element strobe signal for the second data element NOT'd D2 is "0".

[0058] Each strobe logic unit 402 can be implemented using an AND gate, a strobed latch, or any other logically equivalent circuit. Moreover, even though the strobe logic unit 402 is shown as being external to the storage element 202, in other cases the strobe logic unit 402 can be implemented within the storage element 202. For example, in some cases, the storage element can be implemented using a strobed flip-flop that is configured to receive an input clock signal and a corresponding element strobe signal and clock the flip-flop only when both the input clock signal and the element strobe signal are high (i.e., are "1").

[0059] There is an AND gate 404 for each storage element of the first data element and the second data element (Dl and D2). For example, as shown in Figure 4 there are 16 AND gates, one for each bit of the first data element and the second data element (Dl and D2).

[0060] Each AND gate 404 is configured to perform an AND operation on the output of the storage element and the corresponding element gating signal (if the element gating signal is set to "1" to indicate that the corresponding data is to be clocked, after the corresponding element gating signal passes through the NOT gate) and the output of the AND operation is provided to the arithmetic unit 110 or 112. This forces the clocked bit to be zero. Thus, in the example of a zero value data element, instead of clocking the bits of the data element from the storage element to the arithmetic unit, the storage element is clocked (i.e. not clocked) and a zero is provided to the arithmetic unit. Figure 4

[0061] The gating signal generation logic 406 is configured to propagate the element gating information to the next stage of the pipeline. In particular, the gating signal generation logic 406 is configured to determine, based on the element gating signal of the data element input to the arithmetic unit, whether the output data element of the arithmetic unit can be considered to have a zero value and thus be clocked, and to generate an output element gating signal indicating whether the output element can be considered to have a zero value and thus be gated. The output gating element signal is then provided to the clock gating control logic of the next stage in the pipeline.

[0062] In some cases, determining whether the output element should be gated can be based on the type of arithmetic operation performed by the arithmetic unit. For example, in the case that the arithmetic unit is a multiplier, if either input data element has a zero value (or can be considered to have a zero value), the output data element will have a zero value (and thus can be clocked), because zero multiplied by anything is zero. Thus, the gating signal generation logic 406 can be configured to determine that the output element of the multiplier can be considered to have a zero value and thus can be clocked if either input data element of the multiplier can be considered to have a zero value. As mentioned above, in some cases, when at least one of the two data elements has a zero value (or can be considered to have a zero value), the initial element gating signals received by the clock gating control logic 400 are configured to indicate that both data elements can be considered to have a zero value, whereby the two initial element gating signals will be the same - i.e. either they both indicate that the corresponding data element can be considered to have a zero value and thus can be clocked or they both indicate that the corresponding data element cannot be considered to have a zero value and thus cannot be clocked. In these cases, the output element gating signal can be set to either of these initial element gating signals. Thus, the gating signal generation logic 406 for the first stage can be configured to simply output either of the initial element gating signals as the output element gating signal.

[0063] ​Conversely, in the case where the arithmetic unit is an adder, if all of the input data elements have a value of zero or can be considered to have a value of zero, the output data element will only have a value of zero (and thus can be clock gated) because the sum of only zeros equals zero. Thus, as shown in the table of Figure 5 the gating signal generation logic 406 can be configured to indicate that the output element of the adder can be considered to have a value of zero and thus can be clock gated if all of the input data elements can be considered to have a value of zero, and that the output element of the multiplier can be considered to have a value of zero and thus can be clock gated if any of the input data elements can be considered to have a value of zero.

[0064] Reference is now made to Figure 6 which illustrates a second example implementation of clock gating control logic 600, which can be used as the clock gating control logic 302 of Figure 3 In this example, the clock gating control logic 600 is configured to clock gate storage elements before a particular arithmetic unit (e.g., the multiplier 110 or the adder 112) based on the partial data elements based on the bit width of the initial set of data elements. While Figure 6 illustrates clock gating control logic for controlling the clock gating of storage elements before a single arithmetic unit (e.g., the multiplier 110 or the adder 112), the clock gating control logic can be replicated for each arithmetic unit of the same layer and / or for the arithmetic units of all or some other layers.

[0065] In this example, the hardware unit 100 is configured to receive and process data elements including a maximum number of bits, however, the initial data elements of the input can use fewer bits than the maximum number of bits. For example, the hardware unit 100 can be configured to receive and process data elements of 16 bits, however, one of the initial sets of data elements can include data elements of only 4 bits, meaning that the data elements of that initial set of data elements include only 4 bits of relevant information. Thus, 12 bits of the data elements of that initial set of data elements are not used (and thus can be considered to have a value of zero) and thus can be clock gated. Furthermore, different sets of data elements can use different numbers of bits, which reflect the range of values in the set. For example, a first set of data elements (D1) can have a different bit width than a second set of data elements (D2). Thus, in these examples, in addition to receiving the first set of data elements and the second set of data elements as inputs to the hardware unit 100, the hardware unit 100 also receives information indicating the number of bits used for the first set of data elements and the number of bits used for the first set of data elements, which can be referred to herein as the mantissa bit length, bit width, or bit depth. The clock gating control logic 600 is then configured to determine which bits of the data elements can be clock gated based on the bit width of the first set of data elements and the second set of data elements.

[0066] Figure 6 The example clock gating control logic 600 includes bit width control logic 602, gating logic 604, and AND gate 606. The bit width control logic 602 is configured to receive the bit widths of the first and second sets of data elements; determine which bits of the first and second data elements are to be clock-gated (i.e., not clocked) based on the bit widths and the stage of the pipeline in response to determining that at least one bit width is less than the maximum bit width; and output a set of block gating signals for each data element indicating which blocks of data elements are to be clock-gated (i.e., not clocked) and which blocks of data elements are not clock-gated (i.e., clocked).

[0067] In particular, the storage elements 202 of each data element are divided into a plurality of groups or blocks that can be independently clock-gated or clocked. For example, in Figure 2 eight storage elements of each data element (D1 and D2) are divided into groups or blocks, with two bits / two storage elements corresponding to one group or block. However, it will be readily understood by those skilled in the art that this is merely an example and that the storage elements can be divided into blocks in different ways. For example, in other cases, the storage elements of each data element can be divided into groups or blocks each corresponding to four bits / four storage elements.

[0068] The bit width control logic 602 is configured to determine the blocks of data elements that can be clock-gated based on the bit widths of the first and second sets of data elements and the stage of the pipeline. Generally, those bits that are known to have a zero value (or can be considered to have a zero value) can be clock-gated.

[0069] The bits of the data elements input to the multiplier that have a zero value (or can be considered to have a zero value) can be determined according to the bit widths of the respective sets of data elements and the format of the data elements. For example, the bits of the first data elements input to the multiplier that can be considered to have a zero value are determined according to the bit widths of the first set of data elements; and the bits of the second data elements input to the multiplier that can be considered to have a zero value are determined according to the bit widths of the second set of data elements. Generally, if the bit widths of a set of data elements is less than the maximum bit width, then at least some of the bits of the data elements can be considered to have a zero value and thus can be clock-gated. In particular, the number of bits (NumZeroBits) of the data elements input to the multiplier that are zero (or can be considered to have a zero value) is the difference between the maximum bit width (MaxNumBit) and the bit width of the respective set of data elements (BitLength) as shown in equation (2) because the remaining bits can be considered to contain valuable information:

[0070] NumZeroBits = MaxNumBit - BitLength (2)

[0071] In the case where the storage elements are divided into a plurality of groups or blocks of size X, the number of blocks that are clock-gated (NumZeroBlocks) can be expressed as NumZeroBits divided by the size of the block, as shown in Equation (3):

[0072]

[0073] In some cases, in the case where the storage elements are divided into a plurality of groups or blocks of size X, BitLength can be expressed as a multiple of X. Specifically, BitLength can be rounded up to the nearest integer multiple of X.

[0074] The particular bits / blocks that are zero (or can be considered to have zero value) will be based on the format of the data element. For example, if a least significant bit (LSB) alignment format is used for unsigned binary numbers, then the bits containing valuable information will be in the least significant bit positions. For example, if the maximum number of bits is 16 and the data element uses bit width 8, then there will be 8 bits containing valuable information and they will be in the 8 least significant bits (i.e., bits 7 to 0). In these cases, the particular bits / blocks that have zero value (or can be considered to have zero value) can be the highest NumZeroBits or the highest NumZeroBlocks. Conversely, if a most significant bit (MSB) alignment format is used, then the bits containing valuable information will be in the most significant bit positions. For example, if the maximum number of bits is 16 and the data element uses bit depth 8, then there will be 8 bits containing valuable information and they will be in the 8 most significant bits (i.e., bits 15-8). In these cases, the bits of the data element that are not clocked can be the first 8 bits or the last 8 bits, respectively.

[0075] However, if an LSB alignment format is used for signed binary numbers in two’s complement format, then the identification of the bits / blocks that can be considered to have zero value and thus can be clock-gated can vary between positive and negative numbers. This is because for negative numbers in LSB aligned two’s complement format, the sign bit is extended over unused bits so that they include a “1” instead of zero. This means that the negative number bits cannot be clock-gated because there is no guarantee that the bits are zero or can be considered to be zero. If an MSB alignment format is used for signed binary numbers in two’s complement format, then this problem can be avoided and the number of clock-gated bits can be increased. This is because, as shown in Equation (4), the number of bits that can be considered to have zero value and thus can be clock-gated is the same for both positive and negative numbers. Figure 7As shown, the sign bit need not extend to the unused bits of this format, as the unused bits are in the least significant bit position. Thus, if the data elements are signed numbers in two's complement format, it can be advantageous to use the MSB-aligned format. In this case, the least significant NumZeroBits bits according to equation (2) or the least significant NumZeroBits bits according to equation (3) can easily be identified as the bits / blocks that can be clocked for both negative and positive numbers.

[0076] Figure 8 is a table showing for each of a number of different bit width combinations for the first and second sets of data elements based on equation (3), which groups or blocks of the storage elements 202 of the first and second data elements of the first stage of the pipeline can be clocked (e.g., not clocked) where the maximum bit width is 16, the storage elements of the data elements are divided into groups or blocks of 4 storage elements each, and the data elements are MSB-aligned. In this example, the possible bit widths are {4, 8, 12, and 16}, which results in one of {0, 4, 8, 12} bits of each data element not being clocked.

[0077] The bits of the data elements input to the adder that have zero values (or can be considered to have zero values) and thus can be clocked can be determined according to the bit width of the original data elements, but can be determined in different ways. In the first stage of the pipeline (i.e., the multiplier stage), the first data element is multiplied by the second data element, thereby producing a number of multiplication outputs. As is known to those skilled in the art, multiplying a number of x bits by a number of y bits produces a number of x+y bits. Thus, if a number of x bits has a number of a bits that can be considered to have zero values and a number of y bits has a number of b bits that can be considered to have zero values, the total number of bits that can be considered to have zero values is a+b. The number of clocked bits does not increase or decrease in the subsequent additions, so for each adder, the number of bits that can be clocked is the same regardless of the stage of the adder. If the MSB-aligned format is used, the bits of the data elements input to the adder that can be considered to have zero values and thus can be clocked will be the least significant a+b bits of the data elements. In the case where the storage elements are divided into blocks or groups of size N and the MSB-aligned format is used, the blocks that can be considered to have zero values and thus can be clocked will be the least significant (a+b) / N bits.

[0078] Figure 9is a table showing for each of a plurality of different bit width combinations for an initial first data set and a second data set, which storage elements 202 of the first data element and the second data element input to the adder can be considered to have zero values and thus can be clock gated (e.g., not controlled by the clock). In this example, the maximum bit width is 16, the storage elements of a data element are grouped into blocks or groups of 4 storage elements each, and the data elements are MSB aligned. Thus, in this example, the possible bit widths are {4, 8, 12, and 16}, which results in one of {0, 4, 8, 12, 16, 20, 24} bits of each data element being clock gated.

[0079] Once the bit width control logic 602 determines which bits / storage elements or which blocks of bits / storage elements of a data element can be considered to have zero values and thus can be clock gated, the bit width control logic 602 is configured to output a block gating signal for each block of storage elements that indicates whether the storage elements of that block or group can be considered to have zero values and thus be clock gated. For example, in Figure 6 where the storage elements are grouped into blocks of two storage elements, the bit width control logic 602 is configured to generate a control signal for each block of storage elements, where the block numbers are 0 to 3. Specifically, the bit width control logic 602 generates eight block gating control signals - a data element 1 block gating control signal for block 3 (3) (D1BG(3)), a data element 1 block gating control signal for block 2 (2) (D1BG(2)), a data element 1 block gating control signal for block 1 (1) (D1BG(1)), a data element 1 block gating control signal for block 0 (0) (D1BG(0)), a data element 2 block gating control signal for block 3 (3) (D2BG(3)), a data element 2 block gating control signal for block 2 (2) (D2BG(2)), a data element 2 block gating control signal for block 1 (1) (D1BG(1)), and a data element 2 block gating control signal for block 0 (0) (D1BG(0)). The control signals are then provided to the corresponding gating logic units 604 and AND gates 606. For simplicity, Figure 6 only some of the connections between the control signals and the AND gates 606 are shown in

[0080] There is a gating logic unit 604 for each block or group of storage elements associated with a data element. For example, in Figure 6 where the storage elements associated with each data element are grouped into blocks of two storage elements, there are thus four gating logic units 604 for each data element. It will be readily understood by one of skill in the art that this is merely an example and that the storage elements associated with a data element can be grouped in different ways.

[0081] Each strobe logic unit 604 is configured to receive an input clock signal (CLK) and a corresponding block strobe control signal, and to provide an output clock signal to the clock input of each corresponding storage element, which causes the storage element to be clocked when the block strobe signal indicates that the corresponding block is to be clocked and not to be clocked when the block strobe signal indicates that the corresponding block is not to be clocked. In other words, the strobe logic unit 604 effectively generates the output clock signal by performing an AND operation on the input clock signal (CLK) and the corresponding block strobe signal. For example, as shown in Figure 6 the strobe logic unit 604 controlling the storage elements of the two most significant bits of the first data element is configured to generate an output clock signal that causes the storage elements of the two most significant bits of the first data element to be clocked when the Dl block 3 strobe signal (D1BG(3)) is "1" and not to be clocked when the Dl block 3 strobe signal (D1BG(3)) is "0"; and the strobe logic unit 604 controlling the storage elements of the two least significant bits of the first data element is configured to generate an output clock signal that causes the storage elements of the two least significant bits of the first data element to be clocked when the Dl block 0 strobe signal (D1BG(0)) is "1" and not to be clocked when the Dl block 0 strobe signal (D1BG(0)) is "0".

[0082] Each strobe logic unit 604 can be implemented using an AND gate, a strobed latch, or any other logically equivalent circuit. Moreover, even though the strobe logic units 604 are shown as being external to the storage elements 202, in other cases the strobe logic units 604 can be implemented internal to the storage elements 202. For example, in some cases the storage elements can be implemented using strobed flip-flops that are configured to receive an input clock signal and a corresponding block strobe signal, and to clock the flip-flop only when both the input clock signal and the block strobe signal are high (i.e., are "1").

[0083] There is an AND gate 606 for each storage element of the first and second data elements (Dl and D2). For example, as shown in Figure 6 the length of each data element (Dl and D2) can be up to 8 bits, the set of AND gates 606 includes 16 AND gates, one for each bit of the first and second elements (Dl and D2).

[0084] Each AND gate 606 is configured to perform an AND operation on the output of the storage element and the respective block gating signal, and to provide the output of the AND operation to the arithmetic unit 110 or 112. This forces the clock gating bit to be zero. Thus, in the example of Figure 6 if it is determined in advance that a number of bits of a data element are not used and can thus be considered to have a zero value, the storage element for those bits is not clocked to the arithmetic unit, but the storage element for those bits is clocked and a zero is provided to the arithmetic unit.

[0085] Reference is now made to Figure 10 which shows a third example implementation of clock gating control logic 1000, which can be used as the clock gating control logic 302 of Figure 3 The clock gating control logic 1000 is a combination of the clock gating control logic 400 and 600 of Figure 4 and Figure 6 configured to perform operations based on both the per-data-element clock gating and the partial data-element clock gating. Specifically, Figure 10 The clock gating control logic 1000 of Figure 6 includes the bit width control logic 602 of Figure 4 which generates, based on the bit width of the initial data element set, a block gating signal for each storage element block of a data element, which is provided to the respective gating logic unit 604 to cause the storage element of the respective block to be clocked when the block gating signal indicates that it is to be clocked, and which is provided to the respective AND gate 606 to cause a zero to be forced to the arithmetic logic when the respective storage element is clocked. In addition, the clock gating control logic 1000 receives the Dl and D2 element gating signals of Figure 4 The Dl element gating signal is input to the gating logic unit 604 corresponding to the first data element and to the AND gate 606 corresponding to the first data element (for simplicity, Figure 10 only some of the connections between the element gating signals and the AND gates 606 are shown in

[0086] The hardware unit 100 for performing the matrix multiplication described herein can be used in a hardware implementation of a neural network. For example, the hardware unit for performing matrix multiplication can be used in a hardware implementation of a neural network to compute convolutions associated with one or more convolutional layers and / or one or more fully connected layers.

[0087] Now for reference Figures 11 to 13 The flowcharts are exemplary methods 1100, 1200, and 1300, which can be derived from... Figure 3 The clock gating control logic 302 is implemented to clock gating the storage elements of the hardware unit 100 for performing matrix multiplication, wherein the hardware unit 100 includes a plurality of multipliers and a plurality of adders, each multiplier being configured to multiply data elements to produce output multiplied data elements, and the plurality of adders forming an adder tree to produce a sum of multiplied data elements by adding data elements from the multipliers or adders, wherein at least one multiplier and / or at least one adder is preceded by a storage element for each bit of each input data element.

[0088] Method 1100 begins at block 1102, where clock strobe control logic 302 receives information indicating whether each data element input to at least one multiplier and / or at least one adder can be considered to have a zero value and / or indicating whether a portion of each data element input to at least one multiplier and / or at least one adder can be considered to have a zero value. See below for reference. Figure 12 and Figure 13 In a more detailed description, the information indicating whether each data element input to at least one multiplier and / or at least one adder can be considered to have a zero value may include a bit for each data element indicating whether each data element input to at least one multiplier and / or at least one adder can be considered to have a zero value. Once the information indicating whether a data element and / or a portion thereof input to at least one multiplier and / or at least one adder can be considered to have a zero value has been received, method 1100 proceeds to block 1104.

[0089] At box 1104, it is determined whether the received information indicates that at least one data element or a portion thereof input to at least one multiplier and / or at least one adder can be considered to have a zero value. If the information indicates that at least one data element or a portion thereof input to at least one multiplier and / or at least one adder can be considered to have a zero value, then method 1100 proceeds to box 1106. However, if the information does not indicate that at least one data element or a portion thereof input to at least one multiplier and / or at least one adder can be considered to have a zero value, then method 1100 terminates.

[0090] At block 1106, all or a portion of the storage elements corresponding to at least one data element or a portion thereof that can be considered to have a zero value that is input to the at least one multiplier and / or at least one adder are clock gated. In some cases, the method 1100 can end, and in other cases, the method 1100 can proceed to block 1108 and / or block 1110.

[0091] At block 1108, the clock gating control logic 302 causes a zero bit to be provided to the respective adder or multiplier for each storage element that is clock gated.

[0092] At block 1110, the clock gating control logic 302 generates and outputs information indicating whether the output data element of the at least one multiplier and / or at least one adder can be considered to have a zero value based on whether the data element input to the at least one multiplier and / or at least one adder has a zero value or can be considered to have a zero value. In some cases, the clock gating control logic is configured to output information indicating that the output data element of the multiplier can be considered to have a zero value when any input data element of the multiplier can be considered to have a zero value. As described above, this is because if any one of the inputs to the multiplier has a zero value (or can be considered to have a zero value), then the output of the multiplier will be zero (or can be considered to have a zero value). In some cases, the clock gating control logic is configured to output information indicating that the output data of the adder can be considered to have a zero value only when all of the input data elements of the adder can be considered to have a zero value. As described above, this is because the output of the adder has a zero value (or can be considered to have a zero value) only when all of the inputs to the adder have a zero value (or can be considered to have a zero value).

[0093] Reference is now made to Figure 12 which shows Figure 11the first example implementation of the method 1100, in which the storage elements are clock-gated on an element basis (i.e., all storage elements of a data element are clock-gated together). The method 1200 begins at block 1202, in which the clock-gating control logic 302 receives information indicating whether each data element input to the at least one multiplier and / or the at least one adder can be considered to have a zero value. The information indicating whether each data element input to the at least one multiplier and / or the at least one adder can be considered to have a zero value can include a bit for each data element indicating whether each data element input to the at least one multiplier and / or the at least one adder can be considered to have a zero value. In some cases, the information indicating whether each data element input to the multiplier can be considered to have a zero value is configured to indicate that each data element input to the multiplier can be considered to have a zero value if any data element input to the multiplier has a zero value. Once the information indicating whether the data elements input to the at least one multiplier and / or the at least one adder can be considered to have a zero value has been received, the method 1200 proceeds to block 1204.

[0094] At block 1204, a determination is made as to whether the received information indicates that at least one data element input to the at least one multiplier and / or the at least one adder can be considered to have a zero value. If the information indicates that at least one data element input to the at least one multiplier and / or the at least one adder can be considered to have a zero value, the method 1200 proceeds to block 1206. However, if the information does not indicate that at least one data element input to the at least one multiplier and / or the at least one adder can be considered to have a zero value, the method 1200 ends.

[0095] At block 1206, in response to receiving the information indicating that at least one data element can be considered to have a zero value, all storage elements corresponding to the at least one data element are clock-gated. The method 1200 then ends.

[0096] Reference is now made to Figure 13 which shows Figure 11 the second example implementation of the method 1100, in which the storage elements are clock-gated on a partial element basis (i.e., clock-gating can be performed on only a portion of the storage elements of a data element). The method 1300 begins at block 1302, in which the clock-gating control logic 302 receives information indicating whether a portion of each data element input to the at least one multiplier and / or the at least one adder can be considered to have a zero value. The information indicating whether a portion of each data element input to the at least one multiplier and / or the at least one adder can be considered to have a zero value can include a bit width of the initial set of data elements. For example, in the case of a 32-bit data element, the information indicating whether a portion of each data element input to the at least one multiplier and / or the at least one adder can be considered to have a zero value can include a 32-bit value indicating whether each data element input to the at least one multiplier and / or the at least one adder can be considered to have a zero value. Once the information indicating whether a portion of each data element input to the at least one multiplier and / or the at least one adder can be considered to have a zero value has been received, the method 1300 proceeds to block 1304. Figure 1In particular embodiments, the hardware unit 100 receives two sets of data elements - a first set of data elements, which generally corresponds to elements of a first matrix; and a second set of data elements, which generally corresponds to elements of a second matrix. The first set of data elements and the second set of data elements can have a bit width that is less than the maximum bit width, and the bit widths can be the same or different between the two sets of inputs. Once the information indicating whether a portion of each data element input to at least one multiplier and / or at least one adder can be considered to have a zero value has been received, the method 1300 proceeds to block 1304.

[0097] At block 1304, it is determined whether the received information indicates that at least a portion of at least one data element input to at least one multiplier and / or at least one adder can be considered to have a zero value. In some cases, when the bit width of at least one set of initial data elements is less than the maximum bit width, it is determined that the information indicates that at least a portion of at least one data element input to at least one multiplier and / or at least one adder can be considered to have a zero value. If it is determined that the information indicates that at least a portion of at least one data element input to at least one multiplier and / or at least one adder can be considered to have a zero value, the method 1300 proceeds to block 1306. However, if it is determined that the information does not indicate that at least a portion of at least one data element input to at least one multiplier and / or at least one adder can be considered to have a zero value, the method 1300 ends.

[0098] At block 1306, in response to receiving information indicating that a portion of at least one data element input to at least one multiplier and / or at least one adder can be considered to have a zero value, a portion of storage elements corresponding to the data element is clocked. In some cases, when the bit width of the first set of data elements is less than the maximum bit width, N storage elements corresponding to the first data element of the multiplier are clocked, where N is equal to the maximum bit width minus the bit width of the first set of data elements; when the bit width of the second set of data elements is less than the maximum bit width, K storage elements corresponding to the second data element of the multiplier are clocked, where K is equal to the maximum bit width minus the bit width of the second set of data elements. In some cases, when the bit width of at least one set of initial data elements is less than the maximum bit width, a portion of two input data elements of the adder are clocked. This is because the adder receives a data element representing the multiplication of a data element from the first set of data elements and a data element from the second set of data elements, so if the bit width of either set is less than the maximum bit width, then the bit width of the multiplication output will be less than the maximum bit width of the multiplication. As described above, when the data elements are most significant bit aligned, the storage elements that are clocked will be the storage elements corresponding to the least significant bits.

[0099] Figure 14A computer system is shown that can implement the hardware unit described herein for performing matrix multiplication. The computer system includes a CPU 1402, a GPU 1404, memory 1406, and other devices 1414, such as a display 1416, speakers 1418, and a camera 1420. The hardware unit 1410 for performing matrix multiplication (corresponding to the one including...) Figure 3 , Figure 4 , Figure 6 or Figure 10 The hardware unit 100 (control logic 302, 400, 600, or 1000 for performing matrix multiplication) can be implemented on GPU 1404. In other examples, hardware unit 1410 can be implemented on CPU 1402. Components of the computer system can communicate with each other via communication bus 1420.

[0100] Figure 3 , Figure 4 , Figure 6 and Figure 10 The hardware unit and clock gating control unit for performing matrix multiplication are shown as comprising multiple function blocks. This is merely illustrative and is not intended to limit the strict division between the different logic elements of these entities. Each function block can be provided in any suitable manner. It will be understood that the intermediate values ​​formed by the function blocks described herein do not need to be physically generated by the function blocks at arbitrary points, but may simply represent logical values ​​that conveniently describe the processing performed by the function blocks between their inputs and outputs.

[0101] The hardware unit for performing matrix multiplication described herein can be embodied in hardware on an integrated circuit. The hardware unit for performing matrix multiplication described herein can be configured to perform any of the methods described herein. Generally, any of the functions, methods, techniques, or components described above can be implemented in software, firmware, hardware (e.g., fixed logic circuitry), or any combination thereof. The terms “module,” “function,” “component,” “element,” “unit,” “block,” and “logic” are used herein to generally denote software, firmware, hardware, or any combination thereof. In the case of a software implementation, a module, function, component, element, unit, block, or logic represents program code that performs a specified task when executed on a processor. The algorithms and methods described herein can be executed by one or more processors that execute code, causing the one or more processors to perform the algorithm / method. Examples of computer-readable storage media include random access memory (RAM), read-only memory (ROM), optical disk, flash memory, hard disk storage, and other memory devices that can use magnetic, optical, or other technologies to store instructions or other data that can be accessed by a machine.

[0102] The terms "computer program code" and "computer readable instructions" are used herein to refer to any kind of executable code, including code expressed in machine language, interpreted language, or scripting language, for a processor. Executable code includes binary code, machine code, byte code, code that defines integrated circuits (e.g., a hardware description language or netlist), and code expressed in a programming language (e.g., C, Java, or OpenCL). Executable code can be, for example, any kind of software, hardware, script, module, or library that, when executed, processed, interpreted, assembled, executed by a virtual machine or other software environment, causes a processor of a computer system supporting the executable code to perform the tasks specified by the code.

[0103] A processor, computer, or computer system can be any kind of device, machine, or special purpose circuit, or collection or part thereof, having processing capability such that it can execute instructions. A processor can be any kind of general or special purpose processor, such as a CPU, GPU, system on a chip, state machine, media processor, application specific integrated circuit (ASIC), programmable logic array, field programmable gate array (FPGA), etc. A computer or computer system can include one or more processors.

[0104] It is also desirable to cover software that defines hardware configurations described herein, such as HDL (hardware description language) software for designing integrated circuits or for configuring programmable chips to implement desired functions. That is, a computer readable storage medium having computer readable program code encoded thereon in the form of an integrated circuit definition data set that, when processed (i.e., run) in an integrated circuit manufacturing system, configures the system to manufacture a hardware unit for performing matrix multiplication or a hardware unit for performing matrix multiplication that includes any of the apparatuses described herein configured to perform any of the methods described herein can be provided. The integrated circuit definition data set can be, for example, an integrated circuit description.

[0105] Thus, a method of manufacturing a hardware unit for performing matrix multiplication described herein at an integrated circuit manufacturing system is provided. Also, an integrated circuit definition data set that, when processed by an integrated circuit manufacturing system, causes the method of manufacturing a hardware unit for performing matrix multiplication described herein to be performed is provided.

[0106] An integrated circuit definition data set can take the form of computer code, for example, as a netlist, code for configuring a programmable chip, as a hardware description language (including register transfer level (RTL) code) defining hardware suitable for fabrication of an integrated circuit at any level of abstraction, as a high level circuit representation (e.g., Verilog or VHDL), and as a low level circuit representation (e.g., OASIS (RTM) and GDSII). A higher level representation (e.g., RTL) logically defining hardware suitable for fabrication of an integrated circuit can be processed at a computer system configured to generate a fabrication definition of the integrated circuit in the context of a software environment, where the fabrication definition includes definitions of circuit elements and rules for combining the elements to generate a fabrication definition representing the defined integrated circuit. In the execution of software at the computer system to define the general case of a machine, one or more intermediate user steps (e.g., providing commands, variables, etc.) can be required to cause the computer system configured to generate a fabrication definition of an integrated circuit to execute code defining the integrated circuit to generate the fabrication definition of the integrated circuit.

[0107] Reference is now made to Figure 15 , describing an example of processing an integrated circuit definition data set at an integrated circuit fabrication system to configure the system to fabricate a hardware unit for performing matrix multiplication.

[0108] Figure 15 An example of an integrated circuit (IC) fabrication system 1502 configured to fabricate a hardware unit for performing matrix multiplication described in any of the examples herein is shown. Specifically, the IC fabrication system 1502 includes a layout processing system 1504 and an integrated circuit generation system 1506. The IC fabrication system 1502 is configured to receive an IC definition data set (e.g., defining a hardware unit for performing matrix multiplication described in any of the examples herein), process the IC definition data set, and generate an IC from the IC definition data set (e.g., an IC definition data set embodying a hardware unit for performing matrix multiplication described in any of the examples herein). The processing of the IC definition data set configures the IC fabrication system 1502 to fabricate an integrated circuit embodying a hardware unit for performing matrix multiplication described in any of the examples herein.

[0109] The layout processing system 1504 is configured to receive and process the IC definition data set to determine a circuit layout. Methods of determining a circuit layout from an IC definition data set are known in the art, for example, can include synthesizing RTL code to determine a gate level representation of the circuit to be generated (e.g., in terms of logic components (e.g., NAND, NOR, AND, OR, MUX, and FLIP-FLOP components)). The circuit layout can be determined from the gate level representation of the circuit by determining location information for the logic components. This can be implemented automatically or with user involvement to optimize the circuit layout. When the layout processing system 1504 has determined the circuit layout, it can output a circuit layout definition to the IC generation system 1506. The circuit layout definition can be, for example, a circuit layout description.

[0110] The IC generation system 1506 generates an IC from the circuit layout definition, as is known in the art. For example, the IC generation system 1506 can perform a semiconductor device fabrication process to generate the IC, which can include a multi-step sequence of photolithography processes and chemical processing steps during which an electronic circuit is gradually formed on a wafer made of semiconductor material. The circuit layout definition can be in the form of a mask that can be used in a photolithography process to generate the IC from the circuit definition. Alternatively, the circuit layout definition provided to the IC generation system 1506 can be in the form of computer readable code that the IC generation system 1506 can use to form appropriate masks for generating the IC.

[0111] The different processes performed by the IC manufacturing system 1502 can all be implemented at one location, for example, by one party. Alternatively, the IC manufacturing system 1502 can be a distributed system, so some processes can be performed at different locations and can be performed by different parties. For example, some of the following stages can be performed at different locations and / or by different parties: (i) synthesizing RTL code representing the IC definition data set to form a gate level representation of the circuit to be generated, (ii) generating a circuit layout based on the gate level representation, (iii) forming a mask from the circuit layout, and (iv) fabricating an integrated circuit using the mask.

[0112] In other examples, processing of an integrated circuit definition data set at an integrated circuit manufacturing system can configure the system to manufacture a hardware unit for performing matrix multiplication, without processing the IC definition data set to determine a circuit layout. For example, the integrated circuit definition data set can define a configuration of a reconfigurable processor (e.g., FPGA), and processing of the data set can configure the IC manufacturing system to generate a reconfigurable processor having the defined configuration (e.g., by loading configuration data to the FPGA).

[0113] In some embodiments, an integrated circuit manufacturing definition data set, when processed in an integrated circuit manufacturing system, can cause the integrated circuit manufacturing system to generate the apparatus described herein. For example, by an integrated circuit manufacturing definition data set to cause Figure 15 The configuration of the integrated circuit manufacturing system in the manner described can cause the apparatus described herein to be manufactured.

[0114] In some examples, an integrated circuit definition data set can include software running on or in combination with hardware defined at the data set. In Figure 15 In the example shown, the IC generation system can be further configured by the integrated circuit definition data set to load firmware into or provide program code to the integrated circuit for use by the integrated circuit in manufacturing the integrated circuit according to program code defined at the integrated circuit definition data set.

[0115] The concepts set forth in this application can result in performance improvements in the implementation of the apparatuses, devices, modules, and / or systems (and the methods implemented herein) as compared to known implementations. Performance improvements can include one or more of increased computational performance, decreased latency, increased throughput, and / or decreased power consumption. During the manufacturing of these apparatuses, devices, modules, and systems (e.g., in integrated circuits), a balance can be struck between performance improvements and physical implementation, improving the manufacturing methodology. For example, a balance can be struck between performance improvements and layout area, matching the performance of known implementations but using less silicon. This can be achieved, for example, by reusing functional blocks in a sequential manner or sharing functional blocks between elements of the apparatuses, devices, modules, and / or systems. Conversely, the concepts of this application that result in improvements in the physical implementation of the apparatuses, devices, modules, and systems (e.g., a reduction in silicon area) can be traded off for improved performance. This can be achieved, for example, by manufacturing multiple instances of a module within a predetermined area budget.

[0116] Applicant hereby incorporates by reference the entire disclosure of each of the above-cited references. Applicant hereby discloses each of the individual features mentioned in this application individually and in any combination, so that the disclosure can be claimed in any combination of individual features or combinations of features disclosed herein, whether or not the features or combinations of features are presented as being part of a particular embodiment. In view of the descriptions and explanations given herein, one of ordinary skill in the art will be readily able to ascertain the meaning of terms, phrases, and descriptions used in this disclosure.

Claims

1. A hardware unit for performing a matrix multiplication between a first matrix of first data elements and a second matrix of second data elements, the hardware unit comprising: a multiplier stage comprising a plurality of multipliers, each multiplier configured to multiply a first data element and a second data element to produce a multiplied data element; one or more adder stages following the multiplier stage and forming an adder tree to produce a sum of the multiplied data elements, each adder stage comprising one or more adders configured to add at least two data elements output by a preceding stage to produce an added data element; a set of storage elements for each multiplier and for each adder, the set of storage elements for a multiplier or for an adder comprising a storage element for each bit of each data element to be processed by that multiplier or that adder; and clock gating control logic for each multiplier and for each adder, the clock gating control logic configured to: receive an element gating signal for each data element to be processed by a multiplier or an adder, the element gating signal indicating whether that data element can be considered to have a zero value, clock gate all storage elements corresponding to a data element in response to the element gating signal for that data element indicating that the data element can be considered to have a zero value, determine, based on the received element gating signals, whether a data element produced by a multiplier or an adder can be considered to have a zero value, and provide an output element gating signal to the clock gating control logic of a subsequent stage, the output element gating signal indicating whether a data element produced by a multiplier or an adder can be considered to have a zero value, such that element gating information is propagated from one stage to the next.

2. The hardware unit of claim 1, wherein, The clock gating control logic for each adder is further configured to clock gate a portion of the storage elements corresponding to each data element to be processed by the adder when at least one of a bit width of the first data elements and a bit width of the second data elements is less than a maximum bit width.

3. The hardware unit of claim 1 or 2, wherein, The clock gating control logic for each multiplier is further configured to clock gate a portion of the storage elements corresponding to the first data elements to be processed by the multiplier when a bit width of the first data elements to be processed by the multiplier is less than a maximum bit width.

4. The hardware unit of claim 3, wherein, The clocked gated portion of the storage elements corresponding to the first data elements to be processed by the multiplier comprises N storage elements, N being the maximum bit width minus the bit width of the first data elements.

5. The hardware unit of claim 3, wherein, The clock gating control logic for each multiplier is further configured to clock gate a portion of the storage elements corresponding to the second data elements to be processed by the multiplier when a bit width of the second data elements to be processed by the multiplier is less than the maximum bit width.

6. The hardware unit of claim 5, wherein, The clocked gated portion of the storage elements corresponding to the second data elements to be processed by the multiplier comprises K storage elements, K being the maximum bit width minus the bit width of the second data elements.

7. The hardware unit of any one of claims 2, 4, 5, or 6, wherein, The data elements are most significant bit aligned, and the clocked gated portion of the storage elements corresponding to a data element is the storage elements corresponding to the least significant bits of that data element.

8. The hardware unit of any one of claims 2, 4, 5, or 6, wherein, Each clock gating control logic is configured to receive information identifying a bit width of the first data element and / or a bit width of the second data element.

9. The hardware unit of claim 1 or 2, wherein, The clock gating control logic for each multiplier is further configured to clock gate all storage elements corresponding to the first data element and the second data element to be processed by the multiplier when at least one of the received element enable signals indicates that the respective data element can be considered to have a zero value.

10. The hardware unit of claim 1 or 2, wherein, The element enable signals received at the clock gating control logic of a multiplier are configured to indicate that both the first data element and the second data element of the multiplier can be considered to have a zero value if at least one of the first data element and the second data element to be processed by the multiplier has a zero value.

11. The hardware unit of claim 1 or 2, wherein, Each clock gating control logic is further configured to cause zero bits to be provided to the respective multiplier or adder for each storage element that is clock gated.

12. The hardware unit of claim 1 or 2, wherein, The clock gating control logic for each multiplier is configured to generate an output element enable signal indicating that a data element produced by the multiplier can be considered to have a zero value when any of the received element enable signals indicates that the respective data element can be considered to have a zero value.

13. The hardware unit of claim 1 or 2, wherein, The clock gating control logic for each adder is configured to generate an output element enable signal indicating that a data element produced by the adder can be considered to have a zero value when all of the received element enable signals indicate that the respective data elements can be considered to have a zero value.

14. A method of clock gating storage elements in a hardware unit for performing a matrix multiplication between a first matrix of first data elements and a second matrix of second data elements, the hardware unit comprising: a plurality of multipliers each configured to multiply a first data element and a second data element to produce a multiplication data element; a plurality of adders forming an adder tree for producing a sum of the multiplication data elements, each adder configured to produce an addition data element by adding multiplication data elements produced by a group of multipliers or by adding addition data elements produced by a group of adders; and a group of storage elements for each multiplier and for each adder, the group of storage elements for a multiplier or for an adder comprising a storage element for each bit of each data element to be processed by that multiplier or that adder, the method comprising for each multiplier and for each adder: receiving an element enable signal for each data element to be processed by the multiplier or the adder, the element enable signal indicating whether that data element can be considered to have a zero value; clocking out all storage elements corresponding to a data element in response to the element enable signal for that data element indicating that the data element can be considered to have a zero value; determining, based on the received element enable signals, whether a data element produced by the multiplier or the adder can be considered to have a zero value; and generating an output element enable signal indicating whether a data element produced by the multiplier or the adder can be considered to have a zero value; wherein the received element enable signal for a data element to be processed by an adder is the output element enable signal generated for a data element produced by a multiplier or another adder.

15. A method of fabricating a hardware unit as claimed in any of claims 1 to 13 using an integrated circuit fabrication system.

16. A method of fabricating a hardware unit as claimed in any of claims 1 to 13 using an integrated circuit fabrication system, the method comprising: processing a computer readable description of the hardware unit using a layout processing system to generate a circuit layout description of an integrated circuit embodying the hardware unit; and fabricating the hardware unit from the circuit layout description using an integrated circuit generation system.

17. Hardware for implementing a deep neural network, DNN, comprising a hardware unit as claimed in any of claims 1 to 13, the hardware unit being configured to compute a convolution associated with one or more convolutional layers and / or one or more fully connected layers of the DNN.

18. A computer readable storage medium having encoded thereon computer readable code which, when run, causes the method of claim 14 to be performed.

19. A computer readable storage medium having stored thereon a computer readable description of a hardware unit as claimed in any of claims 1 to 13, the computer readable description, when processed in an integrated circuit fabrication system, causing the integrated circuit fabrication system to fabricate an integrated circuit embodying the hardware unit.

20. A computer readable storage medium having stored thereon a computer readable description of a hardware unit as claimed in any of claims 1 to 13, the computer readable description, when processed in an integrated circuit fabrication system, causing the integrated circuit fabrication system to: process a computer readable description of the hardware unit using a layout processing system to generate a circuit layout description of an integrated circuit embodying the hardware unit; and fabricate the hardware unit from the circuit layout description using an integrated circuit generation system.

21. An integrated circuit fabrication system configured to fabricate a hardware unit as claimed in any of claims 1 to 13.

22. An integrated circuit fabrication system comprising: a computer readable storage medium having stored thereon a computer readable description of a hardware unit as claimed in any of claims 1 to 13; a layout processing system configured to process the computer readable description to generate a circuit layout description of an integrated circuit embodying the hardware unit; and an integrated circuit generation system configured to fabricate the hardware unit from the circuit layout description. ​

Images