Systolic array component combining multiple integer and floating point data types

Abstract

Systems and methods are provided for performing multiply-accumulate operations for multiple data types in systolic arrays to improve clock speed and / or reduce the size and number of systolic arrays required to perform multiply-accumulate operations for multiple data types. Each processing element in the systolic array can have a shared multiplier and one or more adders. The shared multiplier can have separate circuitry and / or shared circuitry, where the shared circuitry is capable of performing at least a portion of an integer multiplication and at least a portion of a non-integer multiplication. The one or more adders can be shared adders or separate adders. The shared adders can have separate circuitry and shared circuitry, where the shared circuitry is capable of performing at least a portion of an integer addition and at least a portion of a non-integer addition.

Description

BACKGROUND

[0001] An artificial neural network is a computing system having an architecture based on biological neural networks. Neural networks can be implemented through circuits and data paths such as systolic arrays. Systolic arrays can accelerate the performance of the training and inference phases of artificial neural networks. During the training phase, input data can be provided to train a model. During the inference phase, new inputs can be processed according to the model to obtain a prediction result. User applications typically use the model during the inference phase, so the inference phase can generally be time sensitive, and latency during the inference phase can negatively impact user experience.

[0002] As more and more applications use artificial neural networks, these applications also use a wide range of input data types and input data ranges. Improving data type compatibility and data range compatibility often results in increased complexity, size, and cost of processing elements in systolic arrays. These increases can also affect system processing speed and system power consumption. When a systolic array is required to support multiple data types, the power consumption and size of the systolic array can be an important consideration. BRIEF DESCRIPTION OF DRAWINGS

[0003] Various features will now be described, with reference to the following drawings. In all of the drawings, reference numerals can be repeated between the drawings to indicate correspondence between the referenced elements. The drawings are provided to illustrate examples described herein and are not intended to limit the scope of the disclosure.

[0004] Figure 1 An exemplary 4x4 systolic array is shown.

[0005] Figure 2A A processing element for neural network computation including shared multipliers and shared adders is shown in accordance with certain examples of the disclosed technology.

[0006] Figure 2B A processing element for neural network computation including shared multipliers and separate adders is shown in accordance with certain examples of the disclosed technology.

[0007] Figure 2C A processing element for neural network computation including shared multipliers, shared adders, and delay registers is shown in accordance with certain examples of the disclosed technology.

[0008] Figure 2D A processing element for neural network computation including shared multipliers, separate adders, and delay registers is shown in accordance with certain examples of the disclosed technology.

[0009] Figure 3A device is shown that includes signal modifier and zero detector circuitry for input data elements and weights entering a systolic array for neural network computation according to certain examples of the disclosed technology.

[0010] Figure 4 A device is shown that shows propagation of data types, zero detectors, input data elements, and weight values from one processing element to another according to a first example of the disclosed technology.

[0011] Figure 5 A device is shown that shows propagation of zero indicators, input data elements, and weight values from one processing element to another according to a second example of the disclosed technology.

[0012] Figure 6 A device is shown for neural network computation according to some examples of the disclosed technology.

[0013] Figure 7 A method is shown performed by a processing element for neural network computation according to some examples of the disclosed technology.

[0014] Figure 8 An example of a computing device is shown according to certain aspects of the disclosure.

[0015] Figure 9 A method is shown performed by a processing element for neural network computation involving both integer and non-integer operations.

[0016] Figure 10A An example of a multiply accumulate (MAC) data path is shown. Figure 2A An example of additional details of a multiply accumulate (MAC) data path is shown.

[0017] Figure 10B An example of additional details of a multiply accumulate (MAC) data path is shown. Figure 2B An example of additional details of a multiply accumulate (MAC) data path is shown.

[0018] Figure 10C An example of additional details of a multiply accumulate (MAC) data path is shown. Figure 2C An example of additional details of a multiply accumulate (MAC) data path is shown.

[0019] Figure 10D An example of additional details of a multiply accumulate (MAC) data path is shown. Figure 2D An example of additional details of a multiply accumulate (MAC) data path is shown. DETAILED DESCRIPTION

[0020] In the following description, various examples will be described. For purposes of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the examples. However, it will also be apparent to one skilled in the art that the examples can be practiced without the specific details presented. Furthermore, well-known features can have been omitted or simplified in order not to obscure the example being described.

[0021] Convolutional neural networks (CNNs) are generally feed-forward artificial neural networks that can include multiple intermediate layers and the output from one layer can be used as input to the next layer. A systolic array can be used to accelerate workloads in neural networks by reading data from memory once and reusing it for multiple computations. A systolic array can be implemented using a two-dimensional array of processing elements (PEs).

[0022] Generally, an input dataset (e.g., an input feature map) can be fed into its respective row of a systolic array one input data element at a time and passed from one PE to another PE in a given row starting from the left-most PE. As the proliferation of machine learning applications and neural network applications, the diversity of values and data types of input datasets has likewise proliferated. Some applications use higher precision floating point 32-bit (FP32) or floating point 64-bit (FP64) data types for input. Some applications use floating point 16-bit (FP16), brain floating point 16-bit (bfloatl6 or BF16), or 16-bit integer or 8-bit integer data types for input.

[0023] Providing support for longer inputs (e.g., 64-bit or 32-bit inputs) results in larger amounts of memory, larger communication bandwidths in and between components, larger circuit complexity, larger latencies, and higher costs compared to supporting shorter data types (e.g., 16-bit or 8-bit inputs). In a systolic array of hundreds or thousands of PEs, the increase in support for longer data types results in a multiplicative increase in cost, memory size, communication bandwidth, circuit complexity, and latency.

[0024] According to embodiments of the disclosure, many machine learning applications and neural network applications that are designed or programmed to use longer data types can achieve improved performance speed and execute on lower cost, simpler hardware that supports shorter data types, while sacrificing little to no accuracy of end results such as final inference predictions. This counterintuitive result can occur because simpler hardware can run faster, use less memory, use less bandwidth, use simpler circuits, and use silicon area more cost effectively, and because many applications are not sensitive to intermediate precision.

[0025] In particular, as disclosed herein, longer data type inputs can be executed on hardware that supports shorter data types by quantizing the longer data types to the shorter data types. For many machine learning applications and neural network applications, the loss of intermediate precision due to quantization can have little to no impact on the final result. Thus, simpler hardware can be used to improve the execution of programs written by programmers and designers who use overly precise data types.

[0026] Because of the wide variety of data types used in machine learning applications and neural network applications, including, for example, 16-bit integers, 8-bit integers, FP16, and bfloat16 data types, it is desirable to support a variety of the data types among these data types. It can also be desirable to support a variety of the data types among these data types when the input data is quantized.

[0027] When training a model or performing inference, input values can vary over a wide range. In some applications, the input values are substantially centered around zero. In other applications, the input values are centered around an offset from zero. In many applications, the input data is not symmetrically distributed about zero, and so an asymmetric sampling scheme can be applied to more accurately reflect the distribution of the input data. After asymmetric sampling, the original values of zero can be quantized to values or ranges represented as non-zero binary values.

[0028] Processing elements in systolic arrays can be modified to perform arithmetic operations on asymmetrically quantized numbers. However, such modifications greatly increase the complexity of the PEs. For example, special circuitry can handle multiplication of one or more zero inputs to generate zero as an output. If zero values are shifted according to an asymmetric quantization scheme, then adding this special circuitry increases the complexity of each PE and adds significant cost.

[0029] As an alternative solution, to support processing of asymmetrically quantized data, the data can first be partially dequantized by shifting the data so that zeros are represented as binary zeros, and the arithmetic circuitry can do this by increasing the bit representation of the numbers by one bit to maintain the accuracy of the quantization. For example, to process quantized 8-bit integers, a systolic array can partially dequantize the 8-bit integers to 9-bit integers by shifting the 8-bit integers to a symmetric 9-bit form. This partial dequantization simplifies the arithmetic logic circuitry, such as multipliers, that receive the inputs, because the multipliers can avoid including complex circuitry to support quantized shifted zeros that are not represented as zeros, and the multipliers can also avoid supporting asymmetric representations of positive and negative numbers.

[0030] The dequantizer can provide 9-bit integers to multiple PEs to avoid including dequantization circuitry in each individual PE. The nine-bit integers can accurately represent a quantized 8-bit integer that was shifted to a symmetric representation. Various example systems can include a partial dequantizer configured to shift a quantized number to a slightly longer and lossless enough length symmetric representation and provide the partially dequantized number to multiple PEs without mapping to a next larger standard data type (e.g., without mapping an 8-bit to a 16-bit representation, without mapping a 16-bit to a 32-bit representation).

[0031] The PEs in systolic arrays disclosed herein can perform more efficient operations and support a wide range of data types and data values. The PEs can perform integer operations, floating point operations, other types of non-integer operations, or any combination thereof. For example, the PEs can include a 16-bit arithmetic datapath that performs multiply-accumulate operations. The example 16-bit arithmetic datapath can have multipliers and adders that support all data types or any combination of data types: 8-bit integers, 9-bit integers, FP16, and BF16.

[0032] To perform operations on both floating point inputs and integer inputs, two or more separate circuits can be implemented, where each separate circuit operates on one particular data type. For example, a first multiplier can operate on integer data types, while a second multiplier can operate on floating point data types. As another example, a first adder can operate on integer data types, while a second multiplier can operate on floating point data types.

[0033] In examples disclosed herein, such as discussed with respect to Figure 2A 、 Figure 2B 、 Figure 2C 、 Figure 2D 、 Figure 10A 、 Figure 10B 、 Figure 10C and Figure 10D , the systolic array can have PEs that include shared circuitry that operates on two or more data types. For example, the PEs can operate on integer inputs using a first circuit that includes components that are also used to operate on floating point inputs. As a more specific example, the PEs can include a shared multiplier that can operate on both integer and floating point data types, where the shared multiplier has certain sub-components that are used to handle both integer data types and floating point data types.

[0034] By reusing shared sub-components, examples of the disclosed technology reduce dynamic power consumption in systolic arrays and reduce the size and number of circuits needed to support both integer and non-integer operations. Further, examples of the disclosed technology can include systems and methods that enable systolic arrays to accept both integer and non-integer data, thereby reducing the need for dedicated circuitry for each data type. Individual PEs in a systolic array can perform mathematical operations on both integers and non-integers. In some examples, a PE includes shared arithmetic circuitry that performs at least a portion of an integer arithmetic operation and is reused to perform at least a portion of a non-integer arithmetic operation. The shared arithmetic circuitry enables systolic arrays to reduce power consumption and size compared to systems with separate arithmetic circuitry for different data types.

[0035] In some examples, such as described further with respect to Figure 2A 、 Figure 2B 、 Figure 2C and Figure 2D a PE can include arithmetic circuitry configured to perform both integer and non-integer operations, such as a multiplier and / or an adder. As described further with respect to Figure 10A 、 Figure 10B 、 Figure 10C and Figure 10D a PE in a systolic array can include a first sub-circuit configured to perform at least a portion of one or more integer operations and one or more non-integer operations. The PE in the systolic array can also include a second sub-circuit capable of completing another portion of the non-integer operations.

[0036] A systolic array can receive a data type control signal indicating a selected data type. For example, the data type control signal can select between integer and non-integer data types, such as 8-bit integers, 9-bit integers, FP16, and bfloat16. In some examples, the systolic array can generate the selected data type based on inputs to the systolic array, such as input data elements, input partial sums, weights, or operation codes. In some examples, the selected data type is received as a separate input signal. The control signal indicating the selected data type can be provided to a PE to instruct the PE to perform operations on the selected data type and / or to select outputs having the selected data type.

[0037] In some examples, arithmetic operations can be performed on selected data types while skipping arithmetic operations on unselected data types, such as skipping non-integer operations when the selected data type is an integer. For example, a systolic array can perform multiply-accumulate operations on integer inputs when the selected data type is an integer data type, and then the systolic array can perform multiply-accumulate operations on floating point inputs (or other data types) when the selected data is changed to a floating point data type (or other data type). In other examples, PEs perform both integer and non-integer operations without regard to the selected data type, and a result formatted in the selected data type is selected from the output of the multiply operation and the output of the accumulate operation based on a data type control signal.

[0038] PES in a systolic array can perform multiple arithmetic operations, including but not limited to addition and multiplication. Each of these operations can use shared circuitry or separate circuitry. Each PE in a systolic array can use any combination of shared or separate circuitry when performing addition, multiplication, or any other operation. For example, a PE can use a shared multiplier and separate adders, a PE can use separate multipliers and a shared adder, or a PE can use any combination of one or more shared / separate multipliers with one or more shared / separate adders. Shared circuitry implements at least a portion of one operation on a first data type and at least a portion of a second operation on a second data type. For example, shared circuitry can be a shared multiplier that is used to fully perform an integer multiplication and to perform at least a portion of a floating point multiplication. As a different example, separate circuitry can include a first integer adder and a second floating point adder that is physically separate from the first integer adder.

[0039] A systolic array can include multiple delay registers to pipeline different stages within a PE. Each of these delay registers can receive an input and cause an output to be input delayed by at least one clock cycle. For example, a delay register can be placed between a shared multiplier and one or more adders, such that multiplication and addition can be performed in parallel, and such that the combined delay of multiplication and addition does not limit the clock speed. A shared multiplier can be configured to generate one or more multiplier products to be stored in one or more delay registers for manipulation by one or more adders in a subsequent clock cycle.

[0040] PES in a systolic array can skip certain operations under certain conditions. For example, a multiplication operation can be skipped when a zero is detected on an input data element for a current operation, when a zero is stored as a weight, or when a no operation (NOP) is received by a PE. Operations of other circuitry, such as registers, multiplexers, and the like, can additionally or alternatively be skipped under the same or similar conditions.

[0041] Figure 1An exemplary 4x4 systolic array 100 is shown. For example, the systolic array 100 can include four PEs in each row and four PEs in each column. It should be understood that the systolic array 100 can include any number of PEs in each row and each column. The systolic array 100 can be part of a neural network processor in a computer system. For example, the computer system can be configured to provide a multi-tenant computing service for data processing applications, such as an image recognition service, text-based data processing (e.g., processing of search queries), audio or video data processing, etc.

[0042] Each PE can include a row input bus 102, a column input bus 104, a column output bus 106, and a row output bus 108. The PE can receive input from a left-side PE of the same row (or from an external circuit) via the row input bus 102. The PE can also receive input from a PE above the same column (or from an external circuit) via the column input bus 104. The PE can perform an arithmetic computation based on the inputs and transmit a result of the arithmetic computation to a PE below the same column (or to an external circuit) via the column output bus 106. The PE can also forward the input received via the row input bus 102 to a right-side PE of the same row via the row output bus 108.

[0043] The systolic array 100 can be configured to perform arithmetic computations, including multiplication and addition operations, on processing elements of a neural network. For example, each PE can include an arithmetic unit, such as a multiplier and an adder, a fused multiplier-adder, a pipelined multiplier and adder, or any combination thereof. In Figure 1 In an example, each row of PEs can be configured to process an input data set, and each column of PEs can generate an output data set based on the input data sets received by each PE in the given column.

[0044] In some implementations, the PEs of column 112 (the leftmost column) can receive four input data sets, each processed by a row of PEs. Each PE in column 112 can obtain an input data element and an associated weight value from the corresponding input data set received via row input bus 102 and multiply the input data element by the weight value to generate a scaled input. The scaled inputs generated by the PEs within any column, including column 112, can be accumulated by one or more adders of each PE. For example, PE 112a (of column 112) can generate one or more first scaled inputs (from a first input data set). The one or more adders can generate one or more sums to be selected as output partial sums. PE 112a can be configured to transmit the output partial sums to PE 112b via column output bus 106. PE 112b can also generate second scaled inputs (from a second input data set) and add the second scaled inputs to the partial sums. The updated partial sums are then transmitted to PE 112c via column output bus 106. The partial sums are updated and propagated across column 112, and PE 112d can generate a sum of the scaled inputs from the four input data sets.

[0045] The sum generated by PE 112d can correspond to an output data set and can be fed back to the leftmost PE after passing through an activation function.

[0046] Each PE in column 112 can also propagate the input data sets to other columns of PEs (e.g., column 114), which can scale the input data sets with a different set of weights than column 112. Each column of PEs can perform arithmetic operations (such as multiplication and addition) to generate output data elements in parallel for other processing elements. In Figure 1 In the example of FIG. 1, systolic array 100 can generate output data elements for four PEs corresponding to four columns of systolic array 100.

[0047] Systolic array 100 can perform convolution computations in multiple waves. A wave can be defined as streaming input data elements while reusing the same weights in systolic array 100. For example, the respective weights can have been preloaded in each PE in systolic array 100 sequentially or in parallel before starting the wave computation. The partial sums generated by the PEs can correspond to a single wave. As the PEs of systolic array 100 perform the arithmetic operations of the convolution computation, the dynamic power consumed by all the multipliers in the PEs can be significant. This problem can be further exacerbated for systolic arrays that include a large number of PEs (e.g., thousands). Examples of the arithmetic operations performed by the PEs will be further discussed with reference to Figure 2A Figure 2B Figure 2C and Figure 2D

[0048] Figure 2A ​​​PE 200 for neural network computation is shown in accordance with certain examples of the disclosed technology. Figure 10A Additional details of the multiply-accumulate data path of Figure 2A are shown. PE 200 can be a portion of a systolic array similar to systolic array 100 in Figure 1 Some examples can be described with reference to neural networks, however, it should be understood that certain examples can be used for other applications, such as pattern recognition, image processing, audio processing, video processing, etc., without departing from the scope of the inventive technology.

[0049] With respect to Figure 2A and Figure 10A , PE 200 can include data element load generator 202, input data element register 204, weight register 206, shared multiplier 208, shared adder 210, skip computation generator 212, first multiplexer 213, skip computation register 214, second multiplexer 215, selector circuit 216, input portion and register 218, cached weight register 220, operation decoder 256, and data type register 262. Multiplier 208, products 250 and 251, multiplexer 213, first multiplexer product 255, shared adder 210, and 237 and 238, and multiplexer 215 form exemplary multiply-accumulate data path 209. Register 214 is a portion of skip computation storage data path 207 between skip computation generator 212 and selector 216. According to some examples, PE 200 can be configured to receive input data elements 222, weights 224, zero data element indicator 226, zero weight indicator 228, operation code 230, weight load 232, input portion and 234, and data type input signal 253 to perform a convolution computation.

[0050] PE 200 can be configured to receive input data elements 222 via a first port. Input data elements 222 can correspond to an input data set, or any array of input data elements. PE 200 can receive one input data element from the input data set at a time during a uniform period of time, or systolic interval. For example, the systolic interval can correspond to a clock cycle. The input data set can be similar to an input feature map that includes input feature map elements. For example, the input data set can correspond to an input image, an audio clip, a video clip, a portion of text, or any other data that can be provided for data processing to identify a particular pattern or object. In some cases, the input data set can correspond to an intermediate output data set that has undergone an activation function, such as ReLu or Sigmoid, as described with reference to FIG. 1. Figure 1Each input data element 222 can be formatted as an integer data type, a floating point data type, or any suitable data type. Each input data element 222 can arrive on a bus wide enough to accommodate the largest supported data type, while setting the appropriate number of bits for each data type (e.g., 8 or 9 bits for an integer, 16 bits for FP16 or BF16). The input data elements can be stored in the input data element registers 204 for a period of time.

[0051] The PE 200 can be configured to receive weights 224 via a second port. In some implementations, the weights 224 can belong to a set of weight values corresponding to a convolution filter. The weights 224 can be preloaded in the PE 200 prior to receiving the input data elements 222. In some examples, the PE 200 can receive one weight value at a time from the set of weight values over a uniform period of time to preload each PE in a given row with a respective weight value. The PE can pass the weight value to the next PE in the respective row until each PE in the given row has been preloaded. Each PE can cache the respective weight value for use in computation with the input data elements. Each weight 224 can be an integer data type, a floating point data type, or any suitable data type. Each weight 224 can include 8 bits, 9 bits, 16 bits, or any suitable number of bits. The weights can be stored in the cached weight registers 220 for a period of time.

[0052] The PE 200 can be configured to receive an input partial sum 234 for a current operation via a third port. The input partial sum can be stored in the register 218 and used in the addition operation of the adder 210.

[0053] The PE 200 can be configured to receive a zero data element indicator 226 for the current operation via the fourth port. The zero data element indicator 226 can include a single bit or multiple bits. The zero data element indicator 226 can be used to indicate whether the input data element 222 associated with the zero data element indicator 226 is zero. For example, a value of “1” for the zero data element indicator 226 can indicate that the input data element 222 associated with the zero data element indicator 226 is zero, and a value of “0” for the zero data element indicator 226 can indicate that the input data element 222 associated with the zero data element indicator 226 is not zero. The “0” can correspond to a logical zero or a logical low, and the “1” can correspond to a logical one or a logical high. For example, in some implementations, a logical zero can be represented by a first range of voltage levels (e.g., 0 to 2 volts), and a logical one can be represented by a second range of voltage levels (e.g., 3 to 5 volts). It should be understood that other implementations of representing “0” values and “1” values are possible without departing from the scope of the disclosed technology. The zero data element indicator 226 can be generated by circuitry external to the PE 200 and passed to all PEs in the same row in a uniform time period.

[0054] The PE 200 can be configured to receive a data type input signal 253 via the fifth port for storage in a data type register 262 for use as the data type control signal 235. Alternatively, the data type control signal 235 can be decoded from the opcode 230, the input data element 222, the stored input data element 244, the weight 224, the cached weight 246, the stored weight value 248, the input partial sum and 234, or the stored input partial sum and 236. When decoding the data type control signal 235 from these sources, the register 262 can be implemented or omitted so that the data type control signal 235 correctly reflects the data type of the current operation being performed. The data type control signal 235 can indicate the data type of one or more of the weight 224, the input data element 222, and the input partial sum and 234. The PE 200 can provide the data type control signal 235 to the first multiplexer 213 and the second multiplexer 215 to control the selection of the first multiplexer 213 and the second multiplexer 215.

[0055] The PE 200 can be configured to receive a zero weight indicator 228 via the sixth port. The zero weight indicator 228 can include a single bit or multiple bits. The zero weight indicator 228 can be used to indicate whether the weight 224 associated with the zero weight indicator 228 is zero. For example, a value of “1” for the zero weight indicator 228 can indicate that the weight 224 is zero, and a value of “0” for the zero weight indicator 228 can indicate that the weight 224 is not zero. The zero weight indicator 228 can be generated by circuitry external to the PE 200 and passed to all PEs in the same row in sequence with the weight 224.

[0056] The weight load signal 232 can be used to load the weight 224 into the cached weight register 220 to provide the cached weight 246. The weight load signal 232 can be asserted to cache the weight 224 of the PE 200 in the cached weight register 220 before the input data element 222 is fed into the array. The weight load 232 can be asserted for each PE for a particular time period as the weights are shifted into the array to preload each PE with a respective weight value.

[0057] The operation decoder 256 can be configured to decode the opcode 230 to determine the operation performed by the PE 200 for different instructions represented by different opcode values. In some examples, a first opcode value can correspond to an instruction to shift a weight from one PE to another PE in the systolic array. A second opcode value can correspond to an instruction to start an arithmetic computation by the PE. For example, once the weights have been preloaded into the systolic array, an input data element can be read from memory and an arithmetic computation can be performed as the input data element passes through the array. The second opcode or a separate opcode can also include a selected data type that can be decoded to generate the data type control signal 235. A third opcode value can correspond to an instruction to perform a NOP. The NOP can be used to separate two systolic array instructions or when there is no input data element to read from memory. For example, the NOP can be used to separate an instruction to shift a weight and an instruction to start an arithmetic computation. For example, for a 4x4 array, it can take up to 15 cycles to shift a weight into all of the PEs in the array before starting an arithmetic computation, so 15 NOP cycles can be needed.

[0058] The operation decoder 256 can be configured to decode the operation code 230 to generate the NOP 258 and a start computation signal 260. The operation decoder 256 can be configured to provide the start computation signal 260 to the weight register 206 connected to the shared multiplier 208 and the shared adder 210. The operation decoder 256 can also be configured to provide the start computation signal 260 to the shared multiplier 208. The operation code 230 can include any suitable number of bits, such as two bits, four bits, etc. In some implementations, instead of receiving the data type signal 253, each PE can include an operation decoder 256 configured to decode the operation code 230 to generate the data type control signal 235 for providing to components in the multiply-accumulate data path 209.

[0059] In some implementations, the input data elements 222, the weights 224, the operation code 230, the zero data element indicator 226, the zero weight indicator 228, and the data type input signal 253 can belong to the row input bus 102, as discussed with reference to Figure 1 In some implementations, a splitter (not shown) can be used in the PE 200 to split the row input bus 102 into different internal buses to carry the input data elements 222, the weights 224, the operation code 230, the zero data element indicator 226, the zero weight indicator 228, and the data type input signal 253 within the PE 200.

[0060] The data element load generator 202 can be configured to generate a data load signal 242, which can be used to allow the input data element register 204 to skip storage of the input data element 222 under certain conditions. In some examples, the input data element 222 can be loaded into the input data element register 204 when the data load signal 242 is asserted based on the zero data element indicator 226 and the NOP 258. The data load signal 242 can be asserted when the zero data element indicator 226 corresponding to the input data element 222 is “0” and the operation code 230 does not indicate a NOP (e.g., the NOP 258 is “0”). The data load signal 242 can be de-asserted when the zero data element indicator 226 corresponding to the input data element 222 or the NOP 258 is “1”. The data element load generator 202 can be implemented using OR, NOR, NAND, or any suitable circuit.

[0061] Input data element register 204 can be configured to store input data element 222 or skip storage of input data element 222 based on data load signal 242 of the current operation to provide stored input data element 244. In some implementations, input data element register 204 can store the Din input if the load input is “1” and can save the previous value if the load input is “0.” For example, if data load signal 242 is “1,” input data element register 204 can store the new value of input data element 222 and if data load signal 242 is “0,” input data element register 204 can skip storage of the new value of input data element 222. Thus, in some cases, input data element register 204 can only store non-zero values of input data element 222. According to certain examples, skipping storage of the new value by input data element register 204 can result in no switching of stored input data element 244 and saving the previous value of stored input data element 244.

[0062] Weight register 206 can be configured to store cached weight 246 to provide stored weight value 248 based on start compute signal 260. In some implementations, weight register 206 can store the Din input if the load input is “1” and can save the previous value if the load input is “0.” For example, if start compute signal 260 is asserted (e.g., start compute signal 260 is “1”), cached weight 246 can be loaded into weight register 206, otherwise weight register 206 can save the previous value. Thus, the weight 224 previously loaded into cached weight register 220 using weight load 232 can be shifted into weight register 206 at the start of the arithmetic computation. In some examples, stored weight value 248, once loaded at the start of the arithmetic computation, remains unchanged as input data elements are fed into PE 200 one element at a time for computation by the systolic array corresponding to one or more waves.

[0063] The shared multiplier 208 can be configured to perform a multiplication operation between the stored input data element 244 and the stored weight value 248. The shared multiplier 208 can alternatively be configured to perform two or more multiplication operations between the stored input data element 244 and the stored weight value 248. The shared multiplier 208 can be configured to generate at least a first product 250 and a second product 251 based at least in part on the two or more multiplication operations. In some implementations, the first product 250 can be an 8, 9, or 16-bit integer product, while the second product 251 can be a 16-bit floating point product. In various implementations, the shared multiplier 208 can be configured to generate two or more products that are 8-bit, 9-bit, 16-bit, 32-bit, or any other number of bits.

[0064] As shown and illustrated in additional detail in Figure 2A Figure 10A The shared multiplier 208 includes an integer multiplier sub-circuit 1002 and a floating point multiplier sub-circuit 1004 that share a shared portion 1006. The floating point multiplier sub-circuit 1004 includes a multiplier exponent logic 1008.

[0065] The integer multiplier sub-circuit 1002 is configured to multiply the integer input data element 244 by the integer weight 246 to generate the integer product 250. The integer multiplication can include computing and adding partial products to determine the integer product 250. A non-shared portion of the integer multiplier sub-circuit 1002 can be selectively disabled when the data type control signal 235 indicates that a non-integer data type is selected. In various implementations, the integer product 250 can be an 8-bit integer product, a 9-bit integer product, a 16-bit integer product, or any other type of integer product, signed or unsigned. In some implementations, the integer multiplier sub-circuit 1002 can perform parallel multiplication on shorter integers, such as performing two parallel signed 9-bit integer multiplications. The multiplication can be performed using a variety of logical multiplication techniques.

[0066] ​The floating-point multiplier sub-circuit 1004 is configured to multiply the floating-point input data element 244 by the floating-point weight 246 to generate a floating-point product 251. The floating-point multiplication can include computing and adding partial products to determine the significand of the floating-point product 251. In various implementations, the floating-point product 251 can have an FP16 or BF16 data type, signed or unsigned. In various implementations, the floating-point multiplier sub-circuit 1004 can handle other types of non-integer data types. The floating-point multiplication can also include multiplier exponent logic 1008 to compute an exponent of the floating-point product. At least a portion of the floating-point multiplier sub-circuit 1004, such as the multiplier exponent logic 1008, can be separate from the integer multiplier sub-circuit 1002 because the integer data type does not include an exponent bit. Non-shared portions of the floating-point multiplier sub-circuit 1004 can be selectively disabled when the data type control signal 235 indicates that an integer data type is selected.

[0067] The shared portion of the multiplier 1006 contributes to both floating-point multiplication and integer multiplication. Thus, the shared multiplier 208 can be configured to support multiple data types including integers and non-integers, rather than having multiple separate multipliers each configured to support a different data type. For example, the shared portion 1006 can contribute to both integer multiplication and floating-point multiplication by computing and adding partial products, which can be performed identically or similarly for the significands of integers and floating-point numbers.

[0068] As another example, the shared portion 1006 can include an adder that adds partial products when performing integer multiplication and also adds exponents when performing floating-point multiplication. In another example, a first sub-circuit of the shared multiplier 208 can be configured to perform at least a portion of integer multiplication and floating-point multiplication. In some examples, the shared portion 1006 can perform all or a portion of the functionality of the integer multiplier sub-circuit 1002.

[0069] The multiplier exponent logic 1008 can be configured to compute an exponent value for an exponent field of the floating-point product 251. For example, the multiplier exponent logic 1008 can be configured to add or subtract an exponent from the input data element 244 and the weight 246 when the data type 235 indicates a floating-point data type.

[0070] In various implementations, the shared portion 1006 of the multiplier can perform two, three, any combination thereof, or all of the following: a single 17-bit integer multiplication, two or more parallel 9-bit integer multiplications, a single BF16 multiplication, and a single FP16 multiplication. In some implementations, the shared multiplier 208 can be configured to perform one or more multiplication operations on at least two, three, any combination thereof, or all of the following data types: 8-bit data type, 9-bit data type, 16-bit data type, 32-bit data type, or other data types.

[0071] The shared multiplier 208 can provide at least the first product 250 and the second product 251 to the first multiplexer 213. The first multiplexer 213 can be configured to select between at least the first product 250 and the second product 251 based at least in part on the data type control signal 235 to provide a multiplexer product 255.

[0072] The data type control signal 235 indicates the data type of the input data element 244 provided to the shared multiplier 208 and the data type of the stored weight value 248. In some examples, such as Figure 2A As shown, the data type input signal 253 is received by the PE through a port, cached in a register 262, and read from the register 262 as the data type control signal 235. In other examples, the PE can be configured to detect the data type based on one or more of the input data element 244, the stored weight value 248, the input portion and 236, or the opcode. The data type control signal 235 can be provided to the first multiplexer 213 and the second multiplexer 215 to select an output formatted in the selected data type. The data type control signal 235 can optionally be provided to one or both of the shared multiplier 208 and the shared adder 210 to disable circuitry specific to computing unselected data types, thereby reducing power consumption. For example, when the data type control signal 235 indicates an integer data type, non-shared multiplier exponent logic in the floating point multiplier sub-circuit 1004 can be deactivated. As another example, when the data type control signal 235 indicates an integer data type, non-shared adder exponent logic in the floating point adder sub-circuit 1014 can also be deactivated. In some implementations, the shared multiplier 208 can be configured to output either the first product 250 or the second product 251 based at least in part on the first data type control signal 253.

[0073] As Figure 2A shown and shown in additional detail in Figure 10A The shared adder 210 includes an integer adder sub-circuit 1012 and a floating point adder sub-circuit 1014 that share a shared portion 1016. The floating point adder sub-circuit 1014 also includes adder exponent logic 1018.

[0074] The shared adder 210 can be configured to perform an integer addition of the multiplexer product 255 and the stored input partial sum 236 to provide an integer partial sum 237. The shared adder 210 can be configured to perform a floating point addition of the multiplexer product 255 and the input partial sum 236 to provide a floating point partial sum 238. In various examples, the shared adder 210 can be configured to perform integer addition, floating point addition, or other non-integer addition. The shared adder 210 can be configured as a shared circuit to perform addition operations on different data types. Although the shared adder 210 is shown as a floating point adder, the shared adder can be any type of non-integer adder.

[0075] The integer adder sub-circuit 1012 is configured to add the multiplexer product 255 to the stored input partial sum 236 to generate an integer partial sum 237. The integer addition can include performing binary addition to calculate the integer partial sum 237. The non-shared portion of the integer adder sub-circuit 1012 can be selectively disabled when the data type control signal 235 indicates that a non-integer data type is selected. In various implementations, the integer partial sum 237 can be an 8-bit integer product, a 9-bit integer product, a 16-bit integer product, or any other type of integer product, signed or unsigned.

[0076] In some implementations, the integer adder sub-circuit 1012 can perform parallel addition on integers, such as performing two parallel signed 18-bit integer additions. In such implementations, the shared multiplier 208 can perform two parallel 9-bit multiplications and generate two parallel 18-bit values as the first product 250. The integer adder sub-circuit 1012 can perform parallel addition on integers by adding a first portion of the multiplexer product 255 to a first portion of the input partial sum 236 and by adding a second portion of the multiplexer product 255 to a second portion of the input partial sum 236. For example, the multiplexer product 255 can be composed of two parallel 18-bit values, and the input partial sum 236 can be a 50-bit integer. The integer adder sub-circuit 1012 can perform a first addition on the first parallel 18-bit value and bits 0 to 24 of the input partial sum 236. The integer adder sub-circuit 1012 can perform a second addition on the second parallel 18-bit value and bits 25 to 49 of the input partial sum 236. The first and second additions can be performed in parallel. In some implementations, the first and second additions can be different and can not perform an arithmetic carry from the first portion of the input partial sum 236 to the second portion of the input partial sum 236. The first and second additions can be performed using a variety of logic addition techniques.

[0077] The floating-point adder sub-circuit 1014 is configured to add the multiplexer product 255 to the stored input partial sum 236 to generate a floating-point partial sum 238. The floating-point addition can include shifting the significand to align with the exponent and performing a binary addition on the aligned significands to determine the floating-point partial sum 238. In various implementations, the floating-point partial sum 238 can have an FP16 or BF16 data type, signed or unsigned. In various implementations, the floating-point adder sub-circuit 1014 can handle other types of non-integer data types. The floating-point multiplication can also include adder exponent logic 1018 for calculating the exponent of the floating-point partial sum and for shifting the significand to align. At least a portion of the floating-point adder sub-circuit 1014, such as the adder exponent logic 1018, can be separate from the integer adder sub-circuit 1012 because the integer data type does not include an exponent bit. The non-shared portion of the floating-point adder sub-circuit 1014 can be selectively disabled when the data type control signal 235 indicates that the integer data type is selected.

[0078] The shared portion 1016 of the adder facilitates both floating-point addition and integer addition. Thus, the shared adder 210 can be configured to support multiple data types including integers and non-integers, rather than having multiple separate adders each configured to support a different data type. For example, the shared portion 1016 can facilitate both integer addition and floating-point addition by performing, for example, a binary addition that can be performed identically or similarly for the significands of integers and floating-point numbers.

[0079] The adder exponent logic 1018 can be configured to calculate an exponent value for the exponent field of the floating-point partial sum 238. For example, the multiplier exponent logic 1008 can be configured to add or subtract exponents to normalize the floating-point partial sum 238.

[0080] In various implementations, the shared portion 1016 of the adder 210 can perform or facilitate two, three, any combination, or all of the following: a single 17-bit integer multiplication, two or more parallel 9-bit integer multiplications, a single BF16 addition, and a single FP16 addition. In some implementations, the shared portion 1016 of the adder 210 can perform or facilitate all of the following: a single 17-bit integer addition, two or more parallel 9-bit integer additions, a single BF16 addition, and a single FP16 addition. In some implementations, the shared adder 210 can be configured to perform one or more addition operations on at least two of the following: an 8-bit data type, a 9-bit data type, a 16-bit data type, a 32-bit data type, or other data types.

[0081] The second multiplexer 215 receives the sum generated by the shared adder, including at least the integer portion sum 237 and the floating point portion sum 238. The second multiplexer 215 selects the partial sum 239 from the inputs including the integer portion sum 237 and the floating point portion sum 238 based on the data type control signal 235.

[0082] With continued reference to Figure 2A , the selector circuit 216 can be configured to receive the partial sum 239, the input partial sum 236, and the stored skip computation indicator 254. The selector circuit 216 can be configured to select the partial sum 239 or the input partial sum 236 based on the stored skip computation indicator 254 to provide as the output partial sum 240 via the seventh port. In some implementations, the selector 216 can be combined with the multiplexer 215 to select the integer portion sum 237, the floating point portion sum 238, or the stored input partial sum 236 to provide the output partial sum 240 via the seventh port based on the stored skip computation indicator 254 and the data type control signal 235. According to some embodiments, when the value of the input data element 222 or the weight 224 of the current operation is zero, or the NOP 258 is asserted, the floating point portion sum 238 or the integer portion sum 237 can not provide the correct result of the current operation because the integer product 250 and the floating point product 251 can hold the value of the previous operation. In such cases, the stored skip computation indicator 254 can bypass the floating point portion sum 238 and the integer portion sum 237 and select the stored input partial sum 236 to provide the output partial sum 240. For example, when the stored skip computation indicator 254 provides a skip computation signal of “1,” the input partial sum 236 can be selected by the selector circuit 216 as the output partial sum 240 of the systolic cycle, and when the stored skip computation indicator 254 provides a skip computation signal of “0,” the integer portion sum 237 or the floating point portion sum 238 can be selected by the selector circuit 216 as the output partial sum 240 of the systolic cycle.

[0083] Further, in such cases, the stored skip computation indicator 254 can prevent one or both of the shared multiplier 208 or the shared adder 210 from contributing to the arithmetic computation of the systolic array. For example, the stored skip computation indicator 254 can be used to keep the inputs to the shared multiplier 208 or the shared adder 210 from changing, thereby improving power efficiency by preventing further computation.

[0084] Figure 2B An exemplary PE is shown that is similar to the exemplary PE shown in Figure 2A , except that in Figure 2B , a separate integer adder 211 A and a separate floating point adder 21 IB replace Figure 2AThe shared adder 210 and the first multiplexer 213, and in Figure 2B The source of data type control signal 235 is not shown in the diagram. Figure 10B It shows Figure 2B Further details.

[0085] The separate integer adder 211A receives the first integer product 250. The separate integer adder 211A can generate the integer part sum 237 by adding the integer product 250 to the stored input part sum 236, for example, using a binary adder.

[0086] A separate floating-point adder 211B receives a floating-point product 251. The separate floating-point adder 211B can generate a floating-point partial sum 238 by adding the floating-point product 251 to a stored input partial sum 236. The separate floating-point adder 211B may include adder valid number logic, such as a binary adder, to compute the valid numbers of the floating-point partial sum 238. The floating-point adder 211B may also include adder exponent logic to compute the exponent of the floating-point partial sum 238.

[0087] In other implementations, the individual integer adder 211A and the individual floating-point adder 211B can be configured to perform addition and generate other different data types. For efficiency, the data type control signal 235 can optionally be used to disable one of the individual integer adder 211A or the individual floating-point adder 211B that does not correspond to the selected data type.

[0088] In some implementations, opcode 230 can be configured to disable one or both of the separate integer adder 211A and the separate floating-point adder 211B to improve power efficiency. For example, opcode 230 can be used to disable the separate floating-point adder 211B when the value of opcode 230 indicates an integer data type.

[0089] Figure 2C It shows the relationship with Figure 2A The PE shown is similar to the PE, except that: delay register 263 is coupled in the multiplication-accumulation data path 209 between the first multiplexer 213 and the shared adder 210, delay register 221 is added to the skip calculation storage data path 207, and data type control signal 235 is stored in data type delay register 219 to control the second multiplexer 215 separately. Figure 10C It shows Figure 2C Further details.

[0090] The delay register 263 receives the first multiplexer product 255 from the first multiplexer 213 during the initial pulsation period and provides the stored multiplexer result 257 to the shared adder 210 during subsequent pulsation periods. The delay register 263 can temporarily store the first multiplexer product 255 generated by the first multiplexer 213 during a first pulsation interval of the multiply-delay-accumulate operation for delayed use by the shared adder 210 during a subsequent pulsation interval of the multiply-delay-accumulate operation. For example, during a first pulsation interval, the shared multiplier 208 can generate a first value of the integer product 250 and a first value of the floating point product 251, and the first multiplexer 213 can select one of the product values to store in the delay register 263. During a second pulsation interval after the first pulsation interval, the delay register 263 can provide the first stored multiplexer result 257 to the shared adder 210 and store a second value selected from the second integer product 250 and the second floating point product 251 computed by the shared multiplier 208 as the first multiplexer product 255. During a third pulsation interval after the second pulsation interval, the delay register 263 can provide the second value to the shared adder 210 and store a third value of the first multiplexer product 255 selected by the first multiplexer 213 from the third integer product 250 and the third floating point product 251 computed by the shared multiplier 208. Thus, the delay register 263 functions to separate the multiplier 208 and the adder 210 into different stages of the pipeline, allowing the speed of the pulsation interval to be increased to an interval shorter than the combined processing time of the multiplier 208 and the adder 210.

[0091] The delay register 263 can be enabled during the operation, non-zero condition. If a zero is provided as an input data element or as a weight prior to the first pulsation interval, the delay register 263 can skip the write of the first multiplexer product 255 received from the output of the first multiplexer 213 during the first pulsation interval. During the non-skip condition, the delay register 263 can be activated using the reverse skip computation indicator 217. The reverse skip computation indicator 217 can disable or skip the write to the delay register during the skip condition.

[0092] The delay register 221 can be included in the skip computation storage data path 207 with the delay register 214 such that the stored skip computation indicator 254 is provided to the selector 216 during a pulsation interval synchronized with the corresponding data propagated through the multiply-accumulate data path 209.

[0093] The data type control signal 235 can be stored in the data type delay register 219 for provision to the adder pipeline stage as a delayed data type control signal 241 at the appropriate time. During the first systolic interval, the data type control signal can be stored in the data type delay register 219 for provision to the first multiplexer 213 to select between the integer product 250 and the floating point product 251 generated during the first multiplication operation, and optionally to the shared multiplier 208 to disable circuitry for multiplying the unselected data type. During the second systolic interval, the delayed data type control signal 241 can be provided to the second multiplexer 215 to select between the integer portion sum 237 and the floating point portion sum 238, while a new data type control signal 235 can be provided to the first multiplexer 213 to select the appropriate product generated by the second multiplication operation. The delayed data type control signal 241 can optionally be provided to the shared adder 210 during the second systolic interval to disable circuitry specific to adding the unselected data type.

[0094] Figure 2D A PE similar to that shown in Figure 2B differences. The output of the shared multiplier 208 is stored in a first delay register 263 and a second delay register 273, the data type control signal 235 is delayed by a data type delay register 262, and the delay register 221 is included in the skip computation storage data path 207. Figure 10D Further details of Figure 2D are shown.

[0095] The first delay register 263 is connected between the shared multiplier 208 and the separate integer adder 211 A, while the second delay register 273 is connected between the shared multiplier 208 and the separate floating point adder 21 IB. The first delay register 263 receives the integer product 250 from the shared multiplier 208 during the initial systolic period and provides the first stored integer product 257 to the separate integer adder 211 A during subsequent systolic periods. The second delay register 273 receives the floating point product 251 from the shared multiplier 208 during the initial systolic period and provides the first stored floating point product 259 to the separate floating point adder 21 IB during subsequent systolic periods.

[0096] The first delay register 263 and the second delay register 273 can temporarily store the integer product 250 and the FP product 251 generated by the shared multiplier 208 during a first pulse interval of a multiply-delay-accumulate operation, respectively, for delayed use by the separate integer adder 211 A and the separate floating point adder 211B during a subsequent pulse interval of the multiply-delay-accumulate operation. For example, during a first pulse interval, the first delay register 263 can store a first value of the integer product 250 generated by the shared multiplier 208, and the second delay register 273 can store a first value of the FP product 251 generated by the shared multiplier 208. During a second pulse interval after the first pulse interval, the first delay register 263 can provide the stored first value of the stored integer product 257 to the separate integer adder 211A and store a second value of the integer product 250 generated by the shared multiplier 208. During the second pulse interval, the second delay register 273 can provide the stored first value of the FP product 251 to the separate floating point adder 211B and store a second value of the FP product 251 generated by the shared multiplier 208.

[0097] The first delay register 263 and the second delay register 273 can be enabled during an operation, non-zero condition. If zero is provided as an input data element or as a weight prior to a first pulse interval, the first delay register 263 and the second delay register 273 can skip a write of the first product 250 and the second product 251 received from the output of the shared multiplier 208 during the first pulse interval. During a non-skip condition, the first delay register 263 and the second delay register 273 can be activated using a reverse skip computation indicator 217. The reverse skip computation indicator 217 can disable or skip a write to the first delay register 263 and the second delay register 273 during a skip condition.

[0098] Figure 3 A device 300 is shown that includes a signal modifier and zero detector circuit for input data elements and weights entering a systolic array of neural network computations, in accordance with certain examples of the disclosed technology.

[0099] The device 300 can include a two-dimensional systolic array 302 that includes PEs arranged in rows and columns. The systolic array 302 can be similar to the systolic array 102 Figure 1The systolic array 302 can include a first row of PEs 00, 01, 02,..., 0y, a second row of PEs 10, 11, 12,..., ly, a third row of PEs 20, 21, 22,..., 2y, and an Xth row of PEs x0, xl, x2,..., xy. x and y can include positive integers, such as 32, 64, 128, or any suitable number. Each PE of the systolic array 302 can be similar to the PE 200 and include means for performing arithmetic calculations using power efficient methods, as discussed with reference to Figure 2A , Figure 2B , Figure 2C and Figure 2D .

[0100] In certain examples, a first (e.g., leftmost) PE in each row of the systolic array 302 can be coupled to a respective zero input data detector circuit to detect zero values on input data elements and coupled to a respective zero weight detector circuit to detect zero values on weight values entering the systolic array 302. For example, the PE 00 in the first row can be coupled to a first zero input data detector 306a and a first zero weight detector 308a, the PE 10 in the second row can be coupled to a second zero input data detector 306b and a second zero weight detector 308b, the PE 20 in the third row can be coupled to a third zero input data detector 306c and a third zero weight detector 308c, and the PE x0 in the Xth row can be coupled to an Xth zero input data detector 306x and an Xth zero weight detector 308x. The first zero input data detector 306a, the second zero input data detector 306b, the third zero input data detector 306c,..., and the Xth zero input data detector 306x can be configured to detect zero values on respective input data elements in an input data set 0, an input data set 1, an input data set 2,..., and an input data set x, respectively. Similarly, the first zero weight detector 308a, the second zero weight detector 308b, the third zero weight detector 308c,..., and the Xth zero weight detector 308x can be configured to detect zero values on respective weight values in a filter 0, a filter 1, a filter 2,..., and a filter x, respectively.

[0101] In certain examples, the first (e.g., leftmost) PE in each row of systolic array 302 can be coupled to a respective signal modifier that quantizes, dequantizes, and / or partially dequantizes input data elements, and to a respective zero weight detector circuit that quantizes, dequantizes, and / or partially dequantizes weight values entering systolic array 302. For example, PE 00 in the first row can be coupled to signal modifiers 307a and 309a, PE 10 in the second row can be coupled to signal modifiers 307b and 309b, PE 20 in the third row can be coupled to signal modifiers 307c and 309c, and PE x0 in the Xth row can be coupled to Xth signal modifiers 307x and 309x. Signal modifiers 307a,..., 307x can be configured to quantize, dequantize, and / or partially dequantize respective input data elements in input data set 0, input data set 1, input data set 2,..., and input data set x, respectively. Similarly, signal modifiers 309a,..., 309x can be configured to quantize, dequantize, and / or partially dequantize respective weight values in the respective row.

[0102] Each of input data set 0, input data set 1, input data set 2,..., and input data set x can belong to an image, text, a video clip, an audio clip, or other type of data set that can need to be processed by a neural network processor for convolutional computations. Native input data can be in an overly precise format or in an asymmetrically quantized format. If native input data is in an overly precise format, the native data input data can be quantized to a shorter data format for more efficient computation. If input data is asymmetrically quantized, the input data can be at least partially dequantized by adding extra bits to the data length and shifting the input data into a symmetric representation, allowing processing elements to avoid including complex circuitry to handle non-zero representations of zeros. For example, 8-bit integers can be partially dequantized to 9-bit integers without losing accuracy. Other lengths of data types can similarly be increased in length and partially dequantized without losing accuracy.

[0103] Each of input data set 0, input data set 1, input data set 2,..., and input data set x can be provided to each of signal modifier 307a, signal modifier 307b, signal modifier 307c,..., and signal modifier 307x. Each of the signal modifiers can be configured to quantize, dequantize, and / or partially dequantize input data set 0, input data set 1, input data set 2,..., and input data set x. In some cases, each of the signal modifiers can be configured to partially or wholly shift, quantize, or dequantize some combination thereof. In some cases, each of the signal modifiers can be configured to partially dequantize each of the input data elements of each row of processing elements. For example, each of the signal modifiers can be configured to shift each of the input data elements and add extra bits to the representation of each of the input data elements.

[0104] In some cases, input data set 0, input data set 1, input data set 2,..., and input data set x can be associated with output data set 0, output data set 1, output data set 2,..., output data set y generated by an intermediate layer of a convolution operation. For example, output data set 0, output data set 1, output data set 2,..., output data set y can be fed back to systolic array 302 through an activation function and as input data set 0, input data set 1, input data set 2,..., and input data set x. Filter 0, filter 1, filter 2,..., and filter x can include different sets of weight values to convolve with input data set 0, input data set 1, input data set 2,..., and input data set x. The weight values in filter 0, filter 1, filter 2,..., and filter x can be predetermined using supervised learning, unsupervised learning, or any suitable method of determining convolution filters.

[0105] Each zero-input data detector of a respective row can be configured to detect whether an input data element from an input data set entering the respective row is a "0" and generate a corresponding zero-input data indicator for the input data element. The corresponding zero data element indicator can be passed into the first PE of the respective row along with the input data element. For example, PE 00 can be the first PE of the first row in systolic array 302. PE 00 can be configured to receive an input data element from input data set 0 before other PEs in the first row (e.g., PE 01, PE 02,..., PE 0y). In some examples, the input data elements from input data set 0 can be sequentially fed to PE 00 one at a time over a uniform period of time. First zero-input data detector 306a can be configured to generate a zero data element indicator 226 for each input data element from input data set 0 in each of the uniform period of time (e.g., clock cycles). The zero data element indicator 226 corresponding to each input data element can be sequentially fed to PE 00 along with each input data element over the uniform period of time. PE 00 can store the input data element 222 or skip its storage based on the value of the respective data load signal 242. In some implementations, first zero-input data detector 306a can include a comparator to compare an incoming input data element to zero to assert (e.g., set to "1") or de-assert (e.g., set to "0") the zero data element indicator 226 based on the value of the incoming input data element. For example, the comparator can be implemented using an OR, XOR, NAND, or any suitable circuit.

[0106] Each of filter 0, filter 1, filter 2,..., and filter x can be provided to each of signal modifier 309a, signal modifier 309b, signal modifier 309c,..., and signal modifier 309x. Each of the signal modifiers can be configured to quantize, de-quantize, or partially de-quantize filter 0, filter 1, filter 2,..., and filter x. In some cases, each of the signal modifiers can be configured to some combination of quantizing, de-quantizing, or partially de-quantizing. In some cases, each of the signal modifiers can be configured to partially de-quantize each of the filters of each row processing element. For example, each of the signal modifiers can be configured to shift each of the filters and add extra bits to the representation of each of the filters.

[0107] Each zero weight detector of a respective row can be configured to detect whether a weight value from the set of weight values entering the respective row is zero and generate a corresponding zero weight indicator for the weight value. For example, the first zero weight detector 308a can be configured to detect whether a weight value (e.g., weight 224) from filter 0 includes a zero value and generate a zero weight indicator 228 for the weight. In some implementations, the first zero weight detector 308a can include a comparator to compare the weight value to zero to assert (e.g., set to “1”) or de-assert (e.g., set to “0”) the zero weight indicator 228. The comparator can be implemented using an OR, XOR, NAND, or any suitable circuit, for example. In one example, the weight values from filter 0 can be sequentially fed into PE 00 one at a time over a uniform time period for preloading the respective weight values in PE 0y to PE 00 before starting the arithmetic computation. The first zero weight detector 308a can generate a corresponding zero weight indicator for each of those weight values, which can be sequentially fed into PE 00 along with the corresponding weight values over the uniform time period. PE 00 can pass the respective weight values and corresponding zero weight indicators sequentially to the next adjacent PE until all of the PEs in the first row are preloaded with the respective weight values and corresponding zero weight indicators. The respective weight values and corresponding zero weight indicators can be cached in each PE before feeding the respective input data elements into each row in systolic array 302.

[0108] The second zero input data detector 306b, the third zero input data detector 306c,..., and the Xth zero input data detector 306x can be similar to the first zero input data detector 306a and can generate respective zero data element indicators similar to zero data element indicator 226 for sequential provision to PE 10, PE 20,..., and PE x0 over a uniform time period for power optimization. The respective zero data element indicators generated for each row can be received by the respective first PE in each row via the respective row input bus 102 and propagated by the first PE to all of the PEs in a given row sequentially over the uniform time period. The second zero weight detector 308b, the third zero weight detector 308c,..., and the Xth zero weight detector 308x can be similar to the first zero weight detector 308a and can generate respective zero weight indicators similar to zero weight indicator 228 for sequential provision to PE 10, PE 20,..., and PE x0 for preloading to each PE in the respective row along with the respective weight values before starting the arithmetic computation.

[0109] In some examples, the zero input data detectors 306a-306x and zero weight detectors 308a-308x can be implemented as separate entities outside the systolic array 302. For example, the zero input data detectors 306a-306x and zero weight detectors 308a-308x can be part of the circuit 304. In other examples, the circuit 304 and the systolic array 302 can be part of a compute engine that can be configured to perform arithmetic computations of convolution operations. Some examples of the disclosed technology can provide reduced gate count and dynamic power consumption by detecting zeros on input data elements and weights into the respective first PE in each row of the systolic array and passing zero indicators to all PEs in the array, as compared to using a respective zero detector within each PE in the systolic array 302.

[0110] While Figure 3 Signal modifiers 307a-307x and 309a-309x are shown before the zero weight detectors 306a-306x and 308a-308x, the order of these elements can be changed in other examples. While Figure 3 Signal modifiers 307a-307x and 309a-309x are shown outside the PEs to enable a more efficient layout, the signal modifiers 307a-307x and 309a-309x can also be contained in each individual PE.

[0111] Note that, for ease of illustration, Figure 3 Only the respective zero data element indicators and zero weight indicators into the first PE in each row of the systolic array 302 are shown, however it should be understood that each PE in the respective row of the systolic array 302 can also receive the respective input data element and the respective weight value along with some control signals (e.g., opcode 230, weight load 232, data type, etc.), which can be propagated from left to right for each row of the systolic array 302. This will be explained further with reference to Figure 4 This is further explained.

[0112] Figure 4 A device 400 showing the propagation of zero indicators, input data elements, and weight values from one PE to another PE according to a first example of the disclosed technology is shown.

[0113] In some examples, as Figure 3 Each PE of the systolic array 302 shown is in addition to the zero input data detectors 306a-306x and zero weight detectors 308a-308x as Figure 2A , Figure 2B , Figure 2C and Figure 2DThe components of PE 200 shown can also include components for caching weights and caching zero indicators prior to passing the weights and zero indicators to neighboring PEs in a given row. For example, in addition to the components of PE 200, each PE can include a zero data element indicator register 402, a zero weight indicator register 404, and an output weight register 406. Figure 4 This is further explained with the example of PE 00 and PE 01 in the middle. PE 00 and PE 01 are part of systolic array 302 as discussed with reference to Figure 3

[0114] Zero data element indicator register 402 can be configured to store a zero data element indicator 226 received by PE 00 to provide a stored zero data element indicator 408. Zero data element indicator 226 can correspond to an input data element 222 received by PE 00. As discussed with reference to Figure 3

[0115] ​​Accordingly, in certain examples, if a zero is detected on an input data element 222 received by PE 00 for a current operation, the zero value of the input data element 222 can not propagate to PEs 01-0y because the stored input data element 244 can hold a value from a previous operation or a default value. However, a stored zero data element indicator 408 corresponding to the zero value of the input data element 222 can propagate to neighboring PEs. PE 01 can receive the stored zero data element indicator 408 as a zero data element indicator 226, store the zero data element indicator, and propagate its stored zero data element indicator 408 to a neighboring PE (e.g., PE 02). PE 01 can also propagate its stored input data element 244 from a previous operation to PE 02 along with the stored zero data element indicator 408. Accordingly, the zero data element indicator 226 can be generated only once by the first zero input data detector 306a and passed from PE 00 to PE 0y sequentially over a uniform time period. If the input data element 222 includes a zero value, the respective stored zero data element indicator 408 in each PE can be used to bypass the respective first product 250 and second product 251 in each PE. Accordingly, if the respective input data element 222 or the respective weight 224 includes a zero value or a NOP is received for the operation, the respective output partial sum 240 in each PE can be the respective input partial sum 234.

[0116] The zero weight indicator register 404 can be configured to store a zero weight indicator 228 received by PE 00 to provide a stored zero weight indicator 412. The zero weight indicator 228 can correspond to a weight 224 received by PE 00. PE 00 can be configured to receive weights 224 for preloading the weights into the systolic array 302 prior to starting an arithmetic computation. For example, in one example, the weights can be fed into PE 00 one weight value at a time sequentially over a uniform time period from filter 0. PE 00 can store the received weight values in the output weight register 406 to provide a stored weight value 410 based on a shift weight signal 414. The stored weight value 410 can be shifted into PE 01 as a weight 224. The shift weight signal 414 can be generated by the operation decoder 256 based on the operation code 230. For example, the operation code 230 can include a certain operation code value to indicate that a weight value is to be shifted from one PE to another PE. PE 01 can receive the stored zero weight indicator 412 as a zero weight indicator 228, store the zero weight indicator, and propagate its stored zero weight indicator 412 to a neighboring PE (e.g., PE 02) over a uniform time period. Accordingly, the zero weight indicator 228 can be generated only once by the first zero weight detector 308a and passed from PE 00 to PE 0y sequentially over a uniform time period along with a corresponding weight value.

[0117] In certain examples, the same weight value can be used by all PEs in a given row for convolving with each input data element of an input data set to optimize memory bandwidth. In some examples, instead of preloading weights in a systolic array, the respective weights can be fed into each row along with the input data elements to perform the arithmetic calculations. Reference will be made to Figure 5 This is further explained.

[0118] The multiply-accumulate data path 209 can include components of the multiply- accumulate data path 209 as shown in Figure 4 , Figure 2A , Figure 2B , Figure 2C , Figure 2D , Figure 10A , Figure 10B , Figure 10C or Figure 10D The multiply-accumulate data path 209 can incorporate any of the circuits or components contained therein in any combination. For example, the multiply-accumulate data path 209 can include a shared multiplier, a shared adder, and / or one or more separate adders. The multiply-accumulate data path 209 can be configured to provide multiple partial sums to a multiplexer and / or one or more delay registers. For example, the multiply-accumulate data path 209 can provide an integer partial sum, a non-integer partial sum, or any combination thereof. The multiply-accumulate data path 209 can be configured to receive a stored weight value 248, a stored input data element 244, a start computation signal 260 (not shown in Figure 4 ) and a data type control signal (not shown in Figure 4 ).

[0119] The skip computation storage data path 207 can also include components in addition to those shown in Figure 4 In some examples, the skip computation storage data path 207 can include additional registers 221 such as described with respect to Figure 2C .

[0120] Figure 5 A device 500 showing a zero detector, propagation of input data elements and weight values from one PE to another PE is shown in accordance with a second example of the disclosed technology.

[0121] In a second example, instead of preloading weights in a systolic array, input data elements 222 can be fed into PEs 00 one weight value at a time from filter 0 sequentially with input data elements 222 over a uniform time period. Input data elements 222 and weights 224 can be cached in their respective registers only if a zero is not detected on both input data elements 222 and weights 224. Thus, if a zero is detected on input data elements 222 or weights 224, the shared multiplier input can not switch, resulting in reduced power consumption. Input data elements 222 and weights 224 can be propagated to neighboring PEs with zero data element indicators 226 and zero weight indicators 228.

[0122] In the second example, skip computation generator 502 can be configured to generate a skip computation indicator 526 using zero data element indicators 226 and zero weight indicators 228. Data registers 504 and weight registers 506 can use skip computation indicator 526 to skip storing a zero value for the current operation on input data elements 222 or weights 224, respectively. In some examples, skip computation generator 502 can perform an OR or NOR operation on zero data element indicators 226 and zero weight indicators 228 to generate skip computation indicator 526. Skip computation indicator 526 can be stored in skip computation register 514 to provide a stored skip computation indicator 540 that can be used by selector 516. Skip computation storage data path 507 can also include components in addition to those shown in FIG. 5B. In some examples, skip computation storage data path 507 can include additional registers 221 such as those described with respect to skip computation storage data path 207 of FIG. 2B. Figure 5 Figure 2C

[0123] Data registers 504 can be configured to store input data elements 222 or skip storage of input data elements 222 based on skip computation indicator 526 for the current operation to provide stored input data elements 528. For example, if skip computation indicator 526 is “0,” data registers 504 can store a new value for input data elements 222 and if skip computation indicator 526 is “1,” data registers 504 can skip storage of a new value for input data elements 222. According to certain examples, data registers 504 skipping storage of a new value can result in not switching stored input data elements 528 and preserving a previous value for stored input data elements 528.

[0124] ​​The weight register 506 can be configured to store the weight 224 or skip storage of the weight 224 based on the skip computation indicator 526 of the current operation to provide a stored weight value 530. For example, if the skip computation indicator 526 is “0,” the weight register 506 can store a new value of the weight 224, and if the skip computation indicator 526 is “1,” the weight register 506 can skip storage of a new value of the weight 224. According to certain examples, the weight register 506 skipping storage of a new value can result in not switching the stored weight value 530 and preserving a previous value of the stored weight value 530.

[0125] The multiplier 508 can be configured to perform a multiplication operation between the stored input data element 528 and the stored weight value 530 to provide a multiplication result 532. In some implementations, the skip computation indicator 526 can be used to skip storing both the input data element 222 and the weight 224 in the data register 504 and the weight register 506, respectively, when the value of the input data element 222 or the weight 224 of the current operation is zero. For example, the zero data element indicator 226 or the zero weight indicator 228 can generate a value of “1” for the skip computation indicator 526, which can prohibit loading of the respective input into the data register 504 and the weight register 506. In this case, the stored input data element 528 and the stored weight value 530 can preserve their values from a previous operation and can not switch. As a result, the multiplication result 532 can not change and can reduce dynamic power consumption. Since the multiplication result 532 can not be accurate for the current operation, the multiplication result 532 can not propagate to other PEs in the array.

[0126] The PE 00 can receive the input partial sum 534, which can be stored in the input partial sum register 512 to provide a stored input partial sum 536. The adder 510 can be configured to perform an addition operation on the multiplication result 532 and the stored input partial sum 536 to provide an additional result 538. In some examples, instead of the multiplier 508 and the adder 510, a fused multiplier-adder can be used to perform the multiplication and addition operations in a single step.

[0127] The multiply-accumulate data path 209 can include as Figure 4 、 Figure 2A 、 Figure 2B 、 Figure 2C or Figure 2D 、 Figure 10A 、 Figure 10B 、 Figure 10C or Figure 10DThe components of the multiply-accumulate data path 209 shown. The multiply- accumulate data path 209 can incorporate any of the circuits or components contained therein in any combination. For example, the multiply-accumulate data path 209 can include a shared multiplier, a shared adder, or both a shared multiplier and a shared adder. The multiply-accumulate data path 209 can include a shared multiplier, one or more separate adders, or both a shared multiplier and one or more separate adders. The multiply-accumulate data path 209 can be configured to provide a plurality of partial sums to the selector. For example, the multiply-accumulate data path 209 can provide an integer partial sum, a non-integer partial sum, or any combination thereof. The multiply-accumulate data path 209 can be configured to receive a stored weight value 248, a stored input data element 244, a start computation signal 260 Figure 4 (not shown in FIG. 6). The multiply-accumulate data path 209 can include one or more multiplexers or selectors. The multiply-accumulate data path 209 can include one or more delay registers.

[0128] The selector 516 can be configured to select the adder result 538 or the stored input partial sum 536 based on the stored skip computation indicator 540 to provide an output partial sum 542 via the seventh port. According to some examples, when the value of the input data element 222 or the weight 224 of the current operation is zero, the adder result 538 can not provide the correct result of the current operation because the multiply result 532 can hold the value of the previous operation. In such cases, the stored skip computation indicator 540 can allow bypassing the adder result 538 and selecting the stored input partial sum 536 to provide the output partial sum 542. For example, when the stored skip computation indicator 540 is “1,” the stored input partial sum 536 can be selected as the output partial sum 542, and when the stored skip computation indicator 540 is “0,” the adder result 538 can be selected as the output partial sum 542. The selector 516 can be implemented using a multiplexer or any suitable circuit.

[0129] In some examples, the generation of the skip computation indicator 526 can also be based on the value of the operation to be performed by the PE 00 as determined by the opcode 230 Figure 5 (not shown in FIG. 6). For example, for NOP, the data register 504 and the weight register 506 can hold their values from the previous operation using the skip computation indicator 526, thereby reducing power consumption. The selector 516 can select the stored input partial sum 536 as the output partial sum 542 instead of the adder result 538.

[0130] The zero data element indicator register 518 can be configured to store the received zero data element indicator 226 to provide the stored zero data element indicator 544 to the adjacent PE 01 in the first row in the next time period.

[0131] The zero weight indicator register 524 can be configured to store the zero weight indicator 228 to provide the stored zero weight indicator 550 to the adjacent PE 01 in the first row for the next time period.

[0132] The output data register 520 can be configured to store the input data element 222 to provide the delayed input data element 546 to the adjacent PE 01 in the first row for the next time period.

[0133] The output weight register 522 can be configured to store the weight 224 to provide the delayed weight value 548 to the adjacent PE 01 in the first row for the next time period.

[0134] The stored zero data element indicator 544, the stored zero weight indicator 550, the delayed input data element 546, and the delayed weight value 548 can be provided to the PE 01 via the row output bus 108, as discussed with reference to Figure 1 In some examples, control signals received by the PE 00 from external circuitry can also be cached in the PE 00 and provided to the PE 01 via the row output bus 108. Thus, each respective input data element, weight value, zero data element indicator, and zero weight indicator received by each PE per clock cycle can be cached in the respective PE and passed to the next adjacent PE in the next time period.

[0135] The PE 01 can receive the stored zero data element indicator 544, the stored zero weight indicator 550, the delayed input data element 546, and the delayed weight value 548 via the row input bus 102 as a zero data element indicator, a zero weight indicator, an input data element, and a weight, respectively. According to certain examples, the PE 01 can perform an arithmetic calculation on the delayed input data element 546 and the delayed weight value 548. If the delayed input data element 546 or the delayed weight value 548 includes a zero value based on the stored zero data element indicator 544 and the stored zero weight indicator 550, the PE 01 can skip the multiplication operation, thereby optimizing the dynamic power consumption of the PE 01. The PE 01 can store the stored zero data element indicator 544, the stored zero weight indicator 550, the delayed input data element 546, and the delayed weight value 548 in respective registers of the PE 01 and pass the delayed values to the adjacent PE 02 in the next time period.

[0136] Accordingly, input data set 0 can feed one input data element into the first row of systolic array 302 at each time period and pass from PE 00 to PE 0y in turn. As the input data element 222 passes through the PE, the stored input data element 528 can be multiplied with the stored weight value 530 and accumulated by adder 510 with the stored input partial sum 536. If the input data element 222 or the weight 224 is zero, the input to multiplier 508 can not be changed to reduce power consumption, and the stored input partial sum 536 can be provided as output partial sum 542 via column output bus 106. The output partial sum 542 of PE 00 can be passed as input partial sum 534 to the adjacent PE 10 in the second row. Each row of systolic array 302 can repeat the same operation, and a corresponding output data set can be generated.

[0137] Figure 6 A device 600 for neural network computation is shown in accordance with some examples of the disclosed technology. Device 600 can be part of a computer system (e.g., a host server). For example, the host server can provide multi-tenant computing services for data processing applications such as image recognition services, text-based data processing (e.g., processing of search queries), audio data processing, video data processing, and the like. In some examples, a host device can operate a software application and communicate with device 600 to make predictions based on computation of a predictive model using a neural network processor. For example, the host device can use a predictive model to make predictions by recognizing information included in an input data set of an image, text, audio, video, and the like.

[0138] Device 600 can include a neural network processor 602 coupled to a memory 614, a host interface 616, and a direct memory access (DMA) controller 618 via interconnect 620. Neural network processor 602 can include a compute engine 604, a compute controller 606, a state buffer 608, an output buffer 610, and an activation engine 612. Neural network processor 602 can provide compute resources to support computation of a predictive model. Neural network processor 602 can be implemented as a system on a chip (SoC), a field programmable gate array (FPGA), or any suitable circuit.

[0139] Memory 614 can be configured to store instructions received from a host device, input data sets (e.g., pixel data for images), and weights (e.g., weights corresponding to certain visual and / or non-visual features). Memory 614 can also be configured to store outputs of neural network processor 602 (e.g., one or more image recognition decisions made on input images in the form of output data sets). Memory 614 can include any suitable memory, such as dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate DRAM (DDR DRAM), storage class memory (SCM), flash memory, etc.

[0140] Host interface 616 can be configured to enable communication between a host device and neural network processor 602. For example, host interface 616 can be configured to transmit memory descriptors, including memory addresses of stored data (e.g., input data sets, weights, computation results, etc.), between a host device and neural network processor 602. Host interface 616 can include, for example, a peripheral component interconnect express (PCIe) interface, or any suitable interface for communicating with a host device. The host device can include a host processor and a host memory.

[0141] DMA controller 618 can be configured to perform DMA operations to transfer data between neural network processor 602 and a host device. For example, as described above, a host device can store instructions, input data sets, and weights in memory 614. The host device can provide memory addresses of the stored instructions, data, and weights to neural network processor 602 (e.g., in the form of memory descriptors). Neural network processor 602 can then obtain the stored instructions, data, and weights based on the memory addresses provided by the host device. Neural network processor 602 can also store results of computations (e.g., one or more image recognition decisions) in memory 614, and provide memory addresses of the stored results to the host device.

[0142] State buffer 608 can be configured to provide caching of data used for computations at compute engine 604. Data cached at state buffer 608 can include, for example, input data sets and weights obtained from memory 614, and intermediate outputs of computations at compute engine 604. Caching can reduce the impact of memory access bottlenecks (e.g., caused by latency at memory 614, DMA controller 618, interconnect 620, etc.) on the performance of compute engine 604. State buffer 608 can be an on-chip memory device and can include static random access memory (SRAM) or any suitable memory.

[0143] The compute controller 606 can be configured to provide control over the various components of the neural network processor 602 to perform neural network computations. In some implementations, the compute controller 606 can read instructions stored in the memory 614 and schedule execution of the instructions by the compute engine 604. In a first example, the compute controller 606 can perform a schedule to load weights into the compute engine 604 prior to reading input data elements from the state buffer 608. For example, as discussed with reference to Figure 2A 、 Figure 2B 、 Figure 2C 、 Figure 2D and Figure 4 The compute controller 606 can provide the opcode 230 and weight load 232 to the compute engine 604 based on instructions received from a host device. The compute controller 606 can provide appropriate values for the opcode 230 to the compute engine 604, which can be decoded by each PE in the compute engine to perform a corresponding operation. For example, the compute engine 604 can use the weight load 232 and the opcode 230 to preload weights in all of the PEs in the compute engine 604. Once the weights have been preloaded, the compute controller 606 can perform a schedule to load input data elements from the state buffer 608 into the compute engine 604 sequentially over a uniform period of time to begin the arithmetic computation.

[0144] In a second example, the compute controller 606 can perform a schedule to load weights and input data elements from the state buffer 608 into the compute engine 604 sequentially over a uniform period of time. The compute controller 606 can schedule the loading of weights and input data elements in the respective first PE of each row of the systolic array 302 using respective row data buses. For example, a respective input data element and weight value can be loaded into the first PE of a respective row in each cycle.

[0145] In another example, the compute controller 606 can schedule the loading of weights in the systolic array 302 for each row in parallel using respective column data buses for each PE in a given row. For example, weights for each row can be loaded in parallel in each cycle. In some implementations, the compute controller 606 can determine a data type for the input data set according to instructions received from a host device. The instructions can be in the form of an opcode. The data type can indicate a size and type of the input data elements, e.g., 4-bit, 8-bit, 9-bit, 16-bit, signed, unsigned, or floating point.

[0146] The compute engine 604 can be configured to perform computations of a neural network. In some examples, the compute engine 604 can include a set of PEs configured to perform one or more arithmetic operations involved in neural network computations. Each PE can perform a multiply-accumulate operation using an input data set and an associated weight. For example, the compute engine 604 can include the systolic array 302 and the circuit 304 including the zero input data detectors 306a-306x and the zero weight detectors 308a-308x. In some examples, the zero input data detectors 306a-306x and the zero weight detectors 308a-308x can be external to the compute engine 604. The compute engine 604 can execute instructions scheduled by the compute controller 606 to load weights and input data sets from the state buffer 608 into the compute engine 604 in sequence.

[0147] In a first example, weights can be preloaded prior to reading input data sets from the state buffer 608, as discussed with reference to Figure 4 The respective zero weight indicators corresponding to each weight can be cached locally in each PE, and the cached values can be used to perform arithmetic computations for respective input data elements when the input data elements are fed into the compute engine 604 along with corresponding zero data element indicators. In a second example, weights and input data sets can be read from the state buffer 608 simultaneously, as discussed with reference to Figure 5 The corresponding zero data element indicators and zero weight indicators can be provided by respective zero detector circuits and propagated from one PE to another PE for respective rows in sequence. The weights and input data sets can be obtained from the state buffer 608 using one or more interfaces. In certain examples, the compute engine 604 can perform arithmetic computations to reduce dynamic power consumption of the systolic array 302 using the respective zero data element indicators and zero weight indicators signals, as discussed with reference to FIGS. 2-4, and provide a computation result for storage in the output buffer 610. Figure 5

[0148] ​Output buffer 610 may include a set of registers to store the output dataset generated by computation engine 604. In some implementations, output buffer 610 may also enable additional processing, such as pooling operations, to reduce the size of the stored output. In some implementations, computation engine 604 may be operated to perform computations on a specific neural network layer, and output buffer 610 may process the output of that neural network layer and store the processed output dataset (with or without processing via activation engine 612) at state buffer 608. Computation engine 604 may use the processed output dataset as an intermediate output. In some examples, output buffer 610 may include adders to accumulate partial sums generated for different filter sets and input datasets to generate a convolutional output array. The final output value of the convolutional output array stored in state buffer 608 may be retrieved by computation controller 606 for storage at state buffer 608.

[0149] The activation engine 612 can be configured to apply one or more activation functions (e.g., ReLU functions) to the output of the output buffer 610. 。 For example, activation engine 612 may include one or more lookup tables (e.g., in the form of a multiplexer circuit) that can map inputs to one of candidate outputs representing the result of applying an activation function to the inputs. In some examples, activation engine 612 may also include bypass paths to allow outputs from output buffer 610 to be stored directly at state buffer 608 when no activation function is applied.

[0150] Figure 7 Method 700, performed by a PE for neural network computation, is shown as one of some examples of the disclosed techniques. The PE may be part of a systolic array 302, for example... Figure 3 PE 00. For example... Figure 6 As shown, the pulsating array 302 can be part of the computing engine 604.

[0151] In step 702, PE can receive a zero-weight indicator via the first port, indicating whether weight 224 is zero. Weight 224 may have already been received from the host device into memory 614. PE 00 can receive the zero-weight indicator 228 from the first zero-weight detector 308a via the row input bus 102. (See reference...) Figure 3As discussed, the first zero weight detector 308a can include a comparator or other circuitry to determine that the weight 224 is zero. For example, the first zero weight detector 308a can set the zero weight indicator 228 to "1" when it detects that the weight 224 is zero. Prior to loading of the input data elements into the PEs 00, the compute controller 606 can schedule the sequential loading of the weights 224 corresponding to filter 0 from the state buffer 608 into the PEs 00 of the compute engine 604 over a uniform period of time.

[0152] In step 704, the PE can store the weight value to provide a stored weight value for preloading the weight value into the PE. For example, the PE 00 can include a means for preloading the weights 224 into the PE 00. The compute controller 606 can provide the opcode 230 having a particular opcode value to the compute engine 604 for loading the respective weights into each PE of the compute engine 604. As discussed with reference to Figure 2A 、 Figure 2B 、 Figure 2C and Figure 2D As discussed, the weights 224 can be stored in the cached weight registers 220 using the weight load 232. The cached weights 246 previously loaded into the cached weight registers 220 can be transferred into the weight registers 206 upon initiation of the arithmetic computation based on the start computation signal 260. The stored weight values 248 can be used to perform the arithmetic computation using the stored input data elements 244.

[0153] In step 705, the PE can obtain a data type control signal indicating a selected data type for performing at least one of a multiplication or an addition. The data type control signal can be received from a source external to the PE or can be decoded from one or more signals within the PE.

[0154] In step 706, the PE can receive, via a second port, a zero data element indicator for a current operation indicating whether an input data element associated with the zero data element indicator is zero. As discussed with reference to Figure 3 As discussed, the PE 00 can receive the zero data element indicator 226 from the first zero input data detector 306a via the second port using the row input bus 102. The first zero input data detector 306a can include a comparator or other circuitry to determine that the input data element 222 is zero. For example, the first zero input data detector 306a can set the zero data element indicator 226 to "1" when it detects that the input data element 222 associated with the zero data element indicator 226 is zero.

[0155] At step 708, the PE can skip at least one of a multiplication of the selected data type or an addition of the selected data type based on at least one of a zero data element indicator indicating that an input data element associated with the zero data element indicator is zero or a zero weight indicator indicating that a weight is zero. For example, based on the zero data element indicator 226 indicating that the input data element 222 associated with the zero data element indicator 226 is zero, the multiplication of the input data element 222 by the weight to generate the integer product 250 or 251 can be skipped even if the data type indicator indicates an integer data type.

[0156] At step 710, the PE can resume performing a multiplication operation of the selected data type or performing an addition of the selected data type when a value of either the stored input data element or the stored weight value changes. Thus, dynamic power consumption can be reduced by skipping the multiplication or addition operation.

[0157] Referring back to FIG. 6, Figure 4 The PE 00 can include a shared multiplier 208 to perform a multiplication operation between the stored input data element 244 and the stored weight value 248 to generate the first product 250 and the second product 251. The shared multiplier 208 can perform the multiplication when a value of either the input data element 244 stored in the input data element register 204 or the weight value 248 stored in the weight register 206 changes. As previously described, the value of the input data element 244 stored in the input data element register 204 can change based on the data load signal 242. For example, when the zero data element indicator 226 or the NOP 258 is asserted, the data load signal 242 can prohibit a corresponding new value from being loaded in the input data element register 204, which can avoid switching the stored input data element 244. Since the weight 224 is already preloaded in the PE 00, the stored weight value 248 does not change during the computation. Thus, dynamic power consumption that can occur due to the multiplication operation performed by the shared multiplier 208 can be reduced.

[0158] Examples of the disclosed technology can provide systems and methods for reducing dynamic power consumption in a PE using a zero detector circuit to skip a multiplication operation with a zero value on an input data element. Further, using a respective zero detector circuit to detect a zero on an input data element and a weight into each row of a systolic array and pass a zero indicator to all PEs in the array can minimize gate count and power consumption compared to using a respective zero detector within each PE in the array.

[0159] Figure 8An example of a computing device 800 is shown. The functionality and / or several components of the computing device 800 can be used without limitation with other examples disclosed elsewhere in the present disclosure without limitation. The computing device 800 can perform computations to facilitate the processing of tasks. As an illustrative example, the computing device 800 can be part of a server in a multi-tenant computing service system. Various hardware and software resources of the computing device 800 (e.g., hardware and software resources associated with data processing) can be allocated to clients upon request.

[0160] In one example, the computing device 800 can include processing logic 802, a bus interface module 804, a memory 806, and a network interface module 808. These modules can be hardware modules, software modules, or a combination of hardware and software. In some cases, modules can be used interchangeably with components or engines without departing from the scope of the present disclosure. The computing device 800 can include additional modules, which are not illustrated here for ease of illustration. In some implementations, the computing device 800 can include fewer modules. In some implementations, one or more of the modules can be combined into one module. One or more of the modules can communicate with each other over a communication channel 810. The communication channel 810 can include one or more buses, grids, matrices, fabrics, combinations of these communication channels, or some other suitable communication channel.

[0161] The processing logic 802 can include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a system-on-a-chip (SoC), a network processing unit (NPU), a processor configured to execute instructions, or any other circuit configured to perform logical arithmetic and floating point operations. An example of a processor that can be included in the processing logic 802 can include a processor from the Intel® family of processors, the AMD® family of processors, the ARM® family of processors, the IBM® family of processors, the IBM® POWER® family of processors, the IBM® System z® family of processors, the IBM® z / Architecture®, or any other suitable processor. processors developed by Intel® Corporation, AMD®, ARM®, and others. In some implementations, the processors can include multiple processing cores, where each processing core can be configured to execute instructions independently of the other processing cores. Further, in some implementations, each processor or processing core can implement multiple processing threads executing instructions on the same processor or processing core while maintaining logical separation between the multiple processing threads. Such processing threads executing on a processor or processing core can be exposed to software as separate logical processors or processing cores. In some implementations, multiple processors, processing cores, or processing threads executing on the same core can share certain resources, such as, for example, a bus, a level 1 (LI) cache, and / or a level 2 (L2) cache. Instructions executed by the processing logic 802 can be stored, for example, in the form of a computer program on a computer-readable storage medium. The computer-readable storage medium can be non-transitory. In some cases, the computer-readable medium can be part of the memory 806. The processing logic 802 can also include hardware circuitry for performing artificial neural network computations, including, for example, a neural network processor 602 or the like.

[0162] Access to the processing logic 802 can be granted to clients to provide personal assistant services requested by the clients. For example, the computing device 800 can host a virtual machine on which an image recognition software application can execute. The image recognition software application, when executing, can access the processing logic 802 to predict, for example, objects contained in an image. As another example, access to the processing logic 802 can also be granted as part of a bare-metal instance, where an image recognition software application executing on a client device (e.g., a remote computer, a smart phone, or the like) can directly access the processing logic 802 to perform recognition of an image.

[0163] The memory 806 can include volatile memory or nonvolatile memory, or both. For example, the memory 806 can include random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory, and / or some other suitable type of storage media. In some cases, some or all of the memory 806 can be internal to the computing device 800, while in other cases, some or all of the memory can be external to the computing device 800. The memory 806 can store an operating system including executable instructions that, when executed by the processing logic 802, provide an execution environment for executing instructions that provide functionality for performing convolution computations for the computing device 800. The memory 806 can also store software applications, for example, for performing artificial neural network computations. The memory can also store and maintain several data structures and tables to facilitate the functionality of the computing device 800.

[0164] The bus interface module 804 can enable communication with external entities, such as host devices and / or other components in a computing system, over external communication media. The bus interface module 804 can include physical interfaces for connecting to cables, sockets, ports, or other connections to external communication media. The bus interface module 804 can also include hardware and / or software to manage incoming and outgoing transactions. The bus interface module 804 can implement a local bus protocol, such as a Peripheral Component Interconnect (PCI)-based protocol, Non-Volatile Memory Express (NVMe), Advanced Host Controller Interface (AHCI), Small Computer System Interface (SCSI), Serial Attached SCSI (SAS), Serial AT Attachment (SATA), Parallel ATA (PATA), some other standard bus protocol, or a proprietary bus protocol. The bus interface module 804 can include physical layers for any of these bus protocols, including connectors, power management, and error handling, among others. In some implementations, the computing device 800 can include multiple bus interface modules for communicating with multiple external entities. These multiple bus interface modules can implement the same local bus protocol, different local bus protocols, or a combination of the same and different bus protocols.

[0165] The network interface module 808 can include hardware and / or software for communicating with a network. For example, the network interface module 808 can include physical connectors or physical ports for wired connections to a network, and / or antennas for wireless communication with a network. The network interface module 808 can also include hardware and / or software configured to implement a network protocol stack. The network interface module 808 can communicate with a network using network protocols such as, for example, TCP / IP, Wireless Bandwidth, RoCE, Institute of Electrical and Electronics Engineers (IEEE) 802.11 wireless protocols, User Datagram Protocol (UDP), Asynchronous Transfer Mode (ATM), Token Ring, Frame Relay, High-Level Data Link Control (HDLC), Fiber Distributed Data Interface (FDDI), and / or Point-to-Point Protocol (PPP), among others. In some implementations, the computing device 800 can include multiple network interface modules, each configured to communicate with a different network. For example, in these implementations, the computing device 800 can include network interface modules for communicating with wired Ethernet networks, wireless 802.11 networks, cellular networks, Wireless Bandwidth networks, etc. In some examples, the computing device 800 can receive a set of parameters, such as the above-described weight values for a convolution computation, from a server through the network interface module 808.

[0166] The various components and modules of the computing device 800 described above can be implemented as discrete components, a system on a chip (SoC), ASIC, NPU, FPGA, or any combination thereof. In some examples, the SoC or other components can be communicatively coupled to another computing system to provide various services, such as traffic monitoring, traffic shaping, computing, and the like. In some examples of the technology, the SoC or other components can include multiple subsystems, as disclosed herein.

[0167] The modules described herein can be software modules, hardware modules, or suitable combinations thereof. If the modules are software modules, the modules can be embodied on a non-transitory computer readable medium and processed by a processor in any of the computer systems described herein. It should be noted that the processes and architectures described can be performed in real-time or in an asynchronous mode prior to any user interaction. The modules can be configured in the manner suggested by the 3GPP® Figure 8 The modules can be configured in the manner suggested by the 3GPP®

[0168] Figure 9 A method 900 performed by a PE for neural network computation is shown in accordance with some examples of the disclosed technology. As shown, the PE can be part of a systolic array 302, such as PE 00. Figure 3

[0169] In step 902, the PE can obtain a data type control signal indicative of an integer data type. The PE can obtain the data type control signal via a row input bus, such as row input bus 102 shown in FIG. 2. The PE can additionally or alternatively obtain the data type control signal by decoding an opcode, decoding a format of a weight, decoding a format of an input data element, or decoding some other similar indicator. As discussed with reference to Figure 2C and Figure 2D The data type control signal 235 and its delayed version 241 can be used in PEs configured to perform multiply-delay-accumulate, as discussed with reference to

[0170] In step 904, the PE can perform an integer multiplication operation between a stored input data element and a stored weight value to generate an integer product based on the stored input data element and the stored weight value. In some examples, such as shown in Figure 2A and Figure 2C Non-integer multiplication operations can also be performed to generate a non-integer product, but the non-integer product can be selectively ignored based on the data type control signal indicative of an integer data type. The integer multiplication can be performed using a shared multiplier.

[0171] ​In step 906, the PE can perform an integer addition operation between the stored input partial sum and the integer product to generate an integer partial sum. In some examples, such as described with respect to FIG. 2, the integer addition operation can be performed using a shared adder. Figure 2B and Figure 2D As described, separate adders can also perform unselected non-integer (e.g., floating point or bfloat) additions in parallel. The integer addition can be performed using the shared adder or separate integer adders.

[0172] In step 908, the PE can select the integer partial sum to provide as the output partial sum. For example, the PE can select the integer partial sum while ignoring the non-integer partial sum. The PE can select the integer partial sum based at least in part on the data type control signal indicating an integer data type.

[0173] In step 910, the PE can obtain an altered data type control signal indicating a non-integer data type, such as FP16 or BF16. The PE can obtain the altered data type control signal via a row input bus, such as row input bus 102 shown in FIG. 2. The PE can additionally or alternatively obtain the altered data type control signal by decoding an opcode, decoding a format of the weights, decoding a format of the input data elements, or decoding some other similar indicator.

[0174] In step 912, the PE can perform a non-integer multiplication operation between the stored input data elements and the stored weight values to generate a non-integer product. The non-integer multiplication can be performed using the shared multiplier.

[0175] In step 914, the PE can perform a non-integer addition operation between the stored input partial sum and the non-integer product to generate a non-integer partial sum. The non-integer addition can be performed using the shared adder or separate non-integer adders.

[0176] In step 916, the PE can select the non-integer partial sum to provide as the output partial sum to another processing element. For example, the PE can select the non-integer partial sum while ignoring the integer partial sum. The PE can select the non-integer partial sum based at least in part on the data type control signal indicating a non-integer data type.

[0177] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. However, it will be apparent to those skilled in the art that various modifications and changes can be made thereto without departing from the broader spirit and scope of the disclosure as set forth in the claims.

[0178] Other variations are within the spirit of the present disclosure. Thus, while the disclosed technology is susceptible to various modifications and alternative constructions, certain illustrated examples thereof have been shown in the drawings and described above in detail. It should, however, be understood that there is no intention to limit the disclosure to the specific

[0179] In the context of describing the disclosed examples, particularly in the context of the claims that follow, the use of the terms "a", "an" and "the" and similar referents are to be construed to cover both the singular and plural unless otherwise indicated by context, or explicitly contradicted by context. The terms "comprising", "having", "including" and "containing" are to be construed as open-ended terms (i.e., meaning "including, but not limited to") unless otherwise noted. The term "connected" is to be construed as partly or wholly inclosed, attached to or bound together, even if there is an intervening material or agent. Unless otherwise indicated herein, the description herein of a range of values is merely intended to serve as a shorthand method of referring individually to each separate value falling within the range, and each separate value is incorporated in the specification as if it were individually recited herein. Unless otherwise indicated herein, all methods described herein can be performed in any suitable order unless otherwise indicated herein. The use of any and all examples, or exemplary language (e.g., "such as") provided herein, is intended merely to better illuminate examples of the disclosure and does not pose a limitation on the scope of the disclosure unless otherwise claimed. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

[0180] Unless specifically stated otherwise, the use of certain nomenclature, such as "at least one of X, Y or Z," in the context of certain disclosed examples is used herein to convey the intended meaning that the item, term, etc. can be one of X, Y, or Z, or any combination of X, Y, and / or Z. Thus, such nomenclature is merely an articulation of a possible usage of the items, terms, etc. and is not intended to limit the items, terms, etc. to only those possible uses.

[0181] Various examples of the present disclosure are described herein, including the best mode presently known to the inventors for practicing the present disclosure. Variations of these examples can become apparent to those of ordinary skill in the art upon reading the foregoing description. The inventors expect skilled artisans to employ such variations as appropriate, and the inventors intend for the present disclosure to be practiced otherwise than specifically described herein. Accordingly, the present disclosure includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by law. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the present disclosure unless otherwise indicated herein or otherwise evident to one of ordinary skill in the art.

[0182] Various example embodiments of the present disclosure can be described by the following clauses:

[0183] Clause 1 : A systolic multiply-accumulate processor, comprising:

[0184] a systolic array of processing elements arranged in a first plurality of rows and a second plurality of columns, the systolic array of processing elements configured to operate on an input data set, each of the processing elements comprising:

[0185] a first input port to receive an input data element;

[0186] a weight register to store a stored weight value;

[0187] a second input port to receive an input partial sum;

[0188] an output port to provide an output partial sum;

[0189] a shared multiplier configured to multiply the input data element by the stored weight value, the shared multiplier comprising:

[0190] a first sub-circuit configured to support integer multiplication; and

[0191] a second sub-circuit configured to support floating point multiplication, wherein at least a shared portion of the first sub-circuit is shared with the second sub-circuit;

[0192] one or more adders coupled to the second input port, the output port, and the shared multiplier, wherein the one or more adders are configured to:

[0193] generate an integer partial sum by performing integer addition of an integer product from the shared multiplier and the input partial sum; and

[0194] generating a floating-point partial sum by performing floating-point addition on the floating-point product from the shared multiplier and the input partial sum; and

[0195] a selector circuit configured to select at least between the integer partial sum and the floating-point partial sum to provide to the output port as the output partial sum.

[0196] Clause 2: According to Clause 1. The systolic multiply-accumulate processor, wherein:

[0197] the first sub-circuit is configured to generate a first product by performing integer multiplication on the input data element and the stored weight value;

[0198] the second sub-circuit is configured to generate a second product by performing floating-point multiplication on the input data element and the stored weight value, wherein the second product includes a significand and an exponent; and

[0199] the shared portion of the first sub-circuit that is shared with the second sub-circuit is used to generate the first product and is used to generate the significand of the second product.

[0200] Clause 3: The systolic multiply-accumulate processor of Clause 1 or Clause 2, wherein each processing element in the processing element systolic array further comprises:

[0201] a multiplexer configured to select between a first product and a second product based on a data type control signal that selects a data type to provide to the one or more adders; and

[0202] wherein the multiplexer is configured to activate or de-activate at least a portion of the second sub-circuit based at least in part on the data type control signal.

[0203] Clause 4: The systolic multiply-accumulate processor of any of Clauses 1-3, wherein the one or more adders comprise:

[0204] a shared adder portion configured to do both of the following:

[0205] generate the integer partial sum of an integer data type by adding the generated integer product to the input partial sum; and

[0206] compute one or more portions of the floating-point partial sum; and

[0207] a separate circuit that is separate from the shared adder portion, wherein the shared adder portion and the separate circuit together are used to generate the floating-point partial sum of a floating-point data type.

[0208] Clause 5: A systolic circuit comprising:

[0209] a processing element coupled in a systolic array of processing elements and configured to communicate data with at least one neighboring processing element in the systolic array, the processing element comprising:

[0210] a shared multiplier configured to support both integer multiplication and non-integer multiplication of input data elements and weights, wherein shared sub-circuits of the shared multiplier used to support integer multiplication are also used to support the non-integer multiplication;

[0211] one or more adders configured to receive one or more products from the shared multiplier, the one or more adders configured to support both integer addition and non-integer addition, wherein the one or more adders are configured to add an input partial sum to at least one of the one or more products to produce at least one of an integer partial sum and a non-integer partial sum; and

[0212] a selector circuit configured to select at least between the integer partial sum and the non-integer partial sum to provide to an output port as an output partial sum.

[0213] Clause 6: The systolic circuit of clause 1, wherein the shared multiplier is configured to perform at least floating point multiplication and integer multiplication.

[0214] Clause 7: The systolic circuit of clause 5 or clause 6, further comprising:

[0215] a multiplexer configured to select among a plurality of products generated by the shared multiplier to provide to the one or more adders based on a data type control signal, wherein the plurality of products includes at least an integer product and a floating point product; and

[0216] wherein the shared multiplier is configured to prevent circuitry used for floating point multiplication from contributing to an operation based at least in part on the data type control signal.

[0217] Clause 8: The systolic circuit of any of clauses 5-7, wherein the one or more adders comprise:

[0218] a shared adder configured to perform at least a first portion of a floating point addition and an integer addition; and

[0219] a separate circuit configured to perform at least a second portion of the floating point addition.

[0220] Clause 9: The systolic circuit of any of clauses 5-8, wherein the one or more adders comprise:

[0221] a floating point adder configured to perform floating point addition; and

[0222] an integer adder configured to perform integer addition.

[0223] Clause 10: The systolic circuit of any of clauses 5-9, wherein:

[0224] each of the processing elements further comprises a delay register coupled between the shared multiplier and the one or more adders; and

[0225] the one or more adders are configured to receive one or more products from the shared multiplier in a subsequent systolic cycle, wherein the one or more products are generated by the shared multiplier during a previous systolic cycle.

[0226] Clause 11 : The systolic circuit of any of clauses 5-10, further comprising a skip computation generator configured to propagate a skip computation signal to a plurality of processing elements, wherein the skip computation signal is configured to prevent at least one of the shared multiplier or the one or more adders for a systolic cycle from contributing to an arithmetic computation of the systolic array during the systolic cycle.

[0227] Clause 12: The systolic circuit of any of clauses 5-11, wherein each of the processing elements further comprises a delay register coupled between the shared multiplier and the one or more adders.

[0228] Clause 13: The systolic circuit of any of clauses 5-12, wherein:

[0229] the shared multiplier is selectable for at least one of: a single 17-bit integer multiplication or two parallel 9-bit integer multiplications; and

[0230] the shared multiplier is selectable for at least one of: a single 16-bit brain floating point multiplication or a single 16-bit floating point multiplication.

[0231] Clause 14: The systolic circuit of any of clauses 5-13, wherein the selector circuit is configured to select at least between the integer sum and the non-integer sum based at least in part on a data type control signal.

[0232] Clause 15: The systolic circuit of any of clauses 5-14, further comprising a partial dequantizer configured to partially dequantize the input data elements of a row of the processing elements, wherein partially dequantizing the input data elements comprises shifting the input data elements and adding extra bits to the input data elements.

[0233] Clause 16: The systolic circuit of any of clauses 5-15, wherein the shared multiplier is capable of performing at least selected from: a single 17-bit integer multiplication, two or more parallel 9-bit integer multiplications, a single 16-bit brain floating point multiplication, and a single 16-bit floating point multiplication.

[0234] Clause 17: A method for systolic processing by a processing element, the method comprising:

[0235] obtaining a data type control signal indicative of an integer data type;

[0236] performing, by a shared multiplier, an integer multiplication of a first input data element and an output of a weight register to generate an integer product;

[0237] performing, by one or more adders, an integer addition of a first input partial sum and the integer product to generate an integer partial sum;

[0238] selecting, based at least in part on the data type control signal indicative of the integer data type, the integer partial sum to provide as an output partial sum to another processing element;

[0239] obtaining a changed data type control signal indicative of a non-integer data type;

[0240] performing, by the shared multiplier, a non-integer multiplication of a second input data element and the output of the weight register to generate a non-integer product;

[0241] performing, by the one or more adders, a non-integer addition of a second input partial sum and the non-integer product to generate a non-integer partial sum; and

[0242] selecting, based at least in part on the changed data type control signal indicative of the non-integer data type, the non-integer partial sum to provide as an output partial sum to another processing element.

[0243] Clause 18: The method of clause 12, further comprising:

[0244] performing, by exponent logic in the shared multiplier, a calculation of an exponent bit in the non-integer product.

[0245] Clause 19: The method of clause 12 or clause 18, further comprising:

[0246] receiving the first input data element, wherein the first input data element is a 9-bit or 17-bit integer; and

[0247] performing multiplication on the first input data element and the output of the weight register by the shared multiplier, wherein the shared multiplier is configured to support multiplication of at least one of 9-bit or 17-bit integers.

[0248] Clause 20: The method of any of clauses 17-19, further comprising:

[0249] performing the integer addition by an integer adder; and

[0250] performing the non-integer addition by a non-integer adder separate from the integer adder.

Claims

1. A systolic circuit comprising: a processing element coupled in a processing element systolic array and configured to communicate data with at least one neighboring processing element in the systolic array, the processing element comprising: a shared multiplier configured to support both integer multiplication and non-integer multiplication of input data elements and weights, wherein a data type control signal identifies the integer multiplication or the non-integer multiplication, wherein shared sub-circuits of the shared multiplier are used to support the integer multiplication and the non-integer multiplication, and wherein non-shared sub-circuits of the shared multiplier are disabled based at least in part on the data type control signal; one or more adders configured to receive one or more products from the shared multiplier, the one or more adders configured to support both integer addition and non-integer addition, wherein the one or more adders are configured to add an input partial sum to at least one of the one or more products to produce an integer partial sum and a non-integer partial sum; and a selector circuit configured to select a first output from the integer partial sum and the non-integer partial sum based at least in part on the data type control signal to provide to an output port as an output partial sum and to select a second output from the integer partial sum and the non-integer partial sum to ignore.

2. The systolic circuit of claim 1, wherein the shared multiplier is configured to perform at least floating point multiplication and integer multiplication.

3. The systolic circuit of claim 1 or claim 2, further comprising: a multiplexer configured to select the one or more products from a plurality of products generated by the shared multiplier based on the data type control signal, wherein the plurality of products includes at least an integer product and a floating point product, and wherein the shared multiplier is configured to prevent, based at least in part on the data type control signal, a circuit used for floating point multiplication from contributing to an operation.

4. The systolic circuit of claim 1 or 2, wherein the one or more adders comprise: a floating point adder configured to perform floating point addition; and an integer adder configured to perform integer addition.

5. The systolic circuit of claim 1 or 2, wherein: the processing element further comprises a delay register coupled between the shared multiplier and the one or more adders; and the one or more adders are configured to receive one or more products from the shared multiplier in a subsequent systolic cycle, wherein the one or more products are generated by the shared multiplier during a previous systolic cycle. ​ 6. The systolic circuit of claim 1 or 2, further comprising a skip computation generator configured to propagate a skip computation signal to a plurality of processing elements, wherein the skip computation signal is configured to prevent at least one of the shared multiplier or the one or more adders from contributing to an arithmetic computation of the systolic array during a systolic cycle.

7. The systolic circuit of claim 1 or 2, wherein the processing element further comprises a delay register coupled between the shared multiplier and the one or more adders.

8. The systolic circuit of claim 1 or 2, wherein: the shared multiplier is selectable to perform at least one of: a single 17-bit integer multiplication or two parallel 9-bit integer multiplications; and the shared multiplier is selectable to perform at least one of: a single 16-bit brain floating point multiplication or a single 16-bit floating point multiplication.

9. The systolic circuit of claim 1 or 2, wherein the selector circuit is configured to select the integer sum and or the non-integer sum based at least in part on the data type control signal.

10. The systolic circuit of claim 1 or 2, further comprising a partial dequantizer configured to partially dequantize the input data elements of a row of the processing elements, wherein partially dequantizing the input data elements comprises shifting the input data elements and adding extra bits to the input data elements.

11. The systolic circuit of claim 1 or 2, wherein the shared multiplier is capable of being selected to perform at least one of: a single 17-bit integer multiplication, two or more parallel 9-bit integer multiplications, a single 16-bit brain floating point multiplication, or a single 16-bit floating point multiplication.

12. A method for systolic processing by a processing element, the method comprising: obtaining a data type control signal indicative of an integer data type; performing, by a shared multiplier, an integer multiplication of a first input data element and an output of a weight register to generate an integer product, wherein a first non-shared sub-circuit of the shared multiplier is disabled based at least in part on the data type control signal; performing, by one or more adders, an integer addition of a first input sum and the integer product to generate an integer partial sum; selecting, based at least in part on the data type control signal indicative of the integer data type, a first output corresponding to the integer partial sum to be provided as an output partial sum to another processing element and selecting a second output to be ignored; obtaining a changed data type control signal indicative of a non-integer data type; performing, by the shared multiplier, a non-integer multiplication of a second input data element and the output of the weight register to generate a non-integer product, wherein the first non-shared sub-circuit of the shared multiplier is enabled and a second non-shared sub-circuit of the shared multiplier is disabled based at least in part on the changed data type control signal; performing, by the one or more adders, a non-integer addition of the second input partial sum and the non-integer product to generate a non-integer partial sum; and selecting, based at least in part on the changed data type control signal indicative of the non-integer data type, a third output corresponding to the non-integer partial sum to be provided as an output partial sum to another processing element and selecting a fourth output to be ignored.

13. The method of claim 12, further comprising: performing, by exponent logic of a multiplier in the shared multiplier, a calculation of an exponent bit in the non-integer product.

14. The method of claim 12 or claim 13, further comprising: receiving the first input data element, wherein the first input data element is a 9-bit or 17-bit integer; and performing, by the shared multiplier, a multiplication of the first input data element and the output of the weight register, wherein the shared multiplier is configured to support multiplication of at least one of a 9-bit integer or a 17-bit integer.

15. The method of claim 12 or 13, further comprising: performing the integer addition by an integer adder; and performing the non-integer addition by a non-integer adder separate from the integer adder.

Images