Systolic array with efficient input reduction and expansion array performance

Abstract

Systems and methods are provided for performing reduced-precision multiply-accumulate operations in a systolic array. Each row of the systolic array can receive a reduced input from a respective reducer. The reduced input can include a reduced input data element and / or a reduced weight. The systolic array can lack support for inputs having a first bit length, and the reducer can reduce the bit length of a given input from the first bit length to a shorter second bit length and provide the reduced input to the array. To reduce the bit length, the reducer can reduce the number of trailing bits of the input. Additionally, the systolic array can receive reduced and rounded inputs. The systolic array can propagate the reduced inputs through processing elements in the systolic array. Each processing element can include a multiplier and / or an adder to perform arithmetic operations based on the reduced inputs.

Description

BACKGROUND

[0001] An artificial neural network is a computing system having an architecture based on biological neural networks. Neural networks can be implemented through circuits and data paths such as systolic arrays. Systolic arrays can accelerate the performance of the training and inference phases of artificial neural networks. During the training phase, input data can be provided to train a model. During the inference phase, new inputs can be processed according to the model to obtain a prediction result. User applications typically use the model during the inference phase, so the inference phase can generally be time sensitive, and latency during the inference phase can negatively impact the user experience.

[0002] As more applications use artificial neural networks, these applications also use a wide range of numbers that can include numbers with increased bit lengths (e.g., 32-bit floating point numbers) that can require more computational power or modifications to the neural network. While computational support for numbers with increased bit lengths can provide increased accuracy for mathematical operations, providing support for the increased bit lengths of these numbers can increase the complexity, size, and cost of processing elements in a systolic array. These increases can also impact system processing speed and system power consumption. When a systolic array is required to support a wide range of numbers, power consumption and the size of the systolic array can become very important. BRIEF DESCRIPTION OF DRAWINGS

[0003] Various features will now be described, with reference to the following drawings. In all of the drawings, reference numerals can be repeated between the drawings to indicate correspondence between the referenced elements. The drawings are provided to illustrate examples described herein and are not intended to limit the scope of the disclosure.

[0004] FIG. 1A An exemplary 4x4 systolic array and exemplary column of reducers are shown.

[0005] FIG. 1B An exemplary 1x8 column of a systolic array is shown.

[0006] FIG. 2A A processing element for neural network computation in which inputs enter through separate reducers in accordance with certain examples of the disclosed technology is shown.

[0007] FIG. 2B A processing element for neural network computation in which inputs enter through the same reducer in accordance with certain examples of the disclosed technology is shown.

[0008] FIG. 3 An apparatus including zero detector circuitry for reducing input data elements and reducing weights into a systolic array for neural network computation in accordance with certain examples of the disclosed technology is shown.

[0009] FIG. 4A A reducer showing selection of inputs to be reduced and rounded according to some examples of the disclosed technology.

[0010] FIG. 4B A reducer showing selection of rounded inputs to be reduced according to some examples of the disclosed technology.

[0011] FIG. 4C A reducer showing generation of multiple reduced inputs from selected inputs according to some examples of the disclosed technology.

[0012] FIG. 5 A multiply-accumulate data path for neural network computation according to certain examples of the disclosed technology.

[0013] FIG. 6 An apparatus for neural network computation according to some examples of the disclosed technology.

[0014] FIG. 7 A method performed by a reducer and processing element for neural network computation according to some examples of the disclosed technology.

[0015] FIG. 8 A method performed by a reducer and processing element for neural network computation according to some examples of the disclosed technology.

[0016] FIG. 9A to FIG. 9H An exemplary systolic array processing data over a series of systolic intervals.

[0017] FIG. 10 An example of a computing device according to certain aspects of the disclosure. DETAILED DESCRIPTION

[0018] In general, the present disclosure relates to a systolic array that supports converting an input having a bit length higher than natively supported by elements of the array into one or more reduced inputs. Additionally, the input can be converted into reduced inputs in order to perform a single pass reduced precision computation on an input having a bit length higher than natively supported by elements of the array. For example, elements of the array can natively support performing a single pass computation on an input to the systolic array having a particular bit length, and the systolic array can receive the input from a reducer that reduces the bit length of the input to match the bit length natively supported by the elements during the single pass computation. The input can also be converted into multiple reduced inputs in order to perform a multiple pass full precision computation on an input having a bit length higher than natively supported by elements of the array. As described herein, providing reduced inputs to the systolic array using such reducers can enable an input to be given to the systolic array at an arbitrary bit length and adjusted to a particular bit length (e.g., the highest bit length supported during a single pass computation) in a programmatic manner such that a user does not need to know the particular bit length of the input to the processing elements of the systolic array. While conventional systolic arrays can support different bit lengths, native support for single pass computations on higher bit lengths can increase the size and power consumption of the systolic array. Additionally, this can impact processing of shorter bit lengths. Thus, conventional systolic arrays must balance the ability to perform single pass computations on longer bit lengths and the efficiency of processing shorter bit lengths. This can result in a systolic array not supporting longer bit lengths due to the loss of efficiency in processing shorter bit lengths. Disclosed herein is a systolic array that supports an arbitrary long bit length with reduced precision at a minimal loss of efficiency compared to processing shorter bit lengths. The systolic array can support inputs having an arbitrary bit length through a reducer that can discard excess bits from the significant number of an input having an arbitrary bit length and round the remaining bits. Discarding excess bits can enable the reducer to reduce the bit length of the input to the maximum bit length supported by the systolic array for a single pass computation at the cost of reduced precision for the arbitrary bit length. Additionally, using such a reducer can enable a systolic array receiving inputs having an arbitrary bit length to provide the same performance achieved by a systolic array receiving inputs having a fixed bit length. Allowing a user to provide inputs having an arbitrary (or non-fixed) bit length can allow for the use of lower cost or lower power elements in a systolic array receiving inputs having a larger bit length while maintaining the overall performance of the systolic array due to the reducer reducing the bit length of the input. Additionally, by reducing the bit length of an input (e.g., a 32-bit floating point number), the reducer can provide a reduced precision version of the input (e.g., a 22-bit floating point reduced precision number). Thus, the reducer can generate a reduced input from an input by reducing the bit length of the input.

[0019] The reducer can generate multiple reduced inputs from the input. The systolic array can utilize the multiple reduced inputs in a multi-pass multiply-accumulate operation to preserve accuracy of the input. For example, each combination of the reduced inputs (e.g., input data element 1 and weight 1, input data element 2 and weight 1, input data element 1 and weight 2, and input data element 2 and weight 2 in the case of the reducer generating two reduced inputs for the input data elements and weights) can be passed through the multi-pass multiply-accumulate operation. By generating multiple reduced inputs from the input having a reduced bit length, the reducer can reduce the bit length of the input to a maximum bit length supported by a single-pass computation of the systolic array, but at the cost of reduced performance for arbitrary bit lengths. Additionally, using such a reducer can enable a systolic array receiving multiple reduced inputs (where the bit length is reduced from the original bit length) to provide the same frequency, power advantage, and / or size advantage as a systolic array receiving inputs having a fixed (e.g., standard) bit length, at the cost of lower performance as compared to a systolic array operating on inputs having the original bit length. Allowing a user to provide inputs having arbitrary bit lengths can allow for use of lower cost or lower power components (e.g., power components configured to operate on standard bit lengths) in a systolic array receiving inputs having arbitrary bit lengths, while providing increased precision as compared to a systolic array receiving inputs having standard bit lengths.

[0020] As described herein, a systolic array includes an array of processing elements (PEs) typically arranged in two dimensions (e.g., columns and rows). The PEs of the array can be interconnected to enable data to pass through the PEs, which can perform one or more mathematical operations on the data. For example, each PE can perform a “multiply-accumulate” operation, whereby an input is fed horizontally into the PEs of each row of the array, with each PE multiplying its respective input by a stored weight value and passing the product result to a PE in a subsequent row.

[0021] One illustrative use of a systolic array is to perform the inference phase of a machine learning application. Machine learning generally requires at least two phases: a “learning phase,” in which a model is trained from training data; and an “inference phase,” in which the trained model is applied to production data to predict a result. Inference phase applications are often sensitive to latency, as they are run in production environments. Moreover, inference phase applications (and especially neural network applications) often require intensive algebraic computations, such as matrix multiplication. A systolic array can be used to accelerate inference phase workloads in machine learning applications.

[0022] As described above, the PEs of a systolic array can be divided into rows and columns. Each PE in an input layer can receive an element of an input data set and scale the element using a weight (e.g., a filter) to indicate how much the element contributes to an output. Each PE in an intermediate layer can receive at least one of an element and a weight (or filter) from another PE in the systolic array. Each PE in an intermediate layer can combine elements received from corresponding PEs of the systolic array to compute a set of intermediate outputs. For example, each PE in an intermediate layer can compute a sum of element-weight products and then produce the sum to apply an activation function to the sum (e.g., by a system separate from the PEs of the systolic array).

[0023] Generally, an input dataset (e.g., an input feature map) can be fed into its respective row of the systolic array one input data element at a time, and passed from one PE to another PE in a given row, e.g., starting from the left-most PE. Each row receives a particular input data element and weight that is fed into the first PE in the row, and then passed to an adjacent PE to the right of the first PE in the same row. Additionally, an input partial sum can be fed into its respective column of the systolic array one input partial sum at a time, and passed from one PE to another PE in a given column, starting from the top-most PE. Generally, an input partial sum can be fed in a column from a first PE to an adjacent PE directly below the first PE in the same column. Additionally, each column corresponds to a particular input partial sum that passes through each PE of the given column. Doing so can allow each PE of a given column to perform a mathematical operation on the input partial sum to produce an output partial sum. As an input data element passes through the PEs, the input data element can be multiplied by a weight value and accumulated with the input partial sum. The first PE in a column is provided the input partial sum and generates an output partial sum based on the mathematical operation performed by that PE. The output partial sum is then provided to an adjacent PE in the same column as an input partial sum. The adjacent PE can then perform further mathematical operations before generating an output partial sum and passing the output partial sum to a further adjacent PE. In some embodiments, input data can be fed into the systolic array in a cascading manner, where the PEs in the first column and first row (which can be designated as the position [0, 0], indicating row 0 and column 0) receive an input data element and an input partial sum in a first clock cycle. Thereafter, data can generally flow to subsequent rows and columns at a given rate (e.g., one PE forward per cycle). For example, the output partial sum of the PE at [0, 0] can be fed with an input data element of row 1 to the PE at [1, 0] such that the PE at [1, 0] performs a mathematical operation on that input data element and partial sum during a second clock cycle. Similarly, the input data element of PE [0, 0] can be passed to a PE of a subsequent column (e.g., at position [0, 1]), which can also be fed to an input partial sum such that the PE at [0, 1] performs a mathematical operation on that input partial sum with the input data element during the second clock cycle. Assuming a convention that rows progress downward and columns progress rightward, data can generally flow downward and rightward during the array operation. To facilitate these calculations, weights can be provided to the PEs within the array prior to the first clock cycle, or these PEs can receive the weights in the first clock cycle or during the calculation.

[0024] With the proliferation of machine learning applications and neural network applications, there has been an increased demand for increased processing capabilities (e.g., the ability to process larger numbers and / or more precise numbers) while simultaneously achieving higher precision and maintaining performance. For example, there has been an increased demand to support numbers with increased precision (e.g., the number of decimal places and / or the number of significant numbers). Providing support for numbers with larger bit lengths (e.g., 32-bit floating point numbers) results in a significant increase in integrated circuit die cost, power consumption, and circuit complexity as compared to supporting only numbers with fixed (e.g., specific) bit lengths (e.g., 16-bit floating point numbers) because traditional PEs can not be able to receive numbers with bit lengths greater than a specific length. In systolic arrays of hundreds or thousands of PEs, the additional support for numbers with larger bit lengths can result in an exponential increase in integrated circuit die cost, power consumption, and circuit complexity. In some configurations, PEs support performing mathematical operations on numbers with increased bit lengths (e.g., 32-bit) with specialized circuitry configured for larger bit lengths. For example, a 32-bit floating point systolic array can be dedicated to performing mathematical operations on 32-bit floating point (FP32) numbers. Such modifications can be particularly undesirable, can provide reduced performance, and can be costly and / or time consuming to implement. In other configurations, PEs do not support performing mathematical operations on numbers with bit lengths greater than a given size. For example, a 16-bit floating point systolic array can not be able to perform mathematical operations on numbers other than 16-bit floating point (FP16) numbers. The lack of such capability can be particularly undesirable and can provide reduced precision and / or reduced processing capabilities.

[0025] The present disclosure provides a systolic array with significant advantages over existing implementations. The present disclosure enables a systolic array to support arbitrary bit lengths and maintain performance for shorter bit lengths relative to an array that natively supports single-pass computation on longer bit lengths without significantly increasing the power consumption of the array. Further, the present disclosure can enable the use of numbers with arbitrary bit lengths (e.g., 32-bit floating point numbers) as inputs to a systolic array (e.g., as inputs to a reducer of the array). Additionally, a reducer of a systolic array can programmatically adjust inputs to a particular bit length (e.g., the highest bit length supported during single-pass computation) such that a user does not need to know the particular bit length of inputs received by processing elements of a systolic array. These advantages are provided by the implementations discussed herein and, in particular, by creating a systolic array with one or more reducers that reduce one or more inputs provided to the systolic array. Additionally, the one or more reducers can generate multiple reduced inputs for a particular input in order to preserve the accuracy of the original input.

[0026] A systolic array can support a particular bit length or data type. For example, a systolic array can support a standard bit length and / or data type (e.g., FP16 numbers). A consumer or user can be informed that the systolic array supports a particular bit length or data type. Additionally, a reducer can receive an input having an arbitrary bit length that does not correspond to the supported bit length and / or data type (e.g., FP32 numbers). The reducer can convert the input having the non-supported bit length to a reduced format (e.g., a reduced bit length) and provide the input having the reduced format (e.g., 22-bit floating point numbers) to the systolic array. The reduced format can be a non-standard format, a non-standard bit length, and / or a non-standard data type. The consumer can not be informed that the systolic array supports the input having the reduced format. Additionally, the input having the reduced format can have a higher accuracy or precision than an input having a standard bit length and / or data type, and a higher performance than an input having an arbitrary bit length and / or data type, as an arbitrary bit length and / or data type can require specialized software and / or hardware to use the numbers. Additionally, the internal structure of the systolic array can be a superset of the components for each supported data type. For example, the internal structure of the systolic array can support a standard significand bit length from A to B and a standard exponent bit length from X to Y. Thus, the maximum internal supported bit length of the array can be 1 + B + Y, where B and Y can be any number. Additionally, 1 + B + Y can not correspond to a standard format (e.g., 1 + B + Y can correspond to a 22-bit format), but the reducer can be able to reduce to this format for input to the array. Thus, while the set of data types and / or bit lengths supported by the systolic array can be exposed to the customer, the reduced format (e.g., an intermediate bit length between an arbitrary bit length and a standard bit length) can not be exposed to the customer and can correspond to the maximum format (e.g., bit length) supported by the systolic array. This can enable an increased accuracy relative to an input having a standard bit length and an increased performance relative to an input having an arbitrary bit length.

[0027] As disclosed herein, each reducer (e.g., bit-reducer, zeroer, etc.) assigned to a particular row of the systolic array can reduce one or more inputs provided to the reducer (e.g., change one or more bits to zero) and output one or more reduced inputs based at least in part on the one or more inputs. The inputs provided to the reducer can be numbers represented by a significand and an exponent. For example, the inputs provided can be in floating-point format. The one or more reduced inputs can be represented in a modified format that reduces the significand and extends the exponent. The reduced inputs can have a sign bit, an exponent bit, and a significand bit. The most significant bit of the significand bit can be implicit or hidden. Each reducer can include one or more of the following: a rounder, an exponent extender, a trailing bit reducer, and a multiplexer. The reducer can adjust the inputs provided to the reducer by maintaining the exponent of the original input and reducing the significand of the original input. The reducer can utilize a rounder to round the reduced input generated by the reducer based on the unreduced number. In some embodiments, the inputs can be pre-rounded to a given precision (e.g., the number of bits supported by a single pass computation) and the reducer can discard the resulting trailing zeros to generate the reduced input. The rounder can round the inputs using various rounding techniques (e.g., any standard rounding technique). Additionally, the reducer can utilize an exponent extender to extend the number of bits of the exponent portion of the number and a trailing bit reducer to reduce the number of bits of the significand portion of the number. Each reducer can contain any combination of these components. Each reducer can utilize the components contained in the reducer to produce the reduced input and provide the reduced input to the systolic array or processing elements of the systolic array. By producing the reduced input, the reducer is enabled to reduce or adjust an arbitrary bit length (e.g., an arbitrarily long bit length) to a bit length supported by the processing elements of the array during a single pass computation with a loss of precision of the original input having the arbitrary bit length.

[0028] A reducer can cause a reduction in precision (e.g., corresponding to discarded bits) by discarding bits and providing a single pass through the systolic array. For example, a final output can be a reduced output equal to a reduced weight multiplied by a reduced input data element. This precision can be regained by implementing an additional pass through the array. For example, the reducer can convert the weight into a high-reduced weight and a low-reduced weight, and convert the input data element into a high-reduced input data element and a low-reduced input data element. Additionally, the final output can include greater precision, and can be equal to the low-reduced weight multiplied by the low-reduced input data element plus the low-reduced weight multiplied by the high-reduced input data element plus the high-reduced weight multiplied by the low-reduced input data element plus the high-reduced weight multiplied by the high-reduced input data element. While multiple passes can require a reduction in speed (e.g., based on multiple passes through the array to obtain a single total output), the multiple passes can provide a significant increase in precision compared to a single pass that reduces precision. Thus, the systolic array can be able to utilize hardware that natively supports a maximum bit length that is lower than a higher bit length by receiving inputs from the reducer. Each reducer assigned to a particular row of the systolic array can receive a particular input data element and / or weight, and generate multiple reduced inputs from the received input for multiple passes (e.g., into) of the original input through the systolic array. For example, the reducer can receive an input data element and generate multiple reduced input data elements based on the input data element in order to maintain a higher precision of the original input data element compared to reducing the input to a standard bit length. The multiple reduced inputs can be summed to generate an input. It should be understood that each input can be converted into any number of reduced inputs. The reducer can generate the reduced inputs as a first reduced input (e.g., a high-reduced input) and a second reduced input (e.g., a low-reduced input). The first reduced input can be based on a higher-valued significant digit of the input, and the second reduced input can be based on a lower-valued significant digit. For example, the first reduced input can be based on the left-most digit of the significant number (e.g., the digit with the highest value), and the second reduced input can be based on the right-most digit of the significant number (e.g., the digit with the lowest value). Additionally, the significant number of the input can be divided between the first reduced input and the second reduced input. For example, for a 23-bit significant number, the first reduced input can be based on the first 11 digits of the significant number read from left to right (e.g., digit 22 to digit 12), and the second reduced input can be based on the next 12 digits of the significant number read from left to right (e.g., digit 11 to digit 0).

[0029] The reducer can generate a first reduced input by zeroing a number of low bits of the original input. Additionally, the reducer can generate a second reduced input by zeroing a number of high bits of the original input. In some embodiments, the reducer can determine that the input is a normal (e.g., not subnormal or denormal) number by removing implicit leading bits and renormalizing the reduced significand (e.g., the significand after the number of leading bits is zeroed). The reducer can renormalize the input by shifting the significand a number of bits based on the number of leading zeros. For example, the leading one in the reduced significand can be shifted to the implicit bit position. The reducer can also adjust the exponent based on the number of bits the reducer shifted. Since adjusting the exponent can result in the exponent being outside the range of the current exponent, the reducer can extend the exponent (e.g., from 8 bits to 9 bits) so that the adjusted exponent can be represented by the extended exponent. For example, the range of an 8-bit exponent can achieve exponent values between -126 and +127, and by extending the exponent to a 9-bit exponent, the reducer can achieve exponent values between -254 and +255. Since renormalizing a 32-bit input can require an exponent as low as -149 (-126 - 23) to allow for a shift across all 23 bits of the significand (e.g., where the exponent is “00000000” and the significand is “00000000000000000000001”), the reducer can extend the 8-bit exponent of the input to generate the second reduced input. The reducer can extend the exponents of the first reduced input and the second reduced input. In some embodiments, the reducer can only extend the exponent of the second reduced input.

[0030] Each of the first reduced input and the second reduced input can be represented in a reduced (e.g., compressed) format (e.g., 21-bit length). One or more reducers can generate reduced inputs for input data elements and weights. The one or more reducers can also provide each combination of the reduced inputs to the systolic array for multiply-accumulate operations. The systolic array can implement multiple passes of multiply-accumulate operations for combinations of the reduced inputs to generate a total output. For example, multiply-accumulate operations can be performed on a first reduced weight and a first reduced input data element, a first reduced weight and a second reduced input data element, a second reduced weight and a first reduced input element, and a second reduced weight and a second reduced input data element. For example, a final output can be equal to the first reduced weight multiplied by the first reduced input data element plus the first reduced weight multiplied by the second reduced input data element plus the second reduced weight multiplied by the first reduced input data element plus the second reduced weight multiplied by the first reduced input data element. An adder can sum the outputs of each multiply-accumulate operation (e.g., each partial multiply-accumulate operation) to generate a total output. By generating multiple reduced inputs (e.g., inputs having a reduced bit length) from an input (e.g., an input having an arbitrary bit length), the systolic array can be able to perform multiply-accumulate operations on the input (multiple reduced input versions of the input) without needing to support an arbitrary bit length of the input. The systolic array can have certain frequency constraints, size constraints, etc. in order to maintain performance goals. In view of these limitations, a traditional systolic array can not be able to support an arbitrary bit length. By generating multiple reduced inputs for a particular input, the systolic array can meet these constraints while generating an output based on an input having an arbitrary bit length. It should be understood that any number of reduced inputs can be generated from an original input. For example, a 64-bit floating point number can be converted into 5 21-bit reduced floating point numbers. Each of the reduced inputs can correspond to a portion of a significand portion of the original input. For example, a first reduced input can correspond to a first portion of the significand portion of the original input, a second reduced input can correspond to a second portion of the significand portion, a third reduced input can correspond to a third portion of the significand portion, etc. The particular portion of the significand portion of the original input for a particular reduced input can be identified by zeroing out other portions of the significand portion.

[0031] In some embodiments, a reducer can include or receive a signal from a multiplexer that selects between two or more inputs based on a control signal such as an opcode or a data type indicator. For example, a multiplexer can identify a particular input (e.g., a weight or an input data element) for reduction.

[0032] In some embodiments, the systolic array can have separate reducers that receive one of the input data elements or weights and provide a corresponding reduced version of that input to the systolic array. Each processing element in the initial column of processing elements of the systolic array can receive multiple reduced inputs from one or more reducers. For example, a first processing element of the initial column can receive a reduced input data element from a first reducer and a reduced weight from a second reducer, and a second processing element of the initial column can receive a reduced input data element from a third reducer and a reduced weight from a fourth reducer.

[0033] Each reducer can reduce the bit length of 16-bit numbers, 32-bit numbers, or any bit length. For example, a reducer can reduce the bit length of a 32-bit floating point number to a 22-bit floating point number. In one embodiment, a 32-bit floating point number has 1 bit sign, 8 bits exponent, and 23 bits significand. From this 32-bit floating point number, a reducer can generate a reduced 20-bit floating point number with 1 bit sign, 8 bits exponent, and 11 bits significand. In some embodiments, a reducer can increase the bit length of the exponent of an input in order to adjust the format of the reduced input to a format supported by the processing elements. For example, a reducer can increase the exponent from 8 bits to 10 bits. In some embodiments, to reduce the bit length of a particular number, a reducer can reduce the number of trailing bits of the significand of the number (e.g., the reducer can zero out the lower bits of the significand of the number). For example, a number can be the binary string "10101010101111111111111," and a reducer can zero out the twelve trailing bits of the number to generate the reduced binary string "10101010101000000000000" and / or "10101010101."

[0034] Each reducer can also round the resulting reduced input to the systolic array. The reducer can round the reduced input to a particular precision or number of bits supported by the processing elements of the systolic array. For example, the reducer can round a number to generate a rounded number. By rounding the input to the systolic array, the systolic array can obtain a higher accuracy systolic array computation result. In some embodiments, the reducer can round the reduced input. In other embodiments, the reducer can receive a rounded input (e.g., an input rounded by a separate system) and reduce the rounded input. The rounding can include one or more of random rounding, rounding to the nearest even number, rounding to zero, rounding down, or rounding up. Additionally, a user, system, etc. can specify a rounding method for rounding the input (e.g., via a selection from a user interface).

[0035] The systolic array can have PEs that each include a 22-bit multiplier and a 34-bit adder. The 22-bit multiplier can operate on a 22-bit reduced floating point number reduced from a 32-bit floating point number by a reducer to generate a multiplier product having one sign bit, ten exponent bits, and 23 significand bits. The multiplier product can include 24 significand bits, with the most significant bit being implicit or hidden. The 34-bit adder can operate on 34-bit numbers (e.g., 34-bit multiplier products). Additionally, the adder can operate on 35-bit numbers, with one bit being implicit or hidden. In some implementations, the systolic array can include n-bit multipliers and m-bit adders, where n can be any number and the n-bit multipliers and m-bit adders can operate on x-bit reduced floating point numbers. The variables n, m, x, and y can be any number, where n is greater than x.

[0036] In the following description, various examples will be described. For purposes of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the examples. However, it will also be apparent to one skilled in the art that the examples can be practiced without the specific details presented. Furthermore, well-known features can have been omitted or simplified in order not to obscure the examples being described.

[0037] FIG. 1A An example 4x4 systolic array 100A is shown. The systolic array 100A illustratively includes four columns of PEs and four rows of PEs, with four PEs in each row and four PEs in each column. It should be understood that the systolic array 100A is simplified for purposes of description and that a systolic array 100A according to the present disclosure can include any number of PEs in each row and each column. Additionally, the number of PEs in each row can be different than the number of PEs in each column. It should further be understood that such a systolic array 100A can be logically organized into any number of rows and any number of columns. Additionally, the number of rows can be different than the number of columns. The systolic array 100A can be part of a neural network processor in a computer system. For example, the computer system can provide a multi-tenant computing service for data processing applications, such as an image recognition service, text-based data processing (e.g., processing of search queries), audio or video data processing, etc.

[0038] Each PE can include a respective row input bus 102, a respective column input bus 104, a respective column output bus 106, and a respective row output bus 108. The PE can receive input from a left-side PE of the same row (or from an external circuit) via the row input bus 102. The PE can also receive input from a PE above the same column (or from an external circuit) via the column input bus 104. The PE can perform an arithmetic computation based on the inputs, and transmit a result of the arithmetic computation to a PE below the same column (or to an external circuit) via the column output bus 106. The PE can also forward the input received via the row input bus 102 to a right-side PE of the same row via the row output bus 108.

[0039] The systolic array 100A can perform arithmetic computations, including multiplication and addition operations, for processing elements of a neural network. For example, each PE can include an arithmetic unit such as a multiplier and an adder. In some embodiments, the multiplier and the adder can be a fused multiplier-adder. In FIG. 1A In an example, each row of PEs can process one set of input data, and each column of PEs can generate one set of output data based on the set of input data received by each PE in a given column.

[0040] The column 112 of PEs (the leftmost column) can receive four sets of input data, where each set of input data is processed by a row of PEs. The column 116 of reducers can provide the column 112 of PEs with four sets of reduced input data, where each set of input data is provided by a reducer, which can improve the overall performance of the array compared to a conventional array. It should be understood that the column 116 of reducers can provide the column 112 of PEs with any number of sets of reduced input. For example, the number of reducers and / or the number of sets of reduced input can be based on the number of PEs in a given column. In FIG. 1A In an example, the column 112 of PEs includes four PEs (PE 112a, PE 112b, PE 112c, PE 112d), and the column 116 of reducers includes four corresponding reducers (reducer 116a, reducer 116b, reducer 116c, reducer 116d). It should be understood that the column 116 of reducers can include any number of reducers. Each reducer in the column 116 of reducers can provide a particular PE in the column 112 of PEs with a set of reduced input data, where each set of reduced input data includes two or more reduced inputs. For example, the reducer 116a can provide the PE 112a with a reduced input data element and a reduced weight. Each reducer in the column 116 of reducers can convert an input into a reduced input. For example, the reducer 116a can convert a 32-bit input data element into a reduced 22-bit input data element.

[0041] Each reducer in the column 116 of reducers can further select a reduced input to provide to each PE in the column 112 of PEs. For example, each reducer in the column 116 of reducers can include a multiplexer to select a reduced weight or a reduced input data element to provide to a PE. In some embodiments, each reducer 116a-d can be implemented as multiple reducers (e.g., a first reducer and a second reducer). Additionally, the first and second reducers can provide one or more inputs to the column 112 of PEs. For example, a first reducer of the reducer 116a can provide a reduced input data element to the PE 112a, and a second reducer of the reducer 116a can provide a reduced weight to the PE 112a. In some embodiments, a PE can receive a reduced input (e.g., a reduced input data element) and a non-reduced input (e.g., a non-reduced weight) for an arithmetic operation.

[0042] Each PE in the column 112 can obtain a reduced input data element and a reduced weight from a corresponding input data set received via the row input bus 102. Each PE in the column 112 can multiply the reduced input data element with the reduced weight to generate a scaled input. The scaled inputs generated by the PEs within any column, including the column 112, can be accumulated by an adder of each PE. For example, the PE 112a (of the column 112) can generate a first scaled input (from the first input data set), where the first scaled input can be based on an output of the adder. For example, the adder can generate a first output partial sum, and the PE 112a can generate the first scaled input based at least in part on the first output partial sum. The PE 112a can transmit the first scaled input as a partial sum to the PE 112b via the column output bus 106. The PE 112b can also generate a second scaled input (from the second input data set) and add the second scaled input to the partial sum. The updated partial sum accumulated with the first and second scaled inputs is then transmitted to the PE 112c via the column output bus 106. The partial sum is updated and propagated across the column 112, and the PE 112d can generate a sum of the scaled inputs from the four input data sets.

[0043] The sum generated by the PE 112d can correspond to an output data set, and can be fed back to the leftmost PE after passing through an activation function. Additionally, each PE in the column 112 can also propagate the input data sets to other columns of PEs (e.g., the column 114), which can scale the input data sets using a different set of weights. Each column of PEs can perform arithmetic operations (multiplication and addition) to generate output data elements for other processing elements in parallel. In FIG. 1A In the example of the systolic array 100A, the systolic array 100A can generate output data elements for four PEs corresponding to the four columns of the systolic array 100A.

[0044] Systolic array 100A can perform convolution computations in multiple waves. In one embodiment, a wave represents a stream of input data elements processed while reusing the same weights in systolic array 100A. For example, the respective weights can have been preloaded in each PE in systolic array 100A sequentially or in parallel before starting the wave computation. The partial sums generated by the PEs can correspond to a single wave. As the PEs of systolic array 100A perform the arithmetic operations of the convolution computation, the dynamic power consumed by all the multipliers in the PEs can be significant. For a systolic array including a large number of PEs (e.g., thousands), this problem can be further exacerbated. The arithmetic operations performed by the PEs are further explained with reference to FIG. 2A and FIG. 2B .

[0045] As described above, an input can be reduced to generate a reduced input provided to a systolic array. In addition, for multiple single reduced-precision computations that can be combined into a higher-precision computation, an input can be reduced into multiple reduced inputs. A systolic array can include an aggregator in order to combine partial outputs into a higher-precision output (e.g., a higher-precision output relative to a single-pass computation). FIG. 1B An exemplary configuration of an eight-PE column 120 within a systolic array 100B is shown. Array 100B can be similar to array 100A of FIG. 1A , but illustratively includes 8 rows and one column. Specifically, as shown in FIG. 1B , an input can be converted into multiple reduced inputs, and each PE can perform a multiply-accumulate operation on each combination of reduced inputs and provide a partial output partial sum to a corresponding neighboring PE. By varying the number of reduced inputs, the number of partial output partial sums generated and the number of multiply-accumulate operations can similarly be varied. Thus, each higher-bit-length input can be converted by a reducer for a systolic array into any number of reduced inputs having a lower bit length in order to satisfy the bit lengths natively supported by the systolic array.

[0046] To facilitate the total output sum of a computation column, FIG. 1B column 120 includes an aggregator 130. Aggregator 130 can be located within or outside of array 100B. Aggregator 130 can store and sum partial outputs for each pass of the array for a given input (e.g., each combination of reduced inputs associated with a particular input). Aggregator 130 can add the partial sums generated for each combination of reduced inputs. Aggregator 130 can compute a running sum for the output (e.g., by iteratively adding the partial output sums for a given set of reduced inputs) as the total output sum. For example, aggregator 130 can include a partial sum buffer 132.

[0047] In some implementations, the systolic array can identify a particular order to pass the reduced inputs and reduced weights through the array. For example, the reduced inputs and reduced weights can be passed through the array first in order to preserve accuracy of numbers with lower magnitudes. Thus, the reduced inputs with lower magnitudes can be accumulated first in order to preserve accuracy. For example, the product of a low reduced input data element and a low reduced weight can be added to the product of a high reduced input data element and a low reduced weight (or a low reduced input data element and a high reduced weight) to generate a first partial output. Additionally, the first partial output can be added to the product of a low reduced input data element and a high reduced weight (or a high reduced input data element and a low reduced weight) to generate a second partial output. Additionally, the second partial output can be added to the other of the product of a low reduced input data element and a high reduced weight or the product of a high reduced input data element and a low reduced weight to generate a third partial output. The third partial output can be added to the product of a high reduced input data element and a high reduced weight to generate a total output. By adding the reduced inputs with lower magnitudes first, the precision of the reduced inputs can be maintained in order to minimize loss of precision of the low reduced inputs when added with the high reduced inputs.

[0048] While FIG. 1B The aggregator 130 is shown to provide a pair-wise sum, but the aggregator 130 can instead implement other aggregation techniques. In some implementations, the column 120 of PEs can not include the aggregator 130 and can provide an output data set composed of the partial sums for each combination of reduced inputs. In one implementation, the column 120 can not include the aggregator 130 and the column 120 can provide multiple partial output data sets. In some implementations, the multiple output data sets can each correspond to a partial sum generated for each combination of reduced inputs for the column 120. In another implementation, the aggregator 130 can provide more or fewer output data sets. The aggregator 130 can provide one or more output data sets each corresponding to one or more partial sums. In some cases, the output of the aggregator 130 can be configurable depending on the desired use of the array, and thus can accept instructions as to what output should be provided. In some cases, the aggregator 130 can provide a combination of the outputs described above (e.g., by providing four partial sums corresponding to each combination of reduced inputs and a final sum of the non-reduced inputs). In some implementations, a portion of the aggregation of the partial sums can occur within the systolic array. For example, the systolic array can add (using one or more components) a first partial sum to a second partial sum to generate a third partial sum, and can add a fourth partial sum to a fifth partial sum to generate a sixth partial sum. Additionally, the systolic array can provide the third partial sum and the sixth partial sum for accumulation to the aggregator 130.

[0049] FIG. 2AA PE 00 in a systolic array for neural network computation is shown in accordance with certain embodiments of the disclosed technology. The PE 00 can be a portion of a systolic array similar to the systolic array 100A in FIG. 1A FIG. 4A and FIG. 4B Additional details of the reducers 225, 227 of the systolic array 200 are shown. Some embodiments can be described with reference to a neural network, however, it should be understood that certain embodiments can be used for other applications, such as pattern recognition, image processing, audio processing, video processing, etc., without departing from the scope of the technology. FIG. 2A

[0050] The systolic array 200 includes reducers 225, 227 and a plurality of processing elements including the PE 00 and the PE 01. The PE 00 can include one or more of the following: a data element load generator 202, an input data element register 204, a weight register 206, a multiplier 208, an adder 210, a skip computation generator 212, a skip computation register 214, a selector circuit 216, an input portion and register 218, a cached weight register 220, and an operation decoder 256. According to some embodiments, the PE 00 can receive one or more of the following: a reduced input data element 222, a reduced weight 224, a zero data element indicator 226, a zero weight indicator 228, an operation code 230, a weight load 232, and an input portion and 234 to perform a convolution computation.

[0051] The PE 00 can be connected to a first reducer 225 and a second reducer 227. The first reducer 225 can receive a first input, such as the input data elements 221, and the second reducer 227 can receive a second input, such as the weights 223. The first reducer 225 can convert the first input into a first reduced input, and the second reducer 227 can convert the second input into a second reduced input. The first reducer 225 can provide the reduced input data elements 222 (e.g., a reduced version of the input data elements 221) to the PE 00. Additionally, the second reducer 227 can provide the reduced weights 224 (e.g., a reduced version of the weights 223) to the PE 00. In some embodiments, one or more of the first reducer 225 or the second reducer 227 can round the inputs and / or the reduced inputs. The rounding can be based on a rounding method identified by the system, a user, etc. (e.g., a user input can specify a particular rounding method). In other embodiments, one or more of the first reducer 225 or the second reducer 227 can reduce pre-rounded inputs (e.g., the pre-rounded inputs can be rounded by a local or remote system of the systolic array). Additionally, the first reducer 225 and the second reducer 227 can convert one or more floating point inputs into a reduced representation. The floating point inputs can include a bit length of 32 bits, 64 bits, or any bit length.​​

[0052] In some embodiments, one or more of the first reducer 225 or the second reducer 227 can detect when one or both of the input data element 221 and the weight 223 exceeds a particular bit length. For example, the first reducer 225 can determine whether the input data element 221 exceeds 22 bits, and the second reducer 227 can determine whether the weight 223 exceeds 22 bits. Additionally, a user, system, or the like can provide a particular bit length for comparison with the bit length of the input data element 221 and the weight 223. Upon determining that a particular input (e.g., the input data element 221) exceeds the identified bit length, one or more of the first reducer 225 or the second reducer 227 can generate a reduced input (e.g., the reduced input data element 222).

[0053] To reduce the bit length of the input data element 221 and / or the weight 223, the first reducer 225 and / or the second reducer 227 can reduce the bit length of the significand portion having a particular length. The first reducer 225 and / or the second reducer 227 can reduce the bit length of the significand portion to match a maximum bit length of a significand supported by a component of the systolic array (e.g., a multiplier of each processing element). For example, the first reducer 225 and / or the second reducer 227 can reduce the bit length of the significand portion of the input from 23 bits to 11 bits. In some embodiments, the first reducer 225 and / or the second reducer can expand the exponent portion of the input to a particular format required by the multiplier. For example, the first reducer 225 and / or the second reducer 227 can expand the bit length of the exponent portion of the input from 8 bits to 10 bits.

[0054] With the significand portion of one or both of the input data element 221 and the weight 223 having been reduced, the first reducer 225 and the second reducer 227 can still expand the number of bits used to represent the respective exponent portion. Thus, a subsequent arithmetic circuit, such as the multiplier 208, can perform calculations on numbers of a single format (e.g., a 22-bit floating point format).

[0055] PE 00 can receive reduced input data elements 222 via a first input port. The reduced input data elements 222 can be an input data set, or any array of input data elements. PE 00 can receive one reduced input data element from the input data set at a uniform time period. For example, the uniform time period can correspond to a clock cycle. The input data set can be similar to an input feature map that includes input feature map elements. For example, the input data set can correspond to an input image, an audio clip, a video clip, a section of text, or any other data that can be provided for data processing to identify a particular pattern or object. In some cases, the input data set can be an intermediate output data set that has undergone an activation function, such as ReLu or Sigmoid, as discussed with reference to FIG. 1A Each reduced input data element 222 can be a floating point data type or any suitable data type. Each reduced input data element 222 can include 22 bits, 21 bits, 20 bits, or any suitable number of bits. The reduced input data elements 222 can be stored in the input data element registers 204 for a period of time.

[0056] PE 00 can receive reduced weights 224 via a second input port. In some embodiments, the reduced weights 224 can belong to a set of weight values that correspond to a convolutional filter. The reduced weights 224 can be preloaded in PE 00 prior to receiving the reduced input data elements 222. In some embodiments, PE 00 can receive one reduced weight value from the set of reduced weight values at a uniform time period to preload each PE in a given row with the respective reduced weight value. The PE can pass the reduced weight value to the next PE in the respective row until each PE in the given row has been preloaded. Each PE can cache the respective reduced weight value for use in calculations with the reduced input data elements. Each reduced weight 224 can be a floating point data type or any suitable data type. Each reduced weight 224 can include 22 bits, 21 bits, 20 bits, or any suitable number of bits. The reduced weights 224 can be stored in the cached weight registers 220 for a period of time.

[0057] PE 00 can receive an input partial sum 236 for a current operation via a third input port. In some embodiments, the input partial sum 236 can be a 16-bit, 18-bit, 32-bit, 33-bit, 34-bit number, or have any number of bits.

[0058] PE 00 can receive a zero data element indicator 226 for a current operation via a fourth port. The zero data element indicator 226 can include a single bit or multiple bits. The zero data element indicator 226 can indicate (for indicating) whether the reduced input data element 222 is zero. The zero data element indicator 226 can indicate whether the input data element 221 is zero. For example, a value of “1” for the zero data element indicator 226 can indicate that the reduced input data element 222 associated with the zero data element indicator 226 is zero, and a value of “0” for the zero data element indicator 226 can indicate that the reduced input data element 222 associated with the zero data element indicator 226 is not zero. Additionally, “0” can correspond to a logical zero or a logical low, and “1” can correspond to a logical one or a logical high. For example, a logical zero can be represented by a first range of voltage levels (e.g., 0 to 2 volts), and a logical one can be represented by a second range of voltage levels (e.g., 3 to 5 volts). It should be understood that other implementations of representing “0” values and “1” values are possible without departing from the scope of the disclosed technology. The zero data element indicator 226 can be generated by circuitry external to the PE 00 and sequentially passed to all of the PEs in the same row at a uniform time period.

[0059] The PE 00 can receive a zero weight indicator 228 via a fifth port. The zero weight indicator 228 can include a single bit or multiple bits. The zero weight indicator 228 can indicate whether the reduced weight 224 associated with the zero weight indicator 228 is zero. The zero weight indicator 228 can also indicate whether the weight 223 associated with the zero weight indicator 228 is zero. For example, a value of “1” for the zero weight indicator 228 can indicate that the reduced weight 224 is zero, and a value of “0” for the zero weight indicator 228 can indicate that the reduced weight 224 is not zero. The zero weight indicator 228 can be generated by circuitry external to the PE 00 and sequentially passed to all of the PEs in the same row along with the reduced weight 224.

[0060] The weight load 232 can load the reduced weight 224 into the cached weight register 220 to provide a cached weight 246. The weight load 232 can be asserted to cache the reduced weight 224 of the PE 00 in the cached weight register 220 before the reduced input data element 222 is fed into the array. As the weights are shifted into the array to preload each PE with a respective weight value, the weight load 232 can be asserted for each PE at a particular time period in order to preload each PE with the appropriate weight value.

[0061] The operation decoder 256 can decode the operation code 230 to determine the operation performed by the PE 00 for different instructions represented by different operation code values. In some embodiments, a first operation code value can correspond to an instruction to shift a reduction weight from one PE to another PE in the systolic array. A second operation code value can correspond to an instruction to start an arithmetic computation by the PE. For example, once the reduction weight has been preloaded into the systolic array, a reduction input data element can be read from memory and an arithmetic computation can be performed as the reduction input data element passes through the array. A third operation code value can correspond to an instruction to perform a NOP. NOPS can be used to separate two systolic array instructions, or when there is no reduction input data element to read from memory. For example, a NOP can be used to separate an instruction to shift a reduction weight and an instruction to start an arithmetic computation. For example, for a 4x4 array, it can take up to 15 cycles to shift a reduction weight into all the PEs in the array before starting an arithmetic computation, so 15 NOP cycles can be needed. The operation decoder 256 can decode the operation code 230 to generate a NOP 258 and a start computation signal 260. The operation decoder 256 can provide the start computation signal 260 to the weight register 206 connected to the multiplier 208 and the adder 210. The operation decoder 256 can also provide the start computation signal 260 to the multiplier 208. The operation code 230 can include any suitable number of bits, for example, two, four, etc. In some embodiments, the operation decoder 256 can also decode the operation code to determine the data type to provide a data type control signal.

[0062] In some embodiments, the reduction input data element 222, the reduction weight 224, the operation code 230, the zero data element indicator 226, and the zero weight indicator 228 can belong to the row input bus 102, as discussed with reference to FIG. 1A In other embodiments, a splitter (not shown) can be used in the PE 00 to split the row input bus 102 into different internal buses to carry the reduction input data element 222, the reduction weight 224, the operation code 230, the zero data element indicator 226, and the zero weight indicator 228 within the PE 00. For example, the reduction input data element 222 and the reduction weight 224 can belong to a first row input bus, and the operation code 230, the zero data element indicator 226, and the zero weight indicator 228 can belong to a second row input bus.

[0063] The data element load generator 202 can generate a data load signal 242 that can be used to allow the input data element register 204 to skip storage of the reduced input data element 222 under certain conditions. In some embodiments, the reduced input data element 222 can be loaded into the input data element register 204 when the data load signal 242 is asserted based on the zero data element indicator 226 and the NOP 258. The data load signal 242 can be asserted when the zero data element indicator 226 corresponding to the reduced input data element 222 is "0" and the operation code 230 does not indicate a NOP (e.g., the NOP 258 is "0"). The data load signal 242 can be de-asserted when the zero data element indicator 226 corresponding to the reduced input data element 222 or the NOP 258 is "1." The data element load generator 202 can be implemented using OR, NOR, NAND, or any suitable circuitry.

[0064] The input data element register 204 can store the reduced input data element 222 or skip storage of the reduced input data element 222 based on the current operation of the data load signal 242 to provide a stored input data element 244. In some embodiments, the input data element register 204 can store the Din input if the load input is "1," and can save the previous value if the load input is "0." For example, the input data element register 204 can store a new value of the reduced input data element 222 if the data load signal 242 is "1," and can skip storage of the new value of the reduced input data element 222 if the data load signal 242 is "0." Thus, in some cases, the input data element register 204 can only store non-zero values of the reduced input data element 222. According to certain embodiments, skipping storage of the new value by the input data element register 204 can result in not toggling the stored input data element 244 and maintaining the previous value of the stored input data element 244.

[0065] The weight register 206 can store the cached weight 246 to provide the stored weight value 248 based on the start compute signal 260. In some embodiments, the weight register 206 can store the Din input if the load input is “1” and can save the previous value if the load input is “0.” For example, if the start compute signal 260 is asserted (e.g., the start compute signal 260 is “1”), the cached weight 246 can be loaded into the weight register 206, otherwise the weight register 206 can hold the previous value. Thus, the reduced weight 224 previously loaded into the cached weight register 220 using the weight load 232 can be shifted into the weight register 206 at the start of the arithmetic computation. In some embodiments, the stored weight value 248, once loaded at the start of the arithmetic computation, remains constant as input data elements are fed to the PE 00 through the systolic array one element at a time for computation corresponding to one or more waves.

[0066] The PE 00 can provide the stored input data element 244 to the PE 01 based on the current operation data load signal 242. The PE 01 can receive the stored input data element 244 as the reduced input data element 222 via a first port. In some embodiments, the input data element register 204 can store the Din input if the load input is “1” and can save the previous value if the load input is “0.” The PE 00 can provide the stored weight value 248 to the PE 01 based on the start compute signal 260. The PE 01 can receive the stored weight value 248 as the reduced weight 224 via a second port. In some embodiments, the weight register 206 can store the Din input if the load input is “1” and can save the previous value if the load input is “0.”

[0067] The multiplier 208 can perform a multiplication operation between the stored input data element 244 and the stored weight value 248. The multiplier 208 can generate a product 250 based on the multiplication operation. The multiplier 208 can receive inputs having a fixed bit length. For example, the multiplier 208 can receive 22-bit floating point inputs. Thus, the condenser can enable the systolic array to receive inputs having an arbitrary bit length and provide the multiplier 208 with condensed inputs having a bit length supported by the multiplier 208. In some embodiments, the product 250 can be an integer product, a floating point product, or any other product. Additionally, the multiplier 208 can generate the product 250 having 8 bits, 16 bits, 18 bits, 32 bits, 34 bits, or any other number of bits. The multiplier 208 can be implemented using multiplier circuitry. The multiplier 208 can perform floating point multiplication, integer multiplication, or multiplication involving any other data type. The multiplier 208 can be implemented using a 16-bit multiplier data path, an 18-bit multiplier data path, a 22-bit multiplier data path, or a multiplier data path having any number of bits. The multiplier 208 can support at least n-bit operations, where n is greater than or equal to the number of bits in the inputs (e.g., the input data elements).

[0068] The multiplier 208 can include multiple data paths, for example, as discussed in further detail with respect to FIG. 5 The multiplier 208 can include separate data paths for calculating the sign bit, the mantissa, and the exponent, as discussed in further detail with respect to FIG. 2A It should be appreciated that the mantissa data path and the exponent data path can include data having any number of bits.

[0069] The multiplier 208 can provide the product 250 to an adder 210. The adder 210 can perform an addition operation on the product 250 and the stored input partial sum 236 to provide an addition result 238. The adder 210 can be implemented using adder circuitry. The adder 210 can perform floating point addition, integer addition, or non-integer addition. The adder 210 can perform addition on inputs having 8 bits, 16 bits, 18 bits, 32 bits, 34 bits, or any number of bits. The adder 210 can be implemented using a 16-bit adder data path, an 18-bit adder data path, a 32-bit adder data path, a 34-bit adder data path, or an adder data path having any number of bits. In one embodiment, the adder 210 is implemented with a given bit size (e.g., an adder data path of a given bit size) that can represent a maximum bit size of an intended input of the array. In some embodiments, each processing element can include an adder having a larger bit size and a multiplier having a smaller bit size, as an adder having an increased bit size can be more cost efficient than a multiplier having the same increased bit size. Thus, the present disclosure enables a systolic array to support a larger bit size with reduced precision using a multiplier of a lower bit size. In another embodiment, the adder 210 can be implemented with a bit size that is smaller than a maximum bit size of an intended input of the array. The adder 210 can support at least m-bit operations, where m is equal to or greater than the value of the multiplier data path. The adder data path can be a superset of the multiplier data path.

[0070] The multiplier 208 and the adder 210 can provide a fused multiply-add operation. The multiplier 208 and the adder 210 can be integrated together to perform a single step multiply-add operation. In some embodiments, the output of the multiplier 208 can not be rounded before being provided to the adder 210. Additionally, the multiplier 208 can provide an accurate product 250 to the adder 210. In other embodiments, the PE 00 can perform rounding on the output of the multiplier 208.

[0071] The selector circuit 216 can receive the sum result 238, the input partial sum 236, and the stored skip computation indicator 254. The selector circuit 216 can select the sum result 238 or the input partial sum 236 to provide as the output partial sum 240 via the sixth port. In some embodiments, the selector circuit 216 can include at least one multiplexer that can select the sum result 238 or the input partial sum 236 to be produced. The selector circuit 216 can select the sum result 238 or the input partial sum 236 to provide as the output partial sum 240 via the sixth port based on the stored skip computation indicator 254. According to some embodiments, when the value of the reduced input data element 222 or the reduced weight 224 for the current operation is zero, or the NOP 258 is asserted, the sum result 238 can hold the value of the previous operation because the product 250. In this case, the stored skip computation indicator 254 can allow the sum result 238 to be bypassed and the input partial sum 236 to be selected to provide as the output partial sum 240. For example, when the stored skip computation indicator 254 provides a skip computation signal of "1," the input partial sum 236 can be selected as the output partial sum 240 for the systolic cycle, and when the stored skip computation indicator 254 provides a skip computation signal of "0," or the sum result 238 can be selected as the output partial sum 240 for the systolic cycle.

[0072] FIG. 2B The illustrated diagram shows where the shared reducer 225 replaces the first reducer 225 and the second reducer 227 FIG. 2A The shared reducer 225 can receive the input data element 221 and the weight 223. The shared reducer 225 can also receive the opcode 230. The shared reducer 225 can perform a select operation on the input data element 221 and the weight 223 based at least in part on the opcode 230. In some embodiments, the shared reducer 225 will produce a reduced input based at least in part on the opcode 230. For example, when the opcode 230 is a particular value, the shared reducer 225 can reduce the weight 223 and provide the reduced weight 224 to the PE 00. Additionally, when the opcode 230 provides some other set value, the shared reducer 225 can reduce the input data element 221 and provide the reduced input data element 222 to the PE 00. Thus, the shared reducer 225 can reduce the bit length of the significant portion of both the input data element 221 and the weight 223 to match the maximum bit length of the significant of the significant portion supported by the components of the systolic array (e.g., the multipliers of each processing element). In some embodiments, the shared reducer 225 can receive multiple input data elements and / or multiple weights and produce multiple reduced input data elements and / or multiple reduced weights. For example, the shared reducer 225 can produce any number of (e.g., four) reduced input data elements and / or any number of (e.g., four) reduced weights.

[0073] The shared reducer 225 can use a multiplexer to select between the input data element 221 and the weight 223. In some embodiments, the reduced input data element 222 and the reduced weight 224 can be delivered to the PE 00 on separate buses. In other embodiments, the reduced input data element 222 and the reduced weight 224 can be delivered on the same bus. Additionally, the shared reducer 225 can reduce both the input data element 221 and the weight 223 in the same clock cycle and provide the reduced input data element 222 and the reduced weight 224 to the PE 00. In some embodiments, the shared reducer 225 can reduce the weight 223 and provide the reduced weight 224 to the PE 00 during a clock cycle. The shared reducer 225 can then reduce the input data element 221 and provide the reduced input data element 222 to the PE 00 during a second clock cycle.

[0074] FIG. 3 An apparatus 300 including zero detector circuitry for reducing input data elements and reducing weights into a systolic array for neural network computations is shown in accordance with certain embodiments of the disclosed technology.

[0075] The apparatus 300 can include a two-dimensional systolic array 302 including PEs arranged in rows and columns. The systolic array 302 can be similar to the systolic array 100A in FIG. 1A . The first row of the systolic array 302 can include PE 00, PE 01, PE 02,..., PE 0y, the second row of the systolic array 302 can include PE 10, PE 11, PE 12,..., PE 1y, the third row of the systolic array 302 can include PE 20, PE 21, PE 22,..., PE 2y, and the Xth row of the systolic array 302 can include PE x0, PE x1, PEx2,..., PE xy. x and y can include positive integers, such as 32, 64, 128, or any suitable number. Each PE of the systolic array 302 can be similar to the PE 01 and include an apparatus for performing arithmetic computations on reduced inputs using a power efficient method, as discussed with reference to FIG. 2A , FIG. 2B .

[0076] In certain embodiments, the first (e.g., leftmost) PE in each row of systolic array 302 can be coupled to a respective zero-input-data detector circuit to detect zero values on input data elements and to a respective zero-weight detector circuit to detect zero values on weight values entering systolic array 302. For example, PE 00 in the first row can be coupled to first zero-input-data detector 306a and first zero-weight detector 308a, PE 10 in the second row can be coupled to second zero-input-data detector 306b and second zero-weight detector 308b, PE 20 in the third row can be coupled to third zero-input-data detector 306c and third zero-weight detector 308c, and PE x0 in the Xth row can be coupled to Xth zero-input-data detector 306x and Xth zero-weight detector 308x. First zero-input-data detector 306a, second zero-input-data detector 306b, third zero-input-data detector 306c,..., and Xth zero-input-data detector 306x can detect zero values on respective reduced input data elements in input data set 0, input data set 1, input data set 2,..., and input data set x, respectively. Similarly, first zero-weight detector 308a, second zero-weight detector 308b, third zero-weight detector 308c,..., and Xth zero-weight detector 308x can detect zero values on respective reduced weight values in filter 0, filter 1, filter 2,..., and filter x, respectively.

[0077] Each zero-input-data detector and each zero-weight detector in each row of systolic array 302 can be coupled to a respective reducer to receive a reduced input. Each zero-input-data detector can receive a reduced input data element and each zero-weight detector can receive a reduced weight. For example, first zero-input-data detector 306a can be coupled to first reducer 307a and first zero-weight detector 308a can be coupled to second reducer 309a, second zero-input-data detector 306b can be coupled to third reducer 307b and second zero-weight detector 308b can be coupled to fourth reducer 309b, third zero-input-data detector 306c can be coupled to fifth reducer 307c and third zero-weight detector 308c can be coupled to sixth reducer 309c, and Xth zero-input-data detector 306x can be coupled to Xth reducer 307x and Xth zero-weight detector 308x can be coupled to Yth reducer 309x.

[0078] The reducers 307a-307x and 309a-309x can be implemented as separate entities outside the systolic array 302. For example, the reducers 307a-307x and 309a-309x can be part of a circuit separate from the systolic array. In some embodiments, the circuit and the systolic array 302 can be part of a compute engine that can perform arithmetic computations of convolution operations. In other embodiments, the reducers 307a-307x and 309a-309x can be implemented as part of the systolic array 302.

[0079] In some embodiments, the first reducer 307a and the second reducer 309a can be a first shared reducer, and the third reducer 307b and the fourth reducer 309b can be a second shared reducer, and the fifth reducer 307c and the sixth reducer 309c can be a third shared reducer, and the Xth reducer 307x and the Yth reducer 309x can be an Xth shared reducer. Each shared reducer can provide a reduced input data element and a reduced weight. In some embodiments, each shared reducer can include one output bus and can select a reduced input to be produced. In other embodiments, each shared reducer can include multiple output buses and can output a reduced input data element and a reduced weight.

[0080] The zero input data detectors 306a-306x and / or the zero weight detectors 308a-308x can be arranged before the respective reducers 307a-307x, 309a-309x, such that a zero input can be detected, and if a zero input is detected, the respective reducer 307a-307x, 309a-309x can not work to save power. In some embodiments, both the zero input data detectors 306a-306x and the respective reducers 307a-307x can receive the input data set and operate in parallel rather than sequentially. Additionally, both the zero weight detectors 308a-308x and the respective reducers 309a-309x can receive the filter and operate in parallel rather than sequentially.

[0081] Each of the input data set 0, the input data set 1, the input data set 2, and the input data set x can belong to an image, text, a video clip, an audio clip, or another type of data set that can need to be processed by a neural network processor for convolution computations.

[0082] In some cases, input dataset 0, input dataset 1, input dataset 2,..., and input dataset x can be associated with output dataset 0, output dataset 1, output dataset 2,..., output dataset y generated by an intermediate layer of a convolution operation. For example, output dataset 0, output dataset 1, output dataset 2,..., output dataset y can be fed back to systolic array 302 through an activation function and as input dataset 0, input dataset 1, input dataset 2,..., and input dataset x. Filter 0, filter 1, filter 2,..., and filter x can include different sets of weight values to convolve with input dataset 0, input dataset 1, input dataset 2,..., and input dataset x. The weight values in filter 0, filter 1, filter 2,..., and filter x can be predetermined using supervised learning, unsupervised learning, or any suitable method to determine convolution filters.

[0083] Each zero input data detector of a respective row can detect whether a reduced input data element from an input dataset entering the respective row is “0” and generate a corresponding zero input data indicator for the reduced input data element. Additionally, each zero input data detector of a respective row can also detect whether an input data element from an input dataset entering the respective reducer is “0” and generate a corresponding zero input data indicator for the input data element. The corresponding zero data element indicator can be passed into the first PE of the respective row along with the reduced input data element. For example, PE 00 can be the first PE of the first row in systolic array 302. PE 00 can receive a reduced input data element from input dataset 0 before other PEs (e.g., PE 01, PE 02,..., PE 0y) in the first row. In some embodiments, the reduced input data elements can be sequentially fed to PE 00 from input dataset 0 one at a time within a uniform time period. First zero input data detector 306a can generate a zero data element indicator 226 for each input data element from input dataset 0 within each uniform time period (e.g., clock cycle). The zero data element indicator 226 can be sequentially fed to PE 00 within the uniform time period along with each reduced input data element. PE 00 can or can not store the reduced input data element 222 based on the value of the respective data load signal 242. In some embodiments, first zero input data detector 306a can include a comparator to compare the incoming reduced input data element to zero to assert (e.g., set to “1”) or de-assert (e.g., set to “0”) the zero data element indicator 226 based on the value of the incoming reduced input data element. For example, the comparator can be implemented using OR, XOR, NAND, or any suitable circuit.

[0084] Each zero weight detector of a respective row can detect whether a reduced weight from a set of reduced weights entering the respective row is zero and generate a corresponding zero weight indicator for the reduced weight. In addition, each zero weight detector can also detect whether a weight from a set of filters entering the respective reducer is zero and generate a corresponding zero weight indicator for the weight. For example, the first zero weight detector 308a can detect whether a reduced weight from filter 0 (e.g., reduced weight 224) includes a zero value and generate a zero weight indicator 228 for the reduced weight. In some embodiments, the first zero weight detector 308a can include a comparator to compare the reduced weight to zero to assert (e.g., set to “1”) or de-assert (e.g., set to “0”) the zero weight indicator 228. For example, the comparator can be implemented using an OR, XOR, NAND, or any suitable circuit. In one embodiment, the reduced weights from filter 0 can be sequentially fed to PE 00 one at a time over a uniform time period for preloading the respective reduced weights of PE 0y to PE 0x before starting the arithmetic computation. The first zero weight detector 308a can generate a corresponding zero weight indicator for each of those reduced weights, which can be sequentially fed to PE 00 along with the corresponding reduced weight over the uniform time period. PE 00 can sequentially pass the respective reduced weight and the corresponding zero weight indicator to the next adjacent PE until all PEs in the first row are preloaded with the respective reduced weight and the corresponding zero weight indicator. The respective reduced weight and the corresponding zero weight indicator can be cached in each PE before feeding the respective reduced input data element to each row in the systolic array 302.

[0085] The second zero input data detector 306b, the third zero input data detector 306c,..., and the Xth zero input data detector 306x can be similar to the first zero input data detector 306a and can generate respective zero data element indicators similar to the zero data element indicator 226 for sequential provision to PE 10, PE 20,..., and PEx0 over a uniform time period for power optimization. The respective zero data element indicators generated for each row can be received by the respective first PE in each row via the respective row input bus 102 and sequentially propagated by the first PE to all PEs in a given row over the uniform time period. The second zero weight detector 308b, the third zero weight detector 308c,..., and the Xth zero weight detector 308x can be similar to the first zero weight detector 308a and can generate respective zero weight indicators similar to the zero weight indicator 228 for sequential provision to PE 10, PE 20,..., and PE x0 for preloading to each PE in the respective row along with the respective weight value before starting the arithmetic computation.

[0086] In some embodiments, the zero-input data detectors 306a-306x and zero- weight detectors 308a-308x can be implemented as separate entities outside the systolic array 302. For example, the zero-input data detectors 306a-306x and zero-weight detectors 308a-308x can be part of the circuit 304. In other embodiments, the circuit 304 and the systolic array 302 can be part of a compute engine that can perform arithmetic computations of convolution operations. Some embodiments of the disclosed technology can provide reduced gate counts and dynamic power consumption by detecting zeros on input data elements and weights that enter a respective first PE in each row of the systolic array, and passing zero indicators to all PEs in the array, as compared to using a respective zero detector within each PE in the systolic array 302.

[0087] Note that, for ease of illustration, FIG. 3 Only the respective zero data element indicators and zero weight indicators that enter the first PE in each row of the systolic array 302 are shown, however it should be understood that each PE in the respective row of the systolic array 302 can also receive the respective reduced input data elements and the respective reduced weights along with some control signals (e.g., opcode 230, weight load 232, data type, etc.) that can be propagated from the left to the right of the systolic array 302 for each row.

[0088] FIG. 4A An exemplary reduced system 400A (e.g., a 32-bit floating point (“FP32”) reduced system) is shown in accordance with an example implementation. The reduced system 400A includes a multiplexer 402, a round indicator, and a reducer 405. The reducer 405 can reduce an input having an arbitrary bit length to a maximum bit length supported by the elements of the systolic array during a single pass computation. For example, the reducer 405 can reduce an input to a 22-bit input, where 22 bits is the maximum bit length supported by the multipliers of the systolic array. The reducer 405 can include an exponent spreader 406, a rounder 408, and a tail bit reducer 410. In some embodiments, the reducer 405 can include the exponent spreader 406. In other embodiments, the reducer 405 can not include the exponent spreader 406. For example, the reducer 405 can not spread the exponent of an input to generate a reduced input. In some embodiments, the multiplexer 402 can be separate from the reducer 405. In other embodiments, the reducer 405 can include the multiplexer 402. As previously discussed, the reducer 405 processes the original number 401A to produce the reduced number 403A.

[0089] The reduction system 400A can receive one or more numbers to be reduced. The one or more numbers may include one or more of input data elements 221 and / or weights 223. For example, the reduction system 400A can receive FP32 weights and FP32 input data elements. In some embodiments, the reduction system 400A can receive input data elements 221 or weights 223 without a multiplexer.

[0090] Multiplexer 402 may receive one or more numbers received by reduction system 400A. Multiplexer 402 may also receive opcode 230 or other indicators indicating whether weights or input data elements should be selected. Multiplexer 402 may decode opcode 230 to select the number to be operated on by reduction system 400A. Multiplexer 402 may output different numbers for reduction operations based on the value of opcode 230. In some embodiments, a first opcode value may correspond to an instruction to output weight 223 as multiplexer output 420, and a second opcode value may correspond to an instruction to output input data element 221 as multiplexer output 420. For example, once input data element 221 and weight 223 have been provided to reduction system 400A, multiplexer 402 may output input data element 221 at least in part based on opcode 230, and subsequently output weight 223.

[0091] exist FIG. 4A In the example, the raw number 401A is an FP32 number having a sign bit portion, an exponent bit portion, and a significant bit portion. It should be understood that the raw number 401A can be any bit-length number with any exponent bit length and / or significant bit length. The FP32 format of the raw number 401 includes a 1-bit sign, an 8-bit exponent, and 23 bits of significant bits. In some implementations, the raw number 401A may include more, fewer, or different bits. Additionally, the raw number 401A may include more, fewer, or different bits for the sign bit portion, the exponent bit portion, and / or the significant bit portion.

[0092] Exponent expander 406 may receive an 8-bit exponent 428 from the original number 401A. Exponent expander 406 may increase the number of bits representing exponent 428 from 8 bits to 10 bits. In some embodiments, exponent expander 406 may add 1, 2, 3, or any number of bits to exponent 428. The number of bits added may be sufficient to represent the number in the format expected by PE (e.g., PE may expect a 10-bit exponent). In other embodiments, exponent expander may not add any bits to exponent 428. For example, exponent expander 406 (or another component) may determine that a sufficient (e.g., adequate) number of bits are included in exponent 428 and may not expand exponent 428.

[0093] Exponent spreader 406 can spread exponent 428 and preserve the value of exponent 428. Exponent spreader 406 can spread the exponent using range conversion by copying the most significant bit, appending a second inverted copy of the most significant bit, and appending the other bits of exponent 428 to the end of spread exponent 434. For example, if exponent 428 has a value of "10101010," exponent spreader 406 can copy the most significant bit "1," invert the most significant bit once "0," and append the last seven bits "0101010" so that spread exponent 434 is "100101010." In some embodiments, if the exponent starts with leading zeros, exponent spreader 406 can perform a different operation. Additionally, exponent spreader 406 can spread the exponent using range conversion by copying the most significant bit, appending a second copy of the most significant bit, and appending the other bits of exponent 428 to the end of spread exponent 434. For example, if exponent 428 is "00000000," exponent spreader 406 can spread exponent 428 so that spread exponent 434 is "000000000." In some embodiments, exponent spreader 406 can add extra bits of data to any position in the exponent field depending on the byte order format of the exponent and signed or unsigned representation. Thus, exponent spreader 406 can spread exponent 428 to generate spread exponent 434.

[0094] Exponent spreader 406 can provide spread version 434 of the exponent as a 10-bit spread exponent field of reduced number 403A.

[0095] The reducer 405 can also receive a rounding identifier 404. The rounding identifier 404 can identify a type of rounding to be performed by the reducer 405. For example, the rounding identifier 404 can identify a rounding method, such as a random rounding, a rounding to the nearest even number, a rounding to zero, a rounding down, a rounding up, or any other rounding method. A random rounding can include a random rounding to the next larger or smaller number. For example, a random rounding can include a 50% probability of rounding down and a 50% probability of rounding up. Additionally, in a random rounding, the probability of rounding up or down can be based on the relative position of the number to be rounded. For example, a number x between y and z can have a first probability of rounding up to z equal to (x - y) / (z - y) and a second probability of rounding down to y equal to (z - x) / (z - y), where y and z can be any number and x can be any number between y and z. A rounding to the nearest even number can include a rounding to the nearest even number with a particular number of bits, a rounding to zero can include a rounding of a particular number of bits to zero, a rounding up can include a rounding up of a particular number of bits, and a rounding down can include a rounding down of a particular number of bits. The rounding identifier 404 can be provided by a user (e.g., via a user interface), another system, etc. Additionally, the rounding identifier 404 can be a custom rounding identifier or a default rounding identifier.

[0096] The reducer 405 can include a rounder 408 to round the significand 430. The rounder 408 can perform a rounding based on the rounding method identified by the rounding identifier 404. For example, the rounding method can be a random rounding, a rounding to the nearest even number, a rounding to zero, a rounding down, a rounding up, or any other rounding method. The rounder 408 can perform a rounding based on any bit of the significand. Additionally, the rounder 408 can determine a number of bits to be reduced by the trailing bit reducer 410 (e.g., a number of bits to be zeroed) and can initiate a rounding at the bit immediately preceding the bits to be reduced. Additionally, the rounder 408 can round the bits to be reduced by the trailing bit reducer 410. For example, if the significand 430 includes the bits “1110111” and the trailing bit reducer 410 determines that the trailing bit reducer 410 will reduce three trailing bits (e.g., the first three bits read from left to right), the rounder 408 can perform a rounding based on the “0” in position 4. Additionally, if the rounder 408 determines to perform a rounding to zero, the rounder 408 can produce the rounded significand 432 “1110000,” if the rounder 408 determines to perform a rounding up, the rounder 408 can produce the rounded significand 432 “1111000,” etc. In some embodiments, the rounder 408 can be logically located after the trailing bit reducer 410, and the rounder 408 can round the reduced significand.

[0097] The reducer 405 can also include a trailing bit reducer 410 to reduce the bit representation of the rounded significand 432. The trailing bit reducer 410 can receive the rounded significand 432 as an input. The trailing bit reducer 410 can identify a number of bits to reduce from the rounded significand 432. The number of bits to reduce can be based on a difference between the bit length of the rounded significand 432 and a maximum single-pass computation bit length supported by the elements of the systolic array. Additionally, the number of bits can be based on a user input or a system input (e.g., an input identifying a maximum number of bits supported). The number of bits can be a number of trailing bits (e.g., rightmost bits or least significant bits) of the rounded significand 432. For example, if the trailing bit reducer 410 determines that 3 bits should be reduced from the rounded significand 432, the trailing bit reducer 410 can identify 3 bits from right to left in the rounded significand 432. Additionally, the bits can correspond to bit position 0, bit position 1, and bit position 2 within the original number 401A. The trailing bit reducer 410 can identify the bits and zero out the bits (e.g., reduce, eliminate, push into logical zeros). In FIG. 4A In the example of FIG. 4A, the trailing bit reducer 410 identifies that 12 bits should be reduced from the rounded significand 432 and zeroes out the trailing 12 bits of the rounded significand 432. By reducing the bit representation of the rounded significand 432, the trailing bit reducer 410 can generate a reduced significand 436 that includes only the un-reduced (non-zeroed) bits of the significand 430.

[0098] The trailing bit reducer 410 can provide the reduced significand 436 as an 11-bit rounded significand of the reduced number 403A.

[0099] The reduced number 403A can be a second bit length, where the second bit length is any number of bits that is less than the first bit length. In some embodiments, the second bit length can be a maximum bit length supported by the elements of the systolic array. It should be understood that the reduced number 403A can be any arbitrary bit length number having any exponent bit length and / or significand bit length. In FIG. 4A In the example of FIG. 4A, the reduced number 403A can be a 22-bit floating point number having a sign bit portion, an exponent bit portion, and a significand bit portion, while the original number 401A can be a 32-bit floating point number. The reduced number 403A can include a 1-bit sign (e.g., the sign 426), a 10-bit exponent (e.g., the extended exponent 434), and an 11-bit significand (e.g., the reduced significand 436). The reduction system 400A can provide the reduced number 403A as a reduced output 421. The reduced output 421 can be the reduced input data element 222, the reduced weight 224, or any other reduced number.

[0100] FIG. 4BAn exemplary reduction system 400B (e.g., a 32-bit floating point ("FP32") reduction system) is shown in accordance with an example implementation. The reduction system 400B can include a reducer 405 that can reduce an input having an arbitrary bit length to a maximum bit length supported by the elements of the systolic array during a single pass computation. For example, the reducer 405 can reduce the input to a 22-bit input, where 22 bits is the maximum bit length supported by the multipliers of the systolic array. The reduction system 400B includes components similar to the reduction system 400A, except that in FIG. 4B the original number 401B is rounded by a system prior to being provided to the reduction system 400B.

[0101] In the example of FIG. 4B the original number 401B can be an FP32 number having a sign bit portion, an exponent bit portion, and a significand bit portion. It should be understood that the original number 401B can be any arbitrary bit length number having any exponent bit length and / or significand bit length. The FP32 format of the original number 401B includes a 1-bit sign, an 8-bit exponent, and a 23-bit rounded significand. In some implementations, the original number 401B can include any bit number or be associated with any other bit format. The 23-bit rounded significand can be rounded by a system external or internal to the reduction system 400B.

[0102] The reducer 405 can also include a tail bit reducer 410 to reduce the rounded significand 450. The tail bit reducer 410 can receive the rounded significand 432 as an input and reduce the number of bits representing the rounded significand 450 (e.g., from 23 bits to 11 bits). The tail bit reducer 410 can generate a reduced significand 452 that includes only the non-reduced (non-zeroed) bits of the rounded significand 450. Additionally, the tail bit reducer 410 can provide the reduced significand 452 as the 11-bit rounded significand of the reduced number 403B.

[0103] In some implementations, the reduction system 400B can not receive the rounding identifier 404. For example, the rounding identifier 404 can be provided to a system that generates the rounded significand 450 in order to identify the rounding method. The reduction system 400B can provide the reduced number 403B as a reduced output 441. The reduced output 441 can be a reduced input data element 222, a reduced weight 224, or any other reduced number.

[0104] FIG. 4CAn exemplary reduction system 400C (e.g., a 32-bit floating point ("FP32") reduction system) is shown in accordance with an example implementation. The reduction system 400C can include a reducer 405 that can reduce an input having an arbitrary bit length to a plurality of reduced inputs having a maximum bit length supported by the elements of the systolic array during a single pass computation. For example, the reducer 405 can reduce an input to 21-bit inputs, where 21 bits is the maximum bit length supported by the multipliers of the systolic array. The reduction system 400C includes components similar to the reduction systems 400A and 400B, except that in FIG. 4C the reducer 405 converts the original number 401C to the plurality of reduced inputs.

[0105] In the example of FIG. 4C the original number 401C can be an FP32 number having a sign bit portion, an exponent bit portion, and a significand bit portion. It should be understood that the original number 401C can be any arbitrary bit length number having any exponent bit length and / or significand bit length. The FP32 format of the original number 401C includes a 1-bit sign, an 8-bit exponent, and a 23-bit rounded significand. In some embodiments, the original number 401C can include any bit number or be associated with any other bit format.

[0106] The original number 401C as input 454 can be provided to a format detector 456 for normal and / or denormal detection. For example, the format detector 456 can be a denormal detector and / or a normal detector. The format detector 456 can detect whether the input 454 is normal or denormal based at least in part on at least one of a value of the 1-bit sign, a value of the 8-bit exponent, or a value of the 23-bit significand. For example, the format detector 456 can detect a denormal number when the 8-bit exponent contains a zero in each bit and the significand is non-zero. The format detector 456 can provide an enable signal 458 to the normalizer 455 based at least in part on the detection of a normal number. For example, if the format detector 456 detects that the input 454 is normal, the format detector 456 can provide a first value to the normalizer 455. If the format detector 456 detects that the input 454 is denormal, the format detector 456 can provide a second value to the normalizer 455. In some implementations, the first number can be 1 and the second number can be 0. The detection of a normal number can correspond to a logical high and the detection of a denormal number can correspond to a logical zero. In some embodiments, the format detector 456 can detect a normal number by zeroing (e.g., replacing with zeros) the significand 450 and subtracting the original number 401C with the reduced significand 451 from the original number 401C with the zeroed significand to generate a normal identifier. Additionally, the normal identifier can contain an implied leading bit if the original number 401C is normal and can equal zero if the original number 401C is denormal.

[0107] The reducer 405 can provide the 1-bit sign as the 1-bit sign of the reduced number 403C and the reduced number 403D.

[0108] The reducer 405 can also include a trailing bit reducer 410 and a leading bit reducer 453 to reduce the significand 450. The trailing bit reducer 410 and the leading bit reducer 453 can receive the significand 432 as an input and reduce the number of bits representing the significand 450 (e.g., from 23 bits to 11 bits). The trailing bit reducer 410 can generate a reduced significand 452 that includes only the non-reduced (non-zeroed) bits of the significand 450 by removing the trailing (or low) bits of the significand 450. The leading bit reducer 453 can generate a reduced significand 451 that includes only the non-reduced (non-zeroed) bits of the significand 450 by removing the high bits of the significand 450. Additionally, the trailing bit reducer 410 can provide the reduced significand 452 as the 11-bit reduced significand of the reduced number 403C and the leading bit reducer 453 can provide the reduced significand 451 as an input to the normalizer 455.

[0109] As described above, the reducer 405 can also include exponent expanders 406A and 406B to expand the exponent 428. The exponent expander 406A can generate an expanded exponent 434 and can provide the expanded exponent 434 as an exponent of the reduced number 403C, and the expansion expander 406B can provide the expanded exponent 433 as an input to the exponent adjuster 435.

[0110] The reducer 405 can include a normalizer 455 (e.g., a shifter). The normalizer 455 can be implemented based at least in part on an enable signal 458 received from a format detector 456. The normalizer 455 can receive the reduced significand 451 from the leading bit reducer 453. The normalizer 455 can shift the reduced significand 451 based at least in part on a number of leading zeros of the reduced significand 451 as detected by the normalizer 455. The normalizer 455 can further shift the reduced significand 451 such that a first non-zero number is shifted out of the reduced significand 451 and represented with an implicit bit. The normalizer 455 can shift the reduced significand 451 by adding a bit including a logical low or zero to the right or end of the reduced significand 451. The normalizer 455 can produce a shifted significand 452, where the shifted significand 452 can be the same number of bits as the reduced significand 451. For example, if the reduced significand 451 is 00001100000, the normalizer 455 can count four zeros and further adjust the shift count to five, and the normalizer 455 can shift the reduced significand 451 a total of five times and produce the shifted significand 452, which is 10000000000. The normalizer 455 can then provide the shifted significand 452 as a significand portion of the reduced number 403D. In the case where the format detector 456 does not identify that the original number 40 IC is a normal number (e.g., the original number 40 IC is a denormal number), the normalizer 455 can provide the reduced significand 451 as the significand portion of the reduced number 403D. In some embodiments, if the format detector 456 determines that the original number 40 IC is normal, the reducer 405 can calculate a zeroed number by zeroing the significand of the original number 40 IC. Additionally, the reducer 405 can generate the significand of the reduced number 403D by subtracting the reduced significand from the zeroed number. In other embodiments, the reduced number 403D can be determined by subtracting the reduced number 403C from the original number 40 IC.

[0111] Exponent expander 406B can provide an expanded version of the exponent 433 to exponent adjuster 435 (e.g., a subtractor) based at least in part on enable signal 458 (when format detector 456 detects a normal format for the first input) and signal 437 from normalizer 455 identifying the re-normalized significand 452. Exponent adjuster 435 can receive the expanded exponent 433 from exponent expander 406B and a number of leading zeros from normalizer 455. The number of leading zeros can identify the number of leading zeros removed by normalizer 455 to re-normalize the reduced significand 451. Exponent adjuster 435 can subtract a value from expanded exponent 433 based at least in part on the leading zeros output by normalizer 455. Thus, exponent adjuster 435 can compensate for the shift of the exponent value for the significand. For example, if the number of leading zeros output is equal to 5 and the expanded exponent is equal to 000011111 or 31, then exponent adjuster 435 can subtract 5 from 000011111 or 31 such that the adjusted exponent 439 is equal to 000011010 or 26. Exponent adjuster 435 can provide adjusted exponent 439 as the 9-bit expanded exponent field of reduced number 403D. Otherwise, the expanded version of the exponent 433 can be stored as the 9-bit expanded exponent field of reduced number 403D. In some implementations, exponent expander 406B can expand exponent 433 before normalizer 455 normalizes reduced significand 451. In other implementations, exponent expander 406B can expand exponent 433 after or in parallel with normalizer 455 normalizing reduced significand 451.

[0112] Reducing system 400C can provide reduced number 403C and reduced number 403D as reduced inputs 457 and 459 of original number 401C. Reduced inputs 457 and 459 can be reduced input data elements 222, reduced weights 224, or any other reduced numbers.

[0113] FIG. 5 An exemplary multiply-accumulate data path 500 is shown. Exemplary data path 500 can be implemented as described with respect to FIG. 2A and FIG. 2B multipliers 208 and adders 210 discussed. As FIG. 5As shown, multiplier 208 can receive reduced input data element 222 and reduced weight 224 and provide a product to adder 210. Adder 210 can receive the product and input partial sum 234 and provide an addition result 238. By converting the input to a reduced representation before presenting the input to multiplier 208, multiplier 208 can omit support for numbers having a larger bit length (e.g., 32 bits), instead, multiplier 208 can support numbers having a reduced bit length (e.g., 22 bits). Thus, the systolic array can maintain the performance provided by receiving inputs having a shorter bit length by receiving inputs having an arbitrary bit length and adjusting the input to a particular bit length (e.g., a maximum bit length supported by the processing elements of the systolic array).

[0114] Reduced input data element 222 can be a 22-bit number. In some embodiments, reduced input data element 222 can have any bit length and / or can be any number of bits. Additionally, reduced input data element 222 can be a floating point number. In some embodiments, reduced input data element 222 can be a brain floating point number. Additionally, reduced input data element 222 can be a number of any data type. Reduced input data element 222 can be composed of a sign bit field, an exponent field, and a significand field. Multiplier 208 can support different types of reduced input data elements. For example, reduced input data element 222 can include a 1-bit sign, a 10-bit exponent, and an 11-bit significand. Additionally, reduced input data element 222 can include a 1-bit sign, an 8-bit exponent, and an 11-bit significand. Multiplier 208 can support both of these types of reduced input data elements. In some embodiments, reduced input data element 222 can include an x-bit sign, a y-bit exponent, and a z-bit significand, where x, y, and z can be any number. Reduced input data element 222 can be provided to multiplier 208 via first sign data path 511, first exponent data path 521, and first significand data path 531.

[0115] The reduced weight 224 can be 22 bits. In some embodiments, the reduced weight 224 can have any bit length and / or can be any number of bits. Additionally, the reduced weight 224 can be a floating point number. In some embodiments, the reduced weight 224 can be a brain floating point number. Additionally, the reduced weight 224 can be any data type. The reduced weight 224 can be comprised of a sign bit lane, an exponent bit lane, and a significand bit lane. For example, the reduced weight 224 can include a 1 bit sign, a 10 bit exponent, and an 11 bit significand. Additionally, the reduced weight 224 can include a 1 bit sign, an 8 bit exponent, and a 10 bit significand. In some embodiments, the reduced input data element 222 can include an x bit sign, a y bit exponent, and a z bit significand, where x, y, and z can be any number. The reduced weight 224 can be provided to the multiplier 208 via a second sign data lane 512, a second exponent data lane 522, and a second significand data lane 532.

[0116] The multiplier 208 can include a sign data lane, an exponent data lane, and a significand data lane. The multiplier 208 can receive a first sign data lane 511, a first exponent data lane 521, and a first significand data lane 531 from the reduced input data element 222. The multiplier 208 can receive a second sign data lane 512, a second exponent data lane 522, and a second significand data lane 532 from the reduced weight 224. In some embodiments, the multiplier 208 can also receive a data type control signal. The multiplier 208 can perform a multiplication operation on the received inputs.

[0117] The sign data lane of the multiplier 208 can receive the first sign data lane 511 and the second sign data lane 512. The sign data lane can output a partial sign data lane 513 based at least in part on the first sign data lane 511 and the second sign data lane 512. In some embodiments, the sign data lane can be implemented as an exclusive or (XOR) function. The sign data lane can provide the partial sign data lane 513 to the adder 210.

[0118] The exponent data path of multiplier 208 can receive first exponent data path 521 and second exponent data path 522. The exponent data path of multiplier 208 can include adder 526. In some implementations, the exponent data path of multiplier 208 can include a mapper to adjust the output of multiplier 208 to a format expected by one or more components of systolic array (e.g., an adder separate from adder 526). For example, an adder of systolic array can expect an input (e.g., to operate on) having an 11-bit exponent. Additionally, the mapper can receive first exponent data path 521 and second exponent data path 522 and perform a mapping operation to add one or more bits to the exponent of each of reduced input data element 222 and reduced weight 224

[0119] Adder 526 can receive a mapped or unmapped version of first exponent data path 521 and second exponent data path 522. Adder 526 can perform an addition on the two values received from first exponent data path 521 and second exponent data path 522. Adder 526 can also receive shift / carry information (not shown) from significand data path. Adder 526 can provide partial exponent data path 523 based at least in part on the addition performed on the two values. Partial exponent data path 523 can be 10 bits or other range sufficient to hold the exponent sum without overflow.

[0120] The significand data path of multiplier 208 can receive first significand data path 531 and second significand data path 532. The significand data path of multiplier 208 can include binary multiplier 534 and format adjuster 536. Binary multiplier 534 can multiply the value of first significand data path 531 by the value of second significand data path 532. Binary multiplier 534 can generate a multiplier product based on the multiplication operation. In some implementations, the product can be an integer product, a floating point product, or any other product. Additionally, binary multiplier 534 can generate a product having 8 bits, 16 bits, 32 bits, or any other number of bits. The product can have a bit length of a maximum bit length supported during a single pass computation by elements of systolic array. Thus, systolic array can receive inputs of arbitrary inputs and a reducer can reduce to a bit length corresponding to a maximum bit length supported by elements of systolic array (e.g., multipliers of processing elements). Binary multiplier 534 can also perform floating point multiplication, integer multiplication, or multiplication involving any other data type. Binary multiplier 534 can be implemented using a 16-bit multiplier data path, an 18-bit multiplier data path, or a multiplier data path having any number of bits. Binary multiplier 534 can provide the multiplier product to format adjuster 536. In some implementations, binary multiplier 534 can be implemented using multiplier circuitry.

[0121] The format adjuster 536 can adjust the format of the multiplier product produced by the binary multiplier 534. The significand data path of the multiplier 208 can include the format adjuster 536 to adjust the output of the multiplier 208 to a format expected by one or more components of the systolic array (e.g., an adder separate from the adder 526). For example, an adder of the systolic array can expect an input (e.g., to operate on) having 23 bits of significand. The format adjuster 536 can increase or decrease the number of bits used to represent the multiplier product, e.g., by increasing the bit size to 23 bits. The format adjuster 536 can provide the partial significand data path 533 to the adder 210.

[0122] The adder 210 can include a sign data path, an exponent data path, and a significand data path. The adder 210 can be implemented with a given bit size (e.g., with an adder data path of a given size). In some embodiments, each processing element can include an adder having a larger bit size and a multiplier having a smaller bit size, as an adder having an increased bit size can be more cost efficient than a multiplier having the same increased bit size. Thus, the present disclosure enables a systolic array to support a larger bit size with reduced precision using a multiplier of a lower bit size. The adder 210 can receive the partial sign data path 513, the partial exponent data path 523, and the partial significand data path 533 from the multiplier 208. The adder 210 can also receive the input partial sum and 234. The adder 210 can perform an addition operation on the multiplier product consisting of the partial sign data path 513, the partial exponent data path 523, the partial significand data path 533, and the input partial sum and 234. In some embodiments, the adder 210 can perform an addition operation on both floating-point numbers and brain floating-point numbers. Additionally, the adder 210 can be a 34-bit floating-point adder, a 32-bit floating-point adder, or any other bit length adder.

[0123] The adder 210 can generate an addition result 238 based on the addition operation. The addition result 238 can consist of a sign data path 515, an exponent data path 525, and a significand data path 535. In some embodiments, the addition result 238 can be an integer sum, a floating-point sum, or any other sum. Additionally, the adder 210 can generate a sum having 8 bits, 16 bits, 32 bits, 34 bits, or any other number of bits. In some embodiments, the adder 210 can be implemented using binary adder circuitry.

[0124] FIG. 6An apparatus 600 for neural network computation is shown in accordance with some embodiments of the disclosed technology. The apparatus 600 can be part of a computer system (e.g., a host server). For example, the host server can provide multi-tenant computing services for data processing applications such as image recognition services, text-based data processing (e.g., processing of search queries), audio data processing, video data processing, etc. In some embodiments, the host device can operate a software application and communicate with the apparatus 600 to make predictions based on computations utilizing a prediction model of a neural network processor. For example, the host device can use the prediction model to make predictions by identifying information included in an input data set of an image, text, audio, video, etc.

[0125] The apparatus 600 can include a neural network processor 602 coupled to a memory 614, a host interface 616, and a direct memory access (DMA) controller 618 via an interconnect 620. The neural network processor 602 can include a compute engine 604, a compute controller 606, a state buffer 608, an output buffer 610, and an activation engine 612. The neural network processor 602 can provide compute resources to support computation of a prediction model. The neural network processor 602 can be implemented as a system on a chip (SoC), a field programmable gate array (FPGA), or any suitable circuitry.

[0126] The memory 614 can store instructions received from a host device, input data sets (e.g., pixel data of an image), and weights (e.g., weights corresponding to certain visual and / or non-visual features). The memory 614 can also store outputs of the neural network processor 602 (e.g., one or more image recognition decisions made on an input image in the form of an output data set). The memory 614 can include any suitable memory, such as dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate DRAM (DDR DRAM), storage class memory (SCM), flash memory, etc.

[0127] The host interface 616 can enable communication between a host device and the neural network processor 602. For example, the host interface 616 can transmit memory descriptors between the host device and the neural network processor 602, including memory addresses of stored data (e.g., input data sets, weights, computation results, etc.). The host interface 616 can include, for example, a peripheral component interconnect express (PCIe) interface, or any suitable interface for communicating with a host device. The host device can include a host processor and a host memory.

[0128] The DMA controller 618 can perform DMA operations to transfer data between the neural network processor 602 and a host device. For example, as described above, the host device can store instructions, input data sets, and weights in the memory 614. The host device can provide the memory addresses of the stored instructions, data, and weights to the neural network processor 602 (e.g., in the form of memory descriptors). The neural network processor 602 can then obtain the stored instructions, data, and weights based on the memory addresses provided by the host device. The neural network processor 602 can also store the results of the computations (e.g., one or more image recognition decisions) in the memory 614 and provide the memory addresses of the stored results to the host device.

[0129] The state buffer 608 can provide caching of data used for computations at the compute engine 604. The data cached at the state buffer 608 can include, for example, input data sets and weights obtained from the memory 614, as well as intermediate outputs of computations at the compute engine 604. Caching can reduce the impact of memory access bottlenecks (e.g., caused by delays at the memory 614, DMA controller 618, interconnect 620, etc.) on the performance of the compute engine 604. The state buffer 608 can be an on-chip memory device and can include static random access memory (SRAM) or any suitable memory.

[0130] The compute controller 606 can provide control over the various components of the neural network processor 602 to perform neural network computations. In some implementations, the compute controller 606 can read instructions stored in the memory 614 and schedule execution of the instructions by the compute engine 604. In a first implementation, the compute controller 606 can perform scheduling of loading weights into the compute engine 604 prior to reading input data elements from the state buffer 608. For example, as described with reference to FIGS. 2A-2B, the compute controller 606 can provide the opcode 230 and weight load 232 to the compute engine 604 based on instructions received from the host device. The compute controller 606 can provide appropriate values of the opcode 230 to the compute engine 604, which can be decoded by each PE in the compute engine 604 to perform a corresponding operation. For example, the compute engine 604 can use the weight load 232 and opcode 230 to preload weights in all of the PEs in the compute engine 604. Once the weights have been preloaded, the compute controller 606 can perform scheduling of sequentially loading input data elements from the state buffer 608 into the compute engine 604 over a uniform period of time to begin the arithmetic computations. FIG. 2A , FIG. 2B , FIG. 4A and FIG. 4B The compute controller 606 can provide the opcode 230 and weight load 232 to the compute engine 604 based on instructions received from the host device. The compute controller 606 can provide appropriate values of the opcode 230 to the compute engine 604, which can be decoded by each PE in the compute engine 604 to perform a corresponding operation. For example, the compute engine 604 can use the weight load 232 and opcode 230 to preload weights in all of the PEs in the compute engine 604. Once the weights have been preloaded, the compute controller 606 can perform scheduling of sequentially loading input data elements from the state buffer 608 into the compute engine 604 over a uniform period of time to begin the arithmetic computations.

[0131] In a second implementation, the compute controller 606 can perform a schedule to sequentially load the weights and input data elements from the state buffer 608 into the compute engine 604 within a uniform time period. The compute controller 606 can schedule the loading of the weights and input data elements in the respective first PEs of each row of the systolic array 302 using respective row data buses. For example, a respective input data element and weight value can be loaded into the first PEs of a respective row in each cycle.

[0132] In another implementation, the compute controller 606 can schedule the loading of the weights in the systolic array 302 for each row in parallel using respective column data buses for each PE in a given row. For example, the weights for each row can be loaded in parallel in each cycle. In some implementations, the compute controller 606 can determine a data type of the input data set based on instructions received from the host device. The instructions can be in the form of an opcode. The data type can indicate a size and type of the input data elements, e.g., 4-bit, 8-bit, 16-bit, signed, unsigned, or floating point.

[0133] The compute engine 604 can perform computations of the neural network. For example, the compute engine 604 can reduce the inputs provided to the systolic array to generate reduced inputs. Additionally, the compute engine 604 can determine a maximum supported bit length of the systolic array and generate reduced inputs having the maximum supported bit length. In some implementations, the compute engine 604 can include a set of PEs that perform one or more arithmetic operations involved in the neural network computations. Each PE can perform a multiply-accumulate operation using an input data set and an associated weight. For example, the compute engine 604 can include the systolic array 302 and the circuit 304 including the zero input data detectors 306a-306x and the zero weight detectors 308a-308x. In some implementations, the zero input data detectors 306a-306x and the zero weight detectors 308a-308x can be external to the compute engine 604. The compute engine 604 can execute instructions scheduled by the compute controller 606 to sequentially load the weights and input data sets from the state buffer 608 into the compute engine 604.

[0134] In a first implementation, the weights can be preloaded prior to reading the input data sets from the state buffer 608, as described with reference to FIG. 4. A respective zero weight indicator corresponding to each weight can be locally cached in each PE, and the cached value can be used to perform an arithmetic computation of a respective input data element when the input data element is fed into the compute engine 604 with a corresponding zero data element indicator. In a second implementation, the weights and input data sets can be read from the state buffer 608 simultaneously, as described with reference to FIG. 5. In this implementation, the zero weight indicators and zero data element indicators can be locally cached in each PE, and the cached values can be used to perform an arithmetic computation of a respective input data element when the input data element is fed into the compute engine 604 with a corresponding zero data element indicator. FIG. 5The corresponding zero data element indicator and zero weight indicator can be provided by respective zero detector circuits and sequentially propagated from one PE to another PE for a respective row. The weights and input data sets can be obtained from the state buffer 608 using one or more interfaces. In certain embodiments, the compute engine 604 can perform arithmetic computations to reduce the dynamic power consumption of the systolic array 302 using the respective zero data element indicator and zero weight indicator signals as described with reference to FIGS. 2-5. FIG. 5 The respective zero data element indicator and zero weight indicator signals described can be used to reduce the dynamic power consumption of the systolic array 302 and provide the computed results to be stored in the output buffer 610.

[0135] The output buffer 610 can include a set of registers to store the output data sets generated by the compute engine 604. In some embodiments, the output buffer 610 can also implement additional processing, such as, for example, a pooling operation, to reduce the size of the stored output. Additionally, the compute engine 604 can be operable to perform computations for a particular neural network layer, and the output buffer 610 can process the output of that neural network layer and store the processed output data sets at the state buffer 608 (with or without processing by the activation engine 612). The compute engine 604 can use the processed output data sets as intermediate outputs. In some embodiments, the output buffer 610 can include a summer to accumulate partial sums generated for different sets of filters and input data sets to generate a convolution output array. The final output values of the convolution output array stored in the state buffer 608 can be retrieved by the compute controller 606 to store at the state buffer 608.

[0136] The activation engine 612 can apply one or more activation functions (e.g., a ReLu function) on the output of the output buffer 610. For example, the activation engine 612 can include one or more lookup tables (e.g., in the form of multiplexer circuits) that can map an input to one of the candidate outputs representing the result of applying an activation function to the input. In some examples, the activation engine 612 can also include a bypass path to allow the output from the output buffer 610 to be stored directly at the state buffer 608 without applying an activation function.

[0137] FIG. 7A method 700 performed by a compute engine 604 utilizing a systolic array (e.g., a group of processing elements) is shown in accordance with some examples of the disclosed technology. The array can be similar to, for example, array 100A, and include a plurality of PEs similar to, for example, PE 112a. The systolic array can include a plurality of PEs configured into a plurality of rows and / or a plurality of columns. For example, the systolic array can include 65,536 PEs, which are further divided into 256 rows and 256 columns. The compute engine 604 can be a systolic circuit that includes a systolic array and one or more reducers (e.g., converters) to receive inputs having an arbitrary bit length and convert the arbitrary bit length inputs to inputs having a reduced bit length that corresponds to a maximum supported bit length of elements of the systolic array. For example, the one or more reducers can convert a plurality of input data elements (e.g., 32-bit input data elements) to a plurality of reduced input data elements (e.g., 22-bit input data elements) and / or convert a plurality of weights (e.g., 32-bit weights) to a plurality of reduced weights (e.g., 22-bit weights).

[0138] In block 702, a first reducer receives a first input (e.g., a first number) having a first bit length (e.g., 32 bits). The first input bit length can be an arbitrary bit length. The first input can be represented in a floating point format. Additionally, the first reducer can identify a number of trailing bits of the first input and reduce the number of trailing bits of the first input. The first input can represent an input data element. The first reducer can convert a 32-bit floating point number to a 22-bit floating point number. In some embodiments, the first reducer can convert an m-bit floating point number to an n-bit floating point number, where n and m can be any number, where n is less than m.

[0139] In block 704, the first reducer generates a first reduced input having a second bit length (e.g., 22 bits). The second bit length can be a maximum bit length supported by the elements of the systolic array. For example, the first reduced input can be a 22-bit floating point number. Additionally, the second bit length can be less than the first bit length (e.g., the second bit length can be any bit length that is less than the first bit length). The first reducer can generate the first reduced input based on a number of trailing bits of the first input to reduce. To generate the first reduced input (or any other reduced input), the first reducer can include a trailing bit reducer to reduce a number of trailing bits of a significand portion representing the first input and produce a reduced significand portion of the first input (e.g., a 32-bit first input). For example, the trailing bit reducer can zero out the number of trailing bits. Additionally, the first reducer can include a rounder to round the reduced significand portion of the first input based at least in part on a remainder of bits representing a significand portion of the first input that is not included in the reduced significand portion (e.g., a remainder of non-trailing bits of the first input). For example, rounding the first input can include rounding a portion of the bits of the first input. The rounder can further round the first input to a particular number (e.g., a particular floating point number). In some embodiments, the rounder can round the significand portion and the trailing bit reducer can generate the reduced significand portion from the rounded significand portion (e.g., the first input can be a first rounded input to the trailing bit reducer). In other embodiments, the first reducer can not include a rounder and the significand portion can be pre-rounded (e.g., rounded by another system or not rounded). The rounder can round the input based on one or more of a random round, rounding to the nearest even number, rounding to zero, rounding down, rounding up, or any other rounding method. A random round can include rounding the input up to a first number or down to a second number based on probabilities that are tuned based on a relative distance between the input and the first number and a relative distance between the input and the second number, respectively. In some embodiments, the input can be rounded based on a user input (e.g., a selection of a rounding method). The first reducer can also include an exponent spreader to increase a number of bits of an exponent portion representing the first input. In some embodiments, the first reduced input can be stored in a 24-bit format.

[0140] In some embodiments, the first reducer can generate the second input. In other embodiments, the compute engine 604 can include a second reducer to receive the weight in the floating-point format having the first bit length. The second reducer can identify the number of trailing bits of the weight and reduce the number of trailing bits of the weight. Additionally, the second reducer can generate the weight in the floating-point format having the second bit length based on reducing the number of trailing bits of the weight. For example, the second input can be a second 22-bit floating-point number.

[0141] In block 706, each processing element in at least one row of the systolic array multiplies the first reduced input by a second input (e.g., a second number) to generate a multiplier product. In some embodiments, the second input can be the second reduced input. For example, the second input can be the reduced weight. The first reducer can receive the first input and the weight and generate the first reduced input and the second input. Additionally, the first reducer can select the first reduced input or the second input to provide to each processing element. Each processing element can include a multiplier to multiply the first reduced input by the second input. For example, each processing element can include a 22-bit multiplier. Additionally, each processing element can include a multiplier to multiply at least two inputs by a second bit length (e.g., n bits). Additionally, the multiplier can multiply two 22-bit floating-point numbers. The multiplier can include a 1-bit sign data path, an 11-bit mantissa data path, and a 10-bit exponent data path.

[0142] In block 708, each processing element adds the input portion and the multiplier product to generate an adder portion sum (e.g., an addition result). Each processing element can also include an adder to add the input portion and the multiplier product. For example, each processing element can include a 34-bit adder. Additionally, each processing element can include an adder to add at least two numbers (e.g., p bits, where p is greater than n, the number of bits the multiplier receives) having a third bit length. Additionally, the adder can add two floating-point numbers. The adder can include a 1-bit sign data path, a 23-bit mantissa data path, and a 10-bit exponent data path.

[0143] FIG. 8Method 800 performed by a systolic array utilized by computing engine 604 is shown in accordance with some examples of the disclosed technology. The array can be similar to, for example, array 100A, and include a plurality of PEs similar to, for example, PE 112a. The systolic array can include a plurality of PEs configured into a plurality of rows and / or a plurality of columns. For example, the systolic array can include 65,536 PEs, which are further divided into 256 rows and 256 columns. Computing engine 604 can be a systolic circuit that includes a systolic array and one or more reducers (e.g., converters) to receive inputs having an arbitrary bit length and convert the arbitrary bit length inputs into a plurality of reduced inputs having a reduced bit length that corresponds to a maximum supported bit length of elements of the systolic array. For example, the one or more reducers can convert each of a plurality of input data elements (e.g., 32-bit input data elements) into a plurality of reduced input data elements (e.g., 21-bit input data elements) and / or convert each of a plurality of weights (e.g., 32-bit weights) into a plurality of reduced weights (e.g., 21-bit weights).

[0144] In block 802, a systolic array (e.g., a reducer of the systolic array) receives a first input (e.g., an input data element, a weight, etc.) in a floating point format having a first bit length. For example, the first input can be a 32-bit floating point number. The systolic array can also receive a second input (e.g., an input data element, a weight, etc.) for a multiply-accumulate operation. The reducer can convert an m-bit floating point number into one or more n-bit floating point numbers, where n can be any number less than m. For example, the reducer can convert a 32-bit floating point number into two 21-bit floating point numbers.

[0145] In block 804, the systolic array generates a first reduced input (e.g., a high reduced input) having a second bit length. The first reduced input can correspond to a set of most significant bits of a significand portion of the first input (e.g., leading bits of the significand portion of the first input).

[0146] In block 806, the systolic array generates a second reduced input (e.g., a low reduced input) having a third bit length. The second reduced input can correspond to a set of least significant bits of the significand portion of the first input (e.g., trailing bits of the significand portion of the first input). The first reduced input and the second reduced input can total to the first input. Additionally, the second bit length and the third bit length can be less than the first bit length from the first input. For example, the first reduced input and the second reduced input can each be a 21-bit floating point number. Additionally, the reducer can convert input data elements and weights into respective first reduced numbers and second reduced numbers.

[0147] Each of the first and second reduced inputs can be represented in floating point format. In some embodiments, the reducer can generate the first reduced input and subtract the first reduced input from the first input to generate the second reduced input. For example, if the first input includes a first significand "11111111011010101010101", the first reduced input includes a first significand "11111111011", and by subtracting the first reduced input from the first input, the second reduced input can be determined as "010101010101". The first and second reduced inputs can be a maximum supported bit length of the systolic array and / or a particular processing element. In some embodiments, the reducer can include a first sub-reducer to generate the first reduced input. The first sub-reducer can include a trailing bit reducer to reduce a number of trailing bits of a significand portion of the first input to produce a high reduced significand portion. The first sub-reducer can also include a first exponent spreader to increase a number of bits representing an exponent portion of the first input to produce a first increased exponent portion. Based on the first increased exponent portion and the high reduced significand portion, the first sub-reducer can generate the first reduced input (e.g., a high reduced input). In addition, the reducer can include a second sub-reducer to generate the second reduced input. The second sub-reducer can include a leading bit reducer to reduce a number of leading bits of the significand portion of the first input to produce a low reduced significand portion. The second sub-reducer can also include a second exponent spreader to increase a number of bits representing the exponent portion of the first input to produce a second increased exponent portion. Based on the second increased exponent portion and the low reduced significand portion, the second sub-reducer can generate the second reduced input (e.g., a low reduced input). In some embodiments, the second sub-reducer can also include a format detector to detect whether the first input is denormal or normal, a normalizer to remove an implicit bit of the first input and renormalize the low reduced significand portion to produce a normalized significand portion based on a determination that the first input is normal, and an exponent adjuster to adjust the second increased exponent portion based on the renormalized significand portion to produce an adjusted exponent portion. In addition, the second reduced input can include the adjusted exponent portion and the normalized significand portion.

[0148] In block 808, the systolic array performs a plurality of multiply-accumulate operations on the first reduced input, the second reduced input, and the second input. The first input can be an input data element or a weight, and the second input can be the other of the input data element or the weight. In some embodiments, the second input can not be reduced. In other embodiments, the systolic array can reduce the second input to generate a third reduced input and a fourth reduced input for the plurality of multiply-accumulate operations. To perform the plurality of multiply-accumulate operations, the systolic array can compute a plurality of partial sums. Additionally, for each combination of high / low reduced inputs, the systolic array can compute a partial sum. For example, the systolic array can include processing elements to multiply-accumulate the reduced inputs. The processing elements can each include a multiplier to multiply two 21-bit floating point numbers and an adder to add two floating point numbers. Additionally, the multiplier can include a 1-bit sign data path, an 11-bit mantissa data path, and a 9-bit exponent data path, and the adder can include a 1-bit sign data path, a 23-bit mantissa data path, and a 10-bit exponent data path. Additionally, the reducers can produce the reduced inputs and select the reduced inputs to be provided for processing by the processing elements. The plurality of operations can be a plurality of ordered multiply-accumulate operations (e.g., a plurality of multiply operations and a plurality of accumulate operations for the first input). The processing elements can include a multiplier to multiply at least two n-bit numbers and an adder to add two p-bit numbers, where p can be any number greater than n. For example, the multiplier can be a 21-bit multiplier to multiply two 21-bit numbers, and the adder can be a 34-bit adder. Additionally, to perform the operations, the processing elements can multiply the second reduced input by the second reduced weight to generate a first product, multiply the first reduced input by the second reduced weight to generate a second product, multiply the second reduced input by the first reduced weight to generate a third product, multiply the second reduced input by the first reduced weight to generate a fourth product, add the first product to an input partial sum to generate a first sum, add the first sum to the second product to generate a second sum, add the second sum to the third product to generate a third sum, and add the third sum to the fourth product to generate a total product or output.

[0149] The systolic array can generate a full-precision total output from the plurality of partial sums of the first input and the second input (e.g., input data elements and weights) based on the reduced inputs. In some embodiments, to generate the total output, the systolic array can provide each sub-product to an adder (e.g., an accumulator). The adder can perform block-based accumulation on the output of the systolic array (e.g., each of the sub-products).

[0150] To better illustrate the operations of the systolic array with various combinations of reduced inputs, FIG. 9A to FIG. 9HAn exemplary four-PE column 900 of a systolic array for processing neural network computations of multiply-accumulate operations within systolic intervals 0 through 9 according to certain examples of the disclosed technology is shown. The PE column 900 can be a portion of a systolic array similar to the systolic array 100A in FIG. 1A , which can scale any number of rows and any number of columns. In some embodiments, the systolic array can include a full multiply-accumulate operation for each combination of reduced input (e.g., low input / weight and high input / weight), and the output of each operation can be summed.

[0151] The PE column 900 includes four PEs labeled PE00, PE10, PE20, and PE30 according to their row and column (RC) numbers. In the example of FIG. 9A to FIG. 9H , the column 900 is implementing a two-pass multiply-accumulate operation. For example, an input data element can be converted into two reduced input data elements for the multiply-accumulate operation. Weights can be preloaded into the array, and the weights can be used in multiply-accumulate operations for each reduced input to generate an output. In some embodiments, the weights can also be converted into two (or any number of) reduced weights. A first reduced weight (e.g., a low reduced weight) from the weights can be preloaded for multiply-accumulate operations with the reduced input data elements, and a second reduced weight (e.g., a high reduced weight) from the weights can be subsequently loaded for multiply-accumulate operations with the same reduced input data elements. The output of each combination of reduced input and reduced weight can be summed to generate a total output. It should be understood that the column 900 can implement an n-pass multiply-accumulate operation, where n can be any number. For example, the weights can be converted into any number of reduced weights, and each weight can be iteratively loaded into the systolic array for multiply-accumulate operations with a set of reduced input data elements.

[0152] Each PE illustratively includes a multiplier with a single systolic interval delay (e.g., an input provided at interval n is provided as an output at interval n+1) and an adder with a two interval delay (e.g., an input provided at interval n is provided as an output at interval n+2). Adders with other delays can be implemented. As shown, FIG. 9A to FIG. 9H Each PE of the PE column 900 includes a data register Data RegRC for receiving an input data element, a weight storage register Weight RegRC, a multiplier represented by “X,” and an adder or accumulator represented by “+,” respectively.

[0153] Values provided as input partials and provided along the top row are shown at systolic intervals 0 through 9, with PE00 receiving value Al. (Although value Al is shown for illustrative purposes, in some cases all of the partial input and fed to the top row of the array can be set to the same value, which can be zero.) Values provided as input data elements along the left column are shown at systolic intervals 0 through 9, with PE00 in row 0 receiving values Cl and C2 at the times shown, PE10 in row 1 receiving values Dl and D2 at the times shown, PE20 in row 2 receiving values El and E2 at the times shown, and PE30 in row 3 receiving values Fl and F2 at the times shown. Cl, Dl, El, and Fl can each be a first reduced input data element (e.g., a low reduced input data element) and C2, D2, E2, and F2 can each be a second reduced input data element (e.g., a high reduced input data element). Gl, Hl, Il, and Jl can be weights. In some embodiments, the weights can each be converted to a first reduced weight (e.g., a low reduced weight) and a second reduced weight (e.g., a high reduced weight). When no value is shown, a zero or NOP can be assumed. In cases where indicated, for clarity and to facilitate understanding, the system is initialized with zero values. However, other examples can occur in different states and / or with other internal values.

[0154] FIG. 9A to FIG. 9H The progression of data is shown as the multiply-accumulate operation is performed. The multiply-accumulate operation across the illustrated interval includes (as discussed in more detail below): multiplying weight Gl by input data element Cl and accumulating input partial A1; multiplying weight Gl by input data element C2; multiplying weight Hl by input data element Dl and accumulating input partial Xl from PE00; multiplying weight Hl by input data element D2 and accumulating input partial X2 from PE00; multiplying weight Il by input data element El and accumulating input partial Yl from PE10; multiplying weight Il by input data element E2 and accumulating input partial Y2 from PE10; multiplying weight Jl by input data element Fl and accumulating input partial Zl from PE20; and multiplying weight Jl by input data element F2 and accumulating input partial Z2 from PE20. The techniques disclosed herein can be extended to additional sequences of input data elements and input partials.

[0155] FIG. 9A The state of the PE column 900 at systolic interval 0 is shown. The weights Gl, Hl, Il, and Jl are each preloaded into the respective weight registers. For example, the weights Gl, Hl, Il, and Jl can be preloaded in a weight load operation. In PE00, input data element Cl is received to be written and stored in Data Reg00 for use during the next systolic interval. All other input and other states are initialized to zero.

[0156] FIG. 9B The state of PE column 900 at pulse interval 1 is shown. In PE00, input data element C2 is received to be written and stored in Data Reg00 for use during the next pulse interval. In some embodiments, weight G1 can be preloaded into Weight Reg00 for multiple pulse intervals and can not be preloaded again. For example, weight G1 can be preloaded for multiple multiply-accumulate operations with multiple reduced input data elements. Weight G1 can be subsequently replaced with new weight G2 for multiply-accumulate operations with reduced inputs. For example, G1 and G2 can be reduced weights generated from weights. Thus, weight G1 can only be preloaded into the array once. It should be understood that the combination of inputs or weights can be ordered such that either of the reduced inputs or weights can be stored in the respective data registers for multiple pulse intervals and can not be re-read into the PE. For example, the combination of reduced inputs or weights can be ordered or allocated such that weight G1 is not re-read into the PE. Stored input data element C1 is read from Data Reg00 and provided as input to both the multiplier of PE00 and the data registers of the PEs in the subsequent column. The multiplier in PE00 multiplies C1 by G1 to generate multiply result C1xG1, which is provided to the adder of PE00. Input portion and A1 is also received at the adder of PE00. Each adder is pipelined with a 2-interval delay, and thus processes the respective input portion and and the respective multiply result during a time period corresponding to the delay (e.g., the next 2 intervals).

[0157] In PE10, input data element D1 is received to be written and stored in Data Reg10 for use during the next pulse interval.

[0158] FIG. 9C The state of PE column 900 at pulse interval 2 is shown. In PE00, input data element C2 is read from Data Reg00 and provided as input to both the multiplier of PE00 and the data registers of the PEs in the subsequent column. The multiplier in PE00 multiplies C2 by G1 to generate multiply result C2xG1, which is provided to the adder of PE00 for use in the adder operation. Note that during pulse interval 2, the adder of PE00 continues the add operation between multiply result C1xG1 obtained during interval 1 and input portion and A1.

[0159] In PE10, input data element D2 is received to be written and stored in Data Reg10 for use during the next lapping interval. Stored input data element D1 is read from Data Reg10 and provided as input to both the multiplier in PE10 and the data register of the PE in the subsequent column. The multiplier in PE10 multiplies D1 by H1 to generate a multiplication result D1 x H1, which is provided to the adder of PE10.

[0160] In PE20, input data element E1 is received to be written and stored in Data Reg20 for use during the next lapping interval.

[0161] FIG. 9D The state of PE column 900 at lapping interval 3 is shown. In PE00, the adder completes the addition of A1 and C1 x G1 and generates an addition result A1 + C1 x G1. The addition result A1 + C1 x G1 is passed as an input partial sum to PE10. Additional results of PEs within a given column can be generally referred to herein as a“partial sum.” Note that during lapping interval 3, the adder of PE00 continues the addition operation between multiplication results obtained during interval 2, C2 x G1.

[0162] In PE10, stored input data element D2 is read from Data Reg10 and provided as input to both the multiplier in PE10 and the data register of the PE in the subsequent column. The multiplier in PE10 multiplies D2 by H1 to generate a multiplication result D2 x H1, which is provided to the adder of PE10. The input partial sum C1 x G1 + A1 is received from PE00 and also provided to the adder of PE10 for use in the adder operation. Note that during lapping interval 3, the adder of PE10 continues the addition operation between multiplication result D1 x H1 and the input partial sum (A1 + C1 x G1) from PE00.

[0163] In PE20, input data element E2 is received to be written and stored in Data Reg20 for use during the next lapping interval. Stored input data element E1 is read from Data Reg20 and provided as input to both the multiplier in PE20 and the data register of the PE in the subsequent column. The multiplier in PE20 multiplies E1 by I1 to generate a multiplication result E1 x I1, which is provided to the adder of PE20 for use in the adder operation.

[0164] In PE30, input data element F1 is received to be written and stored in Data Reg30 for use during the next lapping interval.

[0165] FIG. 9EThe state of PE column 900 at systolic interval 4 is shown. The adder completes the addition of 0 and C2xG1 and generates an addend result C2xG1. In some embodiments, the input partial sum can be added to each combination of reduced inputs. For example, where each input is converted into two reduced inputs resulting in four combinations of reduced inputs for each weight and input data element (e.g., four pass multiply accumulate operations for a pair of inputs), the input partial sum can be added to each combination of reduced inputs. In other embodiments, a portion of the input partial sum can be added to each combination of reduced inputs. For example, the input partial sum can be divided across each combination of reduced inputs. The addend result C2xG1 is passed as an input partial sum to PE 10.

[0166] In PE 10, the input partial sum C2xG1 is received from PE 00 and is also provided to the adder of PE 10 for use in the adder operation. Note that during systolic interval 4, the adder of PE 10 continues the add operation between the multiply result D2xH1 and the input partial sum (C2xG1) from PE 00.

[0167] Additionally, in PE 10, the adder completes the addition of D1xH1 + C1xG1 + A1 and generates an addend result X1. The addend result X1 is passed as an input partial sum to PE 20.

[0168] In PE 20, the stored input data element E2 is read from Data Reg 20 and provided as input to both the multiplier of PE 20 and the data register of the PE in the subsequent column. The multiplier in PE 20 multiplies E2 by II to generate a multiply result E2xII, which is provided to the adder of PE 20 for use in the adder operation. The input partial sum X1 is received from PE 10 and is also provided to the adder of PE 20 for use in the adder operation. Note that during systolic interval 4, the adder of PE 20 continues the add operation between the multiply result E1xII and the input partial sum (X1) from PE 10.

[0169] In PE 30, the input data element F2 is received for writing and storage in Data Reg 30 for use during the next systolic interval. The stored input data element Fl is read from Data Reg 30 and provided as input to both the multiplier of PE 30 and the data register of the PE in the subsequent column. The multiplier in PE 30 multiplies Fl by Jl to generate a multiply result FlxJl, which is provided to the adder of PE 30 for use in the adder operation.

[0170] FIG. 9FThe state of PE column 900 at pulse interval 5 is shown. In PE 10, the adder completes the addition of D2xH1 + C2xG1 and generates an addition result X2. The addition result X2 is passed as an input partial sum to PE 20.

[0171] In PE 20, the input partial sum X2 is received from PE 10 and is also provided to the adder of PE 20 for use in the adder operation. Note that during pulse interval 5, the adder of PE 20 continues the add operation between the multiplication result E2xI1 and the input partial sum (X2) from PE 10.

[0172] Additionally, in PE 20, the adder completes the addition of E1xI1 + X1 and generates an addition result Y1. The addition result Y1 is passed as an input partial sum to PE 30.

[0173] In PE 30, the stored input data element F2 is read from Data Reg 30 and provided as input to both the multiplier of PE 30 and the data register of the PE in the subsequent column. The multiplier in PE 30 multiplies F2 by J1 to generate a multiplication result F2xJ1, which is provided to the adder of PE 30 for use in the adder operation. Note that during pulse interval 5, the adder of PE 30 continues the add operation between the multiplication result F1xJ1 obtained during interval 4 and the input partial sum (Y1) from PE 20.

[0174] FIG. 9G The state of PE column 900 at pulse interval 6 is shown. In PE 20, the adder completes the addition of E2xI1 + X2 and generates an addition result Y2. The addition result Y2 is passed as an input partial sum to PE 30.

[0175] In PE 30, the adder of PE 30 continues the add operation between the multiplication result F2xJ1 obtained during interval 5 and the input partial sum (Y2) from PE 20.

[0176] Additionally, in PE 30, the adder completes the addition of F1xJ1 + Y1 and generates an addition result Z1. The addition result Z1 can be passed to another PE and / or aggregator for aggregation with additional combinations of reduced inputs of a particular set of inputs.

[0177] FIG. 9H The state of PE column 900 at pulse interval 7 is shown. In PE 30, the adder completes the addition of F2xJ1 + Y2 and generates an addition result Z2. The addition result Z2 can be passed to another PE and / or aggregator for aggregation with additional combinations of reduced inputs of a particular set of inputs.

[0178] may be performed for one or more start input data elements and any number of start input portions and executions FIG. 9A to FIG. 9H Exemplary states of the data flow shown.

[0179] FIG. 10 An example of a computing device 1000 is shown. The functionality and / or several components of the computing device 1000 can be used without limitation with other embodiments disclosed elsewhere in this disclosure. The computing device 1000 can perform computations to facilitate the processing of tasks. As an illustrative example, the computing device 1000 can be part of a server in a multi-tenant computing service system. Various hardware and software resources of the computing device 1000 (e.g., hardware and software resources associated with data processing) can be allocated to clients upon request.

[0180] In one example, the computing device 1000 can include processing logic 1002, bus interface module 1004, memory 1006, and network interface module 1008. These modules can be hardware modules, software modules, or a combination of hardware and software. In some cases, modules can be used interchangeably with components or engines without departing from the scope of the disclosure. The computing device 1000 can include additional modules, which are not illustrated here for ease of illustration. In some embodiments, the computing device 1000 can include fewer modules. For example, one or more of the modules can be combined into one module. One or more of the modules can communicate with each other over a communication channel 1010. The communication channel 1010 can include one or more buses, grids, matrices, fabrics, combinations of these communication channels, or some other suitable communication channel.

[0181] The processing logic 1002 can include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a system-on-a-chip (SoC), and a network processing unit (NPU), a processor configured to execute instructions, or any other circuit for performing logical arithmetic and floating point operations. Examples of processors that can be included in the processing logic 1002 can include those by processors developed by Intel® Corporation, AMD®, and others. In some embodiments, the processors can include multiple processing cores, and each processing core can execute instructions independently of the other processing cores. Additionally, each processor or processing core can implement multiple processing threads executing instructions on the same processor or processing core while maintaining logical separation between the multiple processing threads. Such processing threads executing on a processor or processing core can be exposed to software as separate logical processors or processing cores. In some embodiments, multiple processors, processing cores, or processing threads executing on the same core can share certain resources, such as, for example, a bus, a level 1 (LI) cache, and / or a level 2 (L2) cache. Instructions executed by the processing logic 1002 can be stored, for example, in the form of a computer program on a computer-readable storage medium. The computer-readable storage medium can be non-transitory. In some cases, the computer-readable medium can be part of the memory 1006. The processing logic 1002 can also include hardware circuitry for performing artificial neural network computations, including, for example, a neural network processor 602 or the like.

[0182] A client can be granted access to the processing logic 1002 to provide a personal assistant service requested by the client. For example, the computing device 1000 can host a virtual machine on which an image recognition software application can execute. The image recognition software application, when executing, can access the processing logic 1002 to predict, for example, objects included in an image. As another example, access to the processing logic 1002 can also be granted as part of a bare-metal instance, where an image recognition software application executing on a client device (e.g., a remote computer, a smart phone, etc.) can directly access the processing logic 1002 to perform recognition of an image.

[0183] The memory 1006 can include volatile memory or nonvolatile memory, or both. For example, the memory 1006 can include random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory, and / or some other suitable type of storage media. In some cases, some or all of the memory 1006 can be internal to the computing device 1000, while in other cases, some or all of the memory can be external to the computing device 1000. The memory 1006 can store an operating system including executable instructions that, when executed by the processing logic 1002, provide an execution environment for executing instructions that provide functionality for performing convolution computations for the computing device 1000. The memory 1006 can also store software applications, for example, for performing artificial neural network computations. The memory can also store and maintain several data structures and tables to facilitate the functionality of the computing device 1000.

[0184] The bus interface module 1004 can enable communication with external entities, such as host devices and / or other components in a computing system, over external communication media. The bus interface module 1004 can include physical interfaces for connecting to cables, sockets, ports, or other connections to external communication media. The bus interface module 1004 can also include hardware and / or software to manage incoming and outgoing transactions. The bus interface module 1004 can implement a local bus protocol, such as a Peripheral Component Interconnect (PCI)-based protocol, Non-Volatile Memory Express (NVMe), Advanced Host Controller Interface (AHCI), Small Computer System Interface (SCSI), Serial Attached SCSI (SAS), Serial AT Attachment (SATA), Parallel ATA (PATA), some other standard bus protocol, or a proprietary bus protocol. The bus interface module 1004 can include physical layers for any of these bus protocols, including connectors, power management, and error handling, among others. In some embodiments, the computing device 1000 can include multiple bus interface modules for communicating with multiple external entities. These multiple bus interface modules can implement the same local bus protocol, different local bus protocols, or a combination of the same and different bus protocols.

[0185] The network interface module 1008 can include hardware and / or software for communicating with a network. For example, this network interface module 1008 can include physical connectors or physical ports for wired connection to a network, and / or an antenna for wireless communication to a network. The network interface module 1008 can also include hardware and / or software that implements a network protocol stack. The network interface module 1008 can communicate with a network using network protocols such as, for example, TCP / IP, Wireless Bandwidth, RoCE, Institute of Electrical and Electronics Engineers (IEEE) 802.11 wireless protocols, User Datagram Protocol (UDP), Asynchronous Transfer Mode (ATM), Token Ring, Frame Relay, High-Level Data Link Control (HDLC), Fiber Distributed Data Interface (FDDI), and / or Point-to-Point Protocol (PPP), among others. In some embodiments, the computing device 1000 can include multiple network interface modules, each configured to communicate with a different network. For example, the computing device 1000 can include network interface modules for communicating with wired Ethernet, wireless 1002.11 networks, cellular networks, InfiniBand technology networks, etc. In some embodiments, the computing device 1000 can receive a set of parameters from a server, such as the weight values described above for a convolution computation, through the network interface module 1008.

[0186] The various components and modules of the computing device 1000 described above can be implemented as discrete components, a system on a chip (SoC), ASIC, NPU, FPGA, or any combination thereof. In some embodiments, the SoC or other components can be communicatively coupled to another computing system to provide various services, such as traffic monitoring, traffic shaping, computing, etc. In some embodiments of the technology, the SoC or other components can include a number of subsystems as disclosed herein.

[0187] The modules described herein can be software modules, hardware modules, or suitable combinations thereof. If the modules are software modules, the modules can be embodied on a non-transitory computer readable medium and processed by a processor in any of the computer systems described herein. It should be noted that the described processes and architectures can be performed in real-time or in an asynchronous mode prior to any user interaction. The modules can be configured in the manner suggested by the 3GPP, and / or the functionality described herein can be provided by one or more modules existing as separate modules, and / or the module functionality described herein can be distributed over multiple modules. FIG. 10

[0188] The specification and drawings should be understood to be illustrative only and not restrictive. It will be apparent, however, that various modifications and variations can be made to the disclosed technology without departing from the spirit and scope of the disclosure as set forth in the following claims.

[0189] Other variations are within the spirit of the present disclosure. Thus, while the disclosed technology is susceptible to various modifications and alternative constructions, certain illustrated embodiments thereof have been shown in the drawings and described above in detail. It should be understood, however, that there is no intention to limit the disclosure to the specific form or forms disclosed, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the disclosure, as defined in the appended claims.

[0190] ​The use of the terms "a" and "an" and "the" and similar referents in the context of describing the disclosed embodiments (especially in the context of the following claims) are to be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. The terms "comprising," "having," "including," and "containing" are to be construed as open-ended terms (i.e., meaning "including, but not limited to,") unless otherwise noted. The term "connected" is to be construed as partly or fully inclusive, unless otherwise indicated herein. Unless otherwise indicated herein, the description herein of values of ranges of values is merely intended to serve as a shorthand method of referring individually to each separate value falling within the range, and each separate value is incorporated in the specification as if it were individually recited herein. Unless otherwise indicated herein, all methods described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The use of any and all examples, or exemplary language (e.g., "such as") provided herein, is intended merely to better illuminate embodiments of the application and does not pose a limitation on the scope of the disclosure unless otherwise claimed. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

[0191] Unless specifically stated otherwise, the use of terms such as "at least one of," "one or more of," or "at least one of X and Y" herein is intended to open up the possibility that "none of X and Y" is also possible. For example, the term "at least one of A and B" means including A, including B, or including both A and B. In other words, this expression creates a possibility that A or B or both A and B are present.

[0192] Various embodiments of the present disclosure are described herein, including the best mode known to the inventors for practicing the present disclosure. Variations of those embodiments can become apparent to those of ordinary skill in the art upon reading the foregoing description. The inventors expect skilled artisans to employ such variations as appropriate, and the inventors intend for the present disclosure to be practiced otherwise than specifically described herein. Accordingly, the present disclosure includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise evident to one of ordinary skill in the art.

[0193] Various example embodiments of the present disclosure can be described by the following clauses:

[0194] Clause 1 : A systolic array processor organized in rows and columns, each row comprising:

[0195] a reducer configured to convert 32-bit input data elements to reduced 22-bit input data elements, the reducer comprising:

[0196] a tail bit reducer configured to reduce a number of bits representing a significant portion of the 32-bit input data elements to produce a reduced significant portion of the 32-bit input data elements;

[0197] a rounder configured to round the reduced significant portion of the 32-bit input data elements to produce a rounded significant portion; and

[0198] an exponent spreader configured to increase a number of bits representing an exponent portion of the 32-bit input data elements to produce an increased exponent portion,

[0199] wherein the reducer produces the reduced 22-bit input data elements based on the rounded significant portion and the increased exponent portion; and

[0200] a plurality of processing elements configured to receive the reduced 22-bit input data elements from the reducer and to receive weights for performing multiply-accumulate operations.

[0201] Clause 2: The systolic array processor of Clause 1, wherein the reducer is further configured to convert 32-bit weights to the weights.

[0202] Clause 3: The systolic array processor of Clause 1 or Clause 2, wherein the reducer further comprises a first reducer, each row further comprising:

[0203] a second reducer configured to convert 32-bit weights to the weights.

[0204] Clause 4: The systolic array processor of any one of Clauses 1-3, wherein the rounder is configured to round the reduced significant portion of the 32-bit input data elements based on one or more of:

[0205] a random rounding;

[0206] rounding to the nearest even number;

[0207] rounding to zero;

[0208] rounding down; or

[0209] rounding up.

[0210] Clause 5: A systolic circuit, comprising:

[0211] a group of processing elements arranged in a plurality of rows; and

[0212] a first converter configured to:

[0213] receive a first input in a floating-point representation having a first bit- length;

[0214] identify a number of trailing bits of the first input;

[0215] reduce the number of trailing bits of the first input; and

[0216] generate, based on reducing the number of trailing bits of the first input, a first reduced input in a floating-point representation having a second bit-length, wherein the second bit-length is less than the first bit-length, wherein the second bit-length corresponds to a bit-length supported by the group of processing elements;

[0217] wherein each processing element in at least one row of the group of processing elements is configured to receive the first reduced input from the first converter and receive a second input for performing a multiply-accumulate operation.

[0218] Clause 6: The systolic circuit of Clause 5, wherein each processing element in the plurality of rows of the group of processing elements comprises:

[0219] a multiplier configured to multiply two 22-bit floating-point numbers, wherein the multiplier is comprised of a 1-bit sign data path, an 11-bit significand data path, and a 10-bit exponent data path; and

[0220] an adder configured to add two floating-point numbers, wherein the adder is comprised of a 1-bit sign data path, a 23-bit significand data path, and a 10-bit exponent data path.

[0221] Clause 7: The systolic circuit of Clause 5 or Clause 6, wherein the first input comprises an input data element and the second input comprises a reduced weight, wherein the first converter is further configured to:

[0222] receive the first input and a weight;

[0223] generate the first reduced input and the second input; and

[0224] select the first reduced input or the second input to provide.

[0225] Clause 8: The systolic circuit of any one of Clauses 5-7, wherein the first converter comprises:

[0226] a tail bit reducer configured to reduce a number of bits representing a significand portion of the first input to produce a reduced significand portion of the first input;

[0227] a rounder configured to round the reduced significand portion of the first input based on a remaining portion of the bits representing the significand portion of the first input that is not included within the reduced significand portion; and

[0228] an exponent spreader configured to increase a number of bits representing an exponent portion of the first input.

[0229] Clause 9: The pulsating circuit of any of clauses 5-8, wherein the first input comprises a first rounded input, wherein the first converter comprises:

[0230] a tail bit reducer configured to reduce a number of bits representing a significand portion of the first input to produce a reduced significand portion of the first input; and

[0231] an exponent spreader configured to increase a number of bits representing an exponent portion of the first input.

[0232] Clause 10: The pulsating circuit of any of clauses 5-9, wherein the first reduced input comprises a first reduced rounded input, wherein the first reduced rounded input is rounded based on one or more of:

[0233] a random rounding;

[0234] to the nearest even number;

[0235] to zero;

[0236] downward;

[0237] upward.

[0238] Clause 11: The pulsating circuit of any of clauses 5-10, wherein the first reduced input comprises a first reduced rounded input, wherein the first reduced rounded input is rounded based on a user input.

[0239] Clause 12: The pulsating circuit of any of clauses 5-11, wherein:

[0240] the first converter is configured to convert a 32-bit floating point number to a 22-bit floating point number,

[0241] wherein each of the processing elements comprises:

[0242] 22-bit multiplier; and

[0243] 34-bit adder.

[0244] Clause 13: The pulsatile circuit of any of clauses 5-12, wherein:

[0245] the first converter is further configured to convert an m-bit floating point number to an n-bit floating point number, where n and m can be any positive integer, where n is less than m,

[0246] wherein each of the processing elements comprises:

[0247] a multiplier configured to multiply at least two n-bit numbers; and

[0248] an adder configured to add two p-bit numbers, where p is greater than n.

[0249] Clause 14: The pulsatile circuit of any of clauses 5-13, wherein to reduce the number of trailing bits of the first input, the first converter is configured to:

[0250] set the number of trailing bits to zero.

[0251] Clause 15: The pulsatile circuit of any of clauses 5-14, further comprising:

[0252] a second converter configured to:

[0253] receive a weight in floating point representation having the first bit length;

[0254] identify a number of trailing bits of the weight;

[0255] reduce the number of trailing bits of the weight; and

[0256] generate the second input in floating point representation having the second bit length based on reducing the number of trailing bits of the weight.

[0257] Clause 16: The pulsatile circuit of any of clauses 5-15, wherein the first reduced input is stored in a 24-bit format.

[0258] Clause 17: A method comprising:

[0259] receiving a first input in floating point representation having a first bit length;

[0260] reducing a number of trailing bits of the first input;

[0261] generating a first reduced input in a floating-point representation having a second bit length based on reducing the number of trailing bits of the first input, wherein the second bit length is less than the first bit length, wherein the second bit length corresponds to a supported bit length; and

[0262] receiving the first reduced input and a second input for performing a multiply- accumulate operation.

[0263] Clause 18: The method of clause 17, wherein:

[0264] the first input comprises a 32-bit floating-point number;

[0265] the first reduced input comprises a first 22-bit floating-point number; and

[0266] the second input comprises a second 22-bit floating-point number.

[0267] Clause 19: The method of clause 17 or clause 18, wherein generating the first reduced input comprises:

[0268] rounding the first input based on a remaining portion of non-trailing bits of the first input to generate the first reduced input, wherein the first input comprises a number of bits, wherein rounding the first input comprises rounding a portion of the number of bits.

[0269] Clause 20: The method of any of clauses 17-19, wherein one or more of the first reduced input or the second input comprises a rounded reduced input, wherein the rounded reduced input is rounded based on one or more of:

[0270] random rounding;

[0271] rounding to the nearest even number;

[0272] rounding to zero;

[0273] rounding down; or

[0274] rounding up.

[0275] Various example embodiments of the present disclosure can be described by the following clauses:

[0276] Clause 1 : A systolic array processor organized in rows and columns, each row comprising:

[0277] a reducer configured to convert a 32-bit input data element into two 21-bit input data elements, the reducer comprising:

[0278] a first sub-reducer configured to convert 32-bit input data elements of the 32-bit input data elements to first 21-bit input data elements, the first 21-bit input data elements corresponding to a set of most significant bits of a significand portion of the 32-bit input data elements, the first sub-reducer comprising:

[0279] a trailing bit reducer configured to reduce a number of trailing bits representing the significand portion of the 32-bit input data elements to generate a first reduced significand portion of the 32-bit input data elements, the first reduced significand portion corresponding to the set of most significant bits; and

[0280] a first exponent spreader configured to increase a number of bits representing an exponent portion of the 32-bit input data elements to generate a first increased exponent portion,

[0281] wherein the first sub-reducer generates the first 21-bit input data elements based on the first reduced significand portion and the first increased exponent portion; and

[0282] a second sub-reducer configured to convert the 32-bit input data elements to second 21-bit input data elements, the second 21-bit input data elements corresponding to a set of least significant bits of the significand portion of the 32-bit input data elements, the second sub-reducer comprising:

[0283] a leading bit reducer configured to reduce a number of leading bits representing the significand portion of the 32-bit input data elements to generate a second reduced significand portion of the 32-bit input data elements, the second reduced significand portion corresponding to the set of least significant bits; and

[0284] a second exponent spreader configured to increase a number of bits representing the exponent portion of the 32-bit input data elements to generate a second increased exponent portion,

[0285] wherein the second sub-reducer generates second 21-bit input data elements based on the second reduced significand portion and the second increased exponent portion; and

[0286] a plurality of processing elements, a processing element of the plurality of processing elements configured to iteratively perform a plurality of pairwise multiply-accumulate operations on the first 21-bit input data elements, the second 21-bit input data elements, and weights to provide a total output, wherein a 21-bit length corresponds to a maximum supported bit length of the processing element.

[0287] Clause 2: The systolic array processor of Clause 1, wherein the first 21-bit input data element and the second 21-bit input data element total the 32-bit input data element.

[0288] Clause 3: The systolic array processor of Clause 1 or Clause 2, wherein the second sub-reducer is further configured to determine that the 32-bit input data element comprises a regular number, the second sub-reducer further comprising:

[0289] a normalizer to remove an implicit bit of the 32-bit input data element and renormalize the second reduced significand portion based on the determination that the 32-bit input data element comprises a regular number to produce a normalized significand portion; and

[0290] an exponent adjuster to adjust the second increased exponent portion based on the renormalization of the second reduced significand portion to produce an adjusted exponent portion,

[0291] wherein the second 21-bit input data element is further based on the normalized significand portion and the adjusted exponent portion.

[0292] Clause 4: The systolic array processor of any of Clauses 1-3, the weights comprising a first reduced weight and a second reduced weight, wherein the processing element is further configured to:

[0293] multiply the second 21-bit input data element by the second reduced weight to generate a first product;

[0294] multiply the first 21-bit input data element by the second reduced weight to generate a second product;

[0295] multiply the second 21-bit input data element by the first reduced weight to generate a third product; and

[0296] multiply the first 21-bit input data element by the first reduced weight to generate a fourth product,

[0297] wherein the systolic array processor further comprises a partial sum buffer configured to:

[0298] add the first product, the second product, the third product, the fourth product, and an input partial sum to generate the total output.

[0299] Clause 5: A systolic circuit, comprising:

[0300] a group of processing elements arranged in a plurality of rows; and

[0301] a first converter configured to:

[0302] receive a first input in a floating-point representation having a first bit length;

[0303] generate a first reduced input in a floating-point representation having a second bit length, the first reduced input corresponding to a set of most significant bits of a significand portion of the first input; and

[0304] generate a second reduced input in a floating-point representation having a third bit length, the second reduced input corresponding to a set of least significant bits of the significand portion of the first input, wherein the first reduced input and the second reduced input total the first input, wherein the second bit length and the third bit length are less than the first bit length, wherein the second bit length and the third bit length correspond to bit lengths supported by the group of processing elements,

[0305] wherein each processing element in at least one row of the group of processing elements is configured to receive the first reduced input and the second reduced input and perform a plurality of multiply-accumulate operations on the first reduced input, the second reduced input, and a second input.

[0306] Clause 6: The systolic circuit of Clause 5, wherein each processing element in the plurality of rows of the group of processing elements comprises:

[0307] a multiplier configured to multiply two 21-bit floating-point numbers, wherein the multiplier is comprised of a 1-bit sign data path, an 11-bit significand data path, and a 9-bit exponent data path; and

[0308] an adder configured to add two floating-point numbers, wherein the adder is comprised of a 1-bit sign data path, a 23-bit significand data path, and a 10-bit exponent data path.

[0309] Clause 7: The systolic circuit of Clause 5 or Clause 6, wherein the first input corresponds to an input data element and the second input corresponds to a weight, wherein the first converter is further configured to:

[0310] receive the second input in a floating-point representation having a fourth bit length;

[0311] generate a third reduced input in a floating-point representation having a fifth bit length, the third reduced input corresponding to a set of most significant bits of a significand portion of the second input;

[0312] generating a fourth reduced input in a floating point representation having a sixth bit length, the fourth reduced input corresponding to a set of least significant bits of the significand portion of the second input, wherein the third reduced input and the fourth reduced input total the second input, wherein the fifth bit length and the sixth bit length are less than the fourth bit length, wherein the fifth bit length and the sixth bit length correspond to the bit lengths supported by the group of processing elements; and

[0313] selecting the first reduced input, the second reduced input, the third reduced input, or the fourth reduced input to provide.

[0314] Clause 8: The pulsatile circuit of any of clauses 5-7, wherein the first converter comprises:

[0315] a first sub-reducer comprising:

[0316] a tail bit reducer configured to reduce a number of the set of least significant bits of the significand portion of the first input to produce a first reduced significand portion of the first input; and

[0317] a first exponent spreader configured to increase a number of bits representing an exponent portion of the first input to produce a first increased exponent portion,

[0318] wherein the first sub-reducer produces the first reduced input based on the first reduced significand portion and the first increased exponent portion; and

[0319] a second sub-reducer comprising:

[0320] a leading bit reducer configured to reduce a number of the set of most significant bits of the significand portion of the first input to produce a second reduced significand portion of the first input; and

[0321] a second exponent spreader configured to increase a number of bits representing the exponent portion of the first input to produce a second increased exponent portion,

[0322] wherein the second sub-reducer produces the second reduced input based on the second reduced significand portion and the second increased exponent portion.

[0323] Clause 9: The pulsatile circuit of clause 8, wherein the second sub-reducer is configured to determine that the first input comprises a normal number, the second sub-reducer further comprising:

[0324] a normalizer to remove implicit bits of the first input and renormalize the second reduced significand portion based on determining that the first input comprises a normal number to produce a normalized significand portion; and

[0325] an exponent adjuster to adjust the second increased exponent portion based on renormalizing the second reduced significand portion to produce an adjusted exponent portion,

[0326] wherein the second reduced input is further based on the normalized significand portion and the adjusted exponent portion.

[0327] Clause 10: The systolic circuit of any of clauses 5-9, wherein the second input corresponds to a first reduced weight and a second reduced weight, wherein to perform the plurality of multiply-accumulate operations, the individual processing elements are configured to:

[0328] multiply the second reduced input by the second reduced weight to generate a first product;

[0329] add the first product to an input portion and to generate a first sum;

[0330] multiply the first reduced input by the second reduced weight to generate a second product;

[0331] multiply the second reduced input by the first reduced weight to generate a third product; and

[0332] multiply the first reduced input by the first reduced weight to generate a fourth product,

[0333] wherein the systolic circuit further comprises a partial sum buffer configured to:

[0334] add the first sum to the second product to generate a second sum;

[0335] add the second sum to the third product to generate a third sum; and

[0336] add the third sum to the fourth product to generate a total output.

[0337] Clause 11 : The systolic circuit of any of clauses 5-10, wherein the plurality of multiply-accumulate operations comprises an ordered plurality of multiply-accumulate operations.

[0338] Clause 12: The systolic circuit of any of clauses 5-11, wherein:

[0339] the first converter is configured to convert 32-bit floating-point numbers to a plurality of 22-bit floating-point numbers,

[0340] wherein each of the processing elements comprises:

[0341] a 22-bit multiplier; and

[0342] a 34-bit adder.

[0343] Clause 13: The systolic circuit of any of clauses 5-12, wherein:

[0344] the first converter is further configured to convert an m-bit floating point number to one or more n-bit floating point numbers, where n and m can be any number, where n is less than m,

[0345] wherein each of the processing elements comprises:

[0346] a multiplier configured to multiply at least two n-bit numbers; and

[0347] an adder configured to add two p-bit numbers, where p is greater than n.

[0348] Clause 14: The systolic circuit of any of clauses 5-13, further comprising:

[0349] a partial sum buffer configured to perform block-based accumulation based on a plurality of outputs of the group of processing elements.

[0350] Clause 15: The systolic circuit of any of clauses 5-14, further comprising:

[0351] a second converter configured to:

[0352] receive the second input in a floating point representation having a fourth bit length, the second input corresponding to a weight;

[0353] generate a third reduced input in a floating point representation having a fifth bit length, the third reduced input corresponding to a set of most significant bits of a significand portion of the second input; and

[0354] generate a fourth reduced input in a floating point representation having a sixth bit length, the fourth reduced input corresponding to a set of least significant bits of the significand portion of the second input, wherein the third reduced input and the fourth reduced input total the second input, wherein the fifth bit length and the sixth bit length are less than the fourth bit length, wherein the fifth bit length and the sixth bit length correspond to the bit lengths supported by the group of processing elements,

[0355] wherein the individual processing elements of the at least one row of the group of processing elements are further configured to receive the third and fourth reduced inputs and perform the plurality of multiply-accumulate operations on the first, second, third, and fourth reduced inputs.

[0356] Clause 16: The systolic circuit of any of clauses 5-15, wherein the group of processing elements performs a first accumulation on a plurality of outputs of the group of processing elements to produce a reduced plurality of outputs, the systolic circuit further comprising:

[0357] a partial and buffer configured to perform a block-based accumulation based on the reduced plurality of outputs to generate an output.

[0358] Clause 17: A method comprising:

[0359] receiving a first input in floating-point representation;

[0360] generating a first reduced input in floating-point representation, the first reduced input corresponding to a set of most significant bits of a significand portion of the first input;

[0361] generating a second reduced input in floating-point representation, the second reduced input corresponding to a set of least significant bits of the significand portion of the first input, wherein the first reduced input and the second reduced input total to the first input, wherein the first reduced input and the second reduced input correspond to a supported bit length; and

[0362] performing one or more operations based on the first reduced input, the second reduced input, and a second input to generate an output.

[0363] Clause 18: The method of clause 17, wherein:

[0364] the first input comprises a 32-bit floating-point number;

[0365] the first reduced input comprises a first 22-bit floating-point number; and

[0366] the second reduced input comprises a second 22-bit floating-point number.

[0367] Clause 19: The method of clause 17 or clause 18, further comprising:

[0368] receiving the second input in floating-point representation;

[0369] generating a third reduced input in floating-point representation, the third reduced input corresponding to a set of most significant bits of a significand portion of the second input; and

[0370] generating a fourth reduced input in floating point representation, the fourth reduced input corresponding to a set of least significant bits of the significant portion of the second input, wherein the third reduced input and the fourth reduced input total the second input,

[0371] wherein the one or more operations are further based on the third reduced input and the fourth reduced input.

[0372] Clause 20: The method of any of clauses 17-19, wherein each of the first input and the second input comprises an input data element or a weight.

[0373] The processes described herein, or in the figures of the disclosure, can begin in response to an event, such as beginning on a predetermined or dynamically determined schedule, on demand when initiated by a user or system administrator, or in response to some other event. When initiating such a process, an executable program instruction set stored on one or more non-transitory computer readable media (e.g., hard disk drives, flash memory, removable media, etc.) can be loaded into the memory (e.g., RAM) of a server or another computing device. The executable instructions can then be executed by the hardware-based computer processor of the computing device. In some embodiments, such a process, or portions of it, can be implemented serially or in parallel on multiple computing devices and / or multiple processors.

[0374] Depending on the embodiment, certain acts, events, or functions of any of the processes described herein can be performed in a different sequence, can be added, merged, or left out altogether (e.g., not all described operations or events are necessary for the practice of the algorithms). Moreover, in certain embodiments, operations or events can occur concurrently, e.g., through multi-threaded processing, interrupt processing, or multiple processors or processor cores or on other parallel architectures, rather than sequentially.

[0375] The various illustrative logical blocks, modules, routines, and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware (e.g., an ASIC or FPGA device), computer software running on computer hardware, or combinations of both. The processor device can be a microprocessor, but in the alternative, the processor device can be a controller, a microcontroller, or a state machine, combinations of the above, or the like. The processor device can include electronic circuitry for processing computer-executable instructions. In another embodiment, the processor device includes an FPGA or other programmable device that performs logic operations without processing computer-executable instructions. The processor device can also be implemented as a combination of computer hardware and software, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Although described herein primarily in the context of digital technology, the processor device can also include primarily analog components. For example, some or all of the rendering techniques described herein can be implemented in analog circuitry or mixed analog and digital circuitry. The computing environment can include any type of computer system, including, but not limited to, a microprocessor-based computer system, a mainframe computer, a digital signal processor, a portable computing device, a device controller or a computational engine within an appliance.

[0376] The elements of a method, process, routine, or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in software modules executed by a processor device, or in combinations of the two. A software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of non-transitory computer-readable storage medium. An exemplary storage medium can be coupled to the processor device such that the processor device can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor device. The processor device and the storage medium can reside in an ASIC. The ASIC can reside in a user terminal. In the alternative, the processor device and the storage medium can reside as discrete components in a user terminal.

[0377] Unless otherwise specified or understood in the context in which they are used, the conditional language used herein (such as “can,” “may,” “may,” “can,” “for example,” etc.) is generally intended to express that certain embodiments include certain features, elements, or steps that are not included in other embodiments. Therefore, this conditional language is not generally intended to imply that a feature, element, or step is necessary for one or more embodiments in any way, or that one or more embodiments necessarily include logic for determining, with or without further input or prompting, whether such features, elements, or steps are included in or will be performed in any particular embodiment. The terms “comprising,” “including,” “having,” etc., are synonymous and used in an open-ended manner as inclusive, without excluding additional elements, features, actions, operations, etc. Furthermore, the term “or” has an inclusive meaning (rather than an exclusive meaning) such that when used, for example, to connect a list of elements, the term “or” means one, some, or all of the elements in the list.

[0378] Unless otherwise specifically stated, disjunctive languages ​​such as “at least one of X, Y or Z” should be understood in the context as generally used to indicate that an item, term, etc., can be X, Y or Z or any combination thereof (e.g., X, Y or Z). Therefore, such disjunctive languages ​​are generally not intended and should not imply that certain embodiments require the presence of at least one of X, at least one of Y, and at least one of Z, respectively.

[0379] While the detailed description above has shown, described, and pointed out novel features applicable to various embodiments, it is understood that various omissions, substitutions, and changes in the form and detail of the illustrated devices or algorithms may be made without departing from the scope of this disclosure. As will be appreciated, some embodiments described herein may be embodied in forms that do not provide all the features and benefits set forth herein, because some features may be used or practiced separately from other features. All changes to the meaning and scope of the equivalent forms of the provisions herein shall be covered within the scope of the provisions.

Claims

1. A systolic array processor organized by rows and columns, each row comprising: Multiple processing elements; as well as A reducer configured to convert a 32-bit input data element into a reduced 22-bit input data element, the reducer comprising: A trailing bit reducer is configured to reduce the number of bits representing the significant portion of the 32-bit input data element based on an input format supported by the plurality of processing elements, to produce a reduced significant portion of the 32-bit input data element. A rounder configured to round the reduced significant portion of the 32-bit input data element to produce a rounded significant portion; and An exponent expander, configured to increase the number of bits representing the exponent portion of the 32-bit input data element based on a format supported by the plurality of processing elements, to produce an increased exponent portion. The reducer generates a reduced 22-bit input data element based on the rounded significant part and the increased exponent part; The plurality of processing elements are configured to receive the reduced 22-bit input data element from the reducer and to receive weights for performing a single-pass multiply-accumulate operation associated with the 32-bit input data element, wherein the bit length of the 32-bit input data element exceeds the maximum bit length supported by the plurality of processing elements for performing the single-pass multiply-accumulate operation.

2. The systolic array processor of claim 1, wherein the reducer is further configured to convert the 32-bit weights into the weights.

3. The systolic array processor of claim 1, wherein the reducer further comprises a first reducer, and each row further comprises: A second reducer is configured to convert 32-bit weights into the weights.

4. The systolic array processor of claim 1, wherein the rounder is configured to round the reduced significant portion of the 32-bit input data element based on one or more of the following: Random rounding; Round to the nearest even number; Round to zero; Round down; or Round up.

5. A pulsating circuit, comprising: A group of processing elements, the group of processing elements being arranged in multiple rows; as well as A first converter, configured as follows: Receive a first input in floating-point representation having a first bit length, wherein the first bit length exceeds the maximum bit length supported by the group of processing elements used to perform a single-pass multiplication-accumulation operation associated with the first input; The number of trailing bits in the first input; The number of trailing bits of the first input is reduced based on the difference between the first bit length and the bit length supported by the group of processing elements. The number of bits representing the exponential portion of the first input is increased based on the input format supported by the group of processing elements. and A first reduced input is generated in floating-point representation with a second bit length based on reducing the number of trailing bits of the first input and increasing the number of bits representing the exponent portion of the first input, wherein the second bit length is less than the first bit length, and wherein the second bit length corresponds to a bit length supported by the group of processing elements. Each processing element in at least one row of the group of processing elements is configured to receive the first reduction input from the first converter and to receive a second input for performing the single-pass multiplication-accumulation operation.

6. The pulsating circuit of claim 5, wherein each processing element in the plurality of rows of the group of processing elements comprises: A multiplier configured to multiply two 22-bit floating-point numbers, wherein the multiplier comprises a 1-bit sign data path, an 11-bit significant data path, and a 10-bit exponent data path; and An adder configured to add two floating-point numbers, wherein the adder comprises a 1-bit sign data path, a 23-bit significant data path, and a 10-bit exponent data path.

7. The pulsating circuit of claim 5, wherein the first input includes input data elements and the second input includes reduction weights, wherein the first converter is further configured to: Receive the first input and weights; Generate the first reduced input and the second input; and Choose either the first reduced input or the second input to provide.

8. The pulsating circuit of claim 5, wherein the first converter comprises: A trailing bit reducer, configured to reduce the number of bits representing the significant portion of the first input to produce a reduced significant portion of the first input. A rounder configured to round the reduced significant number portion of the first input based on the remaining portion of the bits representing the significant number portion that are not included in the reduced significant number portion. as well as An exponent expander configured to increase the number of bits representing the exponent portion of the first input.

9. The pulsating circuit of claim 5, wherein the first input includes a first rounding input, and wherein the first converter includes: A trailing bit reducer, configured to reduce the number of bits representing the significant portion of the first input to produce a reduced significant portion of the first input. as well as An exponent expander configured to increase the number of bits representing the exponent portion of the first input.

10. The pulsating circuit of claim 5, wherein the first reduction input includes a first reduction rounding input, wherein the first reduction rounding input performs rounding based on one or more of the following: Random rounding; Round to the nearest even number; Round to zero; Round down; or Round up.

11. The pulsating circuit of claim 5, wherein the first reduction input includes a first reduction rounding input, wherein the first reduction rounding input performs rounding based on user input.

12. The pulsating circuit as described in claim 5, wherein: The first converter is configured to convert a 32-bit floating-point number to a 22-bit floating-point number. Each of the processing elements said therein includes: 22-bit multiplier; and 34-bit adder.

13. The pulsating circuit as described in claim 5, wherein: The first converter is further configured to convert an m-bit floating-point number into an n-bit floating-point number, where n and m can be any positive integers, and n is less than m. Each of the processing elements said therein includes: A multiplier configured to multiply at least two n-bit numbers; and An adder configured to add two p-bit numbers, where p is greater than n.

14. The pulsating circuit of claim 5, wherein, in order to reduce the number of trailing bits of the first input, the first converter is configured to: Set the number of trailing bits to zero.

15. The pulsating circuit of claim 5, further comprising: The second converter is configured as follows: Receive the weights in floating-point representation having the first bit length; The number of trailing bits that identify the weight; Reduce the number of trailing bits of the weight; and The second input, represented in floating-point format with the second bit length, is generated based on the number of trailing bits of the reduced weights.

16. The pulsating circuit of claim 5, wherein the first reduced input is stored in a 24-bit format.

17. A method implemented by a pulsating circuit, the method comprising: Receive a first input represented in floating-point format with a first bit length, wherein the first bit length exceeds the maximum bit length supported by the processing element for performing a single-pass multiplication-accumulation operation associated with the first input; The number of trailing bits in the first input is reduced based on the difference between the first bit length and the bit length supported by the processing element used for the single-pass multiplication accumulation operation. The number of bits representing the exponential portion of the first input is increased based on the input format supported by the processing element. A first reduced input is generated in floating-point representation with a second bit length based on reducing the number of trailing bits of the first input and increasing the number of bits representing the exponent portion of the first input, wherein the second bit length is less than the first bit length, and wherein the second bit length corresponds to the bit length supported by the processing element. as well as The first reduced input and the second input are received for the processing element to perform the single-pass multiplication-accumulation operation.

18. The method of claim 17, wherein: The first input includes a 32-bit floating-point number; The first reduced input includes a first 22-bit floating-point number; and The second input includes a second 22-bit floating-point number.

19. The method of claim 17, wherein generating the first reduced input comprises: The first input is rounded based on the remaining non-tailed bits of the first input to generate the first reduced input, wherein the first input includes a number of bits, and rounding the first input includes rounding a portion of the number of bits.

20. The method of claim 17, wherein one or more of the first reduction input or the second input includes a rounding reduction input, wherein the rounding reduction input performs rounding based on one or more of the following: Random rounding; Round to the nearest even number; Round to zero; Round down; or Round up.

21. The method of claim 17, wherein the processing element for the single-pass multiplication-accumulation operation does not support the first bit length.

Images