Including pulsating arrays with efficient pre-normalization and extended dynamic range for fused multiplication and accumulation.

Abstract

Systems and methods are provided to perform multiply-accumulate operations on normalized numbers in systolic arrays to achieve greater computational density, reduce the size of the systolic array required to perform multiply-accumulate operations on normalized numbers, and / or achieve higher throughput operations. The systolic array may be provided with normalized numbers by a series of normalizers and may lack support for nonnormal numbers. Each normalizer can normalize the input for each processing element in the systolic array. The systolic array may include multipliers and adders. Multipliers may have multiple data paths corresponding to the data types of the inputs. Multipliers and adders can operate on normalized floating-point numbers using extended exponent ranges and may lack support for nonnormal numbers.

Description

Background Technology

[0001] Artificial neural networks (ANNs) are computational systems with architectures based on biological neural networks. NANs can be implemented using circuits and data pathways, such as systolic arrays. systolic arrays can improve the performance of both the training and inference phases of an ANN. During training, input data is provided to train the model. During inference, new inputs are processed based on the model to obtain predictions. User applications typically use the model during the inference phase, which is often time-sensitive, and delays in the inference phase can negatively impact the user experience.

[0002] As more and more applications utilize artificial neural networks, these applications also use a wide range of numbers that can lead to numerical underflow, such as in floating-point arithmetic calculations. While support for nonnormal number computations can help accommodate numerical underflow, providing support for nonnormal numbers increases the complexity, size, and cost of the processing elements in the systolic array. These increases also affect system processing speed and power consumption. When the systolic array is required to support both normal and nonnormal numbers, power consumption and the size of the systolic array can become critical. Attached Figure Description

[0003] Various features will now be described with reference to the following figures. Throughout the figures, reference numerals may be used repeatedly to indicate the correspondence between the elements referred to. The figures are provided to illustrate the examples described herein and are not intended to limit the scope of this disclosure.

[0004] Figure 1 An example 4×4 systolic array and a normalizer for an example column are shown.

[0005] Figure 2A The diagram illustrates a processing element for neural network computation, in which the input is fed through a separate normalizer, according to certain examples of the disclosed techniques.

[0006] Figure 2B The diagram illustrates a processing element for neural network computation in which the inputs are fed through the same normalizer, according to certain examples of the disclosed technology.

[0007] Figure 3 The apparatus shown is an apparatus, according to certain examples of the disclosed technology, including a zero detector circuit for normalizing input data elements and normalizing weights into a systolic array for neural network computation.

[0008] Figure 4A The illustration shows a normalizer for selecting the input to be normalized (where the input is a floating-point 16-bit number) according to some examples of the disclosed techniques.

[0009] Figure 4BThe illustration shows a normalizer for selecting the input to be normalized (where the input is a 16-bit floating-point brain) according to some examples of the disclosed technology.

[0010] Figure 5 A multiplication-accumulation data path for neural network computation is shown, according to certain examples of the disclosed techniques.

[0011] Figure 6 An apparatus for neural network computation is shown, according to some examples of the disclosed techniques.

[0012] Figure 7 The methods performed by a normalizer and processing elements for neural network computation are shown as examples of some of the disclosed techniques.

[0013] Figure 8 An example of a computing device according to certain aspects of this disclosure is shown. Detailed Implementation

[0014] In the following description, various examples will be described. Specific configurations and details are set forth for illustrative purposes in order to provide a thorough understanding of the examples. However, it will be apparent to those skilled in the art that the embodiments may be practiced without the specific details. Furthermore, well-known features may be omitted or simplified so as not to obscure the described embodiments.

[0015] Convolutional neural networks (CNNs) are typically feedforward artificial neural networks that can include multiple intermediate layers, with the output from one layer serving as the input to the next. Systolic arrays can be used to accelerate workloads in neural networks by reading data from memory once and reusing it across multiple computations. Systolic arrays can be implemented using a two-dimensional array of processing elements (PEs).

[0016] PEs can be divided into rows and columns. Each PE in the input layer receives an element from the input dataset and scales the element using weights (or filters) to indicate how much influence that element has on the output. PEs in intermediate layers can combine the scaled elements received from each PE in the input layer to compute a set of intermediate outputs. For example, each PE in an intermediate layer can compute the sum of the weighted products of the elements and then generate an intermediate output by applying an activation function to the sum. The intermediate output from each PE in an intermediate layer can be viewed as an activation vote (or no vote), associated with weights indicating the influence of the vote, to determine the intermediate output of the next intermediate layer. The output layer can generate a sum of scaled intermediate outputs from the final intermediate layer and generate a binary output (e.g., "yes" or "no") based on whether the sum of the scaled intermediate outputs exceeds a threshold.

[0017] Typically, an input dataset (e.g., an input feature map) can be fed one input data element at a time into the corresponding row of its systolic array, and passed from one PE to another in a given row starting from the leftmost PE. With the proliferation of machine learning and neural network applications, the diversity of input dataset values ​​and data types has also proliferated. Some applications use higher-precision floating-point 32-bit (FP32) or floating-point 64-bit (FP64) data types for input. Some applications require even higher-precision computations but use floating-point 16-bit (FP16), brain-float 16-bit (bfloat16 or BF16), or 16-bit or 8-bit integer data types for input. These higher-precision computations can lead to underflows, where the data type (e.g., FP16) is no longer able to represent the computed value due to the range and precision limitations of that data type. Therefore, some applications or systems allow the use of denormalized numbers to maximize the representable range and precision of numerical data. Denormalized numbers (sometimes called “subnormal” numbers) differ from normalized numbers in that the implicit bit (the most significant bit identified as a significant number) is no longer set to a high-order value or a single value. Furthermore, the exponent field of a nonnormal number can be set to all zeros.

[0018] Compared to supporting only regular numbers, providing support for both regular and non-regular numbers leads to a significant increase in integrated circuit die cost, power consumption, circuit complexity, and latency. In pulsating arrays with hundreds or thousands of PEs, the additional support for non-regular numbers results in an exponential increase in integrated circuit die cost, power consumption, circuit complexity, and latency.

[0019] In some configurations, the PE supports performing mathematical operations on normalized numbers without additional circuitry to support accurate non-regular calculations. For example, for an FP16 systolic array that supports both non-regular and regular number calculations, each processing element in the systolic array may need to be modified to support or perform mathematical operations on non-regular inputs and / or generate non-regular outputs from those operations. Such modifications may be particularly undesirable and may exclude providing arithmetic support for non-regular values.

[0020] Embodiments of the disclosed technology can provide systems and methods to reduce dynamic power consumption in systolic arrays and decrease the size and number of circuits required to perform operations that may involve normal and non-normal numbers. Furthermore, embodiments of the disclosed technology can provide systems and methods to enable systolic arrays to accept normal and non-normal data, thereby reducing the need for dedicated circuitry for each data type. In one embodiment, these problems are addressed by providing a systolic array with a column of normalizers that provide normalized inputs to the systolic array. In some embodiments, each normalizer can be configured to provide more than one normalized input to the systolic array.

[0021] As disclosed herein, each normalizer of a systolic array can normalize one or more inputs provided to the normalizer and output one or more normalized inputs based at least in part on one or more inputs. The one or more normalized inputs can be represented in a modified format with normalized significant numbers. The normalized inputs can have a sign bit, a non-zero exponent bit, and significant bits with a non-zero bit as the most significant bit. The most significant bit of the significant bits can be hidden or concealed. Each normalizer can include one or more of the following: a non-regular detector, an exponent expander, a leading zero encoder, a shifter, a subtractor, and a multiplexer. Each normalizer can contain any combination of these components. Inputs can be provided to the normalizer and normalized based on operators provided to the normalizer. Each normalizer can provide normalized inputs to the systolic array.

[0022] In some implementations, the normalizer may include a multiplexer that selects between two or more inputs based on control signals such as opcodes or data type indicators. The output of the multiplexer can be fed to a second part of the normalizer for normalization.

[0023] In some implementations, the systolic array may have a separate normalizer that receives either an input data element or weights and provides a corresponding normalized version of that input to the systolic array. Each processing element in the initial row of the systolic array's processing elements may be configured to receive normalized input data elements from a first normalizer and normalized weights from a second normalizer. For example, the first normalizer may receive input data elements and provide normalized input data elements to the first processing element, and the second normalizer may receive weights and provide normalized weights to the first processing element.

[0024] The systolic array can be configured to provide an initial column of zero-input data detectors and zero-weight detectors. Each zero-input data detector and each zero-weight detector can be configured to detect when the input to the processing element is zero or empty. A normalized version of each input to the processing element can be provided to each zero-input data detector and each zero-weight detector. For example, a normalized input data element can be provided to each zero-input data detector, and a normalized weight can be provided to each zero-weight detector. Each zero-input data detector can then be configured to provide each normalized input data element to the systolic array, and each zero-weight detector can then be configured to provide each normalized weight to the systolic array. In some implementations, each zero-weight detector and each zero-input data detector can be configured to provide zero or empty input to the systolic array when the normalized input data element or normalized weight is zero. In other implementations, the zero-input data detectors and / or zero-weight detectors can be concatenated before the corresponding normalizers. In other implementations, the zero-input data detectors and / or zero-weight detectors are configured to process the input or weights in parallel with the corresponding normalizers.

[0025] Each normalizer can be configured to normalize 16-bit, 32-bit, or arbitrary-bit numbers. Furthermore, each normalizer can be configured to normalize both floating-point and non-floating-point numbers. For example, a BF16 number with 1 sign bit, 8 exponents, and 7 significant digits can be provided to the normalizer to be normalized to an 18-bit format with 1 sign bit, 10 extended exponents, and 7 shifted significant digits. Alternatively, an FP16 number with 1 sign bit, 5 exponents, and 10 significant digits can be provided to the normalizer to be normalized to an 18-bit format with 1 sign bit, 7 extended exponents, and 10 shifted significant digits. In other implementations, the normalizer can be configured to normalize multiple numbers in different data formats into a single, longer data format with enough bits to represent the longest significant bit and enough exponents to represent the largest exponent after shifting significant digits to the normalized format.

[0026] A systolic array can have a PE including an 18-bit multiplier and a 34-bit adder. The 18-bit multiplier can be configured to operate on 18-bit normalized floating-point numbers normalized from FP16 or BF16 format. The multiplier can be configured to generate a multiplier product with one sign bit, ten exponent bits, and 23 significant bits. The multiplier product can include 24 significant bits, where the most significant bit is implicit or hidden. The 34-bit adder can be configured to operate on 34-bit numbers. A 34-bit adder can also be configured to operate on 35-bit numbers, where one bit is implicit or hidden. In some implementations, the systolic array can be configured to include an 18-bit multiplier and a 32-bit adder, where both can be configured to operate on 18-bit normalized floating-point numbers. In some implementations, the systolic array can be configured to include an n-bit multiplier and an m-bit adder, where n can be any number, and the n-bit multiplier and m-bit adder can be configured to operate on x-bit normalized floating-point numbers and y-bit normalized floating-point numbers. The variables n, m, x, and y can be any numbers, where n is greater than the larger of x and y.

[0027] A systolic array may include multiple delay registers in the PE (Programmable Array). Each of these delay registers can receive inputs and delay the output inputs for at least one clock cycle. For example, a delay register may be placed between a multiplier and an adder. The multiplier may be configured to generate one or more multiplier products and send one or more multiplier products to the delay register. The delay register may be configured to delay the one or more multiplier products to the adder by at least one clock cycle.

[0028] A systolic array may be able to skip multiplication operations in a PE under certain conditions. Multiplication can be skipped when zero is detected on the input data element of the current operation or when the PE receives a no-operation (NOP). For example, each PE can receive an opcode indicating the operation to be performed by the PE. The PE can decode the opcode to determine whether the opcode value corresponds to a NOP. (See reference) Figure 1 Explain the example pulsating array.

[0029] Figure 1 An example 4x4 systolic array 100 is shown. For example, systolic array 100 may include four physical exponents (PEs) in each row and four PEs in each column. It should be understood that systolic array 100 may include any number of PEs in each row and column. It should also be understood that systolic array 100 may be logically organized into any number of rows and any number of columns. systolic array 100 may be part of a neural network processor in a computer system. For example, the computer system may be configured to provide multi-tenant computing services for data processing applications such as image recognition services, text-based data processing (e.g., processing of search queries), audio or video data processing.

[0030] Each PE may include a row input bus 102, a column input bus 104, a column output bus 106, and a row output bus 108. A PE may receive input from the left PE (or from external circuitry) in the same row via the row input bus 102. A PE may also receive input from a PE (or from external circuitry) in the same column via the column input bus 104. A PE may perform arithmetic calculations based on the inputs and transmit the results of the arithmetic calculations to the PE (or external circuitry) in the same column via the column output bus 106. A PE may also forward inputs received via the row input bus 102 to the right PE in the same row via the row output bus 108.

[0031] The systolic array 100 can perform arithmetic calculations, including multiplication and addition, for the processing elements of a neural network. For example, each PE may include an arithmetic unit such as a multiplier and adder, or a fused multiplier and adder. Figure 1 In the example, each row PE can be configured to process a set of input data, and each column PE can generate a set of output data based on the set of input data received by each PE in a given column.

[0032] Column 112 (the leftmost column) of the PE can receive four sets of input data, each set processed by one row of PE. Column 116 of the normalizers can provide four sets of normalized input data to column 112 of the PE, each set provided by a normalizer. Each normalizer in column 116 can provide one set of normalized input data, where each set includes two or more normalized inputs. For example, normalizer 116a can provide normalized input data elements and normalization weights to PE 112a. Each normalizer in column 116 can transform an input into a normalized input. For example, normalizer 116a can transform a 16-bit input data element into a normalized 18-bit input data element.

[0033] Each normalizer in column 116 of the normalizers can further select normalized inputs to provide to each PE in column 112 of the PEs. For example, each normalizer in column 116 of the normalizers can include a multiplexer to select weights or input data elements. In some embodiments, each normalizer 116a-116d can be implemented as a first normalizer and a second normalizer. The first normalizer and the second normalizer can provide one or more inputs to column 112 of the PEs. For example, the first normalizer of normalizer 116a can be configured to provide normalized input data elements to PE 112a, and the second normalizer of normalizer 116a can be configured to provide normalized weights to PE 112a.

[0034] Each PE in column 112 can obtain normalized input data elements and normalized weights from the corresponding input dataset received via row input bus 102. Each PE in column 112 can multiply the normalized input data elements by the normalized weights to generate a scaled input. The scaled input generated by a PE in any column (including column 112) can be accumulated by the adder of each PE. For example, PE 112a (column 112) can generate a first scaled input (from the first input dataset), where the first scaled input can be based on the output of the adder. For example, the adder can generate a first output partial sum, and PE 112a can be configured to generate the first scaled input at least partially based on the first output partial sum. PE 112a can be configured to transmit the first scaled input as a partial sum to PE 112b via column output bus 106. PE 112b can also generate a second scaled input (from a second input dataset) and add the second scaled input to the partial sum. The updated partial sum, accumulated from the first and second scaling inputs, is then transmitted to PE 112c via column output bus 106. The partial sum is updated and propagated across columns 112, and PE 112d can generate the scaling input sum from the four input datasets.

[0035] The sum generated by PE 112d can correspond to the output dataset and can be fed back to the leftmost PE after passing through the activation function. Furthermore, each PE in column 112 can propagate the input dataset to other PE columns (e.g., column 114), which can scale the input dataset using a different set of weights than column 112. Arithmetic operations (multiplication and addition) can be performed on each column of the PEs to generate output data elements in parallel for other processing elements. Figure 1 In the example, the systolic array 100 can generate output data elements for four PEs corresponding to four columns of the systolic array 100.

[0036] The systolic array 100 can perform convolution computations across multiple waves. In one embodiment, a wave represents a stream of input data elements processed while reusing the same weights in the systolic array 100. For example, the corresponding weights may have been preloaded sequentially or in parallel in each PE in the systolic array 100 before the wave computation begins. The partial sums produced by the PEs may correspond to a single wave. Since the PEs of the systolic array 100 perform the arithmetic operations of convolution computations, the dynamic power dissipated by all multipliers in the PEs can be large. This problem can be further exacerbated for systolic arrays comprising a large number of PEs (e.g., thousands). The arithmetic operations performed by the PEs will refer to... Figure 2A and Figure 2B Further explanation.

[0037] Figure 2APE 00 in a systolic array for neural network computation according to certain embodiments of the disclosed technology is shown. PE 00 may be similar to Figure 1 It is a part of the pulsation array 100 in the middle. Figure 4A and Figure 4B It shows Figure 2A Additional details regarding the normalizers 225 and 227 are provided. Some embodiments may be described with reference to neural networks; however, it should be understood that certain embodiments may be used in other applications, such as pattern recognition, image processing, audio processing, video processing, etc., without departing from the scope of this technology.

[0038] The systolic array 200 includes normalizers 225 and 227 and multiple processing elements including PE 00 and PE 01. PE 00 may include a data element loading generator 202, an input data element register 204, a weight register 206, a multiplier 208, an adder 210, a skip calculation generator 212, a skip calculation register 214, a selector circuit 216, an input portion sum register 218, a buffered weight register 220, and an operation decoder 256. PE 00 may be configured to receive normalized input data elements 222, normalized weights 224, a zero data element indicator 226, a zero weight indicator 228, an opcode 230, a weight payload 232, and an input portion sum 234 to perform convolution computations according to some embodiments.

[0039] PE 00 can be connected to a first normalizer 225 and a second normalizer 227. The first normalizer 225 can receive a first input (e.g., input data element 221), and the second normalizer 227 can receive a second input (e.g., weight 223). The first normalizer 225 can convert the first input into a first normalized input, and the second normalizer 227 can convert the second input into a second normalized input. The first normalizer 225 can be configured to provide PE 00 with a normalized input data element 222, where the normalized input data element 222 is a normalized version of input data element 221. Furthermore, the second normalizer 227 can be configured to provide PE 00 with a normalized weight 224, where the normalized weight 224 is a normalized version of weight 223.

[0040] The first normalizer 225 and the second normalizer 227 can detect when one or both of the input data element 221 and the weight 223 are non-normal numbers. For example, the first normalizer 225 can be configured to detect when the input data element 221 is non-normal, while the second normalizer 227 can be configured to detect when the weight 223 is non-normal. When a non-normal number is detected, the corresponding normalizer can provide a normalized version of the non-normal number, at least in part, based on the detection of the non-normal number. Normalization of the non-normal number may include shifting the significant numbers, adjusting the exponent to reflect the shift of the significant numbers, and expanding the number of bits used to represent the adjusted exponent. In some cases, one or both of the input data element 221 and the weight 223 can be normal numbers, and the corresponding normalizer can be configured to provide the corresponding normal number to PE 00 and skip normalization of the corresponding normal number. For example, the first normalizer 225 can detect that the input data element 221 is non-normal, normalize the input data element 221, and provide the normalized input data element 222 to PE 00. Furthermore, the second normalizer 227 can detect that the weight 223 is normal and provide the normalized weight 224 to PE 00, where the weight 223 is equal to the normalized weight 224, and the second normalizer 227 may not perform a normalization operation on the weight 223.

[0041] If one or both of the input data element 221 and weight 223 are already in the normalized representation, the first normalizer 225 and the second normalizer 227 can still expand the number of bits used to represent the adjusted exponent. Therefore, subsequent arithmetic circuitry, such as multiplier 208, can be configured to perform calculations on numbers in a single format, regardless of whether the input was initially a regular or irregular representation. For example, the first normalizer 225 can detect that the input data element 221 is regular, expand the number of bits used to represent the exponent field of the input data element 221, and provide the normalized input data element 222 to PE00.

[0042] The first normalizer 225 and the second normalizer 227 can be configured to convert one or more floating-point inputs into a normalized representation. The floating-point inputs can be 8-bit, 16-bit, 32-bit, or any number of bits.

[0043] PE 00 can receive normalized input data element 222 via a first input port. Normalized input data element 222 can correspond to an input dataset or an array of any input data elements. PE 00 can receive one normalized input data element from the input dataset at a time over a uniform time period. For example, the uniform time period can correspond to a clock cycle. The input dataset can resemble an input feature map that includes input feature map elements. As an example, the input dataset can correspond to an input image, audio clip, video clip, text portion, or any other data that can be provided for data processing to identify specific patterns or objects. In some cases, the input dataset can correspond to an intermediate output dataset that has been subjected to an activation function, such as ReLU or Sigmoid, as referenced. Figure 1 Each normalized input data element 222 can be a floating-point data type or any suitable data type. Each normalized input data element 222 can include 8 bits, 16 bits, or any suitable number of bits. The normalized input data element 222 can be stored in the input data element register 204 for a period of time.

[0044] PE 00 can receive normalized weights 224 via a second input port. In some implementations, the normalized weights 224 may belong to a set of weight values ​​corresponding to a convolutional filter. The normalized weights 224 can be preloaded into PE 00 before receiving normalized input data elements 222. In some embodiments, PE 00 can receive one normalized weight value at a time from the set of normalized weight values ​​over a uniform period of time to preload each PE in a given row with the corresponding normalized weight value. PEs can pass the normalized weight value to the next PE in the corresponding row until each PE in the given row has been preloaded. Each PE can cache its own normalized weight value for computation on normalized input data elements. Each normalized weight 224 can be a floating-point data type or any suitable data type. Each normalized weight 224 can include 8 bits, 16 bits, or any suitable number of bits. The normalized weights 224 can be stored in the cached weight register 220 for a period of time.

[0045] PE 00 can receive the input portion and 236 for the current operation via the third input port. In various implementations, the input portion and 236 can be 16-bit, 18-bit, 32-bit, 33-bit, 34-bit, or have any number of bits.

[0046] PE 00 can receive a zero data element indicator 226 for current operation via a fourth port. The zero data element indicator 226 may include a single bit or multiple bits. The zero data element indicator 226 can be used to indicate whether the normalized input data element 222 is zero. The zero data element indicator 226 can also be used to indicate whether the input data element 221 is zero. For example, a value of "1" for the zero data element indicator 226 can indicate that the normalized input data element 222 associated with the zero data element indicator 226 is zero, and a value of "0" for the zero data element indicator 226 can indicate that the normalized input data element 222 associated with the zero data element indicator 226 is not zero. "0" can correspond to logic zero or logic low, and "1" can correspond to logic 1 or logic high. For example, in some implementations, logic zero can be represented by a first range of voltage levels (e.g., 0-2 volts), and logic one can be represented by a second range of voltage levels (e.g., 3-5 volts). It should be understood that other implementations of representing "0" and "1" values ​​are possible without departing from the scope of the disclosed technology. The zero data element indicator 226 can be generated by circuitry outside of PE 00 and passed sequentially to all PEs in the same row within a uniform time period.

[0047] PE 00 can be configured to receive a zero-weight indicator 228 via a fifth port. The zero-weight indicator 228 may include a single bit or multiple bits. The zero-weight indicator 228 can be used to indicate whether the normalized weight 224 associated with the zero-weight indicator 228 is zero. The zero-weight indicator 228 can also be used to indicate whether the weight 223 associated with the zero-weight indicator 228 is zero. For example, a value of "1" for the zero-weight indicator 228 can indicate that the normalized weight 224 is zero, and a value of "0" for the zero-weight indicator 228 can indicate that the normalized weight 224 is not zero. The zero-weight indicator 228 can be generated by circuitry outside of PE 00 and, along with the normalized weight 224, sequentially passed to all PEs in the same row.

[0048] Weight payload 232 can be used to load normalized weights 224 into cached weight register 220 to provide cached weights 246. Before normalized input data elements 222 are fed into the array, it can be asserted that weight payload 232 caches the normalized weights 224 of PE 00 in cached weight register 220. As weights are shifted into the array to preload each PE with the corresponding weight values, weight payload 232 can be asserted for each PE at specific time intervals to preload each PE with the appropriate weight values.

[0049] The operation decoder 256 can be configured to decode the opcode 230 to determine the operation performed by PE 00 for different instructions represented by different opcode values. In some embodiments, the first opcode value may correspond to an instruction to transfer normalized weights from one PE in the systolic array to another PE. The second opcode value may correspond to an instruction to initiate arithmetic computation by a PE. For example, once the normalized weights are preloaded into the systolic array, normalized input data elements can be read from memory, and arithmetic computation can be performed as the normalized input data elements pass through the array. The third opcode value may correspond to an instruction to perform a NOP. A NOP can be used to separate two systolic array instructions, or when there are no normalized input data elements to be read from memory. For example, a NOP can be used to separate an instruction to shift normalized weights from an instruction to initiate arithmetic computation. For example, for a 4x4 array, it may take up to 15 cycles to shift the normalized weights to all PEs in the array before initiating arithmetic computation, thus potentially requiring 15 NOP cycles. The operation decoder 256 can be configured to decode opcode 230 to generate NOP 258 and a start calculation signal 260. The operation decoder 256 can be configured to provide the start calculation signal 260 to a weight register 206 connected to multiplier 208 and adder 210. The operation decoder 256 can also be configured to provide the start calculation signal 260 to multiplier 208. Opcode 230 can include any suitable number of bits, such as two, four, etc. In some implementations, the opcode decoder 256 can also decode the opcode to determine the data type, thereby providing a data type control signal 235 to indicate the data type to be operated on by the multiplier.

[0050] In some implementations, the normalized input data element 222, the normalized weight 224, the opcode 230, the zero data element indicator 226, and the zero weight indicator 228 can belong to the row input bus 102, as shown in the following reference. Figure 1 In some implementations, a splitter (not shown) can be used in PE 00 to separate the row input bus 102 into different internal buses to carry the normalized input data element 222, normalized weight 224, opcode 230, zero data element indicator 226, and zero weight indicator 228 within PE 00. For example, the normalized input data element 222 and normalized weight 224 may belong to the first row input bus, and the opcode 230, zero data element indicator 226, and zero weight indicator 228 may belong to the second row input bus.

[0051] Data element load generator 202 can be configured to generate a data load signal 242, which can be used to allow input data element register 204 to skip the storage of normalized input data element 222 under certain conditions. In some embodiments, normalized input data element 222 can be loaded into input data element register 204 when data load signal 242 is asserted based on zero data element indicator 226 and NOP 258. Data load signal 242 can be asserted when zero data element indicator 226 corresponding to normalized input data element 222 is "0" and opcode 230 does not indicate NOP (e.g., NOP 258 is "0"). Data load signal 242 can not be asserted when zero data element indicator 226 corresponding to normalized input data element 222 or NOP 258 is "1". Data element load generator 202 can be implemented using OR, NOR, NAND, or any suitable circuitry.

[0052] Input data element register 204 can be configured to store normalized input data element 222, or skip storing normalized input data element 222 to provide a stored input data element 244 for the current operation based on data load signal 242. In some implementations, input data element register 204 can store the Din input if the load input is "1", and can retain the previous value if the load input is "0". For example, if data load signal 242 is "1", input data element register 204 can store the new value of normalized input data element 222, and if data load signal 242 is "0", input data element register 204 can skip storing the new value of normalized input data element 222. Therefore, in some cases, input data element register 204 can only store the non-zero value of normalized input data element 222. According to some embodiments, skipping the storage of a new value in input data element register 204 may result in not switching the stored input data element 244 and retaining the previous value of the stored input data element 244.

[0053] Weight register 206 can be configured to store cached weights 246 to provide stored weight values ​​248 based on the start calculation signal 260. In some implementations, weight register 206 can store the Din input if the load input is "1", and can retain the previous value if the load input is "0". For example, if the start calculation signal 260 is asserted (e.g., the start calculation signal 260 is "1"), cached weights 246 can be loaded into weight register 206; otherwise, weight register 206 can retain the previous value. Thus, normalized weights 224 previously loaded into cached weight register 220 using weighted load 232 can be moved into weight register 206 at the start of arithmetic calculation. In some embodiments, the stored weight values ​​248, once loaded at the start of arithmetic calculation, remain unchanged as input data elements are fed one at a time through the pulsation array to PE00 for calculations corresponding to one or more waves.

[0054] PE 00 can provide the stored input data element 244 to PE 01 based on the data load signal 242 of the current operation. PE 01 can receive the stored input data element 244 as normalized input data element 222 via a first port. In some implementations, if the load input is "1", the input data element register 204 can store the Din input, and if the load input is "0", the previous value can be retained. PE 00 can provide the stored weight value 248 to PE 01 based on the start calculation signal 260. PE 01 can receive the stored weight value 248 as normalized weight 224 via a second port. In some implementations, if the load input is "1", the weight register 206 can store the Din input, and if the load input is "0", the previous value can be retained.

[0055] Multiplier 208 can perform multiplication between stored input data elements 244 and stored weight values ​​248. Multiplier 208 can be configured to generate a product 250 based on the multiplication operation. In some implementations, product 250 can be an integer product, a floating-point product, or any other product. In some implementations, multiplier 208 can generate a product 250 with 8 bits, 16 bits, 18 bits, 32 bits, or any other number of bits. Multiplier 208 can be implemented using multiplier circuitry. Multiplier 208 can be configured to perform floating-point multiplication, integer multiplication, or multiplication involving any other data type. Multiplier 208 can be implemented using a 16-bit multiplier data path, an 18-bit multiplier data path, or a multiplier data path with any number of bits. Multiplier 208 can be configured to support at least n-bit operations, where n is greater than or equal to the number of bits in the input.

[0056] Multiplier 208 may contain multiple data paths, such as those related to... Figure 5 Further discussion is needed. (See reference...) Figure 2A Multiplier 208 may include separate data paths for calculating the sign bit, significant digits, and exponent. Multiplier 208 may adjust the functionality in the data paths based at least in part on data type control signal 235. For example, if data type control signal 235 indicates computation using a normalized BF16 number with an exponent extended from eight bits to ten bits, multiplier 208 may map the shorter 7-bit significant digits to a longer 10-bit significant digit representation and further utilize the normalized 10-bit exponent of the normalized BF16 number. As another example, if data type control signal 235 indicates computation using a normalized FP16 number with an exponent extended from five bits to seven bits, multiplier 208 may map the shorter 7-bit exponent to a 10-bit exponent representation and further utilize the 10 significant bits of the FP16 number. Thus, both BF16 and FP16 numbers can be represented within multiplier 208 in a common format (e.g., 10-bit exponent and 10-bit significant digits). The data type control signal 235 can be configured to provide an interpretation of the data type of at least one of the normalized input data element 222 and the normalized weight 224.

[0057] Multiplier 208 can provide product 250 to adder 210. Adder 210 can perform addition on product 250 and stored input portion 236 to provide addition result 238. Adder 210 can be implemented using adder circuitry. Adder 210 can be configured to perform floating-point addition, integer addition, or non-integer addition. Adder 210 can be configured to perform addition on inputs with 8 bits, 16 bits, 18 bits, 32 bits, 34 bits, or any number of bits. Adder 210 can be implemented using a 16-bit adder data path, an 18-bit adder data path, a 32-bit adder data path, a 34-bit adder data path, or an adder data path with any number of bits. Adder 210 can be configured to support at least m-bit operations, where m is equal to or greater than the value of the multiplier data path. Adder data path can be configured as a superset of multiplier data path.

[0058] Multiplier 208 and adder 210 can provide fused multiplication-accumulation operations. Multiplier 208 and adder 210 can be integrated together to perform single-step multiplication-accumulation operations. PE 00 can be configured to not round the output of multiplier 208. In some implementations, PE 00 can perform rounding on the output of multiplier 208. Multiplier 208 can be configured to provide an accurate product 250 to adder 210.

[0059] Selector circuit 216 may receive addition result 238, input portion sum 236, and a stored skip calculation indicator 254. Selector circuit 216 may select either addition result 238 or input portion sum 236 to provide as output portion sum 240 via a sixth port. In some embodiments, selector circuit 216 may include at least one multiplexer that may select either addition result 238 or input portion sum 236 to be produced. Selector circuit 216 may be configured to select either addition result 238 or input portion sum 236 based on the stored skip calculation indicator 254 to provide as output portion sum 240 via a sixth port. According to some embodiments, when the value of normalized input data element 222 or normalized weight 224 for the current operation is zero, or when NOP 258 is asserted, addition result 238 may retain the value of the previous operation because product 250. In this scenario, the stored skip calculation indicator 254 can allow bypassing the addition result 238 and selecting the input portion 236 as the output portion 240. For example, when the stored skip calculation indicator 254 provides a skip calculation signal of "1", the input portion 236 can be selected as the output portion 240 of the systolic array period, and when the stored skip calculation indicator 254 provides a skip calculation signal of "0", the addition result 238 can be selected as the output portion 240 of the systolic array period.

[0060] Figure 2B This illustrates a scenario where a shared normalizer 225 is used instead of the first normalizer 225 and the second normalizer 227. Figure 2A The diagram shows that input data element 221 and weight 223 can be provided to shared normalizer 225. Opcode 230 can also be provided to shared normalizer 225. Shared normalizer 225 can perform a selection operation on input data element 221 and weight 223, at least partially based on opcode 230. In some implementations, shared normalizer 225 will generate normalized input, at least partially based on opcode 230. For example, when opcode 230 provides a certain set value, shared normalizer 225 can be configured to perform a normalization operation on weight 223 and provide normalized weight 224 to PE 00. Furthermore, when opcode 230 provides some other set value, shared normalizer 225 can be configured to perform a normalization operation on input data element 221 and provide normalized input data element 222 to PE 00.

[0061] In some implementations, the shared normalizer 225 may use a multiplexer to select between input data element 221 and weight 223. Normalized input data element 222 and normalized weight 224 may be transmitted to PE 00 via a separate bus. In some implementations, normalized input data element 222 and normalized weight 224 may be transmitted on the same bus. In some implementations, the shared normalizer 225 may normalize both input data element 221 and weight 223 within the same clock cycle and provide normalized input data element 222 and normalized weight 224 to PE 00. In some implementations, the shared normalizer 225 may normalize weight 223 and provide normalized weight 224 to PE 00 during a clock cycle. The shared normalizer 225 may then normalize input data element 221 and provide normalized input data element 222 to PE 00 during a second clock cycle.

[0062] Figure 3 A device 300 is shown, according to certain embodiments of the disclosed technology, including a zero detector circuit for normalizing input data elements and normalizing weights into a systolic array for neural network computation.

[0063] Device 300 may include a two-dimensional pulsation array 302, which includes PEs arranged in rows and columns. The pulsation array 302 may be similar to... Figure 1 The pulsation array 100 is shown in the image. The first row of the pulsation array 302 may include PE 00, PE 01, PE 02, ..., PE 0y; the second row may include PE 10, PE 11, PE 12, ..., PE 1y; the third row may include PE 20, PE 21, PE 22, ..., PE 2y; and the Xth row of the pulsation array 302 may include PEx0, PE x1, PE x2, ..., PE xy. x and y may include positive integers, such as 32, 64, 128, or any suitable number. Each PE of the pulsation array 302 may be similar to PE 01 and includes, as shown in the reference... Figure 2A and Figure 2B The apparatus for performing arithmetic calculations using a power-efficient method.

[0064] In some embodiments, the first (e.g., the leftmost) PE in each row of the systolic array 302 can be coupled to a corresponding zero-input data detector circuit to detect zero values ​​on the input data elements, and a corresponding zero-weight detector circuit to detect zero values ​​of the weight values ​​entering the systolic array 302. For example, PE 00 in the first row can be coupled to the first zero-input data detector 306a and the first zero-weight detector 308a, PE 10 in the second row can be coupled to the second zero-input data detector 306b and the second zero-weight detector 308b, PE 20 in the third row can be coupled to the third zero-input data detector 306c and the third zero-weight detector 308c, and PE x0 in the Xth row can be coupled to the Xth zero-input data detector 306x and the Xth zero-weight detector 308x. The first zero-input data detector 306a, the second zero-input data detector 306b, the third zero-input data detector 306c, ... and the Xth zero-input data detector 306x can be configured to detect the zero values ​​of the corresponding normalized input data elements in input dataset 0, input dataset 1, input dataset 2, ... and input dataset x, respectively. Similarly, the first zero-weight detector 308a, the second zero-weight detector 308b, the third zero-weight detector 308c, ... and the Xth zero-weight detector 308x can be configured to detect the zero values ​​of the corresponding normalized weight values ​​in filter 0, filter 1, filter 2, ... and filter x, respectively.

[0065] Each zero-input data detector and each zero-weight detector in each row of the pulsating array 302 can be coupled to a corresponding normalizer to receive normalized input. Each zero-input data detector can receive normalized input data elements and each zero-weight detector can receive normalized weights. For example, the first zero-input data detector 306a can be coupled to the first normalizer 307a and the first zero-weight detector 308a can be coupled to the second normalizer 309a, the second zero-input data detector 306b can be coupled to the third normalizer 307b and the second zero-weight detector 308b can be coupled to the fourth normalizer 309b, the third zero-input data detector 306c can be coupled to the fifth normalizer 307c and the third zero-weight detector 308c can be coupled to the sixth normalizer 309c, and the Xth zero-input data detector 306x can be coupled to the Xth normalizer 307x and the Xth zero-weight detector 308x can be coupled to the Yth normalizer 309x.

[0066] Normalizers 307a-307x and 309a-309x can be implemented as separate entities outside of systolic array 302. For example, normalizers 307a-307x and 309a-309x can be part of a circuit. In other embodiments, the circuit and systolic array 302 can be part of a computation engine configured to perform arithmetic computations of convolution operations. In some implementations, normalizers 307a-307x and 309a-309x can be implemented as part of systolic array 302.

[0067] In some implementations, the first normalizer 307a and the second normalizer 309a can be first shared normalizers, the third normalizer 307b and the fourth normalizer 309b can be second shared normalizers, the fifth normalizer 307c and the sixth normalizer 309c can be third shared normalizers, and the Xth normalizer 307x and the Yth normalizer 309x can be Xth shared normalizers. Each shared normalizer can provide normalized input data elements and normalization weights. In some implementations, each shared normalizer can include one output bus, and each shared normalizer must select a normalized input to generate. In some implementations, each shared normalizer can include multiple output buses, and each shared normalizer can output normalized input data elements and normalization weights.

[0068] In some implementations, zero-input data detectors 306a-306x and / or zero-weight detectors 308a-308x can be positioned before the corresponding normalizers 307a-307x and 309a-309x, allowing zero input to be detected first. If zero input is detected, the corresponding normalizers 307a-307x and 309a-309x can be deactivated to save power. In some implementations, both the zero-input data detectors 306a-306x and the corresponding normalizers 307a-307x can receive input datasets and operate in parallel rather than sequentially. In some implementations, both the zero-weight detectors 308a-308x and the corresponding normalizers 309a-309x can receive filters and operate in parallel rather than sequentially.

[0069] Each of the input datasets 0, 1, 2, ..., and x can be an image, text, video clip, audio clip, or another dataset that may need to be processed by a neural network processor for convolutional computation.

[0070] In some cases, input datasets 0, 1, 2, ..., and x can be associated with output datasets 0, 1, 2, ..., y generated by intermediate layers of convolutional operations. For example, output datasets 0, 1, 2, ..., y can be fed back to systolic array 302 via activation functions and as input datasets 0, 1, 2, ..., and x. Filters 0, 1, 2, ..., and x can include different sets of weight values ​​for convolution with input datasets 0, 1, 2, ..., and x. The weight values ​​in filters 0, 1, 2, ..., and x can be pre-determined using supervised learning, unsupervised learning, or any suitable method for determining convolutional filters.

[0071] Each zero-input data detector in the corresponding row can be configured to detect whether a normalized input data element from the input dataset entering the corresponding row is "0" and generate a corresponding zero-input data indicator for that normalized input data element. Each zero-input data detector in the corresponding row can also be configured to detect whether an input data element from the input dataset entering the corresponding normalizer is "0" and generate a corresponding zero-input data indicator for that input data element. The corresponding zero-data element indicator can be passed to the first PE of the corresponding row along with the input data element. For example, PE 00 can be the first PE in the first row of the systolic array 302. PE 00 can be configured to receive normalized input data elements from input dataset 0 before other PEs in the first row (e.g., PE 01, PE 02, ..., PE 0y). In some embodiments, normalized input data elements can be sequentially fed from input dataset 0 to PE00 one at a time within a uniform time period. The first zero-input data detector 306a can be configured to generate a zero-data element indicator 226 for each input data element from input dataset 0 at each of the uniform time periods (e.g., clock cycles). A zero-data element indicator 226 corresponding to each normalized input data element can be sequentially fed to PE 00 along with each normalized input data element in a uniform time period. PE 00 can store or skip storing normalized input data elements 222 based on the value of the corresponding data loading signal 242. In some implementations, the first zero-input data detector 306a may include a comparator to compare an incoming normalized input data element with zero to assert (e.g., set to "1") or deassert (e.g., set to "0") the zero-data element indicator 226 based on the value of the incoming normalized input data element. For example, the comparator can be implemented using OR, XOR, NAND, or any suitable circuitry.

[0072] Each zero-weight detector in a corresponding row can be configured to detect whether a normalized weight value from a set of normalized weight values ​​entering the corresponding row is zero, and generate a corresponding zero-weight indicator for that normalized weight value. Furthermore, each zero-weight detector can also be configured to detect whether a weight value from a set of filters entering the corresponding normalizer is zero, and generate a corresponding zero-weight indicator for that weight value. For example, a first zero-weight detector 308a can be configured to detect whether a normalized weight value from filter 0 (e.g., normalized weight 224) includes a zero value, and generate a zero-weight indicator 228 for the normalized weight. In some implementations, the first zero-weight detector 308a may include a comparator to compare the normalized weight value with zero to assert (e.g., set to "1") or deassert (e.g., set to "0") the zero-weight indicator 228. For example, the comparator can be implemented using OR, XOR, NAND, or any suitable circuitry. In one embodiment, normalized weight values ​​can be fed sequentially from filter 0 to PE 00 one at a time over a uniform time period, preloading the corresponding normalized weight values ​​in PE 00 into PE 0y before starting arithmetic calculations. The first zero-weight detector 308a can generate a corresponding zero-weight indicator for each of the normalized weight values ​​that can be sequentially fed to PE 00 along with their corresponding normalized weight values ​​over a uniform time period. PE 00 can sequentially pass the corresponding normalized weight value and corresponding zero-weight indicator to the next adjacent PE until all PEs in the first row have been preloaded with the corresponding normalized weight value and corresponding zero-weight indicator. The corresponding normalized weight value and corresponding zero-weight indicator can be cached in each PE before feeding the corresponding normalized input data element into each row of the systolic array 302.

[0073] The second zero-input data detector 306b, the third zero-input data detector 306c, ..., and the Xth zero-input data detector 306x can be similar to the first zero-input data detector 306a and can generate corresponding zero-data element indicators (similar to zero-data element indicator 226) to be sequentially provided to PE 10, PE 20, ..., and PE x0 over a uniform time period for power optimization. The corresponding zero-data element indicator generated for each row can be received by the corresponding first PE in each row via the corresponding row input bus 102 and sequentially propagated by the first PE to all PEs in a given row over a uniform time period. The second zero-weight detector 308b, the third zero-weight detector 308c, ..., and the Xth zero-weight detector 308x can be similar to the first zero-weight detector 308a and can generate corresponding zero-weight indicators (similar to zero-weight indicator 228) to be sequentially provided to PE 10, PE 20, ..., and PE x0 to preload each PE in the corresponding row and its corresponding weight value before the arithmetic calculation begins.

[0074] In some embodiments, the zero-input data detectors 306a-306x and the zero-weight detectors 308a-308x can be implemented as separate entities outside of the systolic array 302. For example, the zero-input data detectors 306a-306x and the zero-weight detectors 308a-308x can be part of circuitry 304. In other embodiments, circuitry 304 and the systolic array 302 can be part of a computation engine that can be configured to perform arithmetic computations of convolution operations. Compared to using a corresponding zero detector within each PE in the systolic array 302, some embodiments of the disclosed technology can provide reduced gate count and dynamic power consumption by detecting zeros on input data elements and the weights of the corresponding first PEs in each row of the systolic array, and passing a zero indicator to all PEs in the array.

[0075] Please note that, for ease of explanation, Figure 3 Only the corresponding zero data element indicator and zero weight indicator for the first PE in each row of the systolic array 302 are shown. However, it should be understood that each PE in the corresponding row of the systolic array 302 may also receive the corresponding normalized input data element and the corresponding normalized weight value, as well as some control signals (e.g., opcode 230, weight load 232, data type, etc.), which can propagate from the left to the right side of the systolic array 302 for each row.

[0076] Figure 4AAn example FP16 normalizer 400 is shown according to an example implementation. Normalizer 400 includes a multiplexer 402, a non-regular detector 404, an exponent expander 406, and a non-regular redeformer 405. Non-regular redeformer 405 includes a leading zero encoder 408, a shifter 410, and a subtractor 412. Normalizer 400 processes the original number 401 to produce a normalized number 403.

[0077] Normalizer 400 can be configured to receive one or more numbers to be normalized. The one or more numbers may include input data element 221 and weights 223, which may or may not be non-normalized representations. In some implementations, normalizer 400 may receive either input data element 221 or weights 223 without a multiplexer. In some implementations, the one or more numbers may be normalized numbers. If a normalized number is detected, normalizer 400 may still perform a portion of the normalization operation to increase the number of bits used to represent the exponent.

[0078] Multiplexer 402 can be configured to receive one or more numbers received by normalizer 400. Multiplexer 402 can also receive opcode 230 or other indicators indicating whether weights or input data elements should be selected. Multiplexer 402 can be configured to decode opcode 230 to select the number to be operated on by normalizer 400. Multiplexer 402 can be configured to output different numbers for normalizer operation based on the value of opcode 230. In some embodiments, a first opcode value may correspond to an instruction to output weight 223 as multiplexer output 420. A second opcode value may correspond to an instruction to output input data element 221 as multiplexer output 420. For example, once input data element 221 and weight 223 have been provided to normalizer 400 and multiplexer 402, multiplexer 402 can be configured to output input data element 221 at least partially based on opcode 230, and subsequently output weight 223.

[0079] The primitive number 401 is an FP16 number with a sign bit, an exponent bit, and a significant bit. The FP16 format of primitive number 401 includes 1 sign bit, 5 exponent bits, and 10 significant bits. Other implementations can process other 16-bit floating-point numbers using different bit formats. Other implementations can handle primitive number 401, which may contain 8 bits, 32 bits, or 64 bits. Primitive number 401 can be stored in a register.

[0080] The original number 401 can be provided to the non-regularity detector 404 for non-regularity detection. The non-regularity detector 404 can be configured to detect whether the multiplexer output 420 is non-regular based at least in part on the value of a 1-bit sign, a 5-bit exponent, or a 10-bit significant number. For example, the non-regularity detector 404 can be configured to detect a non-regular number when the 5-bit exponent contains zeros in each bit and the significant number is non-zero. The non-regularity detector 404 can provide an enable signal 424 to the non-regularity reformer 405 based at least in part on the detection of a non-regular number. For example, if the non-regularity detector 404 detects that the multiplexer output 420 is non-regular, the non-regularity detector 404 can provide a first value to the non-regularity reformer 405. If the non-regularity detector 404 detects that the multiplexer output 420 is regular, the non-regularity detector 404 can provide a second value to the non-regularity reformer 405. In some implementations, the first number can be 1, and the second number can be 0. The detection of nonnormal numbers can correspond to logic high, while the detection of normal numbers can correspond to logic zero.

[0081] The irregular reformer 405 can be connected to the enable signal 424 output by the irregular detector 404. The irregular reformer 405 can be operated or skipped at least in part based on the value of the enable signal 424. The irregular reformer 405 may include any one or all of the leading zero encoder 408, shifter 410, and subtractor 412. The irregular reformer 405 can be configured to enable any one or all of the leading zero encoder 408, shifter 410, and subtractor 412. In some implementations, the irregular reformer 405 may include more components.

[0082] Exponent expander 406 can receive a 5-bit exponent 428 from the original number 401. Exponent expander 406 can increase the number of bits representing the exponent 428 from 5 bits to 7 bits. In some implementations, exponent expander 406 can add 1, 2, 3, or any number of bits to the exponent 428. The number of added bits is sufficient to represent the nonnormal number in a normalized format. In the example shown, the exponent expander is configured to expand the 5-bit exponent by two bits.

[0083] Exponent extender 406 can be configured to extend exponent 428 while preserving its value. Exponent extender 406 can be configured to extend the exponent using range conversion by copying the most significant bit, inverting the most significant bit twice, and appending the remaining bits of exponent 428 to the end of the extended exponent 434. For example, if the value of exponent 428 is 01111, exponent extender 406 can copy the most significant bit "0", invert the most significant bit "11" twice, and append the last four bits "1111", resulting in an extended exponent 434 of 0111111. In some implementations, these two data bits can correspond to logic zeros or logic lows added to the end of the exponent data field. For example, if exponent 428 is 00000, exponent extender 406 can extend exponent 428 to result in an extended exponent 434 of 0000000. In various implementations, exponent extender 406 can add additional data bits to any position in the exponent field, depending on the byte order format of the exponent and whether it is signed or unsigned.

[0084] When the irregularity detector 404 detects an irregularity, the exponent expander 406 may provide an expanded version of the exponent to the subtractor 412, at least in part, based on the enable signal 424. Otherwise, the expanded version of the exponent may be stored as a 7-bit expanded exponent field of the normalized number 403.

[0085] The non-regular redo formatter 405 may include a leading zero encoder 408. The leading zero encoder 408 may be enabled at least in part based on an enable signal 424 received from the non-regular detector 404. The leading zero encoder 408 may receive a significant number 430 as input. The leading zero encoder 408 may detect multiple leading zeros in the significant number 430. The number of leading zeros in the significant number 430 may correspond to the number of zeros preceding 1 from left to right. In some implementations, the leading zero encoder 408 may be configured to detect multiple leading logic zeros. For example, if the significant number is 0001000000, the leading zero encoder 408 may detect three leading zeros. Furthermore, if the significant number is 0000001000, the leading zero encoder 408 may detect six leading zeros. The leading zero encoder 408 may be configured to use any means of detecting the number of leading zeros in the digital field. The leading zero encoder 408 can generate the leading zero output 432 based at least in part on the number of leading zeros detected by the leading zero encoder 408. In other implementations, the leading zeros can be adjusted to specify other byte order formats.

[0086] The irregular redistributor 405 may include a shifter 410. The shifter 410 may be enabled at least in part based on an enable signal 424 received from the irregular detector 404. The shifter 410 may receive a leading zero output 432 from the leading zero encoder 408 and a significant number 430 from the original number 401. The shifter 410 may be configured to shift the significant number 430 at least in part based on the leading zero output. The shifter 410 may further shift the significant number 430 such that a first non-zero number is shifted out of the significant number and represented by an implicit bit. The shifter 410 may shift the significant number 430 by adding bits containing logic low or zero to the right or end of the significant number 430. The shifter 410 may produce a shifted significant number 436, wherein the shifted significant number 436 may be the same number of bits as the significant number 430. For example, if the significant number 430 is 0000110000, the leading zero encoder 408 can count four zeros and further adjust the shift count to five, and the shifter 410 can shift the significant number 430 a total of five times to produce the shifted significant number 436, which is 1000000000. The shifter 410 can then provide the shifted significant number 436 to the normalized number 403.

[0087] The irregular redo formatter 405 may include a subtractor 412. The subtractor 412 may be enabled at least in part based on an enable signal 424 received from the irregular detector 404. The subtractor 412 may receive an expanded exponent 434 from the exponent expander 406 and a leading zero output 432 from the leading zero encoder 408. The subtractor 412 may subtract a value from the expanded exponent 434 at least in part based on the leading zero output 432. The subtractor 412 may compensate for shifts in the exponent value for significant digits. For example, if the leading zero output 432 equals 5 and the expanded exponent equals 0011111 or 31, the subtractor 412 may be configured to subtract 5 from 0011111 or 31, such that the adjusted exponent 438 equals 0011010 or 26. The subtractor 412 may provide the adjusted exponent to the normalized number 403 as an output.

[0088] The normalized number 403 can be a second-bit length, where the second-bit length is any number of bits greater than the first-bit length. The second-bit length can be large enough to include the extended exponent. When the original number 401 can be a 16-bit floating-point number, the normalized number 403 can be an 18-bit floating-point number with a sign bit portion, an exponent portion, and a significant bit portion. The normalized number 403 can contain 1 sign bit, 7 exponent bits, and 10 significant bits. The 1 sign bit can be equal to the sign 426. The 7-bit exponent can be equal to the adjusted exponent 438. The 10 significant bits can be equal to the shifted significant bit 436. In some implementations, the sign bit portion, exponent portion, and significant bit portion of the normalized number 403 can be any value. The normalizer 400 can be configured to provide the normalized number 403 as a normalized output 421. The normalized output 421 can be the normalized input data element 222, the normalized weight 224, or any other normalized number.

[0089] Figure 4B An example BF16 normalizer 450 is shown, based on an example implementation. The BF16 normalizer 450 includes features similar to... Figure 4A The components of the FP16 normalizer, in addition to Figure 4B In the original number 407, it begins in BF16 format and is normalized to a normalized number 409, which is in a different format than the normalized number 403.

[0090] The primitive number 407 is a BF16 number with a sign bit, an exponent bit, and a significant bit. The BF16 format of the primitive number 407 includes 1 sign bit, 8 exponent bits, and 7 significant bits. Other implementations can handle other 16-bit floating-point numbers using different bit formats. Other implementations can handle the primitive number 407, which may contain 8 bits, 32 bits, or 64 bits. The primitive number 407 can be stored in a register.

[0091] The normalized number 409 can be an 18-bit floating-point number with a sign part, an exponent part, and a significant part. The normalized number 409 can contain 1 sign bit, 10 extended exponent bits, and 7 shifted significant bits. In some implementations, the normalized number 409 can contain 8 bits, 32 bits, or 64 bits. In some implementations, the normalized number 409 can be an integer or non-integer data type.

[0092] Exponent expander 406 can receive an 8-bit exponent 428 from the original number 407. Exponent expander 406 can increase the number of bits representing the exponent 428 from 8 bits to 10 bits. In some implementations, exponent expander 406 can add 1, 2, 3, or any number of bits to the exponent 428. When the irregularity detector 404 detects an irregularity, exponent expander 406 can provide the expanded version of the exponent to subtractor 412, at least in part, based on enable signal 424. Otherwise, the expanded version of the exponent can be stored as a 10-bit expanded exponent field of the normalized number 409.

[0093] The primitive number 407 can have an exponent length of y and a significant bit length of z. The primitive number 401 can have an exponent length of m and a significant bit length of n, where m is not equal to y and z is not equal to n. For example, the primitive number 407 can have an exponent length of 8 and a significant bit length of 7, while the primitive number 401 can have an exponent length of 5 and a significant bit length of 10. The normalized number 409 can have an exponent length of c and a significant bit length of d. The normalized number 403 has an exponent length of r and a significant bit length of s, where c is not equal to r and d is not equal to s.

[0094] Figure 5 An example multiplication-accumulation data path 500 is shown. The example data path 500 can be implemented as follows: Figure 2A and Figure 2B The multiplier 208 and adder 210 are discussed. For example... Figure 5 As shown, multiplier 208 can be configured to receive normalized input data elements 222 and normalized weights 224 and provide the multiplicative product to adder 210. Adder 210 can be configured to receive the multiplicative product and the sum of the input portions 234 and provide the addition result 238. By converting the input to a normalized representation before presenting it to multiplier 208, multiplier 208 can omit support for non-normalized inputs and non-normalized products. Normalized inputs with extended exponents can provide a larger numerical range for computation, and multiplier 208 may not need to support numerical underflow computation.

[0095] The normalized input data element 222 can be 18 bits. In some implementations, the normalized input data element 222 can be of any number of bits. The normalized input data element 222 can be a floating-point number or a non-floating-point number. In some implementations, the normalized input data element 222 can be of any data type. The normalized input data element 222 can consist of a sign bit field, an exponent field, and a significant bit field. The multiplier 208 can be configured to support different types of normalized input data elements. For example, the normalized input data element 222 can contain 1 sign bit, 10 exponent bits, and 7 significant bits. Alternatively, the normalized input data element 222 can contain 1 sign bit, 7 exponent bits, and 10 significant bits. The multiplier 208 can support both types of 18-bit input data elements. In some implementations, the normalized input data element 222 can contain x sign bits, y exponent bits, and z significant bits. Normalized input data element 222 can be provided to multiplier 208 via first symbol data path 511, first exponent data path 521 and first significant number data path 531.

[0096] The normalized weight 224 can be 18 bits. In some implementations, the normalized weight 224 can be of any number of bits. The normalized weight 224 can be a floating-point number or a non-floating-point number. In some implementations, the normalized weight 224 can be of any data type. The normalized weight 224 can consist of a sign bit path, an exponent bit path, and a significant bit path. For example, the normalized weight 224 can contain 1 sign bit, 10 exponent bits, and 7 significant bits. Furthermore, the normalized weight 224 can contain 1 sign bit, 7 exponent bits, and 10 significant bits. In some implementations, the normalized weight 224 can contain x sign bits, y exponent bits, and z significant bits. The normalized weight 224 can be provided to the multiplier 208 via the second sign data path 512, the second exponent data path 522, and the second significant data path 532.

[0097] Multiplier 208 may include a sign data path, an exponent data path, and a significant number data path. Multiplier 208 may receive a first sign data path 511, a first exponent data path 521, and a first significant number data path 531 from normalized input data element 222. Multiplier 208 may receive a second sign data path 512, a second exponent data path 522, and a second significant number data path 532 from normalized weights 224. Multiplier 208 may also receive a data type control signal 235. Multiplier 208 may perform multiplication operations on floating-point numbers or spiking floating-point numbers. In some implementations, multiplier 208 may perform multiplication operations on floating-point numbers for a certain number of pulsation array cycles, and then subsequently perform multiplication operations on spiking floating-point numbers.

[0098] A first symbolic data path 511 and a second symbolic data path 512 can be provided to the symbolic data path of multiplier 208. The symbolic data path can be configured to output a partial symbolic data path 513, at least partially based on the first symbolic data path 511 and the second symbolic data path 512. In some implementations, the symbolic data path can be implemented as an exclusive OR (XOR) function. The symbolic data path can provide a partial symbolic data path 513 to adder 210.

[0099] A first exponential data path 521, a second exponential data path 522, and a data type control signal 235 can be provided to the exponential data path of multiplier 208. The exponential data path of multiplier 208 may include a 7-to-10 mapper 524 and an adder 526. The 7-to-10 mapper 524 can receive the first exponential data path 521, the second exponential data path 522, and the data type control signal 235. The 7-to-10 mapper 524 can be configured to perform mapping operations at least partially based on the data type control signal 235. The data type control signal 235 can indicate the data type of normalized input data element 222 or normalized weight 224. The data type control signal 235 can be configured to indicate the data type of normalized input data element 222 and / or the data type of normalized weight 224.

[0100] The 7-to-10 mapper 524 can be configured to map either the first exponential data path 521 or the second exponential data path 522 from 7 bits to 10 bits. In some implementations, the 7-to-10 mapper 524 can be configured to map the exponential data path from n bits to m bits, where n and m are any numbers less than m. The 7-to-10 mapper 524 can map the values ​​of the first exponential data path 521 and / or the second exponential data path 522 to a 10-bit range. The mapped value for each exponential data path can correspond to an unmapped value for each of the exponential data paths. For example, the 7-to-10 mapper 524 can receive the first exponential data path 521 with the value 00000001, and the 7-to-10 mapper 524 can map the first exponential data path 521 to a 10-bit field with the value 0000000001. Then, the 7 to 10 mapper 524 can provide the adder 526 with a mapped or unmapped version of the first exponential data path 521 and the second exponential data path 522, at least in part, based on the data type control signal 235.

[0101] If the exponent is already represented as 10 bits, the data type control signal 502 can cause the mapping function to be skipped. For example, as regarding Figure 4A and Figure 4BThe FP16 exponent field is extended to a 7-bit exponent, while the BF16 exponent field is extended to a 10-bit exponent. Therefore, if the data type control signal 502 indicates that the data type is BF16, an extended 10-bit exponent is provided as the exponent field of the 18-bit input data element 222, and the 7-to-10 mapper does not perform any further mapping. Alternatively, if the data type control signal 502 indicates that the data type is FP16, an extended 7-bit exponent is provided as the exponent field of the 18-bit input data element 222, and the 7-to-10 mapper maps the 7-bit exponent to the 10-bit exponent. In other implementations, a shorter exponent field of any length can be mapped to a longer exponent field, where the shorter and longer exponent fields are part of different, supported normalization formats.

[0102] Adder 526 may receive mapped or unmapped versions of the first exponential data path 521 and the second exponential data path 522, at least in part, based on data type control signal 235. Adder 526 may be configured to perform addition on two values ​​received from 7-to-10 mapper 524. Adder 526 may also receive shift / carry information (not shown) from the significant number data path. Adder 526 may provide a partial exponential data path 523, at least in part, based on the addition performed on two values ​​received from adder 210 from 7-to-10 mapper 524. The partial exponential data path 523 may be 10 bits or another range sufficient to accommodate the exponential sum without overflow.

[0103] A first significant number data path 531, a second significant number data path 532, and a data type control signal 235 can be provided to the significant number data path of multiplier 208. The significant number data path of multiplier 208 may include a 7-to-10 mapper 534, an implicit bit parser 536, a binary multiplier 540, and a format adjuster 542. The 7-to-10 mapper 534 can receive the first significant number data path 531, the second significant number data path 532, and the data type control signal 235. The 7-to-10 mapper 534 can be configured to perform mapping operations at least partially based on the data type control signal 235. The data type control signal 235 can indicate the data type of the normalized input data element 222 and / or the normalized weight 224. The 7-to-10 mapper 534 can map either the first significant number data path 531 or the second significant number data path 532 from 7 bits to 10 bits. The 7-to-10 mapper 534 can map both the first significant data path 531 and the second significant data path 532 from 7 bits to 10 bits. In some implementations, the 7-to-10 mapper 534 can be configured to map the significant data path from j bits to k bits, where j and k are any numbers less than k. The 7-to-10 mapper 534 can map the value of either the first significant data path 531 or the second significant data path 532 to a 10-bit range. The mapped value of each of the significant data paths can correspond to an unmapped value of each of the significant data paths. For example, the 7-to-10 mapper 534 can receive the first significant data path 531 with the value 0000001, and the 7-to-10 mapper 534 can map the first significant data path 531 to a 10-bit field with the value 0000000001. Then, the 7 to 10 mapper 534 can provide the implicit bit parser 536 with a mapped or unmapped version of the first valid number data path 531 and the second valid number data path 532, at least in part, based on the data type control signal 235.

[0104] If the exponent is already represented as 10 bits, the data type control signal 502 may cause the mapping function to be skipped. For example, as regarding Figure 4A and Figure 4BThe shifted significant number field for FP16 numbers is 10 bits, while the shifted significant number field for BF16 numbers is 7 bits. Therefore, if the data type control signal 502 indicates that the data type is BF16, the shifted 7 significant bits are processed in the significant number data path without further mapping by the 7-to-10 mapper. Alternatively, if the data type control signal 502 indicates that the data type is FP16, the shifted 7 significant bits are provided as the significant number field of the 18-bit input data element 222, and the 7-to-10 mapper maps the 7 significant bits to 10 significant bits. In other implementations, a shorter exponent field of any length can be mapped to a longer exponent field, where the shorter and longer exponent fields are both part of different, supported normalization formats.

[0105] Implicit bit resolver 536 can provide implicit bits to the output of 7-to-10 mapper 534. Implicit bit resolver 536 can be configured to add the most significant bit to the output of 7-to-10 mapper 534. Implicit bit resolver 536 can provide binary multiplier 540 with a first output corresponding to a first significant data path with implicit bits and a second output corresponding to a second significant data path with implicit bits.

[0106] Binary multiplier 540 can multiply the first output of implicit bit parser 536 by the second output of implicit bit parser 536, where the first and second outputs of implicit bit parser 536 correspond to the first and second significant number data paths 531 and 532. Binary multiplier 540 can be configured to generate a multiplicative product based on multiplication operations. In some implementations, the product can be an integer product, a floating-point product, or any other product. In some implementations, binary multiplier 540 can generate a product of 8 bits, 16 bits, 32 bits, or any other number of bits. Binary multiplier 540 can be implemented using multiplier circuitry. Binary multiplier 540 can be configured to perform floating-point multiplication, integer multiplication, or multiplication involving any other data type. Binary multiplier 540 can be implemented using a 16-bit multiplier data path, an 18-bit multiplier data path, or a multiplier data path with any number of bits. Binary multiplier 540 can provide the multiplicative product to format adjuster 542.

[0107] The format adjuster 542 can adjust the format of the multiplicative product generated by the binary multiplier 540. The format adjuster 542 can, for example, increase or decrease the number of bits used to represent the multiplicative product by increasing the bit size to 23 bits. The format adjuster 542 can provide a partial valid data path to the adder 210.

[0108] Adder 210 may include a sign data path, an exponent data path, and a significant number data path. Adder 210 may be configured to receive a portion of the sign data path 513, a portion of the exponent data path 523, and a portion of the significant number data path 533 from multiplier 208. Adder 210 may also receive an input sum 234. Adder 210 may perform addition on the multiplicative product consisting of the portion of the sign data path 513, the portion of the exponent data path 523, the portion of the significant number data path 533, and the input sum 234. Adder 210 may perform addition on both floating-point numbers and non-floating-point numbers. In some implementations, adder 210 may perform addition on floating-point numbers, and adder 210 is a 32-bit floating-point adder. In some implementations, adder 210 may perform multiplication only on non-floating-point numbers, and adder 210 is a 34-bit floating-point adder.

[0109] Adder 210 can be configured to generate an addition result 238 based on addition operations. The addition result 238 can consist of a sign data path 515, an exponent data path 525, and a significant data path 535. In some implementations, the addition result 238 can be an integer sum, a floating-point sum, or any other sum. In some implementations, adder 210 can generate a sum of 8 bits, 16 bits, 32 bits, 34 bits, or any other number of bits. Adder 210 can be implemented using binary adder circuitry. Adder 210 can be configured to perform floating-point addition, integer addition, or addition involving any other data type.

[0110] Figure 6 An apparatus 600 for neural network computation according to some embodiments of the disclosed technology is shown. Apparatus 600 may be part of a computer system, such as a host server. For example, the host server may provide multi-tenant computing services for data processing applications such as image recognition services, text-based data processing (e.g., search query processing), audio data processing, video data processing, etc. In some embodiments, the host device may operate a software application and communicate with apparatus 600 to make predictions based on computations using a predictive model utilizing a neural network processor. For example, the host device may make predictions by using a predictive model to identify information included in an input dataset of images, text, audio, video, etc.

[0111] Device 600 may include a neural network processor 602 coupled to memory 614, host interface 616, and direct memory access (DMA) controller 618 via interconnect 620. Neural network processor 602 may include a computation engine 604, a computation controller 606, a state buffer 608, an output buffer 610, and an activation engine 612. Neural network processor 602 can provide computational resources to support the computation of predictive models. Neural network processor 602 may be implemented as a system-on-a-chip (SoC), a field-programmable gate array (FPGA), or any suitable circuitry.

[0112] Memory 614 can be configured to store instructions received from the host device, input datasets (e.g., pixel data of an image), and weights (e.g., weights corresponding to certain visual and / or non-visual features). Memory 614 can also be configured to store the output of the neural network processor 602 (e.g., one or more image recognition decisions on the input image in the form of an output dataset). Memory 614 can include any suitable memory, such as dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate DRAM (DDR DRAM), storage-class memory (SCM), flash memory, etc.

[0113] Host interface 616 can be configured to enable communication between a host device and neural network processor 602. For example, host interface 616 can be configured to transfer memory descriptors, including memory addresses of stored data (e.g., input datasets, weights, computation results, etc.), between the host device and neural network processor 602. Host interface 616 may include, for example, a Peripheral Component Interconnect Fast (PCIe) interface, or any suitable interface for communicating with the host device. The host device may include a host processor and host memory.

[0114] The DMA controller 618 can be configured to perform DMA operations to transfer data between the neural network processor 602 and the host device. For example, as described above, the host device can store instructions, input datasets, and weights in memory 614. The host device can provide the memory addresses of the stored instructions, data, and weights to the neural network processor 602 (e.g., in the form of memory descriptors). The neural network processor 602 can then retrieve the stored instructions, data, and weights based on the memory addresses provided by the host device. The neural network processor 602 can also store computation results (e.g., one or more image recognition decisions) in memory 614 and provide the memory addresses of the stored results to the host device.

[0115] State buffer 608 can be configured to provide a cache for data used in computations at computation engine 604. The data cached in state buffer 608 may include, for example, input datasets and weights fetched from memory 614, as well as intermediate outputs of computations at computation engine 604. Caching can reduce the impact of memory access bottlenecks (e.g., latency caused by memory 614, DMA controller 618, interconnect 620, etc.) on the performance of computation engine 604. State buffer 608 may be an on-chip memory device and may include static random access memory (SRAM) or any suitable memory.

[0116] The computation controller 606 can be configured to provide control to various components of the neural network processor 602 to perform neural network computations. In some implementations, the computation controller 606 can read instructions stored in memory 614 and schedule the computation engine 604 to execute the instructions. In a first embodiment, the computation controller 606 can perform the scheduling of loading weights into the computation engine 604 before reading input data elements from the state buffer 608. For example, as referenced... Figure 2A , Figure 2B , Figure 4A and Figure 4B The computation controller 606 can provide opcode 230 and weight payload 232 to the computation engine 604 based on instructions received from the host device. The computation controller 606 can provide the appropriate value of opcode 230 to the computation engine 604, which can be decoded by each PE in the computation engine to perform the corresponding operation. For example, the computation engine 604 can use the weight payload 232 and opcode 230 to preload weights in all PEs in the computation engine 604. Once the weights have been preloaded, the computation controller 606 can execute a schedule to sequentially load input data elements from the state buffer 608 into the computation engine 604 over a uniform time period to begin arithmetic computation.

[0117] In the second embodiment, the computation controller 606 can perform a schedule to sequentially load weights and input data elements from the state buffer 608 into the computation engine 604 over a unified time period. The computation controller 606 can use a corresponding row data bus to schedule the loading of weights and input data elements in the corresponding first PE of each row in the pulsating array 302. For example, the corresponding input data elements and weight values ​​can be loaded into the first PE of the corresponding row in each cycle.

[0118] In another embodiment, the compute controller 606 can schedule the parallel loading of weights in the systolic array 302 for each row using the corresponding column data bus for each PE in a given row. For example, the weights for each row can be loaded in parallel each cycle. In some implementations, the compute controller 606 can determine the data type of the input dataset based on instructions received from the host device. The instructions can be in the form of opcodes. The data type can indicate the size and type of the input data elements, such as 4-bit, 8-bit, 16-bit, signed, unsigned, or floating-point.

[0119] Computation engine 604 can be configured to perform computations in a neural network. In some embodiments, computation engine 604 may include a set of physical entities (PEs) configured to perform one or more arithmetic operations involved in neural network computation. Each PE can perform a multiplication-accumulation operation using an input dataset and associated weights. For example, computation engine 604 may include a systolic array 302 and circuitry 304 including zero-input data detectors 306a-306x and zero-weight detectors 308a-308x. In some embodiments, the zero-input data detectors 306a-306x and zero-weight detectors 308a-308x may be external to computation engine 604. Computation engine 604 can execute instructions scheduled by computation controller 606 to sequentially load weights and input datasets from state buffer 608 into computation engine 604.

[0120] In the first embodiment, the weights can be preloaded before reading the input dataset from the state buffer 608, as shown in the reference. Figure 4A and 4B The corresponding zero-weight indicator for each weight can be cached locally in each PE, and when an input data element is fed into the computation engine 604 along with its corresponding zero-weight indicator, the cached value can be used to perform arithmetic calculations on the corresponding input data element. In the second embodiment, weights and the input dataset can be read simultaneously from the state buffer 608, as described in the reference. Figure 5 The corresponding zero data element indicator and zero weight indicator can be provided by the corresponding zero detector circuit and propagated sequentially from one PE to another for the corresponding row. Weights and input datasets can be obtained from the state buffer 608 using one or more interfaces. In some embodiments, the computation engine 604 can perform arithmetic calculations to use, as described in the reference... Figure 2A and 2B -5 The corresponding zero data element indicator and zero weight indicator signals are used to reduce the dynamic power consumption of the systolic array 302 and to provide the calculation results to be stored in the output buffer 610.

[0121] Output buffer 610 may include a set of registers to store the output dataset generated by computation engine 604. In some implementations, output buffer 610 may also enable additional processing, such as pooling operations, to reduce the size of the stored output. In some implementations, computation engine 604 may be operated to perform computations on a specific neural network layer, and output buffer 610 may process the output of that neural network layer and store the processed output dataset (with or without processing by activation engine 612) in state buffer 608. Computation engine 604 may use the processed output dataset as an intermediate output. In some embodiments, output buffer 610 may include adders to accumulate partial sums generated for different filter banks and input datasets to generate a convolutional output array. The final output value of the convolutional output array stored in state buffer 608 may be retrieved by computation controller 606 for storage in state buffer 608.

[0122] The activation engine 612 can be configured to apply one or more activation functions (e.g., ReLU functions) to the output of the output buffer 610. 。 For example, activation engine 612 may include one or more lookup tables (e.g., in the form of a multiplexer circuit) that can map inputs to one of the candidate outputs representing the result of applying an activation function to the inputs. In some examples, activation engine 612 may also include bypass paths to allow outputs from output buffer 610 to be stored directly in state buffer 608 when no activation function is applied.

[0123] Figure 7 A method 700 for neural network computation performed by a PE according to some embodiments of the disclosed technology is shown. The PE may be part of a systolic array 302, such as PE 00. Figure 6 As shown, the pulsating array 302 can be part of the computing engine 604.

[0124] In step 702, the first normalizer may receive a first input. For example, the first normalizer may receive a first input corresponding to an input data element or weight. The first input may be represented in a floating-point format with a first bit length, where the first bit length can be any number of bits. For example, the first input may be a 16-bit floating-point number. The first input may contain 8 bits, 16 bits, 32 bits, or any number of bits. The first input may be a normal number or a non-normal number.

[0125] In step 704, the first normalizer can generate a first normalized number based at least in part on the first input. The first normalizer can detect whether the first input is normal or nonnormal. When the first input is normal, the first normalizer can generate a first normalized number, where the first normalized number is an extended version of the first input. When the first input is nonnormal, the first normalizer can normalize the first input and generate the first normalized number, where normalizing the first input includes at least one of expanding the exponent, shifting significant numbers, and subtracting a shift value from the exponent. The first normalized number can be represented in a floating-point format with a second bit length, where the second bit length can be any number of bits greater than or equal to the first bit length. For example, the first input can be a 16-bit floating-point number, and the first normalized number can be an 18-bit floating-point number. The first normalizer can include a nonnormal detector, an exponent expander, a leading zero encoder, a shifter, a subtractor, and a multiplexer.

[0126] In step 706, PE 00 may receive a first normalized number and a second normalized number corresponding to the second input. In some implementations, PE 00 may include a second normalizer to receive the second input and generate the second normalized number at least partially based on the second input. The second input may correspond to an input data element or a weight. When the first input corresponds to a weight, the second input may correspond to an input data element, and when the first input corresponds to an input data element, the second input may correspond to a weight. PE 00 may multiply the first normalized number by the second normalized number to produce a multiplier product. PE 00 may include multiple multiplier data paths and may select a multiplier data path at least partially based on at least one of the data types of the first input or the second input. For example, when at least one of the first and second inputs is a 16-bit floating-point data type, PE 00 may select a first multiplier data path, and when at least one of the first and second inputs is a 16-bit floating-point data type, PE 00 may select a second multiplier data path. The multiplier product may be represented in any-bit floating-point format or in a floating-point format.

[0127] In step 708, PE 00 can receive the multiplier product and the input partial sum. PE 00 can add the multiplier product to the input partial sum to produce an addition result. PE 00 can perform addition on floating-point numbers and non-floating-point numbers. The addition result can be represented in either a floating-point format or a non-floating-point format with any number of bits. PE 00 can provide the addition result to a selector circuit, which is configured to select between the addition result and the input partial sum for PE 00 to output as an output partial sum. PE 00 can provide the output partial sum to a subsequent PE. For example, PE 00 can generate an output partial sum and provide it to PE 10, where PE 01 receives the output partial sum as an input partial sum.

[0128] Figure 8 An example of a computing device 800 is shown. The functionality and / or several components of the computing device 800 can be used without limitation with other embodiments disclosed elsewhere in this disclosure. The computing device 800 can perform computations to facilitate the processing of tasks. As an illustrative example, the computing device 800 may be part of a server in a multi-tenant computing service system. Various hardware and software resources of the computing device 800 (e.g., hardware and software resources associated with data processing) can be allocated to clients upon request.

[0129] In one example, computing device 800 may include processing logic 802, bus interface module 804, memory 806, and network interface module 808. These modules may be hardware modules, software modules, or a combination of hardware and software. In some cases, modules may be used interchangeably with components or engines without departing from the scope of this disclosure. Computing device 800 may include additional modules, which are not described herein for ease of illustration. In some implementations, computing device 800 may include fewer modules. In some implementations, one or more modules may be combined into a single module. One or more modules may communicate with each other via communication channel 810. Communication channel 810 may include one or more buses, meshes, matrices, structures, combinations of these communication channels, or some other suitable communication channel.

[0130] Processing logic 802 may include application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), system-on-a-chip (SoCs), network processing units (NPUs), processors configured to execute instructions, or any other circuitry configured to perform logical arithmetic and floating-point operations. Examples of processors that may be included in processing logic 802 may include processors developed by ARM®, MIPS®, AMD®, Intel®, Qualcomm®, etc. In some implementations, the processor may include multiple processing cores, each of which may be configured to execute instructions independently of the other processing cores. Furthermore, in some implementations, each processor or processing core may implement multiple processing threads that execute instructions on the same processor or processing core while maintaining logical separation between the multiple processing threads. Such processing threads executing on a processor or processing core may be exposed to software as separate logical processors or processing cores. In some implementations, multiple processors, processing cores, or processing threads executing on the same core may share certain resources, such as buses, Level 1 (L1) caches, and / or Level 2 (L2) caches. Instructions executed by processing logic 802 may be stored on a computer-readable storage medium, for example, in the form of a computer program. The computer-readable storage medium may be non-transitory. In some cases, the computer-readable medium may be part of the memory 806. The processing logic 802 may also include hardware circuitry for performing artificial neural network calculations, including, for example, a neural network processor 602.

[0131] Access to processing logic 802 can be granted to clients to provide personal assistant services requested by the clients. For example, computing device 800 can host a virtual machine on which image recognition software applications can execute. The image recognition software application, when executing, can access processing logic 802 to predict, for example, objects included in an image. As another example, access to processing logic 802 can also be granted as part of a bare-metal instance, where an image recognition software application executing on a client device (e.g., a remote computer, smartphone, etc.) can directly access processing logic 802 to perform image recognition.

[0132] Memory 806 may include volatile or non-volatile memory, or both. Memory 806 may include, for example, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, and / or some other suitable storage media. In some cases, some or all of memory 806 may be internal to computing device 800, while in others, some or all of memory may be external to computing device 800. Memory 806 may store an operating system including executable instructions that, when executed by processing logic 802, provide an execution environment for performing instructions that provide functionality for computing device 800 to perform convolution calculations. Memory 806 may also store, for example, software applications for performing artificial neural network calculations. Memory may also store and maintain several data structures and tables to facilitate the functionality of computing device 800.

[0133] Bus interface module 804 enables communication with external entities, such as host devices and / or other components in a computing system, via external communication media. Bus interface module 804 may include physical interfaces for connecting to cables, receptacles, ports, or other connections to external communication media. Bus interface module 804 may also include hardware and / or software to manage incoming and outgoing transactions. Bus interface module 804 may implement local bus protocols, such as Peripheral Component Interconnect (PCI) based protocols, Non-Volatile Memory Rapid (NVMe), Advanced Host Controller Interface (AHCI), Small Computer System Interface (SCSI), Serial Attached SCSI (SAS), Serial AT Accessory (SATA), Parallel ATA (PATA), and some other standard bus protocols or proprietary bus protocols. Bus interface module 804 may include a physical layer for any of these bus protocols, including connectors, power management, error handling, etc. In some implementations, computing device 800 may include multiple bus interface modules for communicating with multiple external entities. These multiple bus interface modules may implement the same local bus protocol, different local bus protocols, or a combination of the same and different bus protocols.

[0134] Network interface module 808 may include hardware and / or software for communicating with a network. The network interface module 808 may include, for example, a physical connector or physical port for wired connection to the network, and / or an antenna for wireless communication with the network. Network interface module 808 may also include hardware and / or software configured to implement a network protocol stack. Network interface module 808 can communicate with the network using network protocols such as TCP / IP, wireless bandwidth technology, RoCE, IEEE 802.11 wireless protocol, User Datagram Protocol (UDP), Asynchronous Transfer Mode (ATM), Token Ring, Frame Relay, High-Level Data Link Control (HDLC), Fiber Distributed Data Interface (FDDI), and / or Point-to-Point Protocol (PPP). In some implementations, computing device 800 may include multiple network interface modules, each configured to communicate with a different network. For example, in these implementations, computing device 800 may include network interface modules for communicating with wired Ethernet, wireless 802.11 networks, cellular networks, wireless bandwidth technology networks, etc. In some embodiments, the computing device 800 may receive a set of parameters from a server, such as the aforementioned weight values ​​for convolution calculation, through the network interface module 808.

[0135] The various components and modules of the computing device 800 described above can be implemented as discrete components, system-on-a-chip (SoC), ASIC, NPU, FPGA, or any combination thereof. In some embodiments, the SoC or other components can be communicatively coupled to another computing system to provide various services, such as traffic monitoring, traffic shaping, computation, etc. In some embodiments of this technology, the SoC or other components may include multiple subsystems as disclosed herein.

[0136] The modules described herein can be software modules, hardware modules, or suitable combinations thereof. If a module is a software module, it can be embodied on a non-transitory computer-readable medium and processed by a processor in any computer system described herein. It should be noted that the described processes and architectures can be executed in real-time or asynchronous mode prior to any user interaction. Figure 8 The modules are configured in the manner suggested herein, and / or the functionality described herein may be provided by one or more modules that exist as separate modules, and / or the module functionality described herein may be distributed across multiple modules.

[0137] Therefore, the specification and drawings should be considered illustrative rather than restrictive. However, it will be apparent that various modifications and changes may be made thereto without departing from the broader spirit and scope of this disclosure as set forth in the claims.

[0138] Other variations are also within the spirit of this disclosure. Therefore, while the disclosed technology allows for various modifications and alternative constructions, the specific embodiments shown have been illustrated in the accompanying drawings and described in detail above. However, it should be understood that this disclosure is not intended to be limited to the one or more specific forms disclosed, but rather, the invention is intended to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the invention as defined in the appended claims.

[0139] Unless otherwise indicated herein or obviously contradicted by the context, the use of the terms “a,” “an,” and “the,” and similar designations, in the context of describing the disclosed embodiments (especially in the context of the appended claims), should be interpreted to cover both the singular and plural. Unless otherwise indicated, the terms “comprising,” “having,” “including,” and “containing” should be interpreted as open-ended terms (i.e., meaning “including, but not limited to”). The term “connected” should be interpreted as partially or wholly contained in, connected to, or linked together, even with some interference. Unless otherwise indicated herein, the enumeration of value ranges herein is merely intended as a shorthand method for individually referring to each individual value falling within the range, and each individual value is incorporated into this specification as if separately described herein. Unless otherwise indicated herein or obviously contradicted by the context, all methods described herein can be performed in any suitable order. The use of any and all example or exemplary language (e.g., “such as”) provided herein is intended only to better illustrate embodiments of the invention and, unless otherwise required, does not constitute a limitation on the scope of this disclosure. The language in this specification should not be construed as indicating any non-claimed element as necessary for practicing this disclosure.

[0140] Unless otherwise specifically stated, disjunctive languages ​​such as “at least one of X, Y, Z” should be understood in the context as generally used to indicate that an item, term, etc., may be X, Y, or Z or any combination thereof (e.g., X, Y, and / or Z). Therefore, such disjunctive languages ​​are generally not intended and should not imply that certain embodiments require the presence of at least one of X, at least one of Y, or at least one of Z.

[0141] This document describes various embodiments of this disclosure, including the best mode known to the inventors for carrying out this disclosure. Variations of those embodiments will become apparent to those skilled in the art upon reading the foregoing description. The inventors encourage those skilled in the art to adopt such variations as appropriate, and the inventors intend to practice this disclosure in ways other than those specifically described herein. Therefore, this disclosure includes all modifications and equivalents of the subject matter set forth in the appended claims, where permitted by applicable law. Furthermore, unless otherwise indicated or otherwise clearly contradicted by the context, the scope of this disclosure covers any combination of the foregoing elements in all its possible variations.

[0142] Various exemplary embodiments of this disclosure may be described in accordance with the following terms.

[0143] Item 1: A systolic multiply-accumulate processor organized by row and column logic, each row comprising:

[0144] One or more normalizers, each of the one or more normalizers including a normalizer configured to convert 16-bit input data elements into normalized 18-bit input data elements, each of the one or more normalizers including:

[0145] An irregularity detector is configured to detect that the 16-bit input data element is in an irregular form;

[0146] A shifter configured to shift the effective number of the 16-bit input data element based at least in part on the irregularity detector that detects the irregularity.

[0147] An exponent expander configured to increase the number of bits representing the exponent portion of the 16-bit input data element; and

[0148] A subtractor configured to adjust the value of the exponent portion at least in part based on a shift of the significant digits; and

[0149] A plurality of processing elements are configured to sequentially receive the normalized 18-bit input data elements from the one or more normalizers, each of the plurality of processing elements comprising:

[0150] The first input port is used to receive the normalized 18-bit input data element;

[0151] The second input port is used to receive the normalized weights;

[0152] A weight register is used to store the normalized weights;

[0153] The third input port is used to receive the input portion;

[0154] A multiplier configured to multiply the normalized 18-bit input data elements by the normalized weights to generate a multiplicative product; and

[0155] An adder configured to generate a 34-bit addition result by adding the multiplier product to the input portion.

[0156] Clause 2: Preferably, wherein:

[0157] The normalizer is configured to convert the 16-bit input data element into a first normalized 18-bit input data element when the input data element has a floating-point 16-bit (FP16) format.

[0158] The normalizer is configured to convert the 16-bit input data element into a second normalized 18-bit input data element when the input data element has a Brain Floating-Point 16-bit (BF16) format.

[0159] The first normalized 18-bit input data element and the second normalized 18-bit input data element have different exponent lengths; and

[0160] The first normalized 18-bit input data element and the second normalized 18-bit input data element have different effective number lengths.

[0161] Clause 3: Preferably, it further includes:

[0162] The multiplier consists of two or more multiplier data paths, wherein:

[0163] The first multiplier data path consists of a 1-bit sign data path, a 10-bit significant data path, and a 7-bit exponent data path; and

[0164] The second multiplier data path consists of a 1-bit sign data path, a 7-bit significant number data path, and a 10-bit exponent data path.

[0165] The controller is configured to select from two or more multiplier data paths, at least in part, based on data type control signals; and

[0166] The adder is a 34-bit floating-point adder, wherein the adder consists of 1 sign bit, 23 significant bits and 10 exponent bits.

[0167] Clause 4: Preferably, the normalizer is configured to convert 16-bit input data elements into normalized 18-bit input data elements, and the normalizer is also configured to convert 16-bit weights into normalized 18-bit weights.

[0168] Clause 5: A pulsating circuit comprising:

[0169] A pulsating array comprising processing elements arranged in multiple rows; and

[0170] The first normalizer is configured as follows:

[0171] Receive a first input represented as a floating-point number with a first bit length; and

[0172] Generate a first normalized number in floating-point representation with a second bit length, wherein the second bit length is greater than the first bit length;

[0173] Each processing element in at least one row of the pulsating array is configured as follows:

[0174] Multiply the first normalized number by the corresponding second normalized number to generate a multiplicative product; and

[0175] The corresponding input part is added to the product of the multipliers to generate the corresponding addition result.

[0176] Clause 6: Preferably, each processing element in the multiple rows of the pulsating array comprises:

[0177] A multiplier configured to multiply two 18-bit floating-point numbers, wherein the multiplier comprises a 1-bit sign data path, a 10-bit significant data path, and a 10-bit exponent data path; and

[0178] An adder configured to add two floating-point numbers, wherein the adder consists of a 1-bit sign data path, a 23-bit sign data path, and a 10-bit exponent data path.

[0179] Clause 7: Preferably, the first normalizer is further configured as follows:

[0180] Receive input data elements and weights;

[0181] Generate normalized input data elements and normalized weights; and

[0182] Choose the normalized input data elements or normalized weights to generate.

[0183] Clause 8: Preferably, the first normalizer comprises:

[0184] A first irregular detector is configured to detect whether the first input is irregular based at least in part on the exponent value of the first input and the value of the significant number of the first input;

[0185] A first exponent expander is configured to expand the numerical range of the exponent of the first input; and

[0186] A first shifter is configured to shift the effective number of the first input.

[0187] Clause 9: Preferably, the first normalizer comprises:

[0188] A first leading-zero encoder or counter is configured to detect the number of leading zeros in the significant number of the first input; and

[0189] A first shifter is configured to shift the effective number of the first input at least in part based on the number of leading zeros in the effective number of the first input.

[0190] Clause 10: Preferably, the first normalizer is further configured to receive the first input and expand the exponent of the first input by expanding the exponent of the first input with two additional bits.

[0191] Clause 11: Preferably, each processing element in the multiple rows of the pulsating array further includes:

[0192] Multiplier;

[0193] Adders; and

[0194] The multipliers and adders mentioned therein lack support for inputs provided in an irregular form.

[0195] Clause 12: Preferably, wherein:

[0196] The first normalizer is further configured to convert the first input into a first normalized number, wherein the first normalizer is configured to support 16-bit floating-point numbers and 16-bit floating-point numbers; and

[0197] Each processing element in the multiple rows of the pulsating array further includes:

[0198] 18-bit multiplier; and

[0199] 34-bit adder.

[0200] Clause 13: Preferably, wherein:

[0201] The first normalizer is further configured to convert the first input into a first normalized number, wherein the first normalizer is configured to support at least n-bit floating-point numbers, where n can be any number; and

[0202] Each processing element in the multiple rows of the pulsating array further includes:

[0203] A multiplier configured to multiply at least two n-digit numbers; and

[0204] An adder configured to add two m-bit numbers, where m is greater than n.

[0205] Clause 14: Preferably, it further includes:

[0206] A second normalizer, configured to convert a second input into a second normalized number, the second normalizer comprising:

[0207] A second irregularity detector is configured to detect whether the second input is irregular.

[0208] A second shifter is configured to shift the effective number of the second input based at least in part on the detection of the irregular form by the second irregular detector;

[0209] A second exponent expander is configured to increase the number of bits representing the exponent of the second input; and

[0210] A second subtractor is configured to adjust the exponent value to compensate for the shift of the significant number.

[0211] Clause 15: Preferably, each processing element in the multiple rows of the pulsating array further includes:

[0212] Multiplier;

[0213] Adders; and

[0214] The adder and the multiplier are integrated together to perform a single-step multiply-accumulate operation, wherein the single-step multiply-accumulate operation involves a single rounding.

[0215] Clause 16: A method for performing pulsating array processing by processing elements in a pulsating array of processing elements, the method comprising:

[0216] Receive the first input in floating-point representation with the first bit length;

[0217] Generate a first normalized number in floating-point representation with a second bit length, wherein the second bit length can be any bit length greater than the first bit length;

[0218] Multiply the first normalized number by the second normalized number to generate a multiplicative product; and

[0219] The input portion is added to the product of the multipliers to generate the addition result.

[0220] Clause 17: Preferably, the multiplication is performed using a multiplier that lacks support for inputs provided in an informal form.

[0221] Clause 18: Preferably, it further includes:

[0222] Receive a first input, wherein the first input may be a 16-bit floating-point number or a 16-bit floating-point number;

[0223] Multiply the first normalized number by the second normalized number; and

[0224] The multiplicative product is added to the input portion to produce an addition result, wherein the addition result is a 34-bit floating-point number.

[0225] Clause 19: Preferably, it further includes:

[0226] Receive a first input, wherein the first input may be an n-bit floating-point number, where n may be any number;

[0227] Multiply the first normalized number by the second normalized number; and

[0228] The multiplicative product is added to the input portion to produce an addition result, wherein the addition result is an m-bit floating-point number, where m is greater than n.

[0229] Clause 20: Preferably, it further includes:

[0230] Receive a third input in floating-point representation with a third bit length;

[0231] Generate a third normalized number in floating-point representation with a fourth bit length, wherein the fourth bit length is greater than the third bit length; and

[0232] Choose either the third normalized number or the first normalized number to generate.

Claims

1. A pulsating circuit, comprising: A pulsating array comprising processing elements arranged in multiple rows; as well as A first normalizer includes a first irregular detector, wherein the first normalizer is configured to: Receive the first input in floating-point representation with the first bit length; The first irregularity detector detects that the first input is irregular, at least in part based on the exponent of the first input or the significant number of the first input; as well as Generate a first normalized number in floating-point representation with a second bit length, wherein the second bit length is greater than the first bit length; Each processing element in at least one row of the pulsating array is configured as follows: Multiply the first normalized number by the corresponding second normalized number to generate a multiplier product; as well as The corresponding input part is added to the product of the multipliers to generate the corresponding addition result.

2. The pulsating circuit of claim 1, wherein each processing element in the multiple rows of the pulsating array comprises: A multiplier configured to multiply two 18-bit floating-point numbers, wherein the multiplier comprises a 1-bit sign data path, a 10-bit significant data path, and a 10-bit exponent data path; and An adder configured to add two floating-point numbers, wherein the adder consists of a 1-bit sign data path, a 23-bit sign data path, and a 10-bit exponent data path.

3. The pulsating circuit of claim 1 or claim 2, wherein the first normalizer is further configured to: Receive input data elements and weights; Generate normalized input data elements and normalized weights; as well as Choose the normalized input data elements or normalized weights to generate.

4. The pulsating circuit as described in claim 1 or 2, wherein the first normalizer further comprises: A first exponent expander is configured to expand the numerical range of the exponent of the first input; as well as A first shifter is configured to shift the effective number of the first input.

5. The pulsating circuit as described in claim 1 or 2, wherein the first normalizer further comprises: A first leading zero encoder or counter is configured to detect the number of leading zeros in the valid number of the first input; as well as A first shifter is configured to shift the effective number of the first input at least in part based on the number of leading zeros in the effective number of the first input.

6. The pulsating circuit of claim 1 or 2, wherein the first normalizer is further configured to receive the first input and expand the exponent of the first input by expanding the exponent of the first input with two additional bits.

7. The pulsating circuit as described in claim 1 or 2, wherein: The first normalizer is further configured to convert the first input into the first normalized number, wherein the first normalizer is further configured to support 16-bit floating-point numbers and 16-bit floating-point numbers; as well as Each processing element in the multiple rows of the pulsating array further includes: 18-bit multiplier; as well as 34-bit adder.

8. The pulsating circuit as described in claim 1 or 2, wherein: The first normalizer is further configured to convert the first input into the first normalized number, wherein the first normalizer is further configured to support at least n-bit floating-point numbers, where n is any number; as well as Each processing element in the multiple rows of the pulsating array further includes: A multiplier configured to multiply at least two n-digit numbers; as well as An adder configured to add two m-bit numbers, where m is greater than n.

9. The pulsating circuit as described in claim 1 or 2, further comprising: A second normalizer, configured to convert the second input into a second normalized number, the second normalizer comprising: A second irregularity detector is configured to detect whether the second input is irregular. A second shifter is configured to shift the effective number of the second input based at least in part on the detection of the irregular form by the second irregular detector; A second exponent expander is configured to increase the number of bits representing the exponent of the second input; and A second subtractor is configured to adjust the value of the exponent of the second input to compensate for the shift of the significant number of the second input.

10. The pulsating circuit of claim 1 or 2, wherein each processing element in the multiple rows of the pulsating array further comprises: Multiplier; as well as Adder; The adder and the multiplier are integrated together to perform a single-step multiply-add operation, and the single-step multiply-add operation involves a single rounding.

11. A method for processing a pulsating array by a pulsating circuit, the pulsating circuit comprising a pulsating array of processing elements and a first normalizer, the method comprising: The first input, represented as a floating-point number with a first bit length, is received by the first normalizer; The first input is detected as irregular by the first irregular detector of the first normalizer based at least in part on the exponent of the first input or the significant number of the first input; The first normalizer generates a first normalized number in floating-point representation with a second bit length, wherein the second bit length is any bit length greater than the first bit length; The processing element in the pulsating array multiplies the first normalized number by the second normalized number to generate a multiplier product; as well as The processing element adds the input portion and the product of the multiplier to generate an addition result.

12. The method of claim 11, wherein the multiplier of the processing element lacks support for inputs provided in an irregular form.

13. The method of claim 11 or claim 12, wherein the first input is a 16-bit floating-point number or a 16-bit floating-point number; and The result of the addition is a 34-bit floating-point number.

14. The method of claim 11 or 12, wherein the first input is an n-bit floating-point number, where n is any number; The result of the addition is an m-bit floating-point number, where m is greater than n.

15. The method of claim 11 or 12, further comprising: The third input, represented as a floating-point number with a third bit length, is received by the first normalizer; The first normalizer generates a third normalized number in floating-point representation with a fourth bit length, wherein the fourth bit length is greater than the third bit length; as well as The first normalizer selects either the third normalized number or the first normalized number to be generated.

Images