Unified multiply-accumulate units

Abstract

This specification relates to methods, systems, and apparatus for multiply-accumulate (MAC) units having a unified architecture for performing both floating point MAC operations and integer MAC operations. An example method for performing a MAC operation with a MAC cell having a first unified adder and a second unified adder includes receiving floating point input operands including a first operand, a second operand, a third operand, and a fourth operand. The method further includes performing a pre-multiplication alignment process that aligns a mantissa of one or more of the floating point input operands based on comparing 1) a first sum of exponents of the first operand and the second operand, and 2) a second sum of exponents of the third operand and the fourth operand. The method further includes performing a first multiplication between aligned mantissas of the first operand and the second operand to generate a first mantissa product.

Description

UNIFIED MULTIPLY-ACCUMULATE UNITSBACKGROUND

[0001] This specification generally relates to digital circuitry for computer arithmetic.

[0002] Multiply -Accumulate (MAC) operations are the heart of machine learning (ML) accelerators, which are special-purpose processors designed to efficiently perform common operations of machine learning models, usually with highly parallel architectures. Power consumption and silicon area are important factors to consider for ML accelerators, especially for edge devices that rely on battery power and require small sizes, such as cell phones and tablet computers. MAC units (also referred to as MAC cells) are designed to perform the fundamental mathematical operations of machine learning applications. To handle the increasing complexity and sophistication of modem machine learning models, the number of MAC units built into ML accelerators is increasing rapidly, which is becoming a major source of silicon area and power consumption problems with designing modem ML accelerators.SUMMARY

[0003] This specification describes MAC units having a unified architecture for performing both floating point MAC operations and integer MAC operations. In particular, the unified architecture described in this specification can allow a four-input MAC unit to use two integer adders for both floating point addition and integer point addition, and thus does not require any floating point adders. To support this unified architecture, the described MAC unit can perform a pre-multiplication alignment process prior to multiplication of mantissas. This approach allows the same multipliers and adders to be reused for both floating point and integer data types, thereby resulting in chip area reduction and power savings. The pre-multiplication alignment process can include exponent addition and shifting of mantissa. Specifically, in the pre-multiplication alignment process, the MAC unit can compute sums of exponents, compare the sums, and find the difference between the sums. The difference can be divided equally among two mantissas such that the mantissa of activation and kernel weight gets right shifted by an equal amount, thereby supporting bigger exponent differences without loss of bits.

[0004] One aspect of the subject matter described in this specification can be embodied in a method for performing a MAC operation with a MAC cell having a first unified adder and a second unified adder. The method includes receiving floating point input operands includinga first operand, a second operand, a third operand, and a fourth operand. The method further includes performing a pre-multiplication alignment process that aligns a mantissa of one or more of the floating point input operands based on comparing 1) a first sum of exponents of the first operand and the second operand, and 2) a second sum of exponents of the third operand and the fourth operand. The method further includes: performing a first multiplication between aligned mantissas of the first operand and the second operand to generate a first mantissa product; performing a second multiplication between aligned mantissas of the third operand and the fourth operand to generate a second mantissa product; performing an addition of the first mantissa product and the second mantissa product using the first unified adder; and generating a MAC operation output using an output of the first unified adder.

[0005] These and other implementations can each optionally include one or more of the following features. For example, the method can further include: receiving four integer input operands; performing multiplications between pairs of the four integer input operands to generate a first integer product and a second integer product; and performing an addition of the first integer product and the second integer product using the first unified adder. In some implementations, generating the MAC operation output includes generating the MAC operation output by performing an addition of the output of the first unified adder and an accumulator using a second unified adder.

[0006] In some implementations, generating the MAC operation output can include performing a post-multiplication alignment process that align the output of the first unified adder and a mantissa of an accumulator based on comparing an exponent output of the premultiplication alignment process and an exponent of the accumulator. The exponent output of the pre-multiplication alignment process is a larger of the first sum and the second sum. In some implementations, generating the MAC operation output further includes: generating a mantissa of the MAC operation output by adding outputs of the post-multiplication alignment process using a second unified adder; and generating an exponent of the MAC operation output by selecting a larger of the exponent output of the pre-multiplication alignment process and the exponent of the accumulator.

[0007] The method can further include performing a normalization and rounding process of the MAC operation output and storing an output of the normalizing and rounding process in a storage unit.

[0008] The pre-multiplication alignment process can include shifting mantissas of the first operand and the second operand when the first sum is smaller than the second sum. Thepre-multiplication alignment process can also include shifting mantissas of the third operand and the fourth operand when the first sum is larger than the second sum.

[0009] Another aspect of this specification features a circuit for performing a MAC operation of floating point input operands including a first operand, a second operand, a third operand, and a fourth operand. The circuit includes a pre-multiplication alignment circuit configured to align a mantissa of one or more of the floating point input operands based on comparing 1) a first sum of exponents of the first operand and the second operand, and 2) a second sum of exponents of the third operand and the fourth operand. The circuit further includes: a first multiplier configured to generate a first mantissa product by performing a first multiplication between aligned mantissas of the first operand and the second operand; a second multiplier configured to generate a second mantissa product by performing a second multiplication between aligned mantissas of the third operand and the fourth operand to generate a second mantissa product; and a first unified adder configured to perform an addition of the first mantissa product and the second mantissa product. An output of the MAC operation can be generated based on an output of the first unified adder.

[0010] In some implementations, the circuit is configured to receive four integer input operands. The first multiplier and the second multiplier are configured to perform multiplications betw een pairs of the four integer input operands to generate a first integer product and a second integer product. The first unified adder is configured to perform an addition of the first integer product and the second integer product. The circuit can further include a post-multiplication alignment circuit configured to align the output of the first unified adder and a mantissa of an accumulator based on comparing an exponent output of the pre-multiplication alignment circuit and an exponent of the accumulator. The exponent output of the pre-multiplication alignment circuit is a larger of the first sum and the second sum.

[0011] The circuit can further include a second unified adder. A mantissa of the output of the MAC operation is generated by adding outputs of the post-multiplication alignment circuit using a second unified adder. An exponent of the output of the MAC operation is generated by selecting a larger of the exponent output of the pre-multiplication alignment circuit and the exponent of the accumulator. In some implementations, the circuit further includes a normalization and rounding circuit configured to: generating an output by normalizing and rounding the output of the MAC operation; and store the output of the normalization and rounding circuit in a storage unit.

[0012] The pre-multiplication alignment circuit can include two shifters and can be configured to shift mantissas of the first operand and the second operand when the first sum is smaller than the second sum. The pre-multiplication alignment circuit can be configured to shift mantissas of the third operand and the fourth operand when the first sum is larger than the second sum. For example, the second unified adder is configured to update the mantissa of the accumulator using the mantissa of the output of the MAC operation. The postmultiplication alignment circuit can be configured to update the exponent of the accumulator using the exponent of the output of the MAC operation.

[0013] Another aspect of this specification features a circuit for performing a MAC operation of multiple pairs of floating point input operands. The circuit includes a premultiplication alignment circuit configured to align a mantissa of one or more of the floating point input operands based on comparing a sum of exponents of each pair of the floating point input operands. The circuit can further include multipliers configured to generate mantissa products by performing multiplications between aligned mantissas of each pair of the floating point input operands. The circuit can further include a first unified adder configured to perform an addition of an addition of the mantissa products. An output of the MAC operation is generated based on an output of the first unified adder.

[0014] In some implementations, the circuit is configured to receive multiple pairs integer input operands. The multipliers are configured to perform multiplications between each pair of the integer input operands to generate integer products. The first unified adder is configured to perform an addition of the integer products. Tn some implementations, the first unified adder includes an adder tree formed by multiple integer adders.

[0015] Other implementations of this and other aspects include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices. A system of one or more computers can be so configured by virtue of software, firmware, hardware, or a combination of them installed on the system that in operation causes the system to perform the actions. One or more computer programs can be so configured by virtue of having instructions that, when executed by a data processing apparatus, cause the apparatus to perform the actions.

[0016] The subject matter described in this specification can be implemented in particular embodiments to realize one or more of the following advantages.

[0017] The techniques described in this specification use a unified architecture for performing MAC operations supporting both floating point and integer data types. This approach allows for efficient sharing of hardware resources between the floating point andinteger data paths, thereby reducing silicon area and power consumption compared to architectures using separate data paths for convolution. The described techniques can be applied to perform a pre-multiplication alignment process prior to multiplication of mantissas in a MAC operation. The pre-multiplication alignment process can support bigger exponent differences without loss of bits. The same approach can be used in an accumulating process of the MAC operation. Thus, computing circuits (e.g.. multipliers, adders, and shifters) in a MAC unit can be replaced with smaller and more efficient ones, thereby achieving saves in chip area and power consumption and improvements to the critical path of the MAC unit.

[0018] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Fig. 1 is a block diagram of an example computing system that includes Multiply- Accumulate (MAC) units.

[0020] Fig. 2 illustrates a block diagram of an example MAC unit.

[0021] Figs. 3A-3D illustrate an example MAC unit for performing a MAC operation of four input operands.

[0022] Figs. 4A-4D illustrate an example MAC unit for performing a MAC operation of more than four input operands.

[0023] Fig. 5 illustrates a flow chart of an example method for performing a MAC operation.

[0024] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0025] In machine learning (ML) applications, efficient operation can depend on design and optimization of Multiply-Accumulate (MAC) units. The number of MAC units, data path width, and pipelining can significantly affect performance, chip area, and power consumption of the ML applications. Therefore. MAC units or MAC cells with a unified architecture designed to improve power efficiency and performance and to reduce a size of the chip area are desirable.

[0026] The specification describes techniques related to methods and devices for performing MAC operations using a unified architecture. The described techniques can support both floating point (FP) and integer data types. Furthermore, the techniques allow for efficient sharing of hardware resources between both FP and integer data paths, thereby reducing chip area and power consumption compared to architectures using separate data paths for convolution.

[0027] Fig. 1 is a block diagram of an example computing system 100 that includes MAC units. The system 100 can include a system-on-chip 102 ('‘SoC 102”). The SoC 102 includes a central processing unit 104 (“CPU 104”), a shared memory 106 (“memory 106”), and an IP / circuit block 110. In some implementations, system 100 can include multiple SoCs and any descriptions for the SoC 102 will apply equally to each of the multiple SoCs that may be included at system 100.

[0028] The CPU 104 can be a general-purpose CPU (e.g., a single or multi-coreCPU). The CPU 104 generates one or more indicators, such as an app-launch indicator or a function call that is triggered in response to executing or launching an application at a user device. For example, the application can be a camera application that uses an imaging sensor to generate image data or a gaming application that requires substantial memory and graphics processing resources to render graphical content of the game. The CPU 104 also generates one or more application values, such as pixel values or frame rate. The application values may be associated with a function call, may be descriptive of an event that occurs during execution of the application, or both.

[0029] The memory 106 is a system memory, shared memory’, or both. In the example of Fig. 1, memory 106 is depicted external to circuit block 110. However, memory’ 106 can include portions of memory that are: i) specific to circuit block 110, ii) external to circuit block 110, or iii) both. The memory 106 can be random access memory of the SoC 102, such as static random-access memory (SRAM), dynamic random-access memory (DRAM), a synchronous DRAM (SDRAM), or double data rate (DDR) SDRAM.

[0030] In some implementations, aspects of memory 106 are configured as a shared scratchpad memory that supports parallel access of its memory resources by two or more processors of the circuit block 110. The memory 106 can also include various other types of memory, such as high bandwidth memory’ (HBM), narrow memory’ (e.g., for storing 8-bit values), wide memory (e.g., for storing 16-bit or 32-bit values), etc.

[0031] The circuit block 110 generally includes individual IP devices such as processors, processor cores, or special-purpose processing devices. For example, the circuit block 110can include an image signal processor (ISP) 112, an ML accelerator 114, a digital signal processor (DSP) 116, and a graphics processing unit (GPU) 118. The circuit block 110 is referred to alternatively as an IP block 1 10, where the IP block can include one or more proprietary hardware elements. For example, each of the ISP 112, ML accelerator! 14, DSP 116, and GPU 118 can be a respective proprietary IP block (or IP device) of a particular entity or device manufacturer.

[0032] The CPU 104 can be configured as an instruction and vector data processing engine that processes data obtained from a system memory of the SoC 102, such as memory 106. In some implementations, each processor (e.g., ISP 112, DSP 116, ML accelerator 114, GPU 118) of the SoC 102 includes multiple cores and the CPU 104 can generate control signaling to minimize the processing load at each core of the processors. The control signaling is routed at system 100 using an example bus 122 of the SoC 102. The control signaling can include commands, requests, data, instructions, or a combination of these.

[0033] The ML accelerator 114 is a processing device that is designed to accelerate computations for machine learning applications. In some implementations, the ML accelerator 114 can be a tensor processing unit (TPU) and can include an application-specific integrated circuit (ASIC). In some implementations, the ML accelerator 114 can be a GPU. The ML accelerator! 14 can be configured to train machine learning models such as a deep learning model that uses neural networks (NNs). Training the models usually requires matrix multiplications and convolutions and thus can be computationally intensive. The ML accelerator! 14 can include large arrays of MAC units 120 to perform massive parallel computations. The MAC unit can perform MAC operations of input operands (such as weights and activations in an NN), enabling the ML accelerator! 14 to efficiently perform the large number of matrix calculations that are common in deep learning. While Fig. 1 illustrates MAC units 120 being integrated in the ML accelerator! 14, this example is not intended to be construed in a limiting sense. It is understood that in practice, MAC units 120 can be included in various ty pes of devices and processors, such as those designed for computationally intensive tasks, including but not limited to: DSPs (e.g.. DSP 116), GPUs (e.g.. GPU 118). Field-Programmable Gate Arrays (FPGAs). CPUs (e.g.. CPU 104). Neural Processing Units (NPUs), and ASICs.

[0034] In the example of Fig. 1, system 100 and the SoC 102 is an integrated circuit of an example user / client device 130, consumer electronic device, or mobile device, where each of these devices can include items such as a network server 130a, a smartphone 130b. tablet 130c, or laptop 130d, or even a wearable device or autonomous vehicle. The devices 130 mayalso include other items such as an eNotebook, Netbook, mobile computer, or any device capable of training and / or executing machine learning models. In some implementations, the system 100 and the SoC 102 are integrated circuits of a desktop computer, network server, or related cloud-based asset.

[0035] Fig. 2 illustrates a block diagram of an example MAC unit 200. The MAC unit 200 can be an implementation of the MAC unit 120 of Fig. 1. The MAC unit 200 is configured to perform a MAC operation of integer input operands 212 or floating point input operands 214 to generate a MAC operation output. The MAC unit 200 can include a premultiplication alignment circuit 202, a multiplication circuit 204, an addition circuit 206, a post-multiplication alignment circuit 208, and an addition circuit 210. In some implementations, an integer MAC operation (e.g.. the integer data path) can be performed by the multiplication circuit 204, the addition circuit 206, and the addition circuit 210 of the MAC unit 200. The multiplication circuit 204 can include multipliers. The addition circuit 206 can include a unified adder. The unified adder is also referred to as a shared adder and can perform addition of integer operands (e.g., in the integer data path) or mantissas of floating point operands (e.g., in the floating point data path). The addition circuit 210 can also include a unified adder. During the integer MAC operation, the multiplication circuit 204 is configured to perform multiplications between pairs of the integer input operands 212 to generate integer products. The addition circuit 206 is configured to perform an addition of the integer products to generate an output 222 to the addition circuit 210. The addition circuit 210 is configured to perform an addition of the output 222 and an integer accumulator. In this example, the MAC operation output can be an output 216 of the addition circuit 210.

[0036] In some implementations, a floating point MAC operation (e.g., the floating point data path) can be performed by the MAC unit 200 as described below. The pre-multiplication alignment circuit 202 can include adders and shifters and is configured to perform exponent addition and mantissa shifting of the floating point input operands 214. The premultiplication alignment circuit 202 can generate aligned mantissas (e.g., output 218 as shown in Fig. 2) of the floating point input operands 214 to the multiplication circuit 204 and an exponent output (e.g.. output 220 as shown in Fig. 2) to the post-multiplication alignment circuit 208. The multiplication circuit 204 can perform a multiplication of the output 218 to generate mantissa products. The addition circuit 206 is configured to use its unified adder to perform an addition of the mantissa products generated by the multiplication circuit 204. The addition circuit 206 can generate an output 224 to the post-multiplication alignment circuit 208. The post-multiplication alignment circuit 208 is configured to align the output 224 and amantissa of a floating point accumulator (e.g., 226 provided by the addition circuit 210) and provide aligned mantissas 228 to the addition circuit 210. The post-multiplication alignment circuit 208 is also configured to generate an output 230 as an exponent of the MAC operation output. The addition circuit 210 is configured to perform an addition of the aligned mantissas 228 to generate the output 216 as a mantissa of the MAC operation output. In other words, for the floating point MAC operation, the exponent of the MAC operation output is included in the output 230, and the mantissa of the MAC operation output is included in the output 216.

[0037] Figs. 3A-3D illustrate an example MAC unit 300 for performing a MAC operation of four input operands. The MAC unit 300 can be an example of the MAC unit 120 of Fig. 1 or the MAC unit 200 of Fig. 2. The four input operands can be four floating input operands 314-1, 314-2, 314-3, and 314-4 (as shown in Fig. 3 A) or four integer input operands 312-1, 312-2, 312-3, and 312-4 (as shown in Fig. 3B).

[0038] During normal operation, the MAC unit 300 is configured to generate a sum of two products from the four inputs and add the result to an accumulated value. For example, the MAC unit 300 can multiply a first pair of the input operands such as 314- 1 and 314-2 (or 312-1 and 312-2) to generate a first product and a second pair of the input operands such as 314-3 and 314-4 (or 312-3 and 312-4) to generate a second product. The MAC unit 300 can add the first product and the second product to generate a sum-of-product. When the MAC unit 300 receives new input operands, new sum-of-product can be generated and accumulated. This process can stop when the input operands contributing to one accumulator output are processed. In some implementations, the four input operands can be activation and kernel weights of a deep learning model. For example, the floating input operands 314-1 and 314-3 can be two activation weights and the floating input operands 314-2 and 314-4 can be two kernel weights. In some implementations, an output of the MAC operation (e.g., the accumulator output) can be provided to a non-linear processing function of the deep learning model for further processing. For example, the non-linear processing function can be performed by a Rectified Linear Activation Unit (ReLU) or any other suitable computing unit. In some implementations, the non-linear processing function can be a Swish activation function. After the accumulator output is read out, a present value of the accumulator can be set to 0, and another MAC operation can start.

[0039] The MAC unit 300 can include a pre-multiplication alignment circuit 302 (as shown in Fig. 3 A), a multiplication circuit 304 (as shown in Fig. 3B). an addition circuit 306 (as shown in Fig. 3C), a post-multiplication alignment circuit 308 (which includes a circuit308a as shown in Fig. 3C and a circuit 308b as shown in Fig. 3D), and an addition circuit 310 (as shown in Fig. 3D). In some implementations, the MAC unit 300 can perform a floating point MAC operation. The floating point MAC operation can include three phases. The first phase includes a pre-multiplication alignment process performed by the pre-multiplication alignment circuit 302 (as shown in Fig. 3A) and a multiplication process performed by the multiplication circuit 304 (as shown in Fig. 3B). The pre-multiplication alignment circuit 302 can load the floating point input operands 314-1, 314-2. 314-3, and 314-4 from registers 340- 1, 340-2, 340-3, and 340-4, respectively. The floating point input operands 314-1, 314-2, 314-3, and 314-4 can be of any suitable size and can be in any suitable floating point format. For example, each of the floating point input operands 314-1, 314-2, 314-3, and 314-4 can be a 16-bit brain floating point (BF16) data type and includes a 1-bit sign, a 8-bit exponent, and a 7-bit mantissa. In another example, the floating point input operands 314-1, 314-2, 314-3, and 314-4 can be a 16-bit IEEE floating point (FP 16) data type and each can have a 1-bit sign, a 5-bit exponent, and a 10-bit mantissa.

[0040] Let ei denotes an exponent of the floating point input operand 314-i, and let mi denotes a mantissa of the floating point input operand 314-i (1 < i < 4). As shown in Fig. 3 A, an adder 342 of the pre-multiplication alignment circuit 302 is configured to perform an addition of ei and e2 to generate a sum 344 of exponents of the floating point input operand 314-1 and the floating point input operand 314-2. Another adder 346 of the pre-multiplication alignment circuit 302 is configured to perform an addition of e? and ei to generate a sum 348 of exponents of the floating point input operand 314-3 and the floating point input operand 314-4. A comparator circuit 350 of the pre-multiplication alignment circuit 302 can compare the sum 344 and the sum 348 and generate a maximum 352 of the sum 344 and the sum 348. The maximum 352 can be stored in a register 354.

[0041] The pre-multiplication alignment circuit 302 can use comparator circuits 356 and 358 and shifters 360 and 362 to align mantissas (e.g., mi, m2, m3, and mi) of the floating point input operands 314-1, 314-2, 314-3, and 314-4 based on comparing the sum 344 and the sum 348. As shown in Fig. 3A, when the comparator circuit 356 determines that the sum 344 is smaller than the maximum 352. the shifter 360 is configured to align mi and m2. For example, the shifter 360 can right shift mi by [(maximum 352-sum 344) / 2] bits and right shift m2 by (maximum 352-sum 344) - [(maximum 352-sum 344) / 2] bits, where [ ] represents a round up function or a ceiling function. When the comparator circuit 356 determines that the sum 344 is equal to the maximum 352, mi and m2 are not shifted. In other words, mi and m2 can be considered as already aligned when the sum 344 is equal to the maximum 352.Similarly, when the comparator circuit 358 determines that the sum 348 is smaller than the maximum 352. the shifter 362 is configured to align m3 and m4. For example, the shifter 362 can right shift m3 by [(maximum 352-sum 348) / 2] bits and right shift uu by (maximum 352- sum 348) - [(maximum 352-sum 348) / 2] bits. When the comparator circuit 358 determines that the sum 348 is equal to the maximum 352, m3 and r are not shifted. In other words, m3 and rm can be considered as already aligned when the sum 348 is equal to the maximum 352. The pre-multiplication alignment circuit 302 can generate outputs 364-1, 364-2, 364-3, and 364-4 as aligned mantissas of the floating point input operands 314-1, 314-2, 314-3, and 314- 4. The output 364-1 is the aligned mi, the output 364-2 is the aligned m2, the output 364-3 is the aligned m3, and the output 364-4 is the aligned nu The pre-multiplication alignment circuit 302 can also generate the maximum 352 as an exponent output.

[0042] As shown in Fig. 3B, the multiplication circuit 304 includes multipliers 366 and 368. The multiplier 366 is configured to perform a multiplication between the aligned mantissa mi (e.g., output 364-1) and the aligned mantissa m2 (e.g., output 364-2) to generate a product 370-1. The multiplier 368 is configured to perform a multiplication between the aligned mantissa m3 (e.g., output 364-3) and the aligned mantissa nu (e.g., output 364-4) to generate a product 370-2. In the floating point MAC operation, the products 370-1 and 370-2 are mantissa products. In some implementations, the products 370-1 and 370-2 can be stored in registers 372-1 and 372-2.

[0043] In the second phase of the floating point MAC operation, as shown in Fig. 3C, the addition circuit 306 can load data from the registers 372-1 and 372-2 and perform an addition of the mantissa product 370-1 and the mantissa product 370-2 using an unified adder 374. The unified adder 374 can generate an output 376, which can also be referred to as a sum-of- product mantissa 376. In some implementations, the sum-of-product mantissa 376 can be stored in a register 378. The post-multiplication alignment circuit 308 is configured to align the sum-of-product mantissa 376 and a mantissa of a floating point accumulator based on comparing an exponent of the floating point accumulator with the exponent output 352 (also referred to as the maximum 352) generated by the pre-multiplication alignment circuit 302. In the second phase, the post-multiplication alignment circuit 308 is configured to compare the exponent of the floating point accumulator with the exponent output 352 using the circuit 308a. As shown in Fig. 3C, the circuit 308a can include a comparator circuit 380 and a register 382. The register 382 can be configured to store the exponent of the floating point accumulator (e.g., 384). In other words, the exponent 384 is an exponent of previous accumulated floating point results. The comparator circuit 380 can compare the exponentoutput 352 with the exponent 384. In some implementations, as shown in Fig. 3C, the comparator circuit 380 compares the exponent output 352 with a sum 385 of the exponent 384 and a earn’ overflow 387. The carry overflow 387 can be provided by the addition circuit 310 of Fig. 3D. In other words, the sum 385 represents the exponent 384 incremented by the carry overflow 387. The comparator circuit 380 can generate a larger of the exponent output 352 and the sum 385 as an output 386 and update the register 382 using the output 386. The output 386 can be considered as the exponent of the floating point accumulator being updated based on the exponent output 352. The updated exponent of the floating point accumulator is an exponent of the floating point MAC operation.

[0044] In the third phase of the floating point MAC operation, as shown in Fig. 3D, the post-multiplication alignment circuit 308 is configured to use the circuit 308b to align the sum-of-product mantissa 376 and a mantissa of the floating point accumulator. In some implementations, the mantissa of the floating point accumulator can be an output 388 generated by the addition circuit 310. The circuit 308b can perform the alignment using shifters 390 and 392. In some implementations, the first phase and the third phase can be performed in parallel. In those instances, the circuit 308b may not share shifters with the premultiplication alignment circuit 302. In some other implementations, for example, when the first phase and the third phase are not performed in parallel or when inputs are not coming into the first phase or the third phase every- cycle, the circuit 308b may share shifters with a circuit (e.g., the shifters 360 and 362 of the pre-multiplication alignment circuit 302) that has performed shifting operations in an earlier phase. The alignment can be based on comparison results of the comparator circuit 380. For example, when the comparator circuit 380 determines that the exponent output 352 is smaller than the incremented exponent (e.g., 385) of the floating point accumulator, the shifter 390 can right shift the sum-of-product mantissa 376. When the comparator circuit 380 determines that the exponent output 352 is larger than the incremented exponent (e.g., 385) of the floating point accumulator, the shifter 392 can right shift the mantissa (e.g., 388) of the floating point accumulator. When the comparator circuit 380 determines that the exponent output 352 is equal to the incremented exponent (e.g.. 385) of the floating point accumulator, no shifting operation is performed. In other words, when the exponent output 352 is equal to the incremented exponent (e.g., 385) of the floating point accumulator, the sum-of-product mantissa 376 and the mantissa (e.g., 388) of the floating point accumulator are considered as already aligned. The circuit 308b can generate the aligned sum-of-product mantissa 376 and the aligned exponent 388 as outputs and provide the outputs to the addition circuit 310.

[0045] In the third phase of the floating point MAC operation, the addition circuit 310 (which can include a unified adder 394 and a register 396 as shown in Fig. 3D) is configured to perform an addition of the aligned sum-of-product mantissa 376 and the aligned exponent 388 and generate an output 398. In some implementations, the unified adder 394 can be a 32- bit integer adder and can have a 2-bit carry' part and a 30-bit sum part. The output 398 is an updated mantissa of the floating point accumulator. In the floating point MAC operation, the register 396 is configured to store the mantissa of the floating point accumulator and provide the carry overflow 387 to the circuit 308a. After the output 398 is generated, data in the register 396 can be replaced with the output 398.

[0046] The MAC unit 300 can further include a normalization and rounding circuit 399 coupled to the addition circuit 310 (e.g.. the register 396). To save power and make the computing process more efficient, the normalization and rounding circuit 399 may not be used when the MAC unit 300 performs internal multiply-accumulate (MAC) operations. In some implementations, when the MAC unit 300 generates an output of the floating point MAC operation to an external unit (e.g.. a storage unit or a computing unit for further processing), the normalization and rounding circuit 399 can be configured to normalize and round the output 398 based on format and / or precision requirements of the external unit. For example, the normalization and rounding circuit 399 can shift the mantissa of the floating point MAC operation (e.g., the output 398) and adjust the exponent of the floating point MAC operation (e.g., the output 386) accordingly, so that the output of the floating point MAC operation is in a suitable format (such as a 32-bit IEEE floating point (FP32) format that includes a 1 -bit sign bit, an 8-bit exponent, and a 23-bit mantissa). For precision purposes, the normalization and rounding circuit 399 may also round the output of the floating point MAC operation based on extra bits captured during the alignment step. In some implementations, as shown in Fig. 3D, the addition circuit 310 can include an adder 395, which is configured to perform an addition of the exponent of the floating point MAC operation (e.g., the output 386) and the cany' overflow 387 (e.g., bit

[0030] of the register 396) and provide a sum 397 to the normalization and rounding circuit 399. In this way, the MAC unit 300 can gate operations in the second phase on the last cycle of the accumulation operation, the output 386 available on the last cycle of the accumulation operation is added to the carry overflow 387, and the sum 397 is provided as an input to the normalization and rounding circuit 399. Alternatively (not shown in Fig. 3D), if the operations in the second phase are not gated, the output 386 that is available one cycle after the last cycle of the accumulation operation can be directly provided to the normalization and rounding circuit399. In other words, the adder 395 may not be needed in the latter case. It is understood that the normalization and rounding circuit 399 may apply any suitable normalization and rounding methods.

[0047] In some implementations, the MAC unit 300 can perform an integer MAC operation of the four integer input operands 312-1, 312-2, 312-3, and 312-4. The integer MAC operation can also include three phases. In the first phase of the integer MAC operation, as shown in Fig. 3B, the multiplier 366 is configured to perform a multiplication between the integer input operands 312-1 and 312-2 to generate the product 370-1. The multiplier 368 is configured to perform a multiplication between the integer input operands 312-3 and 312-4 to generate the product 370-2. In some implementations, some 0 bits can be padded on the least significant bit (LSB) side of the input operands if bit-width of the multipliers 366 and 368 can be larger than sizes of the integer input operands 312-1, 312-2, 312-3, and 312-4. For example, the multipliers can be 12x 12 Sint multipliers, and the integer input operands 312-1, 312-2, 312-3, and 312-4 can be SInt8 data type. In this case, three 0 bits can be padded on the LSB side of each of the integer input operands 312-1, 312-2. 312-3, and 312-4. In the integer MAC operation, the products 370-1 and 370-2 are integer products. In some implementations, the integer products 370-1 and 370-2 can be stored in the registers 372-1 and 372-2, respectively. In some implementations, the multipliers 366 and 368 each can be coupled to multiplexers that are configured to load the aligned mantissas (e.g., outputs 364-1 to 364-4) in the floating point MAC operation or load the integer input operands 312-1 to 312-4 in the integer MAC operation. In the second phase of the integer MAC operation, as shown in Fig. 3C, the addition circuit 306 can perform an addition of the integer products 370-1 and 370-2 using the unified adder 374. In this example, the output 376 of the unified adder 374 is an integer sum and can be stored in the register 378. In the third phase of the integer MAC operation, as shown in Fig. 3D, the addition circuit 310 can perform an addition of the output 376 and the output 388 using the unified adder 394. In this example, the output 388 is a current value of an integer accumulator, which is stored in the register 396. The integer accumulator in the register 396 can be updated with the output 398 of the unified adder 394.

[0048] The described techniques allow the MAC unit 300 to support both of the floating point MAC operation and the integer MAC operation. Moreover, the floating point MAC operation and the integer MAC operation can share multiple circuits (e.g., the multiplication circuit 304. the addition circuit 306, and the addition circuit 310) of the MAC unit 300, thereby reducing chip size and achieving efficiency in power consumption. In one example,the floating point input operands 314-1 to 314-4 are BF16 data, and the integer input operands 312-1 to 312-4 are signed integer-8 (SInt8) data. The multipliers 366 and 368 of the multiplication circuit 304 can be 12-bit signed integer multipliers. The registers 372-1 and 372-2 can be 23-bit registers. The unified adder 374 can be a 24-bit integer adder, and the unified adder 394 can be a 32-bit integer adder. It is understood that this example is for illustration purpose, and in practice, multipliers, adders, and registers of any suitable sizes and formats in the shared circuit can be applied to the MAC unit 300.

[0049] Table 1 provides a comparison between computing circuits used in this example and those from a conventional MAC unit that achieves similar computation accuracy. Table 1 also includes improvements of the unified MAC units in terms of Area / Power / Timing metrics.

[0050] Table 1 : Comparison between an example unified MAC unit and an example conventional MAC unit

[0051] Table 1 shows that, in the example of Fig. 3B, 12-bit Sint multipliers (e.g., multipliers 366 and 368) can be used while the example conventional MAC unit uses 9-bit Sint multipliers. The bit-width increase A (e.g., 3 bits) in the multipliers can help avoid loss in computation accuracy. Table 1 also shows that the pre-multiplication alignment circuit 302 can use two smaller shifters (e.g., the shifters 360 and 362 of Fig. 3A which can be 4-bit shifters on 12-bit inputs) instead of a larger shifter (e.g.. a 5-bit shifter on 24-bit inputs) in the example conventional MAC unit. In some implementations, the 12-bit inputs provided to the shifters 360 and 362 can be formed as [1 sign bit, 1 implicit bit, 9 bits of mantissa, 3 bits (e.g., A=3) padded 0],

[0052] In some implementations, the computation accuracy loss can be determined by the exponent difference (e.g., difference between the sum 344 and the sum 348 of Fig. 3 A), the bit-width of the shifters (e.g., shifters 360 and 362), and the bit-width of the multipliers (multipliers 366 and 368). Table 2 illustrates the computation accuracy loss in various scenarios.

[0053] Table 2: Maximum error and multiplier bit-width increase as a function of maximum exponent difference

[0054] In the example of Figs. 3 A-3D, the shifted output of individual mantissa (e.g., the outputs 364-1, 364-2, 364-3, and 364-4) can have at most 12 bits. As shown in the second row of Table 2, this 12-bit limit can ensure negligible precision loss because the mantissa of the BF16 representation has 9 bits. As show n in the third row to the sixth row' of Table 2, as the max exponent difference increases, the computation accuracy loss (the fourth column of Table 2) may occur if 12x 12 multipliers are still used. In this case, the bit-width of the multipliers can be increased to more than 12 bits (e.g., as shown in the fifth column of Table 2) to improve the computation accuracy.

[0055] Figs. 4A-4D illustrate an example MAC unit 400 for performing a MAC operation of more than four input operands. The MAC unit 400 can be an example of the MAC unit 120 of Fig. 1 or the MAC unit 200 of Fig. 2. The MAC unit 400 can include a premultiplication alignment circuit 402 (as shown in Fig. 4A), a multiplication circuit 404 (as shown in Fig. 4B), an addition circuit 406 (as shown in Fig. 4C), a post-multiplication alignment circuit 408 (which include a circuit 408a as shown in Fig. 4C and a circuit 408b as shown in Fig. 4D). and an addition circuit 410 (as shown in Fig. 4D).

[0056] The MAC unit 400 can be configured to perform a floating point MAC operation of N floating input operands 414-1, 414-2, ..., and 414-N (as shown in Fig. 4A), where N is an even integer larger than 4. The floating point MAC operation can include three phases. In the first phase of the floating point MAC operation, the pre-multiplication alignment circuit 402 can load the floating point input operands 414-1, 414-2, ..., and 414-N from registers 440-1, 440-2, ..., and 440-N, respectively. In this example, the floating point input operands 414-1, 414-2, .... and 414-N can be BF16 data. Let ei and mi respectively denote an exponent and a mantissa of the floating point input operand 414-i, (1 < i < N). As shown in Fig. 4A, adders 442 of the pre-multiplication alignment circuit 402 are configured to perform an addition of each pair of exponents (e.g., pair (ei, ez), pair (e3, e4), ... , and pair (ex-i, ex) to generate sums 444-1, 444-2. ..., and 444-N / 2. A comparator circuit 450 of the premultiplication alignment circuit 402 can compare the sums 444-1, 444-2, ..., and 444-N / 2 and generate a maximum 452. The maximum 452 can be stored in a register 454.

[0057] The pre-multiplication alignment circuit 402 can use comparator circuits 456 and shifters 460 to align mantissas (e.g., mi, m2. ..., and mx) of the floating point input operands 414-1, 414-2, ..., and 414-N based on comparing the sum 444-1, 444-2, ..., and 444-N / 2. For example, as shown in Fig. 4A, when one of the comparator circuits 456 determines that the sum 444-N / 2 is smaller than the maximum 452, one of the shifters 460 is configured to align mx-i and mx. The shifter 460 can right shift mx-i by [(maximum 452-sum 444-N / 2) / 2] bits and right shift mx by (maximum 452- sum 444-N / 2) - [(maximum 452- sum 444-N / 2) / 2] bits. When the comparator circuit 456 determines that the sum 444-N / 2 is equal to the maximum 452, IDN-I and mx are not shifted because they are considered as already aligned. The premultiplication alignment circuit 402 can generate outputs 464-1, 464-2. ..., and 464-N as aligned mantissas of the floating point input operands 414-1, 414-2, .... and 414-N. The output 464-i is the aligned mi, (1 < i < N). The pre-multiplication alignment circuit 402 can also generate the maximum 452 as an exponent output.

[0058] As shown in Fig. 4B, the multiplication circuit 404 includes multipliers 466. For example, the multipliers 466 can be 12-bit signed integer multipliers. Each of the multiplier 466 is configured to perform a multiplication between a pair of the aligned mantissas 464-1, 464-2, ..., and 464-N to generate a product 470-i, (1 < i < N / 2). In the floating point MAC operation, the products 470-1, 470-2, ..., and 470-N / 2 are mantissa products. The products 470-1, 470-2, ..., and 470-N / 2 can be stored in registers 472 (e.g.. 23-bit registers).

[0059] In the second phase of the floating point MAC operation, as shown in Fig. 4C, the addition circuit 406 can load data from the registers 472 and perform an addition of themantissa product 470-1, 470-2, ..., and 470-N / 2 using a unified adder 474. For example, the unified adder 474 can be a N / 2-input adder tree. The unified adder 474 can generate an output 476, which can also be referred to as a sum-of-product mantissa 476. In some implementations, the sum-of-product mantissa 476 can be stored in a register 478. The postmultiplication alignment circuit 408 is configured to align the sum-of-product mantissa 476 and a mantissa of a floating point accumulator based on comparing an exponent of the floating point accumulator with the exponent output 452 (also referred to as the maximum 452) generated by the pre-multiplication alignment circuit 402. In the second phase, the postmultiplication alignment circuit 408 is configured to compare the exponent of the floating point accumulator with the exponent output 452 using the circuit 408a. As shown in Fig. 4C, the circuit 408a can include a comparator circuit 480 and a register 482. The register 482 can be configured to store the exponent of the floating point accumulator (e.g., 484). In other words, the exponent 484 is an exponent of previous accumulated floating point results. The comparator circuit 480 can compare the exponent output 452 with the exponent 484. In some implementations, as shown in Fig. 4C. the comparator circuit 480 compares the exponent output 452 with a sum 485 of the exponent 484 and a carry overflow 487. The carry overflow 487 can be provided by the addition circuit 410 of Fig. 4D. In other words, the sum 485 represents the exponent 484 incremented by the carry7overflow 487. The comparator circuit 480 can generate a larger of the sum 485 and the exponent output 452 as an output 486 and update the register 482 using the output 486. The output 486 can be considered as the exponent of the floating point accumulator being updated based on the exponent output 452. The updated exponent of the floating point accumulator is an exponent of the floating point MAC operation.

[0060] In the third phase of the floating point MAC operation, as shown in Fig. 4D, the post-multiplication alignment circuit 408 is configured to use the circuit 408b to align the sum-of-product mantissa 476 and a mantissa of the floating point accumulator. In some implementations, the mantissa of the floating point accumulator can be an output 488 generated by the addition circuit 410. The circuit 408b can perform the alignment using shifters 490 and 492. In some implementations, the first phase and the third phase can be performed in parallel. In those instances, the circuit 408b may not share shifters with the premultiplication alignment circuit 402. In some other implementations, for example, when the first phase and the third phase are not performed in parallel or when inputs are not coming into the first phase or the third phase every cycle, the circuit 408b may share shifters with a circuit (e.g., the shifters 460 of the pre-multiplication alignment circuit 402) that hasperformed shifting operations in an earlier phase. The alignment can be based on comparison results of the comparator circuit 480. For example, when the comparator circuit 480 determines that the exponent output 452 is smaller than the incremented exponent (e.g., 485) of the floating point accumulator, the shifter 490 can right shift the sum-of-product mantissa 476. When the comparator circuit 480 determines that the exponent output 452 is larger than the incremented exponent (e g., 485) of the floating point accumulator, the shifter 492 can right shift the mantissa (e.g.. 488) of the floating point accumulator. When the comparator circuit 480 determines that the exponent output 452 is equal to the incremented exponent (e.g., 485) of the floating point accumulator, the sum-of-product mantissa 376 and the mantissa (e.g., 388) of the floating point accumulator are considered as already aligned, and thus no shifting operation is performed. The circuit 408b can generate the aligned sum-of- product mantissa 476 and the aligned exponent 488 as outputs and provide the outputs to the addition circuit 410.

[0061] In the third phase of the floating point MAC operation, the addition circuit 410 (which can include a unified adder 494 and a register 496 as shown in Fig. 4D) is configured to perform an addition of the aligned sum-of-product mantissa 476 and the aligned exponent 488 and generate an output 498. In some implementations, the unified adder 494 can be a 32- bit integer adder and can have a 2-bit earn- part and a 30-bit sum part. The output 498 is an updated mantissa of the floating point accumulator. The register 496 can be configured to store the mantissa of the floating point accumulator in the floating point MAC operation and provide the carry overflow 487 to the circuit 408a. The output 498 is stored in the register 496. In other words, the register 496 can be updated each time a new output 498 is generated. Similar to the MAC unit 300 of Figs. 3A-3D, the MAC unit 400 can also include a normalization and rounding circuit 499. The normalization and rounding circuit 499 can be coupled to the addition circuit 410 (e.g., the register 496). When the MAC unit 400 generates an output of the floating point MAC operation to an external unit (e.g., a storage unit or a computing unit for further processing), the normalization and rounding circuit 499 can be configured to normalize and round the output 486 and the output 498 based on format (e.g., FP32) and / or precision requirements of the external unit. In some implementations, as shown in Fig. 4D, the addition circuit 410 can include an adder 495, which is configured to perform an addition of the exponent of the floating point MAC operation (e.g., the output 486) and the carry overflow 487 (e.g.. bit

[0030] of the register 496) and provide a sum 497 to the normalization and rounding circuit 499. In this way, the MAC unit 400 can gate operations in the second phase on the last cycle of the accumulation operation, the output 486 available onthe last cycle of the accumulation operation is added to the carry' overflow 487. and the sum 497 is provided as an input to the normalization and rounding circuit 499. Alternatively (not shown in Fig. 4D), if the operations in the second phase are not gated, the output 486 that is available one cycle after the last cycle of the accumulation operation can be directly provided to the normalization and rounding circuit 499. In other words, the adder 495 may not be needed in the latter case.

[0062] In some implementations, the MAC unit 400 can perform an integer MAC operation of N integer input operands 412-1, 412-2, ..., and 412-N. The integer MAC operation can also include three phases. In the first phase of the integer MAC operation, as shown in Fig. 4B, each of the multipliers 466 is configured to perform a multiplication between a pair of the integer input operands 412-1, 412-2, ..., and 412-N to generate the product 470-i, (I < i < N / 2). In the integer MAC operation, the products 470-i (1 < i < N / 2) are integer products. In some implementations, the integer products 470-i (1 < i < N / 2) can be stored in the registers 472. In some implementations, the multipliers 466 each can be coupled to multiplexers that are configured to load the aligned mantissas (e.g., outputs 464-1. 464- 2, ..., and 464-N) in the floating point MAC operation or load the integer input operands 412- 1, 412-2, ..., and 412-N in the integer MAC operation. In the second phase of the integer MAC operation, as shown in Fig. 4C, the addition circuit 406 can perform an addition of the integer products 470-i (1 < i < N / 2) using the unified adder 474. In this example, the output 476 of the unified adder 474 is an integer sum and can be stored in the register 478. In the third phase of the integer MAC operation, as shown in Fig. 4D, the addition circuit 410 can perform an addition of the output 476 and the output 488 using the unified adder 494. In this example, the output 488 is a current value of an integer accumulator, which is stored in the register 496. The integer accumulator in the register 496 can be updated with the output 498 each time the unified adder 494 generates a new output 498.

[0063] Fig. 5 illustrates a flow chart of an example method 500 for performing a MAC operation. The method 500 can be performed by a MAC unit, such as the MAC unit 120 of Fig. 1, the MAC unit 200 of Fig. 2, or the MAC unit 300 of Figs. 3A-3D. At operation 502, the MAC unit receives floating point input operands including a first operand, a second operand, a third operand, and a fourth operand (e.g., the floating input operands 314-1, 314-2, 314-3, and 314-4 of Fig. 3 A).

[0064] At operation 504. the MAC unit performs, e.g., using the pre-multiplication alignment circuit 302 of Fig. 3A, a pre-multiplication alignment process that aligns a mantissa (e.g., mi, m2, m3, and IT described with reference to Fig. 3A) of one or more of thefloating point input operands based on comparing 1) a first sum (e.g., the sum 344) of exponents of the first operand and the second operand, and 2) a second sum (e.g., the sum 348) of exponents of the third operand and the fourth operand.

[0065] At operation 506, the MAC unit performs (e.g., using the multiplier 366 of Fig. 3B) a first multiplication between aligned mantissas (e.g., outputs 364-1 and 364-2 of Figs. 3A-3B) of the first operand and the second operand to generate a first mantissa product (e.g., the output 370-1).

[0066] At operation 508, the MAC unit performs (e.g., using the multiplier 368 of Fig. 3B) a second multiplication between aligned mantissas (e.g., outputs 364-3 and 364-4 of Figs. 3A-3B) of the third operand and the fourth operand to generate a second mantissa product (e.g., the output 370-2).

[0067] At operation 510, the MAC unit performs an addition of the first mantissa product and the second mantissa product using the first unified adder (e.g., the unified adder 374 of Fig. 3C).

[0068] At operation 512. the MAC unit generates a MAC operation output using an output (e.g., the output 376 of Fig. 3C) of the first unified adder.

[0069] In some implementations, the method 500 further includes receiving four integer input operands (e.g., the integer input operands 312-1, 312-2, 312-3, and 312-4 of Fig. 3B). The method 500 further includes performing multiplications betw een pairs of the four integer input operands to generate a first integer product and a second integer product (e.g., the products 370-1 and 370-2 described with reference to Fig. 3B). The method 500 can further include performing an addition of the first integer product and the second integer product using the first unified adder (e.g., the unified adder 374 of Fig. 3C).

[0070] In some implementations, when generating the MAC operation output, the MAC unit performs (e.g., using the post-multiplication alignment circuit 308 of Figs. 3C-3D) a post-multiplication alignment process that align the output (e.g., the sum-of-product mantissa 376) of the first unified adder and a mantissa (e.g., the mantissa 388) of an accumulator based on comparing an exponent output (e g., the exponent output 352 of Fig. 3C) of the premultiplication alignment process and an exponent (e.g.. the exponent 384 of Fig. 3C) of the accumulator. The exponent output of the pre-multiplication alignment process is a larger of the first sum and the second sum.

[0071] In some implementations, when generating the MAC operation output, the MAC unit further generates a mantissa of the MAC operation output by adding outputs (e.g.. the aligned mantissas 376 and 388 of Fig. 3D) of the post-multiplication alignment process usinga second unified adder (e.g., the unified adder 394 of Fig. 3D). The MAC unit can further generate an exponent of the MAC operation output by selecting a larger of the exponent output (e.g., the exponent output 352 of Fig. 3C) of the pre-multiplication alignment process and the exponent (e.g., the exponent 384 of Fig. 3C) of the accumulator.

[0072] In some implementations, the method 500 further includes performing (e.g., using the normalization and rounding circuit 399 of Fig. 3D) a normalization and rounding process of the MAC operation output (e.g., as described with reference to Fig. 3D). The method 500 further includes storing an output of the normalizing and rounding process in a storage unit (e.g., a memory' or a register that supports FP32 format).

[0073] In some implementations, the pre-multiplication alignment process of the method 500 includes shifting (e.g.. using the shifter 360) mantissas of the first operand and the second operand when the first sum is smaller than the second sum (e.g., as described with reference to Fig. 3A).

[0074] In some implementations, the pre-multiplication alignment process of the method 500 includes shifting (e.g.. using the shifter 362) mantissas of the third operand and the fourth operand when the first sum is larger than the second sum (e.g., as described with reference to Fig. 3A).

[0075] In some implementations, the MAC unit can generate the MAC operation output by performing an addition of the output (e.g., the output 376) of the first unified adder and an accumulator (e.g., the output 388) using a second unified adder (e.g., the unified adder 394 of Fig. 3D).

[0076] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory program carrier for execution by, or to control the operation of. data processing apparatus.

[0077] Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storagemedium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0078] The term '‘computing system” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0079] A computer program (which may also be referred to or described as a program, software, a software application, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0080] A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication netw ork.

[0081] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array), an ASIC (application specific integrated circuit), or a GPGPU (General purpose graphics processing unit).

[0082] Computers suitable for the execution of a computer program include, by way of example, can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. Some elements of a computer are a central processing unit for performing or executing instructions and one ormore memory' devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0083] Computer readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0084] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

[0085] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN’’), e.g., the Internet.

[0086] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0087] In addition to the embodiments described above, the following embodiments are also innovative:

[0088] Embodiment 1 is a method for performing a MAC operation with a MAC cell having a first unified adder and a second unified adder, the method comprising:

[0089] receiving floating point input operands including a first operand, a second operand, a third operand, and a fourth operand;

[0090] performing a pre-multiplication alignment process that aligns a mantissa of one or more of the floating point input operands based on comparing 1) a first sum of exponents of the first operand and the second operand, and 2) a second sum of exponents of the third operand and the fourth operand;

[0091] performing a first multiplication between aligned mantissas of the first operand and the second operand to generate a first mantissa product;

[0092] performing a second multiplication between aligned mantissas of the third operand and the fourth operand to generate a second mantissa product;

[0093] performing an addition of the first mantissa product and the second mantissa product using the first unified adder: and

[0094] generating a MAC operation output using an output of the first unified adder.

[0095] Embodiment 2 is the method of embodiment 1, further comprising:

[0096] receiving four integer input operands;

[0097] performing multiplications between pairs of the four integer input operands to generate a first integer product and a second integer product; and

[0098] performing an addition of the first integer product and the second integer product using the first unified adder.

[0099] Embodiment 3 is the method of any one of embodiments 1-2, wherein generating the MAC operation output comprises:

[0100] performing a post-multiplication alignment process that align the output of the first unified adder and a mantissa of an accumulator based on comparing an exponent output of the pre-multiplication alignment process and an exponent of the accumulator, wherein the exponent output of the pre-multiplication alignment process is a larger of the first sum and the second sum.

[0101] Embodiment 4 is the method of any one of embodiments 1-3, wherein generating the MAC operation output further comprises:

[0102] generating a mantissa of the MAC operation output by adding outputs of the post-multiplication alignment process using a second unified adder; and

[0103] generating an exponent of the MAC operation output by selecting a larger of the exponent output of the pre-multiplication alignment process and the exponent of the accumulator.

[0104]

[0105] Embodiment 5 is the method of any one of embodiments 1 -4, further comprising:

[0106] performing a normalization and rounding process of the MAC operation output; and

[0107] storing an output of the normalizing and rounding process in a storage unit.

[0108]

[0109] Embodiment 6 is the method of any one of embodiments 1-5, wherein the premultiplication alignment process comprises:

[0110] shifting mantissas of the first operand and the second operand when the first sum is smaller than the second sum.

[0111] Embodiment 7 is the method of any one of embodiments 1 -6, wherein the premultiplication alignment process comprises:

[0112] shifting mantissas of the third operand and the fourth operand when the first sum is larger than the second sum.

[0113] Embodiment 8 is the method of any one of embodiments 1-7, wherein generating the MAC operation output comprises:

[0114] generating the MAC operation output by performing an addition of the output of the first unified adder and an accumulator using a second unified adder.

[0115] Embodiment 9 is a circuit for performing a MAC operation of floating point input operands including a first operand, a second operand, a third operand, and a fourth operand, the circuit comprising:

[0116] a pre-multiplication alignment circuit configured to align a mantissa of one or more of the floating point input operands based on comparing 1) a first sum of exponents of the first operand and the second operand, and 2) a second sum of exponents of the third operand and the fourth operand;

[0117] a first multiplier configured to generate a first mantissa product by performing a first multiplication between aligned mantissas of the first operand and the second operand;

[0118] a second multiplier configured to generate a second mantissa product by performing a second multiplication between aligned mantissas of the third operand and the fourth operand to generate a second mantissa product; and

[0119] a first unified adder configured to perform an addition of the first mantissa product and the second mantissa product, wherein an output of the MAC operation is generated based on an output of the first unified adder.

[0120] Embodiment 10 is the circuit of embodiment 9, wherein:

[0121] the circuit is configured to receive four integer input operands;

[0122] the first multiplier and the second multiplier are configured to perform multiplications between pairs of the four integer input operands to generate a first integer product and a second integer product; and

[0123] the first unified adder is configured to perform an addition of the first integer product and the second integer product.

[0124] Embodiment 11 is the circuit of any one of embodiments 9-10, further comprising a post-multiplication alignment circuit configured to align the output of the first unified adder and a mantissa of an accumulator based on comparing an exponent output of the premultiplication alignment circuit and an exponent of the accumulator, wherein the exponent output of the pre-multiplication alignment circuit is a larger of the first sum and the second sum.

[0125] Embodiment 12 is the circuit of any one of embodiments 9-11, further comprising a second unified adder, wherein:

[0126] a mantissa of the output of the MAC operation is generated by adding outputs of the post-multiplication alignment circuit using a second unified adder; and

[0127] an exponent of the output of the MAC operation is generated by selecting a larger of the exponent output of the pre-multiplication alignment circuit and the exponent of the accumulator.

[0128] Embodiment 13 is the circuit of any one of embodiments 9-12, further comprising a normalization and rounding circuit configured to:

[0129] generating an output by normalizing and rounding the output of the MAC operation; and

[0130] store the output of the normalization and rounding circuit in a storage unit.

[0131] Embodiment 14 is the circuit of any one of embodiments 9-13, wherein the premultiplication alignment circuit comprises two shifters and is configured to shift mantissas of the first operand and the second operand when the first sum is smaller than the second sum.

[0132] Embodiment 15 is the circuit of any one of embodiments 9-14, wherein the premultiplication alignment circuit is configured to shift mantissas of the third operand and the fourth operand when the first sum is larger than the second sum.

[0133] Embodiment 16 is the circuit of any one of embodiments 9-15, wherein the second unified adder is configured to update the mantissa of the accumulator using the mantissa of the output of the MAC operation.

[0134] Embodiment 17 is the circuit of any one of embodiments 9-16, wherein the postmultiplication alignment circuit is configured to update the exponent of the accumulator using the exponent of the output of the MAC operation.

[0135] Embodiment 18 is a circuit for performing a MAC operation of multiple pairs of floating point input operands, the circuit comprising:

[0136] a pre-multiplication alignment circuit configured to align a mantissa of one or more of the floating point input operands based on comparing a sum of exponents of each pair of the floating point input operands;

[0137] multipliers configured to generate mantissa products by performing multiplications between aligned mantissas of each pair of the floating point input operands; and

[0138] a first unified adder configured to perform an addition of an addition of the mantissa products, wherein an output of the MAC operation is generated based on an output of the first unified adder.

[0139] Embodiment 19 is the circuit of embodiment 18, wherein:

[0140] the circuit is configured to receive multiple pairs integer input operands;

[0141] the multipliers are configured to perform multiplications between each pair of the integer input operands to generate integer products;

[0142] the first unified adder is configured to perform an addition of the integer products.

[0143] Embodiment 20 is the circuit of any one of embodiments 18-19, wherein the first unified adder comprises an adder tree formed by multiple integer adders.

[0144] Embodiment 21 is a system comprising: one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the method of any one of embodiments 1 to 8.

[0145] Embodiment 22 is a computer storage medium encoded with a computer program, the program comprising instructions that are operable, when executed by data processingapparatus, to cause the data processing apparatus to perform the method of any one of embodiments 1 to 8.

[0146] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0147] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0148] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

WHAT IS CLAIMED IS:

1. A method for performing a multiply-accumulate (MAC) operation with a MAC cell having a first unified adder and a second unified adder, the method comprising: receiving floating point input operands including a first operand, a second operand, a third operand, and a fourth operand; performing a pre-multiplication alignment process that aligns a mantissa of one or more of the floating point input operands based on comparing 1) a first sum of exponents of the first operand and the second operand, and 2) a second sum of exponents of the third operand and the fourth operand; performing a first multiplication between aligned mantissas of the first operand and the second operand to generate a first mantissa product; performing a second multiplication between aligned mantissas of the third operand and the fourth operand to generate a second mantissa product; performing an addition of the first mantissa product and the second mantissa product using the first unified adder; and generating a MAC operation output using an output of the first unified adder.

2. The method of claim 1, further comprising: receiving four integer input operands; performing multiplications between pairs of the four integer input operands to generate a first integer product and a second integer product; and performing an addition of the first integer product and the second integer product using the first unified adder.

3. The method of claim 1, wherein generating the MAC operation output comprises: performing a post-multiplication alignment process that align the output of the first unified adder and a mantissa of an accumulator based on comparing an exponent output of the pre-multiplication alignment process and an exponent of the accumulator, wherein the exponent output of the pre-multiplication alignment process is a larger of the first sum and the second sum.

4. The method of claim 3, wherein generating the MAC operation output further comprises:generating a mantissa of the MAC operation output by adding outputs of the postmultiplication alignment process using a second unified adder; and generating an exponent of the MAC operation output by selecting a larger of the exponent output of the pre-multiplication alignment process and the exponent of the accumulator.

5. The method of claim 4, further comprising: performing a normalization and rounding process of the MAC operation output; and storing an output of the normalizing and rounding process in a storage unit.

6. The method of any one of claims 1 to 5, wherein the pre-multiplication alignment process comprises: shifting mantissas of the first operand and the second operand when the first sum is smaller than the second sum.

7. The method of any one of claims 1 to 5, wherein the pre-multiplication alignment process comprises: shifting mantissas of the third operand and the fourth operand when the first sum is larger than the second sum.

8. The method of claim 2, wherein generating the MAC operation output comprises: generating the MAC operation output by performing an addition of the output of the first unified adder and an accumulator using a second unified adder.

9. A circuit for performing a multiply-accumulate (MAC) operation of floating point input operands including a first operand, a second operand, a third operand, and a fourth operand, the circuit comprising: a pre-multiplication alignment circuit configured to align a mantissa of one or more of the floating point input operands based on comparing 1) a first sum of exponents of the first operand and the second operand, and 2) a second sum of exponents of the third operand and the fourth operand; a first multiplier configured to generate a first mantissa product by performing a first multiplication between aligned mantissas of the first operand and the second operand;a second multiplier configured to generate a second mantissa product by performing a second multiplication between aligned mantissas of the third operand and the fourth operand to generate a second mantissa product; and a first unified adder configured to perform an addition of the first mantissa product and the second mantissa product, wherein an output of the MAC operation is generated based on an output of the first unified adder.

10. The circuit of claim 9, wherein: the circuit is configured to receive four integer input operands; the first multiplier and the second multiplier are configured to perform multiplications between pairs of the four integer input operands to generate a first integer product and a second integer product; and the first unified adder is configured to perform an addition of the first integer product and the second integer product.

11. The circuit of claim 9 or claim 10, further comprising a post-multiplication alignment circuit configured to align the output of the first unified adder and a mantissa of an accumulator based on comparing an exponent output of the pre-multiplication alignment circuit and an exponent of the accumulator, wherein the exponent output of the premultiplication alignment circuit is a larger of the first sum and the second sum.

12. The circuit of claim 11, further comprising a second unified adder, wherein: a mantissa of the output of the MAC operation is generated by adding outputs of the post-multiplication alignment circuit using a second unified adder; and an exponent of the output of the MAC operation is generated by selecting a larger of the exponent output of the pre-multiplication alignment circuit and the exponent of the accumulator.

13. The circuit of claim 12, further comprising a normalization and rounding circuit configured to: generating an output by normalizing and rounding the output of the MAC operation; and store the output of the normalization and rounding circuit in a storage unit.

14. The circuit of claim 9, wherein the pre-multiplication alignment circuit comprises two shifters and is configured to shift mantissas of the first operand and the second operand when the first sum is smaller than the second sum.

15. The circuit of claim 14, wherein the pre-multiplication alignment circuit is configured to shift mantissas of the third operand and the fourth operand when the first sum is larger than the second sum.

16. The circuit of claim 12, wherein the second unified adder is configured to update the mantissa of the accumulator using the mantissa of the output of the MAC operation.

17. The circuit of claim 12, wherein the post-multiplication alignment circuit is configured to update the exponent of the accumulator using the exponent of the output of the MAC operation.

18. A circuit for performing a multiply-accumulate (MAC) operation of multiple pairs of floating point input operands, the circuit comprising: a pre-multiplication alignment circuit configured to align a mantissa of one or more of the floating point input operands based on comparing a sum of exponents of each pair of the floating point input operands; multipliers configured to generate mantissa products by performing multiplications between aligned mantissas of each pair of the floating point input operands; and a first unified adder configured to perform an addition of an addition of the mantissa products, wherein an output of the MAC operation is generated based on an output of the first unified adder.

19. The circuit of claim 18, wherein: the circuit is configured to receive multiple pairs integer input operands; the multipliers are configured to perform multiplications between each pair of the integer input operands to generate integer products; the first unified adder is configured to perform an addition of the integer products.

20. The circuit of claim 18 or claim 19, wherein the first unified adder comprises an adder tree formed by multiple integer adders.

Images