Integrated circuit, system and method for multi-precision multiply-accumulate operations - Patents.com

Abstract

To provide an integrated circuit, system, and method for multiple-precision multiply-and-accumulate operation.SOLUTION: In an integrated circuit, multiple-precision multiply-and-accumulate operation is performed by a multiply-and-accumulate (MAC) unit area configured to operate in an integer mode to perform computations on first data-width integer values to produce third data-width integer values and configured to operate in a floating point mode to perform computations on second data-width floating point values to produce third data-width floating point values, in which the second data width is twice the first data width and the third data width is larger than the second data width. The MAC unit area includes a first multiplier configured to multiply two integer values in the integer mode or multiply mantissa values extracted from each of two floating point values in the floating point mode. The MAC unit area further includes a second multiplier configured to multiply two integer values in the integer mode or refrain from using the second multiplier in the floating point mode.SELECTED DRAWING: Figure 1

Description

[Background technology]

[0001] Many neural network accelerators are configured to perform neural network inference using integer formats such as INT8. Many neural network models are designed for floating-point formats such as FP32 and BF16. To perform inference of a neural network model designed for floating-point formats using an accelerator configured for integer formats, the floating-point values ​​are quantized to integer values. [Brief description of the drawings]

[0002] Aspects of the present disclosure are best understood from the following detailed description when read in conjunction with the accompanying drawings, in which: It should be noted that, in accordance with standard industry practice, various features have not been drawn to scale, and in fact, dimensions of various features may be arbitrarily increased or decreased for clarity of discussion.

[0003] [Figure 1] FIG. 1 is a diagram of a system for multi-precision multiply-accumulate operations in accordance with at least one embodiment of the present invention.

[0004] [Diagram 2] FIG. 2 is a diagram of a portion of an integrated circuit for multi-precision multiply-accumulate operations in accordance with at least one embodiment of the present invention.

[0005] [Diagram 3] FIG. 2 is a diagram of a multiply-accumulate (MAC) unit for multi-precision multiply-accumulate operations in accordance with at least one embodiment of the present invention.

[0006] [Figure 4] FIG. 2 is a schematic diagram of a MAC unit performing multiply-accumulate operations in floating-point mode in accordance with at least one embodiment of the present invention.

[0007] [Diagram 5]FIG. 2 is a schematic diagram of a MAC unit performing multiply-accumulate operations in integer mode, in accordance with at least one embodiment of the present invention.

[0008] [Figure 6] FIG. 2 is a diagram of an operational flow for multi-precision multiply-accumulate operations and activation in accordance with at least one embodiment of the present invention.

[0009] [Figure 7] FIG. 2 is an operational flow diagram for performing a multiply-accumulate calculation in accordance with at least one embodiment of the present invention.

[0010] [Figure 8] FIG. 2 is an operational flow diagram for calculating an exponent for a multiply-accumulate calculation of floating-point values ​​in accordance with at least one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The following disclosure provides many different embodiments or examples for implementing different features of the provided subject matter. Specific examples of components, values, operations, materials, arrangements, etc. are described below to simplify the disclosure. Of course, these are merely examples and are not intended to be limiting. Other components, values, operations, materials, arrangements, etc. are contemplated. In addition, the disclosure may repeat reference numerals and / or letters in the various examples. This repetition is for the purposes of brevity and clarity and does not in itself dictate a relationship between the various embodiments and / or configurations discussed.

[0012] Quantization of some neural networks, such as quantization to use INT8 format instead of the native FP32 format, results in significant loss of accuracy and other important parameters when performing inference of the neural network.

[0013] Performing neural network inference using floating-point format has improved accuracy over using integer format, and since many neural network models are designed for floating-point format, performing neural network inference using floating-point format reduces computational resource usage for quantizing such neural network models to integer format.

[0014] At least some embodiments of the integrated circuits described herein support floating-point formats along with integer formats. In at least some embodiments, the integrated circuits described herein support the Brain-Float16 (BF16) format, which requires only half the bandwidth of FP32 but covers the full value range of FP32, which allows for similar accuracy as FP32. The data width of BF16 is twice the bit width of INT8. However, the mantissa, or significand, of BF16 is 8 bits, which matches the bit width of INT8. This means that for every one input of BF16, there are two inputs of INT8, and the amount of terra-operations-per-second (TOPS) will scale by a factor of two from BF16 to INT8.

[0015] 1 is a system for multi-precision multiply-accumulate operations in accordance with at least one embodiment of the present invention. The system includes an integrated circuit 100 and a host computer 102.

[0016] Integrated circuit 100 includes a MAC unit area 110, an integer activation pipeline 112, a floating point activation pipeline 114, a memory 116, and a controller 118. In at least some embodiments, integrated circuit 100 is an application specific integrated circuit (ASIC) that includes dedicated circuitry. In at least some embodiments, the integrated circuit is a field programmable gate array (FPGA).

[0017] The MAC unit area 110 communicates with the memory 116 and the controller 118. In at least some embodiments, the MAC unit area 110 performs a dot product function. In at least some embodiments, the MAC unit area 110 comprises a plurality of multipliers and adders. In at least some embodiments, the MAC unit area 110 comprises a plurality of MAC units and a plurality of accumulating adders. In at least some embodiments, the MAC unit area 110 comprises a plurality of multiplier groups, dedicated multipliers, and accumulating adders. In at least some embodiments, the MAC unit area 110 is configured to operate in an integer mode to perform calculations on first data width integer values ​​to generate third data width integer values ​​and in a floating point mode to perform calculations on second data width floating point values ​​to generate third data width floating point values, where the second data width is twice the first data width and the third data width is greater than the second data width. In at least some embodiments, the MAC unit area is a systolic array, such as systolic array 211 of Figure 2, which is described below. In at least some embodiments, the MAC unit area is a single MAC unit and normalizing adder, such as MAC unit 320 of Figure 3 and normalizing adder 232 of Figure 2, which is described below.

[0018] The integer activation pipeline 112 communicates with the memory 116 and the controller 118. In at least some embodiments, the integer activation pipeline 112 is configured to perform further operations on the output dot products of the systolic array. In at least some embodiments, the integer activation pipeline 112 is configured to activate third data width integer values ​​to generate activated first data width integer values. In at least some embodiments, the integer activation pipeline 112 is configured to perform bias addition, residual addition, residual multiplication, quantization, requantization, and other operations that may be required for inference of a given neural network, in addition to the activation function. In at least some embodiments, the integer activation pipeline 112 comprises at least one look-up table (LUT) for approximation of the activation function. In at least some embodiments, each LUT has a depth of M, and each location in the LUT store is a second data width floating point value.

[0019] The floating-point activation pipeline 114 communicates with the memory 116 and the controller 118. In at least some embodiments, the floating-point activation pipeline 114 is configured to activate a third data width floating-point value to generate an activated second data width floating-point value. In at least some embodiments, the floating-point activation pipeline 114 is configured to perform bias addition, residual addition, residual multiplication, and other operations that may be required for inference of a given neural network, in addition to the activation function. In at least some embodiments, the floating-point activation pipeline 114 performs conversion between the first data width integer format and the second data width floating-point format. In at least some embodiments, the floating-point activation pipeline 114 comprises at least one look-up table (LUT) for approximation of the activation function. In at least some embodiments, each LUT has a depth of M, and each location in the LUT store is a second data width floating-point value.

[0020] In at least some embodiments, memory 116 is configured to store values ​​and transmit stored values. In at least some embodiments, memory 116 comprises one or more banks or blocks of volatile data storage, such as Random Access Memory (RAM), Embedded System Block (ESB), Content Addressable Memory (CAM), or the like. In at least some embodiments, memory 116 is distributed throughout integrated circuit 100. In at least some embodiments, memory 116 communicates with each MAC unit of MAC unit area 110 via one or more data paths, such as an interconnect or data bus. In at least some embodiments, memory 116 is configured to transmit and receive data values ​​through the data paths to MAC unit area 110, integer activation pipeline 112, and floating point activation pipeline 114.

[0021] The controller 118 is in communication with the host computer 102, the MAC unit area 110, the integer activation pipeline 112, and the floating point activation pipeline 114. In at least some embodiments, the controller 118 comprises circuitry configured to receive a program from the host computer 102, such as a program including instructions to perform neural network inference. In at least some embodiments, the controller 118 comprises circuitry configured to transmit the program to one or more sequencers of the MAC unit area 110, the integer activation pipeline 112, and the floating point activation pipeline 114. In at least some embodiments, the controller 118 is configured to operate in an integer mode to perform neural network inference on first data width integer values ​​and in a floating point mode to perform neural network inference on second data width floating point values.

[0022] In at least some embodiments, the host computer 102 is a personal computer, a server, a portion of a cloud computing resource, or other entity capable of sending program instructions to the integrated circuit 100 and storing result data. In at least some embodiments, the host computer 102 is a notebook computer, a tablet computer, a smartphone, a smart watch, an Internet of Things (IoT) device, or the like. The host computer 102 communicates with the integrated circuit 100 through control paths and data paths. In at least some embodiments, the host computer 102 includes external memory, such as a Dynamic Random Access Memory (DRAM), configured to store programs, input data, and result data.

[0023] 2 is a diagram of a portion of an integrated circuit for multi-precision multiply-accumulate operations in accordance with at least one embodiment of the present invention. The portion includes a systolic array 211 and a memory 216.

[0024] Systolic array 211 includes a number of multiply-accumulate (MAC) units, such as MAC unit 220A, MAC unit 220B, MAC unit 220C, and MAC unit 220D, and a number of accumulate adders, such as accumulate adder 230 and normalize accumulate adder 232.

[0025] The memory 216 communicates with each MAC unit in the plurality of MAC units. In at least some embodiments, each MAC unit is configured to perform calculations in an integer mode and is further configured to perform calculations in a floating point mode. In at least some embodiments, each MAC unit is configured to read second data width floating point values ​​from the memory 216, and each MAC unit is further configured to read first data width integer values ​​from the memory 216. In at least some embodiments, each MAC unit in the plurality of MAC units comprises a first multiplier, an exponent adder, a comparator, a subtractor, and a shifter. In at least some embodiments, each MAC unit further comprises a second multiplier. In at least some embodiments, each MAC unit further comprises at least one accumulator adder in the plurality of accumulator adders. In at least some embodiments, each MAC unit includes multiple instances of a first multiplier and second multiplier pair, and each first multiplier in the multiple instances is grouped with an instance of the exponent adder, an instance of the comparator, an instance of the subtractor, and an instance of the shifter.

[0026] Memory 216 is in communication with at least some of the accumulating adders of the plurality of accumulating adders. In at least some embodiments, each accumulating adder is connected to two or more of any combination of multipliers and preceding accumulating adders included in the plurality of MAC units. For example, accumulating adder 230 is connected to MAC unit 220A and MAC unit 220B, and normalizing accumulating adder 232 is connected to MAC unit 220C, MAC unit 220D, and accumulating adder 230, which is the preceding accumulating adder from the perspective of normalizing accumulating adder 232.

[0027] In at least some embodiments, the multiple accumulating adders are collectively configured to accumulate the intermediate mantissa values ​​of the multiple MAC units to generate an accumulated mantissa value, and the multiple accumulating adders are further collectively configured to accumulate the intermediate integer values ​​of the multiple MAC units to generate an accumulated integer value. For example, accumulating adder 230 and normalizing accumulating adder 232 are collectively configured to accumulate the intermediate mantissa values ​​or intermediate integer values ​​of MAC unit 220A, MAC unit 220B, MAC unit 220C, and MAC unit 220D to generate an accumulated mantissa value or accumulated integer value, which is stored on memory 216. In at least some embodiments, each accumulating adder is configured to perform floating point addition and integer addition, where the floating point addition includes shifting the intermediate mantissa value based on a maximum exponent. In at least some embodiments, the plurality of accumulating adders comprises a normalizing accumulating adder further configured to normalize the accumulated mantissa value by the maximum exponent value to generate the third data width floating-point value, the normalizing accumulating adder further configured to generate the third data width integer value without normalization. For example, the normalizing accumulating adder 232 is further configured to normalize the accumulated mantissa value by the exponent value to generate the third data width floating-point value and then store the third data width floating-point value in the memory 216, or to generate the third data width integer value from the accumulated integer value without normalization and then store the third data width integer value in the memory 216.

[0028] In the above example, MAC unit 220A, MAC unit 220B, MAC unit 220C, and MAC unit 220D are multiple MAC units, and accumulate adder 230 and normalize accumulate adder 232 are multiple accumulate adders. In this manner, systolic array 211 may be considered as having multiple instances of multiple MAC units and multiple accumulate adders, each configured to operate in an integer mode to perform calculations on first data width integer values ​​to generate third data width integer values, and in a floating point mode to perform calculations on second data width floating point values ​​to generate third data width floating point values. In at least some embodiments, each instance of the MAC units and accumulate adders is a column of systolic array 211, such as in MAC unit 220A, MAC unit 220B, MAC unit 220C, MAC unit 220D, accumulate adder 230, and normalize accumulate adder 232, or a row of systolic array 211 depending on whether the adders are connected vertically or horizontally.

[0029] 3 is a multiply-accumulate (MAC) unit 320 for multi-precision multiply-accumulate operations according to at least one embodiment of the present invention. The MAC unit 320 includes a multiplier group 321, a dedicated multiplier 324A, and an accumulate adder 330. The multiplier group 321 includes an extractor 323, a group multiplier 324B, an exponent adder 326, a comparator 327, a subtractor 328, and a shifter 329. In at least some embodiments, a memory communicates with each multiplier group in the multiple multiplier groups and each dedicated multiplier in the multiple multipliers. In at least some embodiments, the MAC unit 320 is included in a systolic array of an integrated circuit. In at least some embodiments, the MAC unit 320 is included in an integrated circuit external to any systolic array. In at least some embodiments, the MAC unit area 320 is configured to operate in an integer mode to perform calculations on first data width integer values ​​to generate third data width integer values, and in a floating point mode to perform calculations on second data width floating point values ​​to generate third data width floating point values, where the second data width is twice the first data width and the third data width is greater than the second data width.

[0030] The extractor 323 is coupled to the exponent adder 326 and the group multiplier 324B. In at least some embodiments, the extractor 323 is configured to extract an exponent value and a mantissa value from each of the plurality of second data width floating-point values, the extractor 323 being further configured to pass at least two first data width integer values ​​among the plurality of first data width integer values. In at least some embodiments, the extractor 323 is configured to extract an activation exponent value 342 and an activation mantissa value 344 from the input activation values ​​340 and to extract a weight exponent value 343 and a weight mantissa value 345 from the input weight values ​​341, where the input activation values ​​340 and the input weight values ​​341 are second data width floating-point values. In at least some embodiments, the extractor 323 is configured to receive the input activation values ​​340 and the input weight values ​​341 from a memory. In at least some embodiments, each multiplier group is configured to read a plurality of second data width floating point values ​​from the memory, and each multiplier group is further configured to read a plurality of first data width integer values ​​from the memory. In at least some embodiments, extractor 323 is configured to send activation exponent value 342 and weighted exponent value 343 to exponent adder 326, and to send activation mantissa value 344 and weighted mantissa value 345 to group multiplier 324B. In at least some embodiments, extractor 323 is configured to pass input activation value 340 and input weight value 341 to group multiplier 324B, where input activation value 340 and input weight value 341 are first data width integer values.

[0031] The group multiplier 324B is coupled to the extractor 323 and the shifter 329. In at least some embodiments, the group multiplier 324B is configured to multiply the extracted mantissa values ​​from each of two floating-point values ​​among the plurality of floating-point values ​​to generate a first mantissa product value, and the group multiplier 324B is further configured to multiply two of the at least two first data width integer values ​​among the plurality of first data width integer values ​​to generate a first integer product value. In at least some embodiments, the group multiplier 324B is configured to multiply the activation mantissa value 344 and the weighted mantissa value 345 to generate a mantissa product value 346, where the input activation value 340 and the input weight value 341 are second data width floating-point values. In at least some embodiments, the group multiplier 324B is configured to multiply the input activation value 340 and the input weight value 341 to generate an integer product value, where the input activation value 340 and the input weight value 341 are first data width integer values. In at least some embodiments, group multiplier 324B is configured to receive activation mantissa values ​​344 and weighted mantissa values ​​345 from extractor 323 and is configured to send a mantissa product value 346 to shifter 329. In at least some embodiments, group multiplier 324B is configured to receive input activation values ​​340 and input weight values ​​341 from extractor 323 and is configured to send an integer product value to shifter 329.

[0032] The exponent adder 326 is coupled to the extractor 323, the comparator 327, and the subtractor 328. In at least some embodiments, the exponent adder 326 is configured to add the exponent values ​​extracted from each of two floating-point values ​​in the plurality of floating-point values ​​to generate an exponent sum value. In at least some embodiments, the exponent adder 326 is configured to add the activation exponent value 342 and the weighted exponent value 343 to generate an exponent sum value 347, where the input activation value 340 and the input weight value 341 are second data width floating-point values. In at least some embodiments, the exponent adder 326 is configured to receive the activation exponent value 342 and the weighted exponent value 343 from the extractor 323 and is configured to transmit the exponent sum value 347 to the comparator 327 and the subtractor 328. In at least some embodiments, the exponent adder 326 is in a dormant state when the input activation value 340 and the input weight value 341 are first data width integer values.

[0033] The comparator 327 is coupled to the exponent adder 326 and the subtractor 328. In at least some embodiments, the comparator 327 is configured to determine a maximum exponent value among a plurality of exponent sum values ​​generated from a plurality of multiplier groups of the systolic array. In at least some embodiments, the comparator 327 is configured to determine a maximum exponent value among a plurality of exponent sum values ​​generated from a plurality of MAC units of the systolic array. In at least some embodiments, the comparator 327 is configured to determine a maximum exponent value 348 among a plurality of exponent sum values, including exponent sum value 347, generated from a plurality of MAC units, including MAC unit 320, of the systolic array. In at least some embodiments, the comparator 327 is in communication with a plurality of comparators, each of which is included within a corresponding MAC unit of the plurality of MAC units of the systolic array. In at least some embodiments, the comparator 327 is in communication with an exponent adder and a subtractor included within each MAC unit of the plurality of MAC units of the systolic array. In at least some embodiments, the comparator 327 is configured to determine a maximum exponent value among a plurality of exponent values ​​including the exponent value of the third data width floating-point value resulting from a previous iteration of the calculation. In at least some embodiments, the comparator 327 is configured to receive the exponent sum value 347 from the exponent adder 326 and is configured to send the maximum exponent value 348 to the subtractor 328. In at least some embodiments, the comparator 327 is configured to send the maximum exponent value 348 as an intermediate exponent value 354 to another comparator, another MAC unit, or to an accumulate adder. In at least some embodiments, the comparator 327 is in a dormant state when the input activation value 340 and the input weight value 341 are first data width integer values.

[0034] The subtractor 328 is coupled to the exponent adder 326, the comparator 327, and the shifter 329. In at least some embodiments, the subtractor 328 is configured to subtract the exponent sum value from the maximum exponent value to generate a difference. In at least some embodiments, the subtractor 328 is configured to subtract the exponent sum value 347 from the maximum exponent value 348 to generate an exponent difference value 349. In at least some embodiments, the subtractor 328 is configured to receive the exponent sum value 347 from the exponent adder 326 and the maximum exponent value 348 from the comparator 327, and is configured to send the exponent difference value 349 to the shifter 329. In at least some embodiments, the subtractor 328 is in a dormant state when the input activation value 340 and the input weight value 341 are first data width integer values.

[0035] The shifter 329 is coupled to the subtractor 328, the group multiplier 324B, and the accumulate adder 330. In at least some embodiments, the shifter 329 is configured to shift the first mantissa product value based on the difference to generate an intermediate mantissa value, and the shifter 329 is further configured to pass the first integer product value as the intermediate integer value. In at least some embodiments, the shifter 329 is configured to shift the mantissa product value 346 based on the exponent difference 349 to generate a shifted mantissa value 350, where the input activation value 340 and the input weight value 341 are second data width floating point values. In at least some embodiments, the shifter 329 is configured to pass the first integer product value as the intermediate integer value, where the input activation value 340 and the input weight value 341 are first data width integer values. In at least some embodiments, the shifter 329 is configured to receive the mantissa product value 346 from the group multiplier 324B and the exponent difference value 349 from the subtractor 328, and is configured to transmit the shifted mantissa value 350 to the accumulator adder 330. In at least some embodiments, the shifter 329 is configured to receive the first integer product value from the group multiplier 324B, and is configured to transmit the first integer product value to the accumulator adder 330. In at least some embodiments, the shifter 329 has a larger bit width for improved accuracy. In at least some embodiments, the shifter 329 has a smaller bit width, which reduces accuracy compared to a longer bit width, but requires fewer hardware resources. In this manner, the bit width of the shifter 329 represents a tradeoff between accuracy and hardware resource consumption.

[0036] The dedicated multiplier 324A is coupled to the accumulating adder 330. In at least some embodiments, the dedicated multiplier 324A is configured to multiply two first data width integer values ​​of the plurality of first data width integer values ​​to generate a second integer product value. In at least some embodiments, the dedicated multiplier 324A is configured to multiply a second input activation value and a second input weight value to generate a second integer product value, where the input activation value 340 and the input weight value 341 are first data width integer values. In at least some embodiments, the dedicated multiplier 324A is configured to receive the second input activation value and the second input weight value from the memory. In at least some embodiments, the dedicated multiplier 324A is configured to send the second integer product value to the accumulating adder 330. In at least some embodiments, the dedicated multiplier 324A is in a dormant state when the input activation value 340 and the input weight value 341 are second data width floating point values.

[0037] Accumulation adder 330 is coupled to shifter 329 and dedicated multiplier 324A. In at least some embodiments, accumulation adder 330 is configured to add a first integer product value to a second integer product value to generate an intermediate integer value. In at least some embodiments, accumulation adder 330 is configured to add a first integer product value to a second integer product value to generate an intermediate integer value, where input activation value 340 and input weight value 341 are first data width integer values. In at least some embodiments, accumulation adder 330 is configured to add a shifted mantissa value 350 to another shifted mantissa value from another multiplier group of MAC unit 320 to generate an intermediate mantissa value 352, where input activation value 340 and input weight value 341 are second data width floating point values. In at least some embodiments, the accumulate adder 330 is configured to receive the first integer product value from the shifter 329 and the second integer product value from the dedicated multiplier 324A, and the accumulate adder 330 is configured to send the intermediate integer value to the subsequent accumulate adder. In at least some embodiments, the accumulate adder 330 is configured to receive the shifted mantissa value 350 from the shifter 329 and the other shifted mantissa value from the other multiplier group of the MAC unit 320, and the accumulate adder 330 is configured to send the intermediate mantissa value 352 to the subsequent accumulate adder. In at least some embodiments, the accumulating adder 330 is one of a plurality of accumulating adders, each accumulating adder being connected to two or more of any combination of a shared multiplier in the plurality of multiplier groups, a dedicated multiplier in the plurality of dedicated multipliers, and a preceding accumulating adder, the plurality of accumulating adders being collectively configured to accumulate intermediate mantissa values ​​of the plurality of multiplier groups to generate an accumulated mantissa value, and the plurality of accumulating adders being collectively further configured to accumulate intermediate integer values ​​of the plurality of multiplier groups and the plurality of dedicated multipliers to generate an accumulated integer value. In at least some embodiments, the accumulating adder 330 has a larger bit width for improved accuracy. In at least some embodiments, the accumulating adder 330 has a smaller bit width, which is less accurate compared to a longer bit width, but requires fewer hardware resources.Thus, the bit-width of accumulator adder 330 represents a trade-off between accuracy and hardware resource consumption.

[0038] Figure 4 is a schematic diagram of a MAC unit 420 for performing multiply-accumulate operations in floating-point mode in accordance with at least one embodiment of the present invention. MAC unit 420 comprises multiplier group 421A including group multiplier 424A, dedicated multiplier 424B, multiplier group 421B including group multiplier 424C, dedicated multiplier 424D, and accumulate adder 430. Multiplier groups 421A and 421B, group multipliers 424A and 424C, dedicated multipliers 424B and 424D, and accumulate adder 430 are substantially similar in structure and function to multiplier group 321, group multiplier 324B, dedicated multiplier 324A, and accumulate adder 330 of Figure 3, respectively, except as otherwise indicated.

[0039] In floating point mode, MAC unit 420 receives a plurality of floating point values ​​including activated 0 mantissa 444A, activated 0 exponent 442A, weighted 0 mantissa 445A, weighted 0 exponent 443A, activated 1 mantissa 444B, activated 1 exponent 442B, weighted 1 mantissa 445B, and weighted 1 exponent 443B. In floating point mode, MAC unit 420 directs activated 0 mantissa 444A, activated 0 exponent 442A, weighted 0 mantissa 445A, and weighted 0 exponent 443A to multiplier group 421A and directs activated 1 mantissa 444B, activated 1 exponent 442B, weighted 1 mantissa 445B, and weighted 1 exponent 443B to multiplier group 421B. In floating point mode, dedicated multiplier 424B and dedicated multiplier 424D are inactive. In at least some embodiments, MAC unit 420 is further configured to refrain from using dedicated multiplier 424B and dedicated multiplier 424D in floating point mode. In floating point mode, multiplier group 421A and multiplier group 421B each send a shifted mantissa value to accumulate adder 430. In floating point mode, the accumulate adder adds the shifted mantissa values ​​to generate intermediate mantissa value 452.

[0040] 5 is a schematic diagram of a MAC unit 520 performing multiply-accumulate operations in integer mode in accordance with at least one embodiment of the present invention. MAC unit 520 includes multiplier group 521A including group multiplier 524A, dedicated multiplier 524B, multiplier group 521B including group multiplier 524C, dedicated multiplier 524D, and accumulate adder 530. Multiplier groups 521A and 521B, group multipliers 524A and 524C, dedicated multipliers 524B and 524D, and accumulate adder 530 are substantially similar in structure and function to multiplier group 321, group multiplier 324B, dedicated multiplier 324A, and accumulate adder 330 of FIG. 3, respectively, except as otherwise indicated.

[0041] In integer mode, MAC unit 520 receives a plurality of integer values ​​including activation 0 integer 544A, weight 0 integer 545A, activation 1 integer 544B, weight 1 integer 545B, activation 2 integer 544C, weight 2 integer 545C, activation 3 integer 544D, and weight 3 integer 545D. In integer mode, MAC unit 420 directs activation 0 integer 544A and weight 0 integer 545A to multiplier group 521A, directs activation 1 integer 544B and weight 1 integer 545B to dedicated multiplier 524B, directs activation 2 integer 544C and weight 2 integer 545C to multiplier group 521B, and directs activation 3 integer 544D and weight 3 integer 545D to dedicated multiplier 524D. In integer mode, multiplier group 521A and multiplier group 521B are idle except for group multiplier 524A and group multiplier 524C, respectively. In integer mode, multiplier group 521A, dedicated multiplier 524B, multiplier group 521B, and dedicated multiplier 524D each send integer product values ​​to accumulate adder 530. In integer mode, the accumulate adder adds the integer product values ​​to generate intermediate integer value 556.

[0042] 6 is an operational flow for multi-precision multiply-accumulate operation and activation in accordance with at least one embodiment of the present invention. The operational flow provides a method for multi-precision multiply-accumulate operation and activation. In at least some embodiments, the method is performed by a controller of an integrated circuit, such as controller 118 of FIG. 1.

[0043] At S660, the controller reads the memory. In at least some embodiments, the controller reads integer values ​​from the memory in an integer mode. In at least some embodiments, the controller reads floating point values ​​from the memory in a floating point mode. In at least some embodiments, the controller causes the MAC units to collectively read integer values ​​from the memory in an integer mode and floating point values ​​in a floating point mode.

[0044] In S661, the controller performs a calculation. In at least some embodiments, the controller performs the calculation on values ​​read from the memory. In at least some embodiments, the controller causes the multiple MAC units to calculate an intermediate value from the values ​​read from the memory. In at least some embodiments, the controller causes each MAC unit, in integer mode, to multiply two integer values ​​in the multiple integer values ​​using a second multiplier to generate an intermediate integer value. In at least some embodiments, the controller performs the operations of FIG. 7 described below.

[0045] At S662, the controller accumulates the intermediate values. In at least some embodiments, the controller accumulates the intermediate values ​​to generate an accumulated value. In at least some embodiments, the controller causes the multiple accumulating adders to accumulate intermediate mantissa values ​​of the multiple MAC units to generate an accumulated mantissa value in floating point mode. In at least some embodiments, the controller causes the multiple accumulating adders to accumulate intermediate integer values ​​of the multiple MAC units to generate an accumulated integer value in integer mode. In at least some embodiments where each MAC unit includes an accumulating adder, each MAC unit is further configured to add the intermediate values ​​generated from the group multiplier and the dedicated multiplier in the integer mode.

[0046] In S664, the controller determines whether the accumulated value is a mantissa value. In at least some embodiments, floating point mode is indicated if the accumulated value is a mantissa value. In at least some embodiments, integer mode is indicated if the accumulated value is an integer value. If the accumulated value is a mantissa value, the operational flow proceeds to normalization in S665. If the accumulated value is not a mantissa value, the operational flow proceeds to addition to the previous accumulated value in S666.

[0047] At S665, the controller normalizes the accumulated mantissa value by the maximum exponent value. In at least some embodiments, the controller causes the normalizing accumulate adder to normalize the accumulated mantissa value by the maximum exponent value. In at least some embodiments, the normalizing accumulate adder normalizes the accumulated mantissa value by the maximum exponent value to generate a third data width floating-point value in floating-point mode. In at least some embodiments, the controller replaces the accumulated mantissa value with a two's complement number if the accumulated mantissa value is negative. In at least some embodiments, the controller detects a leading 1 and shifts the accumulated mantissa value left. In at least some embodiments, the controller calculates an exponent value of the floating-point value based on the maximum exponent value, the number of MAC units in the plurality of MAC units, and the number of multipliers in each MAC unit. In at least some embodiments, the controller:

number

[0048] At S666, the controller adds the floating point value to the previously stored floating point value from the previous iteration. In at least some embodiments, the controller causes the normalization accumulate adder to add the floating point value normalized at S665 of the previous iteration to the floating point value normalized at S665 of this iteration. In at least some embodiments, the controller causes the normalization accumulate adder to add the accumulated integer value generated at S665 of the previous iteration to the accumulated integer value generated at S665 of this iteration. In at least some embodiments, if the calculated exponent value is greater than 255 or less than zero, the controller sets the accumulated mantissa value to zero and clips the exponent value to 255 or zero. In at least some embodiments, the resulting value is stored in memory.

[0049] In S668, the controller determines whether all iterations have been completed. In at least some embodiments, the controller is programmed to complete a predetermined number of iterations. If the controller determines that all iterations have not been completed, the operational flow returns to the memory read in S660. If the controller determines that all iterations have been completed, the operational flow proceeds to the accumulation value activation in S669.

[0050] At S669, the controller activates the accumulated value. In at least some embodiments, the controller activates the resulting value stored in memory at S666. In at least some embodiments, the controller causes the integer activation pipeline to activate the third data width integer value to generate an activated first data width integer value. In at least some embodiments, the controller causes the floating point activation pipeline to activate the third data width floating point value to generate an activated second data width floating point value. In at least some embodiments, the activation function includes Sigmoid, GELU, SiLU, RELU, etc. In at least some embodiments, the controller executes the activation function using an approximation function that approximates the value based on the input. In at least some embodiments, the approximation function is a direct approximation, a linear approximation, or a polynomial approximation to second or third order. In at least some embodiments, the approximation function is a piecewise linear approximation. In at least some embodiments, a piecewise linear approximation of Y=MX+C is used, where M is the slope of the line, C is the offset, X is the input value, and Y is the activated value. In at least some embodiments, the approximation function is based on at least one look-up table (LUT), such as one LUT for the slope and one LUT for the offset. In at least some embodiments, when the slope and offset values ​​are returned from the table, the slope is multiplied by the input value and added to the offset value.

[0051] 7 is an operational flow for performing a multiply-accumulate calculation in accordance with at least one embodiment of the present invention. The operational flow provides a method for performing a multiply-accumulate calculation, such as the calculation performed in S661 of FIG 6. In at least some embodiments, the method is performed by a MAC unit of a systolic array, such as MAC unit 320 of FIG 3.

[0052] In S770, the MAC unit determines whether the input value is a floating point value. In at least some embodiments, the MAC unit determines whether to apply floating point mode or integer mode. If the MAC unit determines that the input value is a floating point value, the operational flow proceeds to extraction in S772. If the MAC unit determines that the input value is not a floating point value, the operational flow proceeds to integer multiplication in S777.

[0053] At S772, the MAC unit extracts a mantissa and an exponent from each input floating point value. In at least some embodiments, the MAC unit causes the extractor to extract an exponent value and a mantissa value from each of a plurality of floating point values, each floating point value in the plurality of floating point values ​​having a second data width. In at least some embodiments, the MAC unit causes the extractor to separate the mantissa and exponent from the concatenation of values ​​that make up the second data width floating point value. In at least some embodiments, the MAC unit causes the extractor to separate bits 6-0 as the mantissa from bits 14-8 as the exponent. In at least some embodiments, the MAC unit causes the extractor to replace bits 6-0 in the mantissa with a two's complement number.

[0054] At S774, the MAC unit calculates an exponent value. In at least some embodiments, the MAC unit calculates an exponent sum value, compares the exponent sum value to another exponent sum value, and determines a difference value from which to shift the mantissa product value. In at least some embodiments, the MAC unit performs the operational flow of FIG. 8, which is described below.

[0055] At S776, the MAC unit multiplies the mantissa values. In at least some embodiments, the MAC unit causes a multiplier, such as a group multiplier, to multiply the mantissa values ​​extracted from each of two floating-point values ​​in the plurality of floating-point values ​​to generate a first mantissa product value. In at least some embodiments, the MAC unit causes a group multiplier of each multiplier group to multiply the mantissa values ​​to generate a mantissa product value.

[0056] At S777, the MAC unit multiplies the integer values. In at least some embodiments, the MAC unit causes a multiplier, such as a group multiplier, a dedicated multiplier, or any combination thereof, to multiply two integer values ​​in the plurality of integer values ​​to generate an intermediate integer value, each integer value in the plurality of integer values ​​having a first data width. In at least some embodiments, the MAC unit causes a group multiplier and each dedicated multiplier of each multiplier group to multiply the integer values ​​to generate the intermediate integer value.

[0057] At S778, the MAC unit shifts the mantissa product value. In at least some embodiments, the MAC unit causes a shifter to shift the first mantissa product value based on the difference value to generate an intermediate mantissa value. In at least some embodiments, the MAC unit causes a shifter of each multiplier group to shift the mantissa product value to generate an intermediate mantissa value.

[0058] 8 is an operational flow for computing an exponent for a multiply-accumulate calculation of floating-point values, in accordance with at least one embodiment of the present invention. The operational flow provides a method for computing an exponent for a multiply-accumulate calculation of floating-point values, such as the calculation performed at S774 of FIG 7. In at least some embodiments, the method is performed by a MAC unit of a systolic array, such as MAC unit 320 of FIG 3.

[0059] At S880, the MAC unit adds the exponent values. In at least some embodiments, the MAC unit causes the exponent adders to add the exponent values ​​extracted from each of the two floating-point values ​​in the plurality of floating-point values ​​to generate an exponent sum value. In at least some embodiments, the MAC unit causes the exponent adders of each multiplier group to add the exponent values ​​to generate an exponent sum value.

[0060] At S884, the MAC unit determines a maximum exponent value. In at least some embodiments, the MAC unit causes the comparator to determine a maximum exponent value among a plurality of exponent sum values ​​generated from a plurality of MAC units of the systolic array. In at least some embodiments, the MAC unit causes the comparator to determine a maximum exponent value among a plurality of exponent values ​​including an exponent value of a third data width floating-point value resulting from a previous iteration of the calculation.

[0061] At S888, the MAC unit subtracts the exponent sum value from the maximum exponent value. In at least some embodiments, the MAC unit causes a subtractor to subtract the exponent sum value from the maximum exponent value to generate a difference value. In at least some embodiments, the MAC unit causes a subtractor in each multiplier group to subtract an exponent value from the maximum exponent value to generate a difference value.

[0062] At least some embodiments are described with reference to flow charts and block diagrams whose blocks represent (1) stages of a process where an operation is performed or (2) sections of a controller responsible for performing an operation. In at least some embodiments, certain stages and sections are implemented by dedicated circuitry, programmable circuitry provided with computer readable instructions stored on a computer readable medium, and / or a processor provided with computer readable instructions stored on a computer readable medium. In at least some embodiments, the dedicated circuitry includes digital and / or analog hardware circuitry, including integrated circuits (ICs) and / or discrete circuits. In at least some embodiments, the programmable circuitry includes reconfigurable hardware circuitry, e.g., field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), etc., including logical AND, OR, XOR, NAND, NOR, and other logic operations, flip-flops, registers, memory elements, etc.

[0063] In at least some embodiments, a computer readable storage medium includes a tangible device capable of maintaining and storing instructions for use by an instruction execution device. In some embodiments, a computer readable storage medium includes, but is not limited to, for example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer readable storage media includes the following: portable computer diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), static random access memories (SRAMs), portable compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or raised structures in grooves with instructions recorded thereon, and any suitable combination of the foregoing. Computer-readable storage medium, as used herein, should not be construed as a transitory signal per se, such as electric waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted through wires.

[0064] In at least some embodiments, the computer readable program instructions described herein can be downloaded from a computer readable storage medium to the respective computing / processing device or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. In at least some embodiments, the network includes copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. In at least some embodiments, a network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium in the respective computing / processing device.

[0065] In at least some embodiments, the computer readable program instructions that perform the operations described above are either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk, C++, etc., and traditional procedural programming languages ​​such as the "C" programming language or similar programming languages. In at least some embodiments, the computer readable program instructions execute entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In at least some embodiments, in the latter scenario, the remote computer is connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or the connection is made to an external computer (e.g., through the Internet using an Internet Service Provider). In at least some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), executes computer readable program instructions by utilizing state information of the computer readable program instructions to individualize the electronic circuitry to perform aspects of the invention.

[0066] Although the embodiments of the present invention have been described, the scope of any claimed subject matter is not limited to the above-described embodiments. Those skilled in the art will understand that various modifications and improvements to the above-described embodiments are possible. Those skilled in the art will also understand that the scope of the claims will include embodiments to which such modifications or improvements have been made.

[0067] The operations, steps, phases, and stages of each process performed by the apparatus, system, program, and method shown in the claims, embodiments, or figures can be performed in any order, unless the order is indicated by "prior to" or "before," etc., and unless output from a previous process is used in a later process. Even if a process flow is described in the claims, embodiments, or figures using phrases such as "first" or "next," such description does not necessarily mean that the process must be performed in the order described.

[0068] In at least some embodiments, the multi-precision multiply-accumulate operation is performed by a multiply-accumulate (MAC) unit configured to operate in an integer mode to perform calculations on first data width integer values ​​to generate third data width integer values, and in a floating-point mode to perform calculations on second data width floating-point values ​​to generate third data width floating-point values, where the second data width is twice the first data width and the third data width is greater than the second data width.

[0069] The foregoing outlines features of some embodiments so that those skilled in the art may better understand the aspects of the present disclosure. Those skilled in the art should appreciate that the present disclosure can be readily used as a basis for designing or modifying other processes and structures for carrying out the same purposes and / or achieving the same advantages as the embodiments introduced herein. Those skilled in the art should also recognize that such equivalent constructions do not depart from the spirit and scope of the present disclosure, and that various changes, substitutions, and alterations herein are possible without departing from the spirit and scope of the present disclosure.

Claims

1. a multiply-accumulate (MAC) unit configured to operate in an integer mode to perform calculations on first data width integer values ​​to generate third data width integer values ​​and configured to operate in a floating point mode to perform calculations on second data width floating point values ​​to generate third data width floating point values, wherein a second data width of the second data width floating point values ​​is twice a first data width of the first data width integer values ​​and a third data width of the third data width integer values ​​and the third data width floating point values ​​is greater than the second data width, the MAC unit comprising an extractor, a first multiplier, a second multiplier, an exponent adder, a comparator, a subtractor, and a shifter; the extractor is configured to extract an exponent value and a mantissa value from each of a plurality of input values ​​to the MAC unit when the input values ​​are a plurality of second data width floating point values, and to pass at least two first data width integer values ​​among the plurality of input values ​​when the input values ​​are a plurality of first data width integer values; the MAC unit is configured in the integer mode to perform a calculation including multiplying two integer values ​​of the plurality of integer values ​​passed through the extractor using the first multiplier to generate an intermediate integer value and multiplying two other integer values ​​of the plurality of integer values ​​using the second multiplier to generate an intermediate integer value, each integer value of the plurality of integer values ​​having the first data width; The MAC unit, in the floating point mode, extracting, with the extractor, an exponent value and a mantissa value from each of a plurality of floating point values, each floating point value in the plurality of floating point values ​​having the second data width. multiplying, using the first multiplier, a mantissa value extracted from each of two floating-point values ​​among the plurality of floating-point values ​​using the extractor to generate a first mantissa product value, and refraining from using the second multiplier; adding, with the exponent adder, exponent values ​​extracted from each of the two floating point values ​​in the plurality of floating point values ​​to generate an exponent sum value; determining, using the comparator, a maximum exponent value among a plurality of exponent sum values ​​generated from the MAC unit; subtracting the exponent sum value from the maximum exponent value using the subtractor to generate a difference value; and shifting the first mantissa product value based on the difference value to generate an intermediate mantissa value using the shifter; The device is further configured to perform calculations including: an accumulator adder connected to two or more multipliers included in the MAC unit, the accumulator adder comprising: accumulating intermediate mantissa values ​​of the MAC unit to generate an accumulated mantissa value in the floating point mode; and accumulating intermediate integer values ​​of the MAC units to generate an accumulated integer value in the integer mode; and a normalization accumulating adder connected to the accumulating adder, the normalization accumulating adder comprising: normalizing the accumulated mantissa value with the maximum exponent value to generate a third data width floating-point value in the floating-point mode; and generating a third data width integer value without normalization in said integer mode; 4. The method of claim 3, Equipped with a number of integer values ​​among the plurality of first data width integer values ​​is twice the number of floating point values ​​among the plurality of second data width floating point values, and the MAC unit is configured to generate, per unit time, in the integer mode, a number of intermediate integer values ​​that is twice the number of intermediate mantissa values ​​in the floating point mode. Integrated circuits.

2. further comprising a memory in communication with the MAC unit; The computing in the integer mode further includes reading the input values ​​comprising the first data width integer values ​​from the memory; 2. The integrated circuit of claim 1, wherein the computing in the floating-point mode further comprises reading the input values ​​comprised of the second data width floating-point values ​​from the memory.

3. 2. The integrated circuit of claim 1, further comprising: a controller in communication with the MAC unit, the controller configured to operate in the integer mode to perform neural network inference on first data width integer values ​​and configured to operate in the floating point mode to perform neural network inference on second data width floating point values.

4. 3. The integrated circuit of claim 2, further comprising an integer activation pipeline in communication with the memory, the integer activation pipeline configured to activate the third data width integer value to generate an activated first data width integer value.

5. 3. The integrated circuit of claim 2, further comprising a floating-point activation pipeline in communication with the memory, the floating-point activation pipeline configured to activate the third data width floating-point value to generate an activated second data width floating-point value.

6. A systolic array comprising: a plurality of instances of the MAC unit, where each instance of the MAC unit is further configured to perform, in the floating-point mode, determining, using the comparator, the maximum exponent value among the plurality of exponent sum values ​​generated from the plurality of instances of the MAC unit of the systolic array; and a plurality of accumulating adders, each accumulating adder being connected to two or more of any combination of a multiplier and a preceding accumulating adder included within said plurality of instances of a MAC unit; and a systolic array having: The plurality of accumulating adders include accumulating the intermediate mantissa values ​​of the multiple instances of the MAC unit to generate the accumulated mantissa value in the floating point mode; and accumulating the intermediate integer values ​​of the multiple instances of the MAC unit to generate the accumulated integer value in the integer mode. are collectively configured to: the plurality of accumulating adders includes the normalizing accumulating adder; 2. The integrated circuit of claim 1, wherein in the floating-point mode, an exponent value of the third data width floating-point value is calculated based on the maximum exponent value, a number of instances of the MAC unit in the multiple instances of the MAC unit, and a number of the first multipliers in each instance of the MAC unit.

7. the MAC unit includes a plurality of instances of the pair of the first multiplier and the second multiplier, each first multiplier in the plurality of instances being grouped with an instance of the exponent adder, an instance of the comparator, an instance of the subtractor, and an instance of the shifter; The integrated circuit of claim 6 , wherein the MAC unit is further configured to perform the calculation in the floating point mode for each first multiplier.

8. A systolic array comprising: a plurality of instances of the MAC unit, where each instance of the MAC unit is further configured to perform, in the floating-point mode, determining, using the comparator, the maximum exponent value among the plurality of exponent sum values ​​generated from the plurality of instances of the MAC unit of the systolic array; and a plurality of accumulating adders, each accumulating adder being connected to two or more of any combination of a multiplier and a preceding accumulating adder included within said plurality of instances of a MAC unit; and a systolic array having: The plurality of accumulating adders include accumulating the intermediate mantissa values ​​of the multiple instances of the MAC unit to generate the accumulated mantissa value in the floating point mode; and accumulating the intermediate integer values ​​of the multiple instances of the MAC unit to generate the accumulated integer value in the integer mode. are collectively configured to: The integrated circuit of claim 1 , wherein the plurality of accumulating adders includes the normalizing accumulating adder.

9. a multiply-accumulate (MAC) unit configured to operate in an integer mode to perform calculations on first data width integer values ​​to generate third data width integer values ​​and configured to operate in a floating point mode to perform calculations on second data width floating point values ​​to generate third data width floating point values, wherein a second data width of the second data width floating point values ​​is twice a first data width of the first data width integer values ​​and a third data width of the third data width integer values ​​and the third data width floating point values ​​is greater than the second data width, and the MAC unit is configured to: an extractor configured to extract an exponent value and a mantissa value from each of a plurality of input values ​​to the MAC unit when the plurality of input values ​​are a plurality of second data width floating point values, the extractor being further configured to pass at least two first data width integer values ​​among the plurality of input values ​​when the plurality of input values ​​are a plurality of first data width integer values. a first multiplier connected to the extractor, the first multiplier configured to multiply a mantissa value extracted from each of two floating-point values ​​among a plurality of floating-point values ​​using the extractor to generate a first mantissa product value, the first multiplier further configured to multiply two of the at least two first data width integer values ​​among the plurality of first data width integer values ​​passed through the extractor to generate a first integer product value; an exponent adder coupled to the extractor, the exponent adder configured to add the extracted exponent values ​​from each of the two floating-point values ​​in the plurality of floating-point values ​​to generate an exponent sum value; a second multiplier configured to multiply other two first data width integer values ​​among the plurality of input values ​​to generate a second integer product value as an intermediate integer value when the plurality of input values ​​are a plurality of first data width integer values; and the MAC unit refrains from using the second multiplier when the plurality of input values ​​are a plurality of second data width floating point values. a comparator coupled to the exponent adder, the comparator configured to determine a maximum exponent value among a plurality of exponent sum values ​​generated from the MAC unit; a subtractor coupled to the exponent adder and the comparator, the subtractor configured to subtract the exponent sum value from the maximum exponent value to generate a difference value; and a shifter coupled to the subtractor and the first multiplier, the shifter configured to shift the first mantissa product value based on the difference value to generate an intermediate mantissa value, the shifter further configured to pass the first integer product value as an intermediate integer value. having an accumulator adder configured to accumulate intermediate mantissa values ​​of the MAC units to generate accumulated mantissa values, the accumulator adder further configured to accumulate intermediate integer values ​​of the MAC units to generate accumulated integer values; and a normalizing accumulate adder coupled to the accumulating adder, the normalizing accumulate adder configured to normalize the accumulated mantissa value with the maximum exponent value to generate a third data width floating-point value, the normalizing accumulate adder further configured to generate a third data width integer value without normalization; Equipped with a number of integer values ​​among the plurality of first data width integer values ​​is twice the number of floating point values ​​among the plurality of second data width floating point values, and the MAC unit is configured to generate, per unit time, in the integer mode, a number of intermediate integer values ​​that is twice the number of intermediate mantissa values ​​in the floating point mode. Integrated circuits.

10. further comprising a memory in communication with the MAC unit; 10. The integrated circuit of claim 9, wherein the MAC unit is configured to read the plurality of input values ​​comprised of the plurality of second data width floating point values ​​from the memory, and the MAC unit is further configured to read the plurality of input values ​​comprised of the plurality of first data width integer values ​​from the memory.

11. 10. The integrated circuit of claim 9, further comprising a controller in communication with the MAC unit, the controller configured to operate in the integer mode to perform neural network inference on first data width integer values ​​and in the floating point mode to perform neural network inference on second data width floating point values.

12. 11. The integrated circuit of claim 10, further comprising an integer activation pipeline in communication with the memory, the integer activation pipeline configured to perform activation on a third data width integer value to generate a first data width integer value.

13. 11. The integrated circuit of claim 10, further comprising a floating-point activation pipeline in communication with the memory, the floating-point activation pipeline configured to perform activation on third data width floating-point values ​​to generate second data width floating-point values.

14. A systolic array comprising: a plurality of instances of the MAC unit, where the comparator of each instance of the MAC unit is configured to determine the maximum exponent value among the plurality of exponent sum values ​​generated from the plurality of instances of the MAC unit of the systolic array; and a plurality of accumulating adders, each accumulating adder connected to two or more of any combination of multipliers and preceding accumulating adders included within the plurality of instances of the MAC unit, the plurality of accumulating adders collectively configured to accumulate the intermediate mantissa values ​​of the plurality of instances of the MAC unit to generate the accumulated mantissa value, the plurality of accumulating adders further collectively configured to accumulate the intermediate integer values ​​of the plurality of instances of the MAC unit to generate the accumulated integer value; and a systolic array having: the plurality of accumulating adders includes the normalizing accumulating adder; 14. The integrated circuit of claim 9, wherein in the floating-point mode, an exponent value of the third data width floating-point value is calculated based on the maximum exponent value, a number of instances of the MAC unit in the multiple instances of the MAC unit, and a number of the first multipliers in each instance of the MAC unit.

15. A systolic array comprising: a plurality of instances of the MAC unit, where the comparator of each instance of the MAC unit is configured to determine the maximum exponent value among the plurality of exponent sum values ​​generated from the plurality of instances of the MAC unit of the systolic array; and a plurality of accumulating adders, each accumulating adder connected to two or more of any combination of multipliers and preceding accumulating adders included within the plurality of instances of the MAC unit, the plurality of accumulating adders collectively configured to accumulate the intermediate mantissa values ​​of the plurality of instances of the MAC unit to generate the accumulated mantissa value, the plurality of accumulating adders further collectively configured to accumulate the intermediate integer values ​​of the plurality of instances of the MAC unit to generate the accumulated integer value; and a systolic array having: The integrated circuit of claim 9 , wherein the plurality of accumulating adders includes the normalizing accumulating adder.

16. An integrated circuit comprising: a plurality of multiplier groups configured to operate in an integer mode to perform calculations on first data width integer values ​​to generate third data width integer values ​​and configured to operate in a floating point mode to perform calculations on second data width floating point values ​​to generate third data width floating point values, wherein a second data width of the second data width floating point values ​​is twice a first data width of the first data width integer values ​​and a third data width of the third data width integer values ​​and the third data width floating point values ​​is greater than the second data width, each multiplier group: an extractor configured to extract an exponent value and a mantissa value from each of a plurality of input values ​​to the plurality of multiplier groups when the plurality of input values ​​are a plurality of second data width floating-point values, the extractor being further configured to pass at least two first data width integer values ​​among the plurality of input values ​​when the plurality of input values ​​are a plurality of first data width integer values. a shared multiplier connected to the extractor, the shared multiplier configured to multiply a mantissa value extracted from each of two floating-point values ​​among a plurality of floating-point values ​​using the extractor to generate a first mantissa product value, the shared multiplier further configured to multiply two of the at least two first data width integer values ​​among the plurality of first data width integer values ​​passed through the extractor to generate a first integer product value. an exponent adder coupled to the extractor, the exponent adder configured to add the extracted exponent values ​​from each of the two floating-point values ​​in the plurality of floating-point values ​​to generate an exponent sum value; a comparator coupled to the exponent adder, the comparator configured to determine a maximum exponent value among a plurality of exponent summation values ​​generated from the plurality of multiplier groups; a subtractor coupled to the exponent adder and the comparator, the subtractor configured to subtract the exponent sum value from the maximum exponent value to generate a difference value; and a shifter coupled to the subtractor and the shared multiplier, the shifter configured to shift the first mantissa product value based on the difference value to generate an intermediate mantissa value, the shifter further configured to pass the first integer product value as an intermediate integer value. having a plurality of dedicated multipliers, each dedicated multiplier configured to multiply two other first data width integer values ​​among the plurality of first data width integer values ​​to generate a second integer product value as an intermediate integer value when the plurality of input values ​​are a plurality of first data width integer values, the integrated circuit refraining from using the plurality of dedicated multipliers when the plurality of input values ​​are a plurality of second data width floating point values; an accumulating adder coupled to a shared multiplier among the multiple multiplier groups and a dedicated multiplier among the multiple dedicated multipliers, the accumulating adder configured to accumulate the intermediate mantissa values ​​of the multiple multiplier groups to generate an accumulated mantissa value, the accumulating adder further configured to accumulate the intermediate integer values ​​of the multiple multiplier groups and the multiple dedicated multipliers to generate an accumulated integer value; and a normalizing accumulate adder configured to normalize the accumulated mantissa value by the maximum exponent value to generate a third data width floating-point value, the normalizing accumulate adder further configured to generate a third data width integer value without normalization. Equipped with a number of integer values ​​among the plurality of first data width integer values ​​is twice the number of floating-point values ​​among the plurality of second data width floating-point values, and the plurality of multiplier groups and the plurality of dedicated multipliers are configured to generate, per unit time, in the integer mode, a number of intermediate integer values ​​that is twice the number of intermediate mantissa values ​​in the floating-point mode. Integrated circuits.

17. a memory in communication with each multiplier group in the plurality of multiplier groups and each dedicated multiplier in the plurality of dedicated multipliers; 17. The integrated circuit of claim 16, wherein each multiplier group is configured to read the plurality of input values ​​comprised of the plurality of second data width floating point values ​​from the memory, and each multiplier group is further configured to read the plurality of input values ​​comprised of the plurality of first data width integer values ​​from the memory.

18. 17. The integrated circuit of claim 16, further comprising a controller in communication with the plurality of multiplier groups, the controller configured to operate in the integer mode to perform neural network inference on first data width integer values ​​and in the floating point mode to perform neural network inference on second data width floating point values.

19. 20. The integrated circuit of claim 17, further comprising a floating-point activation pipeline in communication with the memory, the floating-point activation pipeline configured to perform activation on third data width floating-point values ​​to generate second data width floating-point values.

20. A systolic array comprising: the plurality of multiplier groups; the plurality of dedicated multipliers; and a plurality of accumulating adders, each accumulating adder connected to two or more of any combination of a shared multiplier in the plurality of multiplier groups, a dedicated multiplier in the plurality of dedicated multipliers, and a preceding accumulating adder, the plurality of accumulating adders collectively configured to accumulate the intermediate mantissa values ​​of the plurality of multiplier groups to generate the accumulated mantissa value, the plurality of accumulating adders further collectively configured to accumulate the intermediate integer values ​​of the plurality of multiplier groups and the plurality of dedicated multipliers to generate the accumulated integer value; and a systolic array having:

20. The integrated circuit of claim 16, wherein the plurality of accumulating adders includes the normalizing accumulating adder.

Images