A reconfigurable floating-point multiply-accumulate device and processor

Abstract

This application relates to the field of integrated circuit technology, providing a reconfigurable floating-point multiply-accumulate device and processor. The multiply-accumulate device includes a control unit and multiple processing units. The control unit generates a mode selection signal based on the type identifier of the weighted data. The multiple processing units reconstruct the data path based on the mode selection signal. For a first floating-point format, they reuse the multiplexed multiplier array corresponding to a second floating-point format to perform operations. For the second floating-point format, they call the multiplexed multiplier array to perform operations, bypassing the group scaling factor multiplication path. For a third floating-point format, they split the mantissa of the weighted data into high-order and low-order segments, inputting them respectively into the corresponding sub-multipliers of the multiplexed multiplier array to perform parallel multiplication operations, bypassing the group scaling factor multiplication path. This application achieves compatibility with three floating-point formats by using a multiplexed multiplier array in conjunction with the mode selection signal to reconstruct the path, effectively reducing multiplier area and power consumption, and avoiding redundant design of independent paths.

Description

Technical Field

[0001] This application relates to the field of integrated circuit technology, specifically to a reconfigurable floating-point multiply-accumulate device and processor. Background Technology

[0002] Existing neural network processors typically support only a single or a few fixed-precision formats for their multiply-accumulate mechanisms. With the rise of low-bit floating-point formats, especially the introduction of new number systems such as NVFP4 (NVIDIA 4-bit Floating Point) and MXFP4 (Microscaling 4-bit Floating Point), existing devices face significant challenges.

[0003] For example, in existing technologies, to support the 16-input channel group quantization feature of NVFP4 weights and the 32-input channel group quantization feature of MXFP4, numerical conversion and additional scaling factor multiplication operations need to be performed outside the multiply-accumulate unit. This not only increases the bandwidth pressure of data transfer but also leads to an increase in chip area and power consumption. Summary of the Invention

[0004] This application provides a reconfigurable floating-point multiply-accumulate device and processor to solve the problems of existing multi-precision multiply-accumulate devices requiring independent computing paths and having high area and power consumption.

[0005] The first aspect of this application provides a reconfigurable floating-point multiply-accumulate device, comprising: The control unit is used to generate a mode selection signal based on the type identifier of the received weight data; Multiple processing units, coupled to the control unit, are configured as follows: The data path is reconstructed based on the mode selection signal to support the execution of floating-point multiplication and accumulation operations with weighted data in the first, second, or third floating-point formats as the first operand, and to generate the operation result. When the mode selection signal represents the weight data in the first floating-point format, the multiplexed multiplier array corresponding to the second floating-point format is used to perform multiply-accumulate operations and multiply the group scaling factor. When the mode selection signal represents the weight data in the second floating-point format, the multiplexed multiplier array is called to perform multiply-accumulate operations, and the multiplication operation path of the group scaling factor is bypassed. When the mode selection signal represents the weight data in the third floating-point format, the control multiplexer array splits the weight mantissa of the weight data into a high-order segment and a low-order segment. The high-order segment and the low-order segment are respectively input into the corresponding sub-multipliers of the multiplexer array to perform parallel multiplication operations, and the multiplication operation path of the group scaling factor is bypassed.

[0006] In this way, by reusing the multiplier array and reconstructing the data path with the mode selection signal, compatibility of the first floating-point format, the second floating-point format, and the third floating-point format can be achieved, reducing the area of ​​the multiplier and the power consumption of the multiplier, and avoiding redundant design of independent paths.

[0007] In one implementation, the first floating-point format is a 4-bit floating-point format, the second floating-point format is an 8-bit floating-point format, and the third floating-point format is a 16-bit floating-point format.

[0008] In this way, by clearly defining the specific types and quantization characteristics of the three floating-point formats, the encoding rules and group quantization requirements of different formats can be accurately adapted. For example, for NVFP4's 16IC group and MXFP4's 32IC group, the group processing logic can be flexibly switched through the control unit, avoiding the external data conversion overhead caused by format incompatibility in existing technologies and improving the versatility of multi-format support.

[0009] In one implementation, the first floating-point format includes NVFP4 and MXFP4 formats; Multiple processing units are also configured as follows: When the weight data representing the mode selection signal is in NVFP4 format, the calculation result is multiplied by a scaling factor based on E4M3 format to generate a multiplication result. When the mode selection signal represents the weight data in MXFP4 format, the scaling factor in E4M3 format is passed to the subsequent accumulation module. In the floating-point multiplication unit of the subsequent accumulation module, the scaling factor is applied to the weight data by multiplying the mantissa and adding the exponents, and then the multiply-accumulate operation is performed.

[0010] In this way, considering the scaling factor characteristics of MXFP4, the scaling factor processing is delayed to the subsequent accumulation module, and exponential addition is used to replace multiplication, thereby reducing the amount of computation inside the multiply-accumulate device.

[0011] In one implementation, the multiple processing units include a pixel-level processing array, and the pixel-level processing array includes multiple pixel processing circuits. Each pixel processing circuit includes multiple output channel processing sub-circuits, and each output channel processing sub-circuit includes multiple basic computing units; the basic computing units are configured to receive input feature data as second operands and decode the second operands to output mantissa product and exponent sum.

[0012] In this way, the parallel architecture of the pixel-level processing array can handle the computational tasks of multiple output channels simultaneously, thereby improving data parallelism.

[0013] In one implementation, the basic computing unit includes a multiplexed multiplier array, a decoding module, and an exponential adder; The decoding module is used to decode the weight data to obtain the weight mantissa and weight exponent; The multiplexer array is used to multiply the mantissa of the weight by the mantissa of the second operand to obtain the mantissa product; The exponential adder is used to add the weighted exponent to the exponent of the second operand to obtain the exponential sum.

[0014] In this way, the multiplexer array can share hardware resources by adapting to the mantissa bit width of different formats, avoiding the need to design a separate multiplier for each format. The integration of the decoding module and the exponential adder enables the basic computing unit to independently complete the multiplication and addition operations of a single set of weights and features, reducing the transmission latency of data between different modules.

[0015] In one implementation, the output channel processing sub-circuit further includes: an alignment and addition tree module and a scaling and merging module; The alignment and addition tree module is coupled with the basic computing unit to perform exponential alignment and accumulation of the mantissa product and exponential sum output by multiple basic computing units, and output the first intermediate result and the second intermediate result. The scaling and merging module is coupled to the alignment and addition tree module, which is used to perform scaling factor multiplication and group merging on the first intermediate result and the second intermediate result respectively when the mode selection signal characterization weight data is in NVFP4 format, and output a 37-bit partial sum. The 37-bit partial sum includes a 9-bit exponent field and a 28-bit signed fixed-point mantissa field.

[0016] In this way, the alignment and addition tree module achieves multiplication and accumulation operations on multiple input channels through exponential alignment and addition tree accumulation, reducing the storage overhead of intermediate results. The integrated processing of the scaling and merging module under the NVFP4 format means that group scaling correction does not rely on external circuitry, avoiding bandwidth pressure during data transfer.

[0017] In one implementation, the scaling and merging module is also configured as follows: When the mode selection signal characterization weight data is in MXFP4 format or second floating-point format, bypass scaling factor multiplication and grouping and merging operations are performed, and the first intermediate result and the second intermediate result are output as a 37-bit partial sum.

[0018] In this way, by bypassing unnecessary operations, the data path length is shortened and the logic inversion rate is reduced in both MXFP4 and FP8 formats. Furthermore, redundant multiplier calls are avoided in the MXFP4 format, allowing for dynamic optimization of the data path based on format characteristics and improving operational efficiency across different data formats.

[0019] In one implementation, the control unit includes a rearrangement module, which is configured to: In response to the mode selection signal representing the weight data as a first floating-point format, the weight data is transposed and rearranged according to the first dimension order; In response to the mode selection signal representing the weight data as a second floating-point format, the weight data is transposed and rearranged according to the second dimension order, wherein the data interleaving granularity corresponding to the first dimension order is greater than the data interleaving granularity corresponding to the second dimension order. In response to the mode selection signal representing the weight data as a third floating-point format, the operation of bypass transpose rearrangement is performed.

[0020] In this way, by using differentiated rearrangement strategies, we can match the parallel access requirements of different formats and improve data reuse efficiency.

[0021] In one implementation, the control unit further includes a scaling factor cache module, which includes multiple first-in-first-out queues; The scaling factor caching module is coupled to the scaling and merging module and is configured to cache and distribute scaling factors in E4M3 format to multiple processing units when the mode selection signal characterization weight data is in NVFP4 format, in order to match the pipeline timing of multiple processing units.

[0022] In this way, the scaling factor cache module solves the problem of matching the scaling factor reading timing with the processing unit pipeline under the NVFP4 format by using multiple first-in-first-out queues.

[0023] In one implementation, the basic computing unit is further configured to: when the mode selection signal represents the weight data in a third floating-point format, control the multiplexer array to split the weight mantissa into a high-order segment and a low-order segment for multiplication operations respectively. The output channel processing sub-circuit is also configured to output valid data for the lower 8 channels and set the data for the higher 8 channels to zero when the mode selection signal represents the weight data in the third floating-point format.

[0024] In this way, by splitting the mantissa and multiplexing the multiplier array, no additional multiplier hardware is needed in the third floating-point format. Multiplication can be achieved through logic control alone, saving multiplier area. The design of setting the high 8 channels to zero simplifies the output control logic and avoids the overhead of subsequent processing of invalid data.

[0025] A second aspect of this application provides a processor including the above-described reconfigurable floating-point multiply-accumulate device, and a subsequent accumulation module; The subsequent accumulation module is configured to receive the partial sum output by the reconfigurable floating-point multiply-accumulate device and perform offset compensation on the exponent field of the partial sum according to the weight data type identifier. The subsequent accumulation module is also configured to add the scaling factor corresponding to the first floating-point format to the exponential field when the weight data type identifier represents the weight data as the first floating-point format and the scaling factor is in pure exponential format.

[0026] This moves the exponential offset compensation and scaling factor processing to the subsequent accumulator module, avoiding the repeated execution of compensation and addition operations for each input channel batch within the multiply-accumulator. It also reduces the number of exponential adders, especially in the MXFP4 format, where the subsequent accumulator module only needs to perform exponential addition once, whereas doing so within the multiply-accumulator would require one operation for each input channel, thus reducing overall power consumption.

[0027] This application provides a reconfigurable floating-point multiply-accumulate device and processor. The multiply-accumulate device includes a control unit and multiple processing units. The control unit generates a mode selection signal based on the type identifier of the received weight data. The multiple processing units are configured to: reconstruct the data path based on the mode selection signal to support floating-point multiply-accumulate operations using weight data in a first floating-point format, a second floating-point format, or a third floating-point format as the first operand, and generate an operation result; when the mode selection signal indicates that the weight data is in the first floating-point format, reuse the corresponding second floating-point format. The multiplexed multiplier array performs multiply-accumulate operations and multiplies the group scaling factor. When the mode selection signal represents the weight data in the second floating-point format, the multiplexed multiplier array is invoked to perform multiply-accumulate operations, bypassing the multiplication path of the group scaling factor. When the mode selection signal represents the weight data in the third floating-point format, the multiplexed multiplier array is controlled to split the weight mantissa of the weight data into high-order and low-order segments. The high-order and low-order segments are respectively input into the corresponding sub-multipliers of the multiplexed multiplier array to perform parallel multiplication operations, bypassing the multiplication path of the group scaling factor. The technical solution provided in this application, by using a multiplexed multiplier array in conjunction with the mode selection signal to reconstruct the data path, achieves compatibility between the first, second, and third floating-point formats, reduces the area of ​​the multiplier, lowers the power consumption of the multiplier, and avoids redundant design of independent paths. Attached Figure Description

[0028] To more clearly illustrate the implementation methods in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0029] Figure 1 Schematic diagram of the reconfigurable floating-point multiply-accumulate device provided in the embodiments of this application Figure 1 .

[0030] Figure 2This is a schematic diagram of the pixel processing circuit provided in an embodiment of this application.

[0031] Figure 3 Schematic diagram of the reconfigurable floating-point multiply-accumulate device provided in the embodiments of this application Figure 2 .

[0032] Figure 4 Schematic diagram of the reconfigurable floating-point multiply-accumulate device provided in the embodiments of this application Figure 3 .

[0033] Figure 5 Schematic diagram of the reconfigurable floating-point multiply-accumulate device provided in the embodiments of this application Figure 4 . Detailed Implementation

[0034] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of this application.

[0035] It should be noted that many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Therefore, the scope of protection of this application is not limited to the specific embodiments disclosed below.

[0036] In this application, the use of terms such as "first," "second," etc., is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features.

[0037] The various embodiments of this application will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0038] This application provides a reconfigurable floating-point multiply-accumulate device and processor to solve the problems of existing multi-precision multiply-accumulate devices requiring independent computing paths and having high area and power consumption.

[0039] The first aspect of this application provides a reconfigurable floating-point multiply-accumulate device, which is suitable for the convolution calculation path of a neural network processor, such as... Figure 1As shown, the multiply-accumulate device includes a control unit and multiple processing units (Pixel0 to Pixel7).

[0040] The control unit generates a mode selection signal based on the type identifier of the received weight data. For example, the control unit is equipped with a 3-bit configuration register, which stores the type identifier of the weight data. The control unit identifies the type identifier and generates the mode selection signal based on the type identifier. For example, the weight data formats include five types: NVFP4, MXFP4, FP8E4M3, FP8E5M2, and BF16.

[0041] The mode selection signal is a 2-bit signal, corresponding to the first, second, and third floating-point formats, respectively. These three formats are distinct floating-point format categories. For example, the first floating-point format is a format with a bit width less than 8 bits and supports block quantization, where every N input channels share a scaling factor, and N is an integer greater than 1. The second floating-point format is an 8-bit format without block quantization, where each input channel corresponds to an independent scaling factor of 1. The third floating-point format is a format with a bit width greater than 8 bits and a mantissa bit width greater than the native bit width of the multiplexer array.

[0042] Multiple processing units are coupled to the control unit; for example, each processing unit corresponds to one pixel. These multiple processing units work in parallel, for example... Figure 1 As shown, it includes 8 processing units operating in parallel. When the mode selection signal represents the weight data in the first floating-point format, the multiplexed multiplier array corresponding to the second floating-point format is used to perform multiply-accumulate operations, while simultaneously performing multiplication operations on the group scaling factor.

[0043] Taking the first floating-point format as NVFP4 as an example, the weight data in NVFP4 format is 4 bits. The multiplier array designed using the 8-bit floating-point format is reused. The 4-bit weight mantissa is extended and then input into the multiplier array. At the same time, the scaling and merging module is called to perform the scaling factor multiplication operation of E4M3 format.

[0044] When the mode selection signal represents the weight data in the second floating-point format, the multiplexed multiplier array is directly invoked to perform multiply-accumulate operations. In this case, the multiplication path of the group scaling factor is bypassed. Taking the second floating-point format as FP8 as an example, the weight data does not require a scaling factor, and the scaling factor multiplication path is bypassed.

[0045] When the mode selection signal represents the weight data in third floating-point format, the multiplexed multiplier array splits the weight mantissa into high-order and low-order segments for multiplication. For example, in BF16 format, the weight mantissa is 7 bits, split into high-order 2 bits and low-order 5 bits. The high-order and low-order segments are then input into the corresponding sub-multipliers of the multiplexed multiplier array for parallel multiplication, i.e., input into two 4-bit sub-multipliers of the multiplexed multiplier array respectively, bypassing the multiplication path of the group scaling factor.

[0046] In this way, by reusing the multiplier array and reconstructing the data path with the mode selection signal, compatibility of the first floating-point format, the second floating-point format, and the third floating-point format can be achieved, reducing the area of ​​the multiplier and the power consumption of the multiplier, and avoiding redundant design of independent paths.

[0047] In some embodiments, the first floating-point format is a 4-bit floating-point format. For example, the first floating-point format is NVFP4 or MXFP4.

[0048] In the 4-bit floating-point format, the NVFP4 format uses 16-input channel (IC) block quantization, with each block corresponding to an E4M3 scaling factor. The MXFP4 format uses 32-input channel block quantization, with each block corresponding to an E8M0 scaling factor.

[0049] The second floating-point format is an 8-bit floating-point format, for example, the second floating-point format is FP8E4M3 format (8-bit floating-point E4M3 format) and FP8E5M2 format (8-bit floating-point E5M2 format).

[0050] In the 8-bit floating-point format, the FP8E4M3 format has a 4-bit exponent width, a 3-bit mantissa width, and an offset of 7. The FP8E5M2 format has a 5-bit exponent width, a 2-bit mantissa width, and an offset of 15.

[0051] The third floating-point format is a 16-bit floating-point format, for example, the third floating-point format is BF16 format (BrainFloating-Point 16).

[0052] The 16-bit floating-point BF16 format has an 8-bit exponent width, a 7-bit mantissa width, and an offset value of 127.

[0053] In this way, by clearly defining the specific types and quantization characteristics of the three floating-point formats, the encoding rules and group quantization requirements of different formats can be accurately adapted. For example, for 16IC groups in NVFP4 format and 32IC groups in MXFP4 format, the group processing logic can be flexibly switched through the control unit, avoiding the external data conversion overhead caused by format incompatibility in existing technologies and improving the versatility of multi-format support.

[0054] In some embodiments, the first floating-point format includes NVFP4 format and MXFP4 format, and multiple processing units employ differentiated scaling factor processing strategies for NVFP4 format and MXFP4 format.

[0055] When the mode selection signal characterization weight data is in NVFP4 format, the scaling and merging modules in multiple processing units multiply the result of the multiplication and accumulation operation with the scaling factor based on E4M3 format to generate the multiplication result.

[0056] The specific process is as follows: After decoding the weight data in NVFP4 format, the weight mantissa and weight exponent are obtained. The mantissa and exponent of the input feature data are multiplied and added in the processing unit to obtain the intermediate result. The intermediate result is then multiplied with the scaling factor based on E4M3 format to complete the group quantization correction.

[0057] When the mode selection signal represents the weight data in MXFP4 format, multiple processing units pass the scaling factor in E4M3 format to the subsequent accumulator (ACC). In the floating-point multiplication unit of the ACC, the scaling factor is applied by multiplying the mantissa and adding the exponents. Since the scaling factor in MXFP4 format is in E8M0 format (containing only the exponent field and no mantissa field), there is no need to perform multiplication operations within the multiply-accumulate device. The scaling factor information only needs to be passed to the subsequent accumulator, which then compensates for the exponent field of the accumulation result by adding it in the exponent path.

[0058] In this way, considering the scaling factor characteristics of the MXFP4 format, the scaling factor processing is delayed to the subsequent accumulation module, and exponential addition is used to replace multiplication, thereby reducing the amount of computation inside the multiply-accumulate device.

[0059] In some embodiments, the plurality of processing units include a pixel-level processing array. For example, such as Figure 2 As shown, the pixel-level processing array contains eight pixel processing circuits (Pixel0 to Pixel7). Each pixel processing circuit includes multiple output channel processing sub-circuits (OC0 to OC7), for a total of eight output channel processing sub-circuits. Each output channel processing sub-circuit corresponds to one output channel (OC). Each output channel processing sub-circuit includes 16 basic computing units (unit0 to unit15), for a total of 128 basic computing units per pixel processing circuit.

[0060] The basic computing unit is configured to receive input feature data as a second operand, wherein the input feature data is in a third floating-point format (BF16 format). The basic computing unit decodes the second operand.

[0061] For example, a 16-bit input feature data in BF16 format is split into a sign bit (1 bit), an exponent field (8 bits), and a mantissa field (7 bits). If the exponent field is non-zero, a 1 is padded to the highest bit of the mantissa field (hidden bit). If the exponent field is zero, it is treated as zero or a subnormal number, and the final output is the mantissa product and the exponent sum. For example, if the input feature data is 0x3F80 in BF16 format (corresponding to decimal 1.0), the decoded mantissa is 9'b1_0000000 and the exponent is 8'd127.

[0062] In this way, the parallel architecture of the pixel-level processing array can handle the computational tasks of multiple output channels simultaneously, thereby improving data parallelism.

[0063] In some embodiments, such as Figure 3 As shown, the basic computing unit includes a multiplexed multiplier array, a decoding module, and an exponential adder.

[0064] The decoding module is used to decode the weight data to obtain the weight mantissa and weight exponent. Taking NVFP4 format weight data as an example, the 4-bit weight data is split into a sign bit (1 bit), an exponent field (2 bits), and a mantissa field (1 bit). The decoding module converts the exponent field into an internal exponent value according to the bias rule of NVFP4 format (bias value is 1), thus obtaining the weight mantissa and weight exponent.

[0065] If the first floating-point format is Microscaling 4-bit Floating Point (MXFP4) format, the 4-bit weight data is divided into 1 bit sign bit, 2 bits exponent field, and 1 bit mantissa field, with no extra reserved bits. The bit order is as follows: the highest bit is the sign bit, the second highest bit to the second lowest bit is the exponent field, and the lowest bit is the mantissa field.

[0066] When the 2-bit exponent field is all 1s (i.e., exp = 2'b11), regardless of the value of the mantissa field, it is considered an outlier. After decoding, the internal special exponent 9'd455 is output, corresponding to infinity (Inf) or not-a-Number (NaN). In the RTL (Register Transfer Level) implementation, infinity and not-a-Number outliers of various floating-point formats are uniformly mapped to the 9-bit internal special exponent 9'd455 and passed to the subsequent module. The subsequent module does not need to make differentiated judgments based on the weight format, avoiding differences in outlier handling branches for different formats.

[0067] When the 2-bit exponent field is all 0 (i.e., exp=2'b00) and the mantissa field is 0 (frac=1'b0), the decoding result is the value 0, and the output exponent and mantissa are 0.

[0068] When the 2-bit exponent field is all 0 (i.e., exp=2'b00) and the mantissa field is 1 (frac=1'b1), it is determined to be a subnormal number. The mantissa is not padded with a hidden bit, and the internal exponent bias value is 0. That is, the internal exponent of the decoded output is 0, and no bias correction is required.

[0069] When the 2-bit exponent field is not all zeros (i.e., exp ≠ 2'b00), it is determined to be a normal number, and the highest bit of the mantissa is padded with a hidden bit of 1 to form the 2-bit valid mantissa of 2'b1_frac. The internal exponent bias value is 1, that is, the internal exponent of the decoded output is the exponent field value plus 1, without the need to subtract the format preset bias, and can be directly added to the exponent field of the input feature data.

[0070] For weighted data in NVFP4 and MXFP4 formats, the decoding differences lie in the differences in group quantization granularity and scaling factor binding logic.

[0071] The differences in quantization granularity are as follows: NVFP4 format weight data uses 16 input channels as one quantization group, with each group corresponding to one E4M3 format scaling factor. MXFP4 format weight data uses 32 input channels as one quantization group, with each group corresponding to one E8M0 format scaling factor.

[0072] The difference in scaling factor binding logic is as follows: After the weight data in NVFP4 format is decoded, the weight data is directly bound to the scaling factor in E4M3 format for the corresponding group, and they both enter the scaling and merging module to perform scaling factor multiplication. After the weight data in MXFP4 format is decoded, the weight data only carries the index identifier of its group. The scaling factor in E8M0 format does not enter the data path inside the multiply-accumulate device; it only passes the group index to the subsequent accumulation module. After the subsequent accumulation module completes the exponent field compensation, it retrieves the corresponding E8M0 scaling factor according to the group index and completes the scaling process by adding the exponents. In this way, by utilizing the characteristic that the scaling factor in E8M0 format has no mantissa field, redundant multiplier calls inside the multiply-accumulate device are avoided.

[0073] Multiplexed multiplier arrays are used to multiply the mantissa of the weights by the mantissa of the second operand, resulting in a mantissa product. For example, in 4-bit format, a multiplexed multiplier array reuses an 8-bit multiplier structure, extending the 4-bit weight mantissa to 8 bits before input. In 8-bit format, the 8-bit multiplier is used directly. In 16-bit format, it is split into two 4-bit sub-multipliers to process the high and low mantissa segments respectively. Exponential adders are used to add the weight exponent to the exponent of the second operand, resulting in an exponential sum. For example, if the weight exponent is 3'd2 and the input feature exponent is 8'd127, the exponential adder outputs an exponential sum of 9'd129.

[0074] In this way, the multiplexer array can share hardware resources by adapting to the mantissa bit width of different formats, avoiding the need to design a separate multiplier for each format. The integration of the decoding module and the exponential adder enables the basic computing unit to independently complete the multiplication and addition operations of a single set of weights and features, reducing the transmission latency of data between different modules.

[0075] In some embodiments, such as Figure 4 As shown, the output channel processing sub-circuit also includes an alignment and addition tree module and a scaling and merging module.

[0076] The alignment and addition tree module is coupled with the basic computing units to perform exponential alignment and accumulation of the mantissa products and exponential sums output by multiple basic computing units, outputting a first intermediate result and a second intermediate result. For example, the alignment and addition tree module receives the mantissa products and exponential sums output by 16 basic computing units. First, it compares the exponential sums of the 16 sets of data, finds the largest exponential sum as the alignment benchmark, and right-shifts the other mantissa products according to the difference in exponential sums. Then, it accumulates the 16 aligned mantissa products through the addition tree, outputting a first intermediate result (OC0 path) and a second intermediate result (OC1 path).

[0077] The scaling and merging module is coupled to the alignment and addition tree module. When the mode selection signal indicates that the mode is NVFP4 format, it performs scaling factor multiplication and group merging on the first intermediate result and the second intermediate result respectively, and outputs a 37-bit partial sum. The 37-bit partial sum includes a 9-bit exponent field and a 28-bit signed fixed-point mantissa field.

[0078] Specifically, the scaling factor in E4M3 format is read, and the first and second intermediate results are multiplied by the scaling factor respectively. Then, the grouped results from the 16 input channels are merged into a 37-bit partial sum. This 37-bit partial sum includes a 9-bit exponent field and a 28-bit signed fixed-point mantissa field. The 9-bit exponent field is the result of aligning and scaling the exponent of the adder tree output, and the 28-bit mantissa field is the signed two's complement fixed-point number after scaling multiplication.

[0079] In this way, the alignment and addition tree module achieves multiplication and accumulation operations on multiple input channels through exponential alignment and addition tree accumulation, reducing the storage overhead of intermediate results. The integrated processing of the scaling and merging module under the NVFP4 format means that group scaling correction does not rely on external circuitry, avoiding bandwidth pressure during data transfer.

[0080] In some embodiments, the scaling and merging module is further configured to bypass the scaling factor multiplication and grouping merging operations when the mode selection signal characterization weight data is in MXFP4 format or a second floating-point format, and output the first intermediate result and the second intermediate result as a 37-bit partial sum.

[0081] Specifically, when the weight data is in MXFP4 format, since the scaling factor is in E8M0 format, there is no need to perform multiplication inside the multiply-accumulate device. The scaling and merging module directly outputs the first and second intermediate results from the alignment and addition tree module as a 37-bit partial sum, retaining only the original values ​​of the exponent and mantissa fields. The scaling factor related information is transmitted to the subsequent accumulation module through an independent control signal.

[0082] When the weighted data is in FP8 format (including E4M3 and E5M2 formats), the weighted data itself does not require a scaling factor. The scaling and merging modules also bypass the scaling factor multiplication and grouping and merging operations, directly mapping the intermediate results to a 37-bit partial sum format. Among them, the 9-bit exponent field is the sum of the weight exponent and the feature exponent, and the 28-bit mantissa field is the mantissa multiplication and accumulation result.

[0083] In this way, by bypassing unnecessary operations, the data path length is shortened and the logic inversion rate is reduced in both MXFP4 and FP8 formats. Furthermore, redundant multiplier calls are avoided in the MXFP4 format, allowing for dynamic optimization of the data path based on format characteristics and improving operational efficiency across different data formats.

[0084] In some embodiments, such as Figure 5 As shown, the control unit includes a rearrangement module configured to transpose and rearrange the weight data according to a mode selection signal. Specifically, in response to the mode selection signal indicating that the weight data is in a first floating-point format, the weight data is transposed and rearranged according to the first dimension order. For example, the data interleaving granularity corresponding to the first dimension order is 32 input channels, and the specific rearrangement logic is an 8, 2, 2, 16-dimensional transpose operation, that is, first reshaping the weight data into an 8×2×2×16 four-dimensional tensor, and then exchanging the order along the 0th and 3rd dimensions to achieve data interleaving, so as to meet the 32IC packet access requirements of NVFP4 and MXFP4 formats.

[0085] In response to the mode selection signal representing the weight data as a second floating-point format, the rearrangement module transposes and rearranges the weight data according to the second-dimensional order. The data interleaving granularity corresponding to the first-dimensional order is greater than that corresponding to the second-dimensional order. For example, if the data interleaving granularity corresponding to the second-dimensional order is 16 input channels, the specific rearrangement logic is an (8, 2, 16)-dimensional transpose operation, i.e., reshaping it into an 8×2×16 three-dimensional tensor and then exchanging the order along the 0th and 2nd dimensions, adapting to the 16IC parallel computing requirements of the FP8 format.

[0086] In response to the mode selection signal representing that the weight data is in a third floating-point format, the rearrangement module bypasses the transpose rearrangement operation, and the weight data is directly distributed to the processing unit in the original storage order.

[0087] In this way, differentiated rearrangement strategies can match the parallel access requirements of different formats and improve data reuse efficiency. For example, 32IC granularity rearrangement in 4-bit format allows the simultaneous reading of weight data from 32 input channels in each cycle, doubling the computing power when combined with the parallel computing capabilities of the processing unit. 16IC granularity rearrangement in 8-bit format optimizes the data throughput in FP8 format, enabling FP8 format to achieve a computing power of 4 TOPS / s at a frequency of 1GHz.

[0088] In some embodiments, such as Figure 5 As shown, the control unit also includes a scaling factor cache module. The scaling factor cache module comprises multiple first-in-first-out (FIFO) queues. For example, the scaling factor cache module includes eight FIFO queues, each corresponding to one of the eight pixel processing circuits. Each FIFO queue has a bit width of 256 bits and a depth of 3. The scaling factor cache module is coupled to the scaling and merging module and is activated when the mode selection signal represents the weight data in NVFP4 format, caching and distributing the E4M3 scaling factor to multiple processing units.

[0089] Specifically, the external scaling factor buffer, based on the k_rden (scaling factor read enable) signal sent by the control unit, returns 256 bits of k_data (scaling factor data) in each clock cycle. This scaling factor data is then written to the corresponding pixel's FIFO queue. Since the NVFP4 format scaling factors are grouped in sets of 16 input channels, the 256 bits of data contain 32 8-bit E4M3 scaling factors, which can meet the needs of the two output channels (OC0 and OC1) of the 16 basic computing units within a pixel. The FIFO queue is designed with a depth of 3, buffering scaling factor data for 3 clock cycles, matching the 9-stage pipeline timing of the pixel processing circuit, and avoiding scaling factor waiting due to pipeline delays.

[0090] In this way, the scaling factor cache module solves the problem of matching the scaling factor reading timing with the processing unit pipeline under the NVFP4 format by using multiple first-in-first-out queues.

[0091] In some embodiments, the basic computing unit is further configured to control the multiplexer array to split the weight mantissa into high-order and low-order segments for multiplication when the mode selection signal indicates a third floating-point format.

[0092] For example, the third floating-point format is BF16 format. The weight mantissa of BF16 format is 7 bits (excluding hidden bits), which is split into high 2 bits (high bit segment) and low 5 bits (low bit segment). Two 4-bit sub-multipliers in the multiplexer array receive the high bit segment and the low bit segment respectively, multiply them with the mantissa of the input feature, and obtain the mantissa product of the two parts. Then, they are merged according to the bit weight in the alignment and addition tree module.

[0093] The output channel processing sub-circuit is also configured to output valid data for the lower 8 channels and set the data for the higher 8 channels to zero when the mode selection signal represents the weight data in BF16 format. Since each output channel only needs to process the weight data of 10 input channels (ICs) in BF16 format, and the output channel processing sub-circuit supports 16 ICs, in BF16 format, only the lower 8 channels from OC0 to OC7 (a total of 8 OCBs, each OCB containing 10 valid ICs) output valid calculation results, while the data for the 8 channels from OC8 to OC15 are forcibly set to zero.

[0094] In this way, by splitting the mantissa and multiplexing the multiplier array, no additional multiplier hardware is needed in the third floating-point format. Multiplication can be achieved through logic control alone, saving multiplier area. The design of setting the high 8 channels to zero simplifies the output control logic and avoids the overhead of subsequent processing of invalid data.

[0095] A second aspect of this application provides a processor, which includes the aforementioned reconfigurable floating-point multiply-accumulate device and a subsequent accumulation module.

[0096] The subsequent accumulator module receives the partial sum of the output of the reconfigurable floating-point multiply-accumulator, for example, a 37-bit partial sum, which includes a 9-bit exponent field and a 28-bit signed fixed-point mantissa field.

[0097] The subsequent accumulation module performs offset compensation on the exponent field of the partial sum based on the weight data type identifier. Specifically, for the difference in bias values ​​between different floating-point formats, the exponent field output by the multiplying and accumulating device (without subtracting the bias) is subtracted from the corresponding format's bias value to obtain the actual exponent. For example, the bias value of the FP8E4M3 format is 7, so the subsequent accumulation module subtracts 7 from the exponent field. The bias value of the BF16 format is 127, so the exponent field is subtracted from the format's bias value of 127 to obtain the actual exponent value.

[0098] The subsequent accumulation module is also configured to add the scaling factor corresponding to the first floating-point format to the exponent field when the weight data type identifier represents the weight data in the first floating-point format and the scaling factor is in pure exponent format. For example, if the first floating-point format is MXFP4, since the scaling factor in MXFP4 format only contains the exponent field (E8M0), the subsequent accumulation module, after completing the exponent offset compensation, directly adds the exponent value of the scaling factor in E8M0 format to the compensated exponent field without performing a multiplication operation. If the scaling factor in MXFP4 format is 0x7F in E8M0 format (exponent value is 0), the compensated exponent field is 9'd130, and the exponent field after addition is still 9'd130. If the scaling factor is 0x81 (exponent value is 2), the exponent field after addition is 9'd132.

[0099] This moves the exponential offset compensation and scaling factor processing to the subsequent accumulator module, avoiding the repeated execution of compensation and addition operations for each input channel batch within the multiply-accumulator. It also reduces the number of exponential adders, especially in the MXFP4 format, where the subsequent accumulator module only needs to perform exponential addition once, whereas doing so within the multiply-accumulator would require one operation for each input channel, thus reducing overall power consumption.

Claims

1. A reconfigurable floating-point multiply-accumulate device, characterized in that, include: The control unit is used to generate a mode selection signal based on the type identifier of the received weight data; Multiple processing units, coupled to the control unit, are configured to: The data path is reconstructed based on the mode selection signal to support the execution of floating-point multiplication and accumulation operations with weight data in the first floating-point format, the second floating-point format, or the third floating-point format as the first operand, and to generate the operation result. When the mode selection signal indicates that the weight data is in the first floating-point format, the multiplexed multiplier array corresponding to the second floating-point format is used to perform multiply-accumulate operations and perform multiplication operations of the group scaling factor. When the mode selection signal indicates that the weight data is in the second floating-point format, the multiplexer array is invoked to perform multiply-accumulate operations, and the multiplication operation path of the group scaling factor is bypassed. When the mode selection signal indicates that the weight data is in the third floating-point format, the multiplexer array is controlled to split the weight mantissa of the weight data into a high-order segment and a low-order segment. The high-order segment and the low-order segment are respectively input into the corresponding sub-multipliers of the multiplexer array to perform parallel multiplication operations, and the multiplication operation path of the group scaling factor is bypassed.

2. The reconfigurable floating-point multiply-accumulate device according to claim 1, characterized in that, The first floating-point format is a 4-bit floating-point format, the second floating-point format is an 8-bit floating-point format, and the third floating-point format is a 16-bit floating-point format.

3. The reconfigurable floating-point multiply-accumulate device according to claim 1 or 2, characterized in that, The first floating-point format includes NVFP4 format and MXFP4 format; The plurality of processing units are further configured to: When the mode selection signal represents the weight data as the NVFP4 format, the calculation result is multiplied by a scaling factor based on the E4M3 format to generate a multiplication result; When the mode selection signal indicates that the weight data is in the MXFP4 format, the scaling factor of the E4M3 format is passed to the subsequent accumulation module. In the floating-point multiplication unit of the subsequent accumulation module, the scaling factor is applied to the weight data by multiplying the mantissa and adding the exponents, thereby performing a multiply-accumulate operation.

4. The reconfigurable floating-point multiply-accumulate device according to claim 1, characterized in that, The plurality of processing units include a pixel-level processing array, and the pixel-level processing array includes a plurality of pixel processing circuits; Each pixel processing circuit includes multiple output channel processing sub-circuits, and each output channel processing sub-circuit includes multiple basic computing units; the basic computing units are configured to receive input feature data as a second operand, and decode the second operand to output the mantissa product and exponent sum.

5. The reconfigurable floating-point multiply-accumulate device according to claim 4, characterized in that, The basic computing unit includes a multiplexed multiplier array, a decoding module, and an exponential adder; The decoding module is used to decode the weight data to obtain the weight tail and weight exponent; The multiplexer array is used to multiply the weight mantissa by the mantissa of the second operand to obtain the mantissa product; The exponential adder is used to add the weighted exponent to the exponent of the second operand to obtain the exponential sum.

6. The reconfigurable floating-point multiply-accumulate device according to claim 5, characterized in that, The output channel processing sub-circuit further includes: an alignment and addition tree module and a scaling and merging module; The alignment and addition tree module is coupled to the basic computing unit and is used to perform exponential alignment and accumulation on the mantissa product and exponent sum output by the multiple basic computing units, and output the first intermediate result and the second intermediate result. The scaling and merging module is coupled to the alignment and addition tree module, and is used to perform scaling factor multiplication and group merging on the first intermediate result and the second intermediate result respectively when the mode selection signal represents the weight data as NVFP4 format, and output a 37-bit partial sum. The 37-bit portion includes a 9-bit exponent field and a 28-bit signed fixed-point mantissa field.

7. The reconfigurable floating-point multiply-accumulate device according to claim 6, characterized in that, The scaling and merging module is also configured to: When the mode selection signal indicates that the weight data is in MXFP4 format or the second floating-point format, the scaling factor multiplication and grouping / merging operations are bypassed, and the first intermediate result and the second intermediate result are used as the 37-bit partial sum output.

8. The reconfigurable floating-point multiply-accumulate device according to claim 1, characterized in that, The control unit includes a rearrangement module, which is configured to: In response to the mode selection signal indicating that the weight data is in the first floating-point format, the weight data is transposed and rearranged according to the first dimension order; In response to the mode selection signal indicating that the weight data is in the second floating-point format, the weight data is transposed and rearranged according to the second dimension order, wherein the data interleaving granularity corresponding to the first dimension order is greater than the data interleaving granularity corresponding to the second dimension order. In response to the mode selection signal indicating that the weight data is in the third floating-point format, the transpose rearrangement operation is bypassed.

9. The reconfigurable floating-point multiply-accumulate device according to claim 3, characterized in that, The control unit also includes a scaling factor cache module, which includes multiple first-in-first-out queues; The scaling factor caching module is coupled to the scaling and merging module and is configured to cache and distribute E4M3 format scaling factors to the plurality of processing units when the mode selection signal indicates that the weight data is in the NVFP4 format, so as to match the pipeline timing of the plurality of processing units.

10. The reconfigurable floating-point multiply-accumulate device according to claim 4, characterized in that, The basic computing unit is also configured to: when the mode selection signal indicates that the weight data is the third floating-point format, control the multiplexer array to split the weight mantissa into a high-order segment and a low-order segment and perform multiplication operations on them respectively. The output channel processing sub-circuit is further configured to: when the mode selection signal indicates that the weight data is in the third floating-point format, output the valid data of the lower 8 channels and set the data of the higher 8 channels to zero.

11. A processor, characterized in that, It includes a reconfigurable floating-point multiply-accumulate device as described in any one of claims 1 to 10, and a subsequent accumulation module; The subsequent accumulation module is configured to receive the partial sum output by the reconfigurable floating-point multiply-accumulate device, and to perform offset compensation on the exponent field of the partial sum according to the weight data type identifier. The subsequent accumulation module is further configured to add the scaling factor corresponding to the first floating-point format to the exponential field when the weight data type identifier indicates that the weight data is in a first floating-point format and the scaling factor is in a pure exponential format.

Images