Multiple low-bit-width floating-point fusion dot product addition calculation device and method

Abstract

This invention aims to solve the optimization problem of fused dot product addition in support of multiple low-bit-width floating-point numbers with correct rounding under the same input length, and proposes a fused dot product addition calculation device and method for multiple low-bit-width floating-point numbers. The calculation device includes the following modules: input segmentation and detection module, sign processing module, exponent processing module, mantissa processing and calculation module, anomaly detection module, mantissa data path module, leading zero detection module, mantissa normalization module, rounding module, exponent adjustment module, normal output module, anomaly output module, and output module. The method is based on the analysis of the mantissa data path, and proposes and uses a new mantissa shifting method for FP4 and FP6; for FP8, it proposes and uses a mantissa allocation scheme that can achieve correct rounding. In addition, a comparison and sorting scheme for the exponent of low-bit-width floating-point numbers is proposed, thereby enabling fused dot product addition calculation of multiple low-bit-width floating-point numbers with correct rounding.

Description

Technical Field

[0001] This invention relates to the field of low-bit-width floating-point number fusion dot product addition calculation, and specifically proposes a device and method for correct rounding multi-low-bit-width floating-point number fusion dot product addition calculation based on mantissa data path analysis. Background Technology

[0002] The floating-point bit width used by deep learning models in training and inference is continuously shrinking. This aims to optimize computational resources, accelerate model operation, and reduce energy consumption. The floating-point type used for model training is gradually transitioning from the traditional FP32 to FP16 and BF16, and further exploring low-bit-width floating-point numbers such as FP8, FP6, and FP4. Currently, the main implementation of low-bit-width floating-point numbers is the long dot product method, which lacks flexibility and faces difficulties in supporting multiple low-bit-width floating-point formats.

[0003] Among low-bit-width floating-point numbers, FP6 is particularly unique. Its bit width is not a power of 2. Multi-precision arithmetic unit designs typically use a halving approach to support floating-point numbers of different bit widths, but this method is difficult to implement with FP6. Furthermore, FP6 is also unique in data transmission with 8 bits as the smallest unit, which may lead to decreased data transmission efficiency. Dot product addition is a feasible method, which can break down a long dot product into multiple shorter dot products and higher-precision additions. By using dot product forms of different lengths, the input can be fully utilized at different precisions, achieving component reuse.

[0004] Correct rounding is the method for converting infinitely precise results into floating-point numbers. Conventional multiplication, addition, and multiply-plus floating-point operations can achieve correct rounding, but operations such as dot product, multi-operand addition, and dot product-plus often fail to achieve this goal. Floating-point operations require shifting the mantissa according to the exponent's value, while dot product and multi-operand addition require accumulating multiple shifted mantissas. Data shifted beyond the data path width is not included in the operation, and errors in the rounding process often stem from this, primarily from two problems: sticky bits and catastrophic cancellation. Sticky bits refer to the mantissa being shifted beyond the data path width, but the shifted-out portion participates in the operation, affecting the sticky bits in the rounding. Catastrophic cancellation occurs when the sum of the numbers retained in the data path is zero, but existing data has been shifted out, causing the final result to be incorrectly rounded to zero.

[0005] For low-bit-width floating-point numbers, the difference in bit width between the exponent and mantissa leads to significant variations in the bit width used in the mantissa data path for different floating-point numbers. Inappropriate bit width can result in incorrect rounding and performance loss. A commonly used correct rounding data path is full-size accumulation, ensuring that shifts do not extend beyond the mantissa data path. However, the overhead increases dramatically as the exponent's bit width increases. Another method is compressed accumulation, which compresses unnecessary zeros between mantissas before accumulation. However, this requires exponent adjustment and decompression after leading zero calculation, increasing latency.

[0006] Based on this, the present invention proposes a correct rounding calculation device and corresponding method for low-bit-width floating-point number fusion dot product addition by analyzing the mantissa data path. Summary of the Invention

[0007] This invention aims to solve the optimization problem of fused dot product addition in supporting multiple low-bit-width floating-point numbers with correct rounding under the same input length, and proposes a fused dot product addition calculation device and method for multiple low-bit-width floating-point numbers. The dot product addition unit is optimized through analysis of the mantissa data path. Specifically, the following technical solution is adopted:

[0008] The first aspect of this invention proposes a multi-low-bit-width floating-point fusion dot product addition calculation device, comprising:

[0009] The input segmentation and detection module has its input connected to an external memory and its output connected to a sign processing module, an exponent processing module, a mantissa processing and operation module, and an anomaly detection module. It is used to segment the input data and detect the segmented data.

[0010] The sign processing module, whose output is connected to the mantissa processing operation module, the mantissa data path module, the normal output module, and the output module, is used to perform XOR calculation on the sign bit;

[0011] The exponent processing module, whose output is connected to the exponent adjustment module and the mantissa data path module, is used to calculate the exponent and shift amount required for shift alignment.

[0012] The mantissa processing module, whose output is connected to the mantissa data path module, is used to perform denormalized number processing, mantissa multiplication and mantissa two's complement operation, and outputs the mantissa in two's complement form.

[0013] The mantissa data path module, whose output is connected to the leading zero detection module and the mantissa normalization module, is used to perform mantissa shifting, mantissa compression and mantissa addition to obtain intermediate results of the mantissa;

[0014] The leading zero detection module, whose output is connected to the mantissa normalization module, is used to detect how many bits are 0 before the most significant bit in the intermediate result of the mantissa.

[0015] The mantissa normalization module outputs to the rounding module, which is used to shift the mantissa according to the number of leading zeros and normalize it to the result before rounding.

[0016] The rounding module, whose output is connected to the normal output module and the exception output module, is used to select the final rounding result according to the GRS rules, and at the same time determine whether there is a rounding carry, and output the relevant results.

[0017] The exponent adjustment module, whose output is connected to the normal output module, is used to perform calculations based on the exponent of the order during shifting and alignment, the number of leading zeros, and whether there is a carry during rounding, in order to obtain the adjusted exponent.

[0018] An anomaly detection module, whose output is connected to an anomaly output module, is used to determine whether an anomaly has occurred in the operation based on the received input segmentation and the detection results output by the detection module, and to output an anomaly flag.

[0019] The output module, whose output is connected to the output module, is used to generate the output of the usual calculation based on the sign processing result, mantissa output, and exponent adjustment result.

[0020] An exception output module, whose output is connected to the output module, is used to generate an exception output based on the exception flag in the exception detection.

[0021] The output module, whose output is connected to an external memory, is used to generate the final output based on the normal output and abnormal output results.

[0022] Furthermore, the computing device performs the fusion dot product calculation based on the following formula:

[0023]

[0024] Among them (A) i ) i∈[0,N-1] and (B) i ) i∈[0,N-1] Using low-bit-width floating-point numbers, C uses BF16 format floating-point numbers, and N represents the length of the dot product; the operation represented by this formula is denoted as FDPNA;

[0025] The computing device supports the calculation of floating-point numbers in three formats: FP8, FP6, and FP4, and performs the calculations in the forms of FDP3A, FDP4A, and FDP6A, respectively. FP8 includes two formats: E4M3 and E5M2; FP6 includes two formats: E2M3 and E3M2; and FP4 is in the E2M1 format. E represents the exponent and M represents the mantissa.

[0026] Furthermore, the input segmentation and detection module segments the input data and detects the segmented data, including:

[0027] The input data is 64 bits, where A i &B i Using 48 bits, C uses 16 bits. Based on the floating-point data type used, the input data is split into each floating-point number float_i, and the float_i is split into sign, exponent and mantissa.

[0028] The data after splitting is checked to see if the exponent and mantissa are all 0 or all 1, in order to determine whether it is a denormalized number, NaN, Infinity or zero;

[0029] The sign bit is output to the sign processing module, the exponent bit is output to the exponent processing module, the denormalized number detection result and the mantissa bit are output to the mantissa processing and operation module, and the detection result is output to the anomaly detection module.

[0030] The symbol processing module performs an XOR operation on the sign bit, A i B i Perform an XOR operation with C, and perform mantissa operations based on the sign of C.

[0031] Furthermore, the index processing module includes an index summation unit, an index comparison and sorting unit, and an index difference unit;

[0032] Among them, the exponential summation unit, when performing exponential summation, for A i and B i The index is calculated directly. For the exponent of C, calculate according to the floating-point type. Note as bias dif Then calculate E C -bias dif This ensures that the exponent of C is consistent with the sum of exponents, where bias represents the floating-point bias.

[0033] For FP4 and FP6, the exponent comparison and sorting unit obtains the maximum value through comparison; in FP8 mode, E is... C -bias dif By comparing with the sum of the three indices, the position is determined and the final sorting result is obtained;

[0034] The exponent difference unit calculates the difference between the sum of the largest exponents for FP4 and FP6; for FP8, it calculates the difference between adjacent exponents after sorting.

[0035] Furthermore, the mantissa processing module includes a denormalized number processing unit, a mantissa multiplication unit, and a mantissa two's complement operation unit;

[0036] The denormalized number processing unit is based on the denormalized number formula V = (-1). S ×2 E-bias ×(m y- 1.m y-2 …m0) is used for calculation, where y represents the mantissa width of the floating-point number;

[0037] The mantissa multiplication unit uses four 4-bit multipliers and two 2-bit multipliers to perform mantissa multiplication operations. FP8 uses three 4-bit multipliers, FP6 uses four 4-bit multipliers, and FP4 uses all multipliers.

[0038] After the mantissa multiplication is completed, the mantissa complement operation unit performs a complement operation on the mantissa according to the sign, and fills the shifted bits with 0 or 1 according to the sign in the subsequent shift operation.

[0039] Furthermore, the mantissa data path module includes a mantissa shifting unit, a mantissa compression unit, and a mantissa addition unit;

[0040] The mantissa shifting unit uses a barrel shifter for shifting and alignment.

[0041] The mantissa compression unit uses a 4-2 compressor and a 3-2 compressor to compress the mantissa into sum and carry;

[0042] The mantissa addition unit performs addition on sum and carry, and determines whether the final result needs to be performed using two's complement based on the sign.

[0043] Furthermore, the mantissa normalization module performs conventional rounding based on the following formula, and will be further extended in FP8 mode:

[0044] M R =z0z -1 z -2 …z -(L-1) G&M B =M R RS

[0045] M O =M R or(M R +1) according to GRS

[0046] Where z0 represents the most significant bit and L represents the mantissa width of the output;

[0047] For BF16, L is 7, and there are three additional bits used for the retention bit (G), rounding bit (R), and sticky bit (S). The sticky bit indicates whether there is a 1 in the following bits; the final rounding result is M. O From M R and M R +1 is selected according to the GRS rules;

[0048] The mantissa normalization module normalizes the result to M. B form;

[0049] The rounding module calculates M. R and M R+1, and select M according to the GRS rules. O It also checks for rounding carry-over and non-precise anomalies, and finally outputs the relevant results.

[0050] The second aspect of this invention also proposes a multi-low-bit-width floating-point number fusion dot product addition calculation method, based on the computing device described in the first aspect above, optimizing the shift method for FP4 and FP6 precision, and using the calculation method of shifting to the maximum exponent sum for mantissa shift calculation.

[0051] Furthermore, for FP8, an additional mantissa data path is added outside the regular mantissa data path for mantissa allocation to ensure correct rounding. The specific method is as follows:

[0052] Additional shift, compression, and addition units are added to the mantissa data path module to form an additional data path parallel to the regular data path;

[0053] Sort the exponents and allocate mantissas based on the differences between adjacent exponents. Divide the four mantissas into two data paths, placing the mantissa with the largest exponent in the regular data path, and placing the remaining mantissas according to the differences between adjacent exponents. When sorting the exponents, the mantissa corresponding to the i-th sorted exponent is represented as M. i And calculate D ij It represents the difference between the i-th and j-th sorted indices, and the difference relative to M. i M that needs to be shifted j The quantity; for M i When D is 0, the exponent is considered to be minimal; when D is 0, the exponent is considered to be minimal. ij The number of leading zeros plus M R When the number of R is added to the carry amount, the subsequent mantissas are placed in an additional data path;

[0054] When the exponential difference in the extra data path exceeds the preset value, a sticky bit is added to the extra data path to retain the overflow bit.

[0055] When performing rounding calculations, the rounding formula is expanded as follows:

[0056] M R =z0z -1 z -2 …z -(L-1) G&M B =M R RSC

[0057] M O =M R or(M R +1) according to GRSC

[0058] Where C is the sign agreement bit, used to indicate whether the sign of S matches that of M. O Symbol matching.

[0059] Furthermore, the comparison and sorting method for low-bit-width floating-point exponents is as follows:

[0060] Using different inputs and different hierarchical operations, six 3-bit comparators are used to implement comparison or sorting operations for three types of low-bit-width floating-point numbers. The input of each 3-bit comparator is two unsigned 3-bit numbers, and the output is used to determine whether input one is greater than input two and whether input one is equal to input two. FP4 and FP8 use all 3-bit comparators, while FP6 uses 3 comparators. Both FP4 and FP6 use comparators in two stages. The input of the second-stage comparator is obtained by selecting the larger term from the output and input of the first-stage comparator. FP8 does not use hierarchical operations and directly uses all comparators.

[0061] The beneficial effects of the present invention are as follows:

[0062] The present invention provides a multi-low-bit-width floating-point number fusion dot product addition calculation device and method. Based on the analysis of the mantissa data path, a new mantissa shifting method is proposed and used for FP4 and FP6; and a mantissa allocation scheme that enables correct rounding is proposed and used for FP8. In addition, a comparison and sorting scheme for low-bit-width floating-point number exponents is proposed, thereby enabling correct rounding multi-low-bit-width floating-point number fusion dot product addition calculation. Attached Figure Description

[0063] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0064] Figure 1 This is a schematic diagram of an embodiment of the multi-low-bit-width floating-point fusion dot product addition calculation device of the present invention.

[0065] Figures 2(a)-2(c) This is a comparative diagram of data paths corresponding to different shifting methods in FP4 mode in an embodiment of the present invention.

[0066] Figure 3 This is a schematic diagram of the mantissa allocation method for two mantissa data paths in FP8 mode in an embodiment of the present invention.

[0067] Figure 4 This is a schematic diagram of a multi-low-bit-width floating-point exponent comparison and sorting component in an embodiment of the present invention. Detailed Implementation

[0068] The present invention will be further described below with reference to specific embodiments, but the scope of protection of the present invention is not limited thereto.

[0069] Example 1

[0070] like Figure 1 As shown in the figure, this embodiment illustrates a multi-low-bit-width floating-point fusion dot product addition calculation device 100, comprising:

[0071] The input segmentation and detection module 102 has its input connected to the external memory 200 and its output connected to the sign processing module 104, the exponent processing module 106, the mantissa processing and operation module 108, and the anomaly detection module 110. It is used to segment the input data and detect the segmented data.

[0072] The sign processing module 104, whose output is connected to the mantissa processing operation module 108, the mantissa data path module 112, the normal output module 122 and the output module 126, is used to perform XOR calculation on the sign bit;

[0073] The exponent processing module 106, whose output is connected to the exponent adjustment module 120 and the mantissa data path module 112, is used to calculate the exponent and shift amount required for shift alignment.

[0074] The mantissa processing module 108, whose output is connected to the mantissa data path module 112, is used to perform denormalized number processing, mantissa multiplication and mantissa two's complement operation, and outputs the mantissa in two's complement form.

[0075] The mantissa data path module 112, whose output is connected to the leading zero detection module 114 and the mantissa normalization module 116, is used to perform mantissa shifting, mantissa compression and mantissa addition to obtain intermediate results of the mantissa;

[0076] Leading zero detection module 114, whose output is connected to mantissa normalization module 116, is used to detect how many bits are 0 before the most significant bit in the intermediate result of the mantissa.

[0077] The mantissa normalization module 116, whose output is connected to the rounding module 118, is used to shift the mantissa according to the number of leading zeros and normalize it to the result before rounding.

[0078] The rounding module 118, whose output is connected to the normal output module 122 and the exception output module 124, is used to select the final rounding result according to the GRS rules, and at the same time determine whether there is a rounding carry, and output the relevant results.

[0079] The exponent adjustment module 120, whose output is connected to the general output module 122, is used to perform calculations based on the exponent of the order during shifting and alignment, the number of leading zeros, and whether there is a carry during rounding, in order to obtain the adjusted exponent.

[0080] An anomaly detection module 110, whose output is connected to an anomaly output module 124, is used to determine whether an anomaly has occurred in the operation based on the received input segmentation and detection results output by the detection module, and to output an anomaly flag.

[0081] Normal output module 122, whose output is connected to output module 126, is used to generate normally calculated output based on the sign processing result, mantissa output and exponent adjustment result;

[0082] An abnormal output module 124, whose output is connected to the output module 126, is used to generate an abnormal output based on the abnormal flag in the abnormal detection.

[0083] The output module 126, whose output is connected to the external memory 200, is used to generate the final output based on the normal output and abnormal output results.

[0084] The external memory 200 stores the input data used for the fusion dot product addition calculation and the output data after the calculation is completed. The external memory 200 is prior art and will not be described in detail.

[0085] The fusion point product calculation performed by the aforementioned computing device can be expressed as the following formula, where (A i ) i∈[0,N-1] and (B) i ) i∈[0,N-1] Using low-bit-width floating-point numbers, C uses BF16 format floating-point numbers, and N represents the length of the dot product. The operation represented by this formula is denoted as FDPNA.

[0086]

[0087] Floating-point numbers consist of three parts: a sign bit, an exponent bit, and a mantissa bit. This invention uses the form EXMY, where E is the exponent, M is the mantissa, and X and Y are the quantity. For FP8, there are two formats: E4M3 and E5M2; for FP6, there are two formats: E2M3 and E3M2; and for FP4, it is E2M1. The computing device of this invention supports FP8, FP6, and FP4 formats, performing operations in the forms FDP3A, FDP4A, and FDP6A, respectively.

[0088] The input segmentation and detection module 102 segments the input data and detects the segmented data. The input is 64 bits, where A... i &B iUsing 48 bits and C using 16 bits, the input data is split into individual floating-point numbers (float_i) based on the datatype used. Each float_i is then split by sign, exponent, and mantissa. The split data is then checked to see if the exponent and mantissa are all 0s or all 1s to determine if it is a denormalized number, NaN, Infinity, or zero. Because some special numbers in low-bit-width floating-point numbers are unencoded, analysis must be performed based on the precision used. In FP6-E2M3, FP6-E3M2, and FP4, Infinity and NaN are not encoded; in FP8-E4M3, Infinity is unencoded, and NaN is encoded with only one bit (S1111 111); while the FP8-E5M2 encoding format is consistent with the IEEE 754 standard. All of these floating-point formats support zero and denormalized numbers. The sign bit is input to the sign processing module, the exponent bit is input to the exponent processing module, the denormalized number detection result and the mantissa bit are input to the mantissa processing and calculation module, and the detection result is input to the anomaly detection module.

[0089] The sign processing module 104 performs an XOR operation on the sign bit, A i B i Perform an XOR operation with C, and perform mantissa operations based on the sign of C.

[0090] The exponent processing module 106 is divided into three parts: an exponent summation unit, an exponent comparison and sorting unit, and an exponent difference unit. This module is used to calculate the exponent and shift amount (i.e., the exponent difference) required for shift alignment. Since different floating-point numbers have different biases, the final output is in BF16 format. i ×B i Exponentiation is However, performing this operation in practice is quite expensive, so it is directly calculated in the exponential summation. After the exponents are calculated, they are compared to determine the maximum value. For FP4 and FP6, the maximum value is obtained through comparison; in FP8 mode, the exponents are sorted. Simultaneously, the exponent of C needs to be calculated, based on the floating-point type. Note as bias dif Then calculate E. C -bias dif This ensures that the exponent of C aligns with the sum of its exponents. In FP8 mode, this data is compared to the sums of the three exponents to determine its position and obtain the final sorting result. Finally, the exponent differences are calculated. For FP4 and FP6, the difference between the largest sums of exponents is calculated; for FP8, the difference between adjacent exponents after sorting is calculated. In FP8 mode, when the exponent difference is large, additional exponent difference calculations are performed to ensure that the accuracy of the result is sufficiently high.

[0091] The mantissa processing module 108 is divided into three parts: denormalized number processing, mantissa multiplication, and mantissa two's complement arithmetic, outputting the mantissa in two's complement form. The formula for calculating denormalized numbers is V = (-1). S ×2 E-bias+1 ×(0.M), where M=m y- 1m y-2 …m0, y represents the mantissa width of the floating-point number. Since this operation requires adding an extra 1 to the exponent of the denormalized number, resulting in significant overhead, it is modified to V = (-1). S ×2 E-bias ×(m y-1 .m y-2 …m0). To avoid the denormalized number affecting the mantissa data path allocation of FP8, leading zero detection is required for the denormalized number in FP8. In mantissa multiplication, four 4-bit multipliers and two 2-bit multipliers are used for the mantissa multiplication operation. FP8 uses three 4-bit multipliers, FP6 uses four 4-bit multipliers, and FP4 uses all multipliers. Since the exponent processing part is the critical path, to avoid excessive waiting time for the mantissa multiplication result, after the mantissa multiplication is completed, the mantissa is complemented according to the sign, and in subsequent shift operations, it is padded with 0 or 1 according to the sign.

[0092] The anomaly detection module 110 determines whether an anomaly has occurred in the operation based on the detection results of the received denormalized number, NaN, Infinity and zero, and outputs an anomaly flag.

[0093] The mantissa data path module 112 is divided into three parts: a mantissa shifting unit, a mantissa compression unit, and a mantissa addition unit, to obtain the intermediate result of the mantissa. The mantissa shifting uses a barrel shifter for alignment. The mantissa compression section uses a 4-to-2 compressor and a 3-to-2 compressor to compress the mantissa into sum and carry. The mantissa addition section performs addition on sum and carry and determines whether the final result requires two's complement operation based on the sign. For FP4 and FP6, a regular mantissa data path is used; while for FP8, an additional mantissa data path is added in addition to the regular mantissa data path to achieve lower accumulation overhead and correct rounding.

[0094] The leading zero detection module 114 is responsible for detecting how many bits are 0 before the most significant bit (i.e., the bit with the highest value of 1) in the intermediate result of the mantissa.

[0095] The mantissa normalization module 116 shifts the mantissa based on the number of leading zeros and normalizes it to the result before rounding. The standard rounding formula is as follows: z0 represents the most significant bit, and L represents the output mantissa width. For BF16, L is 7, with three additional bits used for the retention bit (G), rounding bit (R), and sticky bit (S). The sticky bit indicates whether subsequent bits are 1, and the final rounded result M... O From M R and M R +1 is selected according to the GRS rule. The mantissa normalization module normalizes the result to M. B form.

[0096] M R =z0z -1 z -2 …z -(L-1) G&M B =M R RS

[0097] M O =M R or(M R +1) according to GRS

[0098] Rounding module 118 calculates M R and M R +1, and select M according to the GRS rules. O It also checks for rounding carry-over and non-precise anomalies, and finally outputs the relevant results.

[0099] The exponent adjustment module 120 performs calculations based on the exponent of the order during shifting, the number of leading zeros, and whether there is a carry during rounding, to obtain the adjusted exponent.

[0100] Typically, output module 122 outputs M based on the sign processing result and the mantissa. O The results of the index adjustment generate the output that is usually calculated.

[0101] The abnormal output module 124 generates abnormal output based on the abnormal flags in the abnormal detection.

[0102] Output module 126 generates the final output based on the normal output and abnormal output results.

[0103] Example 2

[0104] This embodiment demonstrates a correct rounding multi-low-bit width floating-point number fusion dot product calculation method based on mantissa data path analysis. It optimizes the shifting method for FP4 and FP6 precision, and uses the shifting method to the maximum exponent sum for calculation.

[0105] Specifically, in FP4 and FP6 modes, a shift method that shifts to the maximum exponent sum is used to achieve correct rounding while reducing overhead.

[0106] For FP4-E2M1 and FP6-E2M3, the maximum value of the exponent sum is 4, and the minimum value is -2; while for FP6-E3M2, the maximum value of the exponent sum is 8, and the minimum value is -6. BF16 has an 8-bit exponent with a maximum value of 127 and a minimum value of -126, allowing it to represent a much wider range than FP4 and FP6. Therefore, if the traditional method of shifting to the maximum exponent value is still used in FP4 or FP6, it will lead to significant mantissa imbalance. Although the mantissa of the dot product is relatively concentrated, full-size accumulation requires a larger size due to the exponent of BF16.

[0107] The purpose of mantissa shifting is to align the mantissas according to the exponent difference for addition operations. Common shifting methods include right-shifting the smaller exponent to match the largest exponent, right-shifting operands with smaller exponents according to the exponent difference, or selecting the exponent of one operand and left-shifting or right-shifting the other operands. In mantissa data paths involving multiple mantissa accumulations, the method of selecting the largest exponent for shifting is usually adopted. This design employs a novel shifting method that combines the previous two methods, namely, selecting the sum of the largest exponents.

[0108] Example 3

[0109] This embodiment demonstrates a correct rounding method for low-bit-width floating-point number fusion dot product addition based on mantissa data path analysis. For the two mantissa data paths of FP8, a mantissa allocation method is designed to ensure that correct rounding can be achieved.

[0110] Specifically, in FP8 mode, the FDP3A involves four mantissas. Additional shift, compression, and addition units are added to achieve correct FP8 rounding. These additional modules are labeled as extra data paths, while the regular modules are labeled as regular data paths.

[0111] Because the exponent bits of FP8-E4M3 and FP8-E5M2 are 4 and 5 bits respectively, and the maximum exponent and difference are 28 and 58 respectively, the full-size accumulation overhead of FP8 is relatively high. In order to achieve correct rounding, FP8 must be specially optimized.

[0112] If full-size calculations are used, correct rounding can be achieved by analyzing the rounding formula. However, due to the high cost of full-size accumulation, reduced-size calculations are typically used, which involve truncation during the shift process. In reduced-size calculations, using the same formula may not yield accurate results due to catastrophic cancellation and multiple sticky bits. Compact accumulation, on the other hand, increases latency. To strike a balance between latency and area, this design employs two mantissa data paths to achieve correct rounding.

[0113] To determine which mantissas are calculated in the regular data path and which in the extra data path, the exponents are first sorted, and the mantissas are allocated based on the difference between adjacent exponents, dividing the four mantissas into two data paths. The mantissa with the largest exponent is placed in the regular data path, and the remaining mantissas are placed according to the difference between adjacent exponents. When the difference between adjacent exponents exceeds a certain range, the remaining mantissas are placed in the extra data path. If all mantissas are in the data paths, it is guaranteed that there will be no issues with sticky bits or catastrophic cancellation. When there are four mantissas in the regular data path, there will be no issues with sticky bits or catastrophic cancellation; when there are three mantissas in the regular data path and one mantissa in the extra data path, there will also be no problem. If there are two mantissas in the regular data path, the exponent difference may be too large in the extra data path, so sticky bits need to be added in the extra data path to retain overflow bits. When cancellation occurs in the regular data path, the extra data path can achieve correct rounding. When there is only one mantissa in the regular data path, no cancellation will occur. In this case, the result of the extra data path only affects the sticky bits. By calculating the extra data path, the problem of multiple sticky bits can be avoided.

[0114] Since rounding is designed for infinitely precise results, the rounding formula must be revised when allocating mantissas. However, when the mantissa is allocated to two parts, S is no longer limited to 0 and 1; it can also take the value -1. Therefore, in the following formula, we introduce a sign agreement bit (C) to indicate whether the sign of S matches that of M. O Symbol matching.

[0115] M R =z0z -1 z -2 …z -(L-1) G&M B =M R RSC

[0116] M O =M R or(M R +1) according to GRSC

[0117] According to the rounding formula, M must be calculated to ensure correct rounding.R R. Those that do not affect M R The mantissa of R is put into an additional module for calculation to obtain S.

[0118] To correctly allocate the mantissa, the exponents need to be sorted. After sorting, this invention represents the mantissa corresponding to the i-th sorted exponent as M. i And calculate D ij It represents the difference between the i-th and j-th sorted indices, and the difference relative to M. i M that needs to be shifted j The quantity of M. i When the value is 0, the exponent is considered to be minimal in order to avoid M. i All zeros and M i+1 There are cases where D has a non-zero value. ij The number of leading zeros plus M R When the number of R is added to the carry amount, the subsequent mantissa digits are placed in an additional module to achieve correct rounding.

[0119] For denormalized numbers, the number of leading zeros in M ​​will be detected, thereby modifying the constraints to avoid the extra data path width caused by denormalized numbers.

[0120] Example 4

[0121] This embodiment demonstrates a correct rounding method for multi-low-bit width floating-point number fusion dot product addition based on mantissa data path analysis, which employs a multi-low-bit width floating-point number exponent comparison and sorting method.

[0122] Specifically, using different inputs and different hierarchical operations, six 3-bit comparators are used to implement comparison or sorting operations for three types of low-bit-width floating-point numbers.

[0123] In the exponent part, FP4 and FP6 need to be compared to obtain the maximum exponent of the dot product, while FP8 requires sorting the dot product components. The calculation is done in the exponent addition. After obtaining the exponent sum, FP4 ultimately yields six 3-bit results, FP6 yields four 4-bit results, and FP8 yields three 6-bit results. In the exponent comparison section, FP4 and FP6 need to determine the maximum value of the exponent, while FP8 needs to sort them.

[0124] The comparison section uses six 3-bit comparators. Each 3-bit comparator takes two unsigned 3-bit numbers as input and outputs whether input 1 is greater than or equal to input 2. Since the condition that input 1 is less than input 2 can be derived from the results of greater than and equal to, the corresponding output is omitted. FP4 and FP8 will use all 3-bit comparators, while FP6 will use three comparators. FP4 and FP6 use comparators in two stages; the input of the second-stage comparator is obtained by selecting the larger of the output and input of the first-stage comparator. FP8, however, does not use a staged approach and uses all comparators directly.

[0125] For FP4, the six inputs in the first stage are six 3-bit exponent sums. The larger input is selected based on the comparator output, resulting in three larger 3-bit exponent sums. These three exponent sums are then compared pairwise in the second-stage comparator. For FP6, only three comparators are used; the most significant bit of the four 4-bit exponent sums is not compared. The input data is determined by whether the most significant bit is 1. For FP8, the 6-bit exponent sum is divided into high 3 bits and low 3 bits. The three high 3 bits and three low 3 bits are compared pairwise, requiring exactly six comparators.

[0126] Example 5

[0127] This embodiment illustrates a specific implementation of a shift method that uses a shift to the maximum sum of exponents in mantissa shifting. Figure 2 shows the mantissa data path bit width required for several different shift methods at FP4 precision.

[0128] Figure 2(a) illustrates the shifting method of the present invention, in which the mantissa of the dot product is shifted right, while the mantissa of C is shifted left or right depending on the difference from the maximum exponent sum. During this process, the shift aligns the seven 8-bit mantissas, which are then accumulated in the normal data path. Additionally, the left shift of C is added together with the carry input. Figure 2(b) illustrates the shifting method with the maximum exponent and full-size accumulation. In this case, there are two possibilities: the maximum exponent is the exponent of C, or some other exponent. These two possibilities are marked with different colors in the figure. The large exponent range of BF16 requires a considerably large data path width. Figure 2(c) illustrates the shifting method with the maximum exponent and compressed accumulation. Regarding the compressed accumulation method, there are also two possibilities, one of which is to compress additional zeros between the mantissas. The required data path width is much greater than that proposed in this invention. Therefore, it is clear that choosing the maximum exponent sum is reasonable and efficient in FP4 mode. The situation is similar in FP6 mode.

[0129] In FP4 and FP6 modes, no additional data path is used; computation is performed directly within the regular data path. To ensure correct shift operations within the regular data path, such as... Figure 4As shown, in FP4 mode, six 3-bit comparators are used. The six inputs of the first stage are the 3-bit exponent sums. The three largest values ​​are selected from the comparator outputs, and these three values ​​are input pairwise to the second-stage comparator. The output of the second-stage comparator selects the largest exponent sum. In FP6 mode, only three comparators are used. The first stage uses two comparators to process four 3-bit inputs. The most significant bit of the four 4-bit exponent sums is not compared; it is checked whether the most significant bit is 1. If the most significant bit is not all 0, and it is 1, the comparator input is the last three bits; if it is 0, the comparator input is 0. If the most significant bit is all 0, the last three bits are used normally. After the two comparators obtain two larger values, the comparison is completed in the second-stage comparator.

[0130] Example 6:

[0131] This embodiment illustrates a correct rounding multi-low-bit width floating-point number fusion dot product calculation device based on mantissa data path analysis, operating in FP8 mode.

[0132] The computing device in FP8 mode has the same structure as that in Embodiment 1, and both data paths in the mantissa data path module 112 are used. Through the special design of FP8 mode, effective mantissa processing and correct rounding are ensured, thereby improving computing performance and accuracy.

[0133] FP8 mode has two precision levels: FP8-E4M3 and FP8-E5M2. During calculations, it is uniformly classified as E5M3 format. The mantissa multiplication result is 8 bits, and the mantissa of C is also 8 bits; therefore, they are treated uniformly during analysis. For normalized numbers, at least one of the first two bits of the mantissa must be 1. For example... Figure 3 As shown, for D 12 The analysis assumes that the second bit of M1 is 1, and the first bit of M2 is also 1. When calculating M2, M3, and M4, a two-bit carry-over will occur. Therefore, when D... 12 When D > 1 + 9 + 2, M2, M3, and M4 are placed in an additional module. Similarly, when D 12 ≤12 and D 23 When >11, M3 and M4 are placed in an additional module; when D 12 ≤12、D 23 ≤11 and D 34 When the value is >10, M4 is placed in an additional module.

[0134] For denormalized numbers, the formula used is V = (-1) S ×2 E-bias ×(m y-1 .m y-2…m0). By detecting leading zeros in denormalized numbers, the number of leading zeros in the result of multiplication of the denormalized numbers compared to the result of ordinary multiplication is determined, and the result is sorted and denoted as Subd. i Constraint D in the case of normalized numbers ij >N, in the case of denormalized numbers, is changed to D. ij >N+Subd i -Subd j By modifying the constraints, the additional computational overhead for denormalized numbers is avoided.

[0135] like Figure 4 As shown, in FP8 mode, all comparators are used to divide the 6-bit exponent sum into high 3 bits and low 3 bits. The three high 3 bits and three low 3 bits are compared pairwise, which uses exactly 6 comparators. For the three exponent sums, the high 3 bits and low 3 bits are sorted simultaneously. When the high 3 bits are equal, the sorted result of the low 3 bits is used.

[0136] Finally, it is important to note that the above examples are merely specific embodiments of the present invention. Obviously, the present invention is not limited to these embodiments. All modifications that can be directly derived or conceived by those skilled in the art from the disclosure of this invention are considered to be within the scope of protection of this invention.

Claims

1. A multi-low-bit-width floating-point fusion dot product addition calculation device, characterized in that, include: The input segmentation and detection module has its input connected to an external memory and its output connected to a sign processing module, an exponent processing module, a mantissa processing and operation module, and an anomaly detection module. It is used to segment the input data and detect the segmented data. The sign processing module, whose output is connected to the mantissa processing operation module, the mantissa data path module, the normal output module, and the output module, is used to perform XOR calculation on the sign bit; The exponent processing module, whose output is connected to the exponent adjustment module and the mantissa data path module, is used to calculate the shift amount required for shift alignment and the base exponent required for exponent adjustment. The mantissa processing module, whose output is connected to the mantissa data path module, is used to perform denormalized number processing, mantissa multiplication and mantissa two's complement operation, and outputs the mantissa in two's complement form. The mantissa data path module, whose output is connected to the leading zero detection module and the mantissa normalization module, is used to perform mantissa shifting, mantissa compression and mantissa addition to obtain intermediate results of the mantissa; The leading zero detection module, whose output is connected to the mantissa normalization module, is used to detect how many bits are 0 before the most significant bit in the intermediate result of the mantissa. The mantissa normalization module outputs to the rounding module, which is used to shift the mantissa according to the number of leading zeros and normalize it to the result before rounding. The rounding module, whose output is connected to the normal output module and the exception output module, is used to select the final rounding result according to the rounding rules, determine whether there is a rounding carry, and output the relevant results. The exponent adjustment module, whose output is connected to the general output module, is used to perform calculations based on the base exponent during shift alignment, the number of leading zeros, and whether there is a carry during rounding, in order to obtain the adjusted exponent. An anomaly detection module, whose output is connected to an anomaly output module, is used to determine whether an anomaly has occurred in the operation based on the received input segmentation and the detection results output by the detection module, and to output an anomaly flag. The output module, whose output is connected to the output module, is used to generate the output of the usual calculation based on the sign processing result, mantissa output, and exponent adjustment result. An exception output module, whose output is connected to the output module, is used to generate an exception output based on the exception flag in the exception detection. The output module, whose output is connected to an external memory, is used to generate the final output based on the normal output and abnormal output results.

2. The multi-low-bit-width floating-point fusion dot product addition calculation device as described in claim 1, characterized in that, The computing device performs the fusion dot product calculation based on the following formula: Among them (A) i ) i∈[0,N-1] and (B) i ) i∈[0,N-1] Using low-bit-width floating-point numbers, C uses BF16 format floating-point numbers, and N represents the length of the dot product; the operation represented by this formula is denoted as FDPNA; The computing device supports the calculation of floating-point numbers in three formats: FP8, FP6, and FP4, and performs the calculations in the forms of FDP3A, FDP4A, and FDP6A, respectively. FP8 includes two formats: E4M3 and E5M2; FP6 includes two formats: E2M3 and E3M2; and FP4 is in the E2M1 format. E represents the exponent and M represents the mantissa.

3. The multi-low-bit-width floating-point fusion dot product addition calculation device as described in claim 2, characterized in that, The input segmentation and detection module segments the input data and detects the segmented data, including: The input data is 64 bits, where A i &B i Using 48 bits, C uses 16 bits. Based on the floating-point data type used, the input data is split into each floating-point number float_i, and the float_i is split into sign, exponent and mantissa. The data after splitting is checked to see if the exponent and mantissa are all 0 or all 1, in order to determine whether it is a denormalized number, NaN, Infinity or zero; Output the sign bit to the sign processing module, the exponent bit to the exponent processing module, output the denormalized number detection result and the mantissa bit to the mantissa processing and operation module, and output the detection result to the anomaly detection module. The symbol processing module performs an XOR operation on the sign bit, A i B i Perform an XOR operation with C, and perform mantissa operations based on the sign of C.

4. The multi-low-bit-width floating-point fusion dot product addition calculation device as described in claim 3, characterized in that, The exponent processing module includes an exponent summation unit, an exponent comparison and sorting unit, and an exponent difference unit. Among them, the exponential summation unit, when performing exponential summation, for A i and B i The index is calculated directly. For the exponent of C, calculate according to the floating-point type. Note as bias dif Then calculate E C -bias dif This ensures that the exponent of C is consistent with the sum of exponents, where bias represents the floating-point bias. For FP4 and FP6, the exponent comparison and sorting unit obtains the maximum value through comparison; in FP8 mode, E is... C -bias dif By comparing with the sum of the three indices, the position is determined and the final sorting result is obtained; The exponent difference unit calculates the difference between the sum of the largest exponents for FP4 and FP6; for FP8, it calculates the difference between adjacent exponents after sorting.

5. The multi-low-bit-width floating-point fusion dot product addition calculation device as described in claim 4, characterized in that, The mantissa processing module includes a denormalized number processing unit, a mantissa multiplication unit, and a mantissa complement operation unit. The denormalized number processing unit is based on the modified denormalized number formula V = (-1). S ×2 E-bias ×(m y-1 .m y-2 …m0) is used for calculation, where y represents the mantissa width of the floating-point number; The mantissa multiplication unit uses four 4-bit multipliers and two 2-bit multipliers to perform mantissa multiplication operations. FP8 uses three 4-bit multipliers, FP6 uses four 4-bit multipliers, and FP4 uses all multipliers. After the mantissa multiplication is completed, the mantissa complement operation unit performs a complement operation on the mantissa according to the sign, and fills the shifted bits with 0 or 1 according to the sign in the subsequent shift operation.

6. The multi-low-bit-width floating-point fusion dot product addition calculation device as described in claim 5, characterized in that, The mantissa data path module includes a mantissa shift unit, a mantissa compression unit, and a mantissa addition unit, and has two sets of parallel modules. The conventional module is used for FP4, FP6, and FP8 operations, while the additional module is only used for FP8 operations. The mantissa shifting unit uses a barrel shifter for shifting and alignment. The mantissa compression unit uses a 4-2 compressor and a 3-2 compressor to compress the mantissa into sum and carry; The mantissa addition unit performs addition on sum and carry, and determines whether the final result needs to be performed using two's complement based on the sign.

7. The multi-low-bit-width floating-point fusion dot product addition calculation device as described in claim 6, characterized in that, The mantissa normalization module performs conventional rounding based on the following formula, and further extends the formula in FP8 mode: M R =z0z -1 z -2 …z -(L-1) G&M B =M R RS M O =M R or(M R +1)according GRS Where z0 represents the most significant bit and L represents the mantissa width of the output; For BF16, L is 7, and there are three additional bits used for the retention bit (G), rounding bit (R), and sticky bit (S). The sticky bit indicates whether there is a 1 in the following bits; the final rounding result is M. O From M R and M R +1 is selected according to the GRS rules; The mantissa normalization module normalizes the result to M. B form; The rounding module calculates M. R and M R +1, and select M according to the GRS rules. O It also checks for rounding carry-over and non-precise anomalies, and finally outputs the relevant results.

8. A method for calculating low-bit-width floating-point fusion dot product addition, based on the computing device described in any one of claims 1-7, characterized in that, The shifting method for FP4 and FP6 precision is optimized by using the calculation method of shifting to the maximum exponent sum for mantissa shifting calculation.

9. The method for calculating low-bit-width floating-point numbers by fusion dot product addition as described in claim 8, characterized in that, For FP8, an additional mantissa data path is added outside the regular mantissa data path for mantissa allocation to ensure correct rounding. The specific method is as follows: Additional shift, compression, and addition units are added to the mantissa data path module to form an additional data path parallel to the regular data path; Sort the exponents and allocate mantissas based on the differences between adjacent exponents. Divide the four mantissas into two data paths, placing the mantissa with the largest exponent in the regular data path, and placing the remaining mantissas according to the differences between adjacent exponents. When sorting the exponents, the mantissa corresponding to the i-th sorted exponent is represented as M. i And calculate D ij It represents the difference between the i-th and j-th sorted indices, and the difference relative to M. i M that needs to be shifted j The quantity; for M i When D is 0, the exponent is considered to be minimal; when D is 0, the exponent is considered to be minimal. ij The number of leading zeros plus M R When the number of R is added to the carry amount, the subsequent mantissas are placed in an additional data path; When the exponential difference in the extra data path exceeds the preset value, a sticky bit is added to the extra data path to retain the overflow bit. When performing rounding calculations, the rounding formula is extended as follows: M R =z0z -1 z -2 …z -(L-1) G&M B =M R RSC N O =M R or(M R +1)according GRSC Where C is the sign agreement bit, used to indicate whether the sign of S matches that of M. O Symbol matching.

10. The low-bit-width floating-point number fusion dot product addition calculation method as described in claim 8 or 9, characterized in that, The comparison and sorting method for low-bit-width floating-point exponents is as follows: Using different inputs and different hierarchical operations, six 3-bit comparators are used to implement comparison or sorting operations on three types of low-bit-width floating-point numbers. The input of each 3-bit comparator is two unsigned 3-bit numbers, and the output is used to determine whether input one is greater than input two and whether input one is equal to input two. FP4 and FP8 use all 3-bit comparators, while FP6 uses 3 comparators. Both FP4 and FP6 divide the comparators into two stages, and the input of the second-stage comparator is obtained by selecting the larger term from the output and input of the first-stage comparator. FP8 does not hierarchically and directly uses all comparators.

Images