Multi-low-bit-width floating-point number fusion dot product addition calculation device and method

Abstract

The invention aims to solve the optimization problem of fusion dot product addition in support of various correctly rounded low-bit-width floating-point numbers under the same-length input, and provides a multi-low-bit-width floating-point number fusion dot product addition calculation device and method. The computing device comprises the following modules: an input segmentation and detection module, a symbol processing module, an index processing module, a mantissa processing and computing module, an anomaly detection module, a mantissa data path module, a leading zero detection module, a mantissa normalization module, a rounding module, an index adjustment module, a normal output module, an anomaly output module and an output module. According to the method, based on analysis of mantissa data paths, a new mantissa shifting mode is provided and used for FP4 and FP6; for FP8, a mantissa allocation scheme capable of realizing correct rounding is provided and used. In addition, a comparison and sorting scheme for low-bit-width floating-point number indexes is also provided, so that correctly rounded multi-low-bit-width floating-point number fusion dot product addition calculation can be realized.

Description

Technical Field

[0001] The present invention relates to the field of low-bit-width floating-point fusion dot product addition calculation, and specifically proposes a correctly rounded multiple low-bit-width floating-point fusion dot product addition calculation device and method based on mantissa data path analysis. Background Art

[0002] The floating point bit width used in deep learning models for training and reasoning is experiencing a continuous reduction. The goal is to optimize computing resources, accelerate model operation and reduce energy consumption. The floating point type for model training has gradually transitioned from the traditional FP32 to FP16 and BF16, and further explored low-bit-width floating point numbers such as FP8, FP6 and FP4. At present, the main implementation method of low-bit-width floating point numbers is the long dot product method, which is not flexible enough and has difficulties in supporting multiple low-bit-width floating point formats.

[0003] Among low-bit-width floating-point numbers, FP6 is particularly unique because its bit width is not an exponential multiple of 2. Multi-precision arithmetic unit designs usually use a halved split method to support floating-point numbers of different bit widths, but FP6 is difficult to use this method. In addition, FP6 is also special in data transmission with 8 bits as the minimum unit, which may lead to reduced data transmission efficiency. Dot product addition is a feasible method that can split a long dot product into multiple short dot products and higher-precision additions. Through dot product forms of different lengths, the input can be fully utilized at different precisions to achieve component reuse.

[0004] Correct rounding is a method of converting infinitely accurate results into floating-point numbers. Conventional multiplication, addition, and multiply-add floating-point operations can achieve correct rounding, but operations such as dot product, multi-operand addition, and dot product addition often fail to achieve this goal. Floating-point operations require the mantissa to be shifted according to the size of the exponent, while dot product and multi-operand addition require the accumulation of multiple shifted mantissas. Data that is shifted beyond the data path width will not be calculated. Errors in the rounding process often come from this, mainly from two problems: multiple sticky bits and catastrophic cancellation. Multiple sticky bits refer to the mantissa shift exceeding the data path width, but the shifted part participates in the calculation, affecting the sticky bits in the rounding. Catastrophic cancellation refers to the accumulation of the numbers retained in the data path. The result is zero, but the existing data is shifted out, causing the final result to be incorrectly rounded to zero.

[0005] For low-bit-width floating-point numbers, due to the difference in bit width between the exponent and the mantissa, different floating-point numbers use different bit widths on the mantissa data path. Inappropriate bit width may lead to incorrect rounding and performance loss. The commonly used correct rounding data path is full-size accumulation, which ensures that the shift does not move out of the mantissa data path, but as the bit width of the exponent increases, the overhead rises sharply. In addition, there is a compressed accumulation method that compresses the useless 0s between the mantissas before accumulation, but after the leading zero calculation is completed, the exponent adjustment decompression is required, which will increase the latency overhead.

[0006] Based on this, the present invention proposes a correct rounding calculation device and a corresponding method for low-bit width floating point number fusion dot product addition through analysis of the mantissa data path. Summary of the invention

[0007] The present invention aims to solve the optimization problem of fused dot product addition in the support of multiple low-bit-width floating-point numbers with correct rounding under the same length input, and proposes a multiple low-bit-width floating-point fused dot product addition calculation device and method. By analyzing the mantissa data path, the dot product addition unit is optimized. The following technical solutions are specifically adopted:

[0008] The first aspect of the present invention provides a multiple low-bit-width floating-point fusion dot product addition calculation device, comprising:

[0009] An input segmentation and detection module, whose input is connected to an external memory, and whose output is connected to a sign processing module, an exponent processing module, a mantissa processing operation module and an anomaly detection module, is used to segment the input data and detect the segmented data;

[0010] A sign processing module, whose output is connected to the mantissa processing operation module, the mantissa data path module, the normal output module and the output module, for performing an XOR calculation on the sign bit;

[0011] An exponential processing module, whose output is connected to the exponential adjustment module and the mantissa data path module, is used to calculate the exponent and shift amount required when shifting the right order;

[0012] A mantissa processing operation module, whose output is connected to the mantissa data path module, is used for performing denormalized number processing, mantissa multiplication and mantissa complement operation, and outputting the mantissa in the form of complement;

[0013] A mantissa data path module, whose output is connected to the leading zero detection module and the mantissa normalization module, is used for performing mantissa shift, mantissa compression and mantissa addition to obtain an intermediate result of the mantissa;

[0014] A leading zero detection module, whose output is connected to a mantissa normalization module, is used to detect how many bits before the most significant bit in the mantissa intermediate result are 0;

[0015] A mantissa normalization module, whose output is connected to the rounding module, is used to shift the mantissa according to the number of leading zeros and normalize it to the result before rounding;

[0016] The rounding module, whose output is connected to the normal output module and the abnormal output module, is used to select the final rounding result according to the GRS rule, and at the same time determine whether there is a rounding carry, and output the relevant result;

[0017] An exponent adjustment module, whose output is connected to the normal output module, is used to calculate according to the exponent of the order when shifting the order, the number of leading zeros and whether there is a carry when rounding, so as to obtain an adjusted exponent;

[0018] The abnormality detection module, whose output is connected to the abnormality output module, is used to determine whether the operation is abnormal based on the received input segmentation and the detection result output by the detection module, and output an abnormality flag;

[0019] A common output module, whose output is connected to the output module, is used to generate a common calculated output according to the sign processing result, the mantissa output and the exponent adjustment result;

[0020] An abnormal output module, whose output is connected to the output module, is used to generate abnormal output according to the abnormal flag in the abnormal detection;

[0021] The output module, whose output is connected to the external memory, is used to generate the final output according to the normal output and abnormal output results.

[0022] Furthermore, the computing device performs fusion dot product addition calculation based on the following formula:

[0023]

[0024] Among them (A i ) i∈[0,N-1] and (B i ) i∈[0,N-1] Use low-width floating-point numbers, C uses a floating-point number in BF16 format, and N represents the length of the dot product; the operation represented by this formula is recorded as FDPNA;

[0025] The computing device supports the calculation of floating point numbers in three formats: FP8, FP6 and FP4, and performs operations in the form of FDP3A, FDP4A and FDP6A respectively; wherein FP8 includes two formats: E4M3 and E5M2; FP6 includes two formats: E2M3 and E3M2; FP4 is E2M1 format; E is the exponent bit, and M is the mantissa bit.

[0026] Furthermore, the input segmentation and detection module segments the input data and detects the segmented data, including:

[0027] The input data is 64 bits, where A i &B i Use 48 bits, C uses 16 bits, and according to the floating point type datatype used, split the input data into each floating point number float_i, and split float_i into sign, exponent and mantissa;

[0028] Check whether the exponent and mantissa of the split data are all 0 or all 1 to determine whether it is a denormalized number, NaN, Infinity or zero;

[0029] The sign bit is output to the sign processing module, the exponent bit is output to the exponent processing module, the non-normalized number detection result and the mantissa bit are output to the mantissa processing operation module, and the detection result is output to the anomaly detection module.

[0030] The symbol processing module performs XOR calculation on the sign bit, A i , B i Perform an XOR operation with C and perform mantissa data calculation based on the sign of C.

[0031] Further, the index processing module includes an index summing unit, an index comparing and sorting unit, and an index differencing unit;

[0032] Among them, when the exponential summation unit performs exponential summation, for A i and B i Direct calculation of the exponential For the exponent of C, calculate it according to the floating point type Note as bias dif , then calculate E C -bias dif , so that the exponent of C is consistent with the exponent sum, where bias represents the bias of the floating point number;

[0033] The index comparison and sorting unit obtains the maximum value through comparison for FP4 and FP6; in FP8 mode, E C -bias dif Compare with the sum of the three indices to determine the position and obtain the final sorting result;

[0034] The exponent difference unit calculates the difference between the maximum exponent sums for FP4 and FP6, and the difference between sorted adjacent exponents for FP8.

[0035] Further, the mantissa processing operation module includes a denormalized number processing unit, a mantissa multiplication unit and a mantissa complement operation unit;

[0036] The denormalized number processing unit is based on the denormalized number formula V=(-1) S ×2 E-bias ×(m y- 1.m y-2 …m0) is used for calculation, and y represents the mantissa width of the floating point number;

[0037] The mantissa multiplication unit uses four 4-bit multipliers and two 2-bit multipliers to perform mantissa multiplication operations, wherein FP8 uses three 4-bit multipliers, FP6 uses four 4-bit multipliers, and FP4 uses all multipliers;

[0038] The mantissa complement operation unit performs a complement operation on the mantissa according to the sign after the mantissa multiplication is completed, and performs shift filling using 0 or 1 according to the sign in a subsequent shift operation.

[0039] Further, the mantissa data path module includes a mantissa shift unit, a mantissa compression unit and a mantissa addition unit;

[0040] Among them, the mantissa shift unit uses a barrel shifter to perform shift alignment;

[0041] The mantissa compression unit uses a 4-2 compressor and a 3-2 compressor to compress the mantissa into sum and carry;

[0042] The mantissa addition unit adds sum and carry and determines whether the final result requires a complement operation based on the sign.

[0043] Furthermore, the mantissa normalization module performs conventional rounding based on the following formula, and performs additional expansion in FP8 mode:

[0044] M R =z0z -1 z -2 …z -(L-1) G&M B =M R RS

[0045] M O =M R or(M R +1) according to GRS

[0046] Among them, z0 represents the most significant bit, and L represents the mantissa width of the output;

[0047] For BF16, L is 7, and there are three extra bits for the reserved bit (G), the rounding bit (R), and the sticky bit (S). The sticky bit indicates whether there is a 1 in the subsequent bit; the final rounded result M O From M R and M R +1 is selected according to GRS rules;

[0048] The mantissa normalization module normalizes the result to M B form;

[0049] The rounding module calculates M R and M R+1, and select M according to GRS rules O , and at the same time determine whether there is rounding carry, detect whether there is an inexact exception, and finally output the relevant results.

[0050] The second aspect of the present invention further proposes a method for calculating dot product addition by fusing multiple low-bit-width floating-point numbers, which is based on the computing device described in the first aspect above, optimizes the shifting method for FP4 and FP6 precision, and uses the calculation method of shifting to the maximum exponent sum to perform mantissa shift calculation.

[0051] Furthermore, for FP8, an additional mantissa data path is added in addition to the regular mantissa data path to perform mantissa allocation to ensure correct rounding. The specific method is as follows:

[0052] Adding additional shift, compression and addition units in the mantissa data path module to form an additional data path in parallel with the regular data path;

[0053] Sort the exponents and assign mantissas according to the difference between adjacent exponents. Divide the four mantissas into two data paths. The mantissa with the largest exponent is placed in the regular data path, and the remaining mantissas are placed according to the difference between adjacent exponents. When sorting the exponents, the mantissa corresponding to the i-th sorted exponent is represented as M. i , and calculate D ij , which represents the difference between the i-th and j-th sorted indices, and relative to M i M that needs to be shifted j The number of i When D is 0, the index is considered to be the minimum; ij The number greater than the leading zero plus M R When the number of R is added to the number of carries, the subsequent mantissa bits are placed in the extra data path;

[0054] When the exponent difference in the additional data path exceeds a preset value, a sticky bit is added in the additional data path to retain the overflowed bit;

[0055] When performing rounding calculations, the rounding formula is expanded:

[0056] M R =z0z -1 z -2 …z -(L-1) G&M B =M R RSC

[0057] M O =M R or(M R +1) according to GRSC

[0058] Among them, C is the sign consistency bit, which is used to indicate whether the sign of S is consistent with M O The symbol matches.

[0059] Furthermore, the comparison and sorting methods for low-width floating-point exponents are as follows:

[0060] Using different inputs and different classification operations, 6 3-bit comparators are used to implement comparison or sorting operations on three low-width floating-point numbers. The input of each 3-bit comparator is two unsigned 3-bit numbers, and the output is used to determine whether input one is greater than input two and whether input one is equal to input two; FP4 and FP8 use all 3-bit comparators, and FP6 uses 3 comparators; FP4 and FP6 both divide the comparators into two levels, and the input of the second-level comparator is obtained by selecting the larger one of the output and input of the first-level comparator; FP8 is not classified and all comparators are used directly.

[0061] The beneficial effects of the present invention are as follows:

[0062] The device and method for calculating multiple low-bit-width floating-point fusion dot product addition of the present invention propose and use a new mantissa shifting method for FP4 and FP6 based on the analysis of the mantissa data path; and propose and use a mantissa allocation scheme that can achieve correct rounding for FP8. In addition, a comparison and sorting scheme for low-bit-width floating-point exponents is also proposed, so that multiple low-bit-width floating-point fusion dot product addition calculations with correct rounding can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0064] Figure 1 It is a schematic diagram of the structure of an embodiment of a multiple low-bit-width floating-point fusion dot product addition calculation device of the present invention.

[0065] Figure 2(a)-Figure 2(c) Schematic diagram of comparison of data paths corresponding to different shifting modes in FP4 mode in an embodiment of the present invention.

[0066] Figure 3 Schematic diagram of the mantissa allocation method of two mantissa data paths in FP8 mode in an embodiment of the present invention.

[0067] Figure 4 Schematic diagram of multiple low-bit-width floating-point exponent comparison and sorting components in an embodiment of the present invention. DETAILED DESCRIPTION

[0068] The present invention is further described below in conjunction with specific embodiments, but the protection scope of the present invention is not limited thereto.

[0069] Example 1

[0070] like Figure 1 As shown, this embodiment shows a multiple low-bit-width floating-point fusion dot product addition calculation device 100, including:

[0071] An input segmentation and detection module 102, whose input is connected to the external memory 200, and whose output is connected to the sign processing module 104, the exponent processing module 106, the mantissa processing operation module 108 and the abnormality detection module 110, is used to segment the input data and detect the segmented data;

[0072] The sign processing module 104, whose output is connected to the mantissa processing operation module 108, the mantissa data path module 112, the normal output module 122 and the output module 126, is used to perform an XOR calculation on the sign bit;

[0073] The exponent processing module 106, whose output is connected to the exponent adjustment module 120 and the mantissa data path module 112, is used to calculate the exponent and shift amount required when shifting the order;

[0074] A mantissa processing operation module 108, whose output is connected to the mantissa data path module 112, is used to perform denormalized number processing, mantissa multiplication and mantissa complement operation, and output the mantissa in the form of complement;

[0075] A mantissa data path module 112, whose output is connected to a leading zero detection module 114 and a mantissa normalization module 116, for performing mantissa shifting, mantissa compression and mantissa addition to obtain an intermediate result of the mantissa;

[0076] A leading zero detection module 114, whose output is connected to a mantissa normalization module 116, is used to detect how many bits before the most significant bit in the mantissa intermediate result are 0;

[0077] A mantissa normalization module 116, whose output is connected to a rounding module 118, for shifting the mantissa according to the number of leading zeros and normalizing it to a result before rounding;

[0078] The rounding module 118, whose output is connected to the normal output module 122 and the abnormal output module 124, is used to select the final rounding result according to the GRS rule, and at the same time determine whether there is a rounding carry, and output the relevant result;

[0079] An exponent adjustment module 120, whose output is connected to a normal output module 122, is used to perform calculations according to the exponent of the order when shifting the order, the number of leading zeros, and whether there is a carry when rounding, so as to obtain an adjusted exponent;

[0080] The abnormality detection module 110, whose output is connected to the abnormality output module 124, is used to determine whether the operation is abnormal according to the received input segmentation and the detection result output by the detection module, and output an abnormality flag;

[0081] A normal output module 122, whose output is connected to an output module 126, for generating a normal calculated output according to the sign processing result, the mantissa output and the exponent adjustment result;

[0082] An abnormal output module 124, whose output is connected to an output module 126, is used to generate an abnormal output according to an abnormal flag in abnormal detection;

[0083] The output module 126, whose output is connected to the external memory 200, is used to generate a final output according to the normal output and abnormal output results.

[0084] The input data for the fusion dot product addition calculation and the output data after the calculation are stored in the external memory 200. The external memory 200 belongs to the prior art, so it will not be described in detail.

[0085] The fusion dot product addition calculation performed by the above-mentioned computing device can be expressed as the following formula, where (A i ) i∈[0,N-1] and (B i ) i∈[0,N-1] Use low-width floating-point numbers, C uses a floating-point number in BF16 format, and N represents the length of the dot product. The operation represented by this formula is denoted as FDPNA.

[0086]

[0087] A floating point number consists of a sign bit, an exponent bit, and a mantissa bit. The present invention uses the form of EXMY, where E is the exponent bit, M is the mantissa bit, and X and Y are quantities. For FP8, there are two formats, E4M3 and E5M2; FP6 includes two formats, E2M3 and E3M2; and FP4 is E2M1. The computing device of the present invention supports three formats, FP8, FP6, and FP4, and performs operations in the form of FDP3A, FDP4A, and FDP6A, respectively.

[0088] The input segmentation and detection module 102 segments the input data and detects the segmented data. The input is 64 bits, where A i &B iUsing 48 bits, C uses 16 bits. According to the floating point type datatype used, the input data is divided into each floating point float_i, and float_i is divided into signs, exponents, and mantissas. After the division, the exponent and mantissa of the data are checked to see if they are all 0 or all 1 to determine whether they are denormalized numbers, NaN, Infinity, or zero. Since some special numbers in low-width floating point numbers are uncoded, the detection needs to be analyzed according to the precision used. In FP6-E2M3, FP6-E3M2, and FP4, Infinity and NaN are not coded; in FP8-E4M3, Infinity is not coded, and NaN only has one bit coded S1111 111; while the FP8-E5M2 coding format is consistent with the IEEE 754 standard. The above floating point formats all support zero and denormalized numbers. The sign bit is input into the sign processing module, the exponent bit is input into the exponent processing module, the non-normalized number detection result and the mantissa bit are input into the mantissa processing operation module, and the detection result is input into the anomaly detection module.

[0089] The symbol processing module 104 performs an XOR calculation on the sign bit, A i , B i Perform an XOR operation with C and perform mantissa data calculation based on the sign of C.

[0090] The exponential processing module 106 is divided into three parts: exponential summing unit, exponential comparison and sorting unit, and exponential difference unit. This module is used to calculate the exponent and shift amount (i.e., exponential difference) required when shifting the order. Since different floating point numbers have different biases, the final output is in BF16 format. i ×B i The exponential operation is However, actually performing this operation is expensive, so we compute it directly in the exponential summation. After the exponential sum is calculated, a comparison is performed to determine the maximum value. For FP4 and FP6, the maximum value is obtained by comparison; in FP8 mode, the exponential sum is sorted. At the same time, the exponent of C needs to be calculated, and the floating point type is calculated. Note as bias dif Then calculate E C -bias dif , so that the index of C is consistent with the index sum. In FP8 mode, this data is compared with the three index sums to determine the position and obtain the final sorting result. Finally, the index difference is calculated. For FP4 and FP6, the difference between the maximum index sums is calculated; for FP8, the difference between the sorted adjacent indexes is calculated. In FP8 mode, when the index difference is large, an additional index difference operation is performed to ensure that the accuracy of the operation result is high enough.

[0091] The mantissa processing module 108 is divided into three parts: denormalized number processing, mantissa multiplication and mantissa complement operation, and outputs the mantissa in the form of complement. The calculation formula for the denormalized number is V = (-1) S ×2 E-bias+1 ×(0.M), where M = m y- 1m y-2 …m0, y represents the mantissa width of the floating point number. Since this operation requires adding 1 to the exponent of the denormalized number, which results in a large overhead, it is modified to V = (-1) S ×2 E-bias ×(m y-1 .m y-2 …m0). To prevent the denormalized number from affecting the mantissa data path allocation of FP8, the leading zero detection of the mantissa of the denormalized number of FP8 is required. In the mantissa multiplication, four 4-bit multipliers and two 2-bit multipliers are used for mantissa multiplication, of which FP8 uses three 4-bit multipliers, FP6 uses four 4-bit multipliers, and FP4 uses all multipliers. Since the exponent processing part is the critical path, in order to avoid the waiting time for the mantissa multiplication result too long, after the mantissa multiplication is completed, the mantissa is complemented according to the sign, and in the subsequent shift operation, 0 or 1 is used for shift filling according to the sign.

[0092] The abnormality detection module 110 determines whether an abnormality occurs in the operation according to the received detection results of denormalized numbers, NaN, Infinity and zero, and outputs an abnormality flag.

[0093] The mantissa data path module 112 is divided into three parts: a mantissa shift unit, a mantissa compression unit, and a mantissa addition unit to obtain the intermediate result of the mantissa. The mantissa shift uses a barrel shifter for shift alignment. The mantissa compression part uses a 4-2 compressor and a 3-2 compressor to compress the mantissa into sum and carry. The mantissa addition part adds sum and carry, and determines whether the final result requires a complement operation based on the sign. For FP4 and FP6, a conventional mantissa data path is used; for FP8, an additional mantissa data path is added to the conventional mantissa data path to achieve lower accumulation overhead and achieve the purpose of correct rounding.

[0094] The leading zero detection module 114 is responsible for detecting how many bits are 0 before the most significant bit (ie, the bit with the highest value of 1) in the intermediate result of the mantissa.

[0095] The mantissa normalization module 116 shifts the mantissa according to the number of leading zeros and normalizes it to the result before rounding. The conventional rounding formula is as follows: z0 represents the most significant bit, and L represents the mantissa width of the output. For BF16, L is 7, and there are three additional bits for reserved bits (G), rounding bits (R), and sticky bits (S). The sticky bit indicates whether there is a 1 in the subsequent bit, and the final rounded result M O From M R and M R +1 is selected according to the GRS rule. The mantissa normalization module normalizes the result to M B form.

[0096] M R =z0z -1 z -2 …z -(L-1) G&M B =M R RS

[0097] M O =M R or(M R +1) according to GRS

[0098] The rounding module 118 calculates M R and M R +1, and select M according to GRS rules O , and at the same time determine whether there is rounding carry, detect whether there is an inexact exception, and finally output the relevant results.

[0099] The exponent adjustment module 120 performs operations according to the exponent of the order when shifting the order, the number of leading zeros, and whether there is a carry when rounding to obtain an adjusted exponent.

[0100] Usually the output module 122 outputs M according to the symbol processing result and the mantissa O and exponent-adjusted results generate the normally calculated output.

[0101] The abnormality output module 124 generates an abnormality output according to an abnormality flag in abnormality detection.

[0102] The output module 126 generates a final output based on the normal output and abnormal output results.

[0103] Example 2

[0104] This embodiment shows a correctly rounded multiple low-bit-width floating-point fusion dot product addition calculation method based on mantissa data path analysis, which optimizes the shifting method for FP4 and FP6 precision and uses the method of shifting to the maximum exponent sum for calculation.

[0105] Specifically, in FP4 and FP6 modes, in order to achieve correct rounding while reducing overhead, a shift method to the maximum exponent sum is adopted.

[0106] For FP4-E2M1 and FP6-E2M3, the maximum value of the sum of the exponents is 4 and the minimum value is -2; while for FP6-E3M2, the maximum value of the sum of the exponents is 8 and the minimum value is -6. BF16 has an 8-bit exponent with a maximum exponent of 127 and a minimum exponent of -126, which enables it to represent a range far beyond FP4 and FP6. Therefore, if the traditional method of shifting to the maximum exponent value is still used in FP4 or FP6, it will result in a significant imbalance in the mantissa. Although the mantissa of the dot product is relatively concentrated, the full-size accumulation requires a larger size due to the exponent of BF16.

[0107] The purpose of mantissa shift is to align the mantissas according to the exponent difference for addition. Common shifting methods include shifting the smaller exponent right to match the largest exponent, shifting the operand with the smaller exponent right according to the exponent difference, and selecting the exponent of one operand and shifting the other operands left or right. In the mantissa data path of multiple mantissa accumulation, the method of selecting the largest exponent for shifting is usually adopted. In this design, a new shifting method is adopted, which combines the first two methods, that is, the method of selecting the maximum exponent sum.

[0108] Example 3

[0109] This embodiment shows a correctly rounded low-bit width floating point fusion dot product addition calculation method based on mantissa data path analysis. A mantissa allocation method is designed for the two mantissa data paths of FP8 to ensure correct rounding.

[0110] Specifically, in FP8 mode, FDP3A involves four mantissas. In order to achieve correct rounding for FP8, additional shift, compression, and addition units are added. These additional modules are marked as additional data paths, while regular modules are marked as regular data paths.

[0111] Since the exponent bits of FP8-E4M3 and FP8-E5M2 are 4 and 5 bits respectively, and the maximum exponent and difference are 28 and 58 respectively, the full-size accumulation overhead of FP8 is high. In order to achieve correct rounding, special optimization must be performed for FP8.

[0112] If full-size calculations are used, correct rounding can be achieved by analyzing the rounding formula. However, due to the high cost of full-size accumulation, reduced-size calculations are often used, which truncate during the shift process. Using the same formula in the reduced-size calculation may not produce an exact result due to catastrophic cancellation and multiple sticky bits. Computing with compressed accumulations adds latency overhead. To strike a balance between latency and area, this design uses two mantissa data paths to achieve correct rounding.

[0113] In order to determine which mantissas are calculated in the regular data path and which are calculated in the extra data path, the exponents are first sorted and the mantissas are allocated according to the differences of adjacent exponents, dividing the four mantissas into two data paths. The mantissa with the largest exponent is placed in the regular data path, and the remaining mantissas are placed according to the differences of adjacent exponents. When the difference of adjacent exponents is out of range, the remaining mantissas are placed in the extra data path. If all mantissas are in the data path, it is guaranteed that there will be no problems with multiple sticky bits and catastrophic cancellation. When there are four mantissas in the regular data path, there will be no problems with multiple sticky bits and catastrophic cancellation; when there are three mantissas in the regular data path and there is one mantissa in the extra data path, there will be no problem. If there are two mantissas in the regular data path, the exponent difference may be too large in the extra data path, so it is necessary to add sticky bits in the extra data path to retain the overflowed bits. When the regular data path has cancellation, the extra data path can achieve correct rounding. When there is only one mantissa in the regular data path, no cancellation occurs. In this case, the result of the extra data path only affects the sticky bit. By calculating the extra data path, the problem of multiple sticky bits can be avoided.

[0114] Since rounding is designed for infinitely accurate results, the rounding formula must be revised when the mantissa is distributed. However, when the mantissa is distributed into two parts, S is no longer limited to 0 and 1, it may also take the value -1. Therefore, in the following formula, we introduce the sign consistency bit (C) to indicate whether the sign of S is consistent with M. O The symbol matches.

[0115] M R =z0z -1 z -2 …z -(L-1) G&M B =M R RSC

[0116] M O =M R or(M R +1) according to GRSC

[0117] According to the rounding formula, in order to ensure correct rounding, M must be calculatedR R. Those that do not affect M R The mantissa of R is put into an extra module for calculation to obtain S.

[0118] In order to correctly allocate the mantissa, the exponents need to be sorted. After sorting, the present invention represents the mantissa corresponding to the i-th sorted exponent as M i , and calculate D ij , which represents the difference between the i-th and j-th sorted indices, and relative to M i M that needs to be shifted j For M i When it is 0, the index is considered to be the minimum to avoid M i All zeros and M i+1 There are non-zero values. When D ij The number greater than the leading zero plus M R When the number of R is added to the number of carries, the subsequent mantissa bits are put into extra modules to achieve correct rounding.

[0119] For denormalized numbers, the number of leading zeros in M ​​is detected and the constraints are modified to avoid the additional data path width caused by the denormalized numbers.

[0120] Example 4

[0121] This embodiment shows a correctly rounded multiple low-bit-width floating-point number fusion dot product addition calculation method based on mantissa data path analysis, and adopts a multiple low-bit-width floating-point number exponent comparison and sorting method.

[0122] Specifically, using different inputs and different classification operations, six 3-bit comparators are used to implement comparison or sorting operations on three low-bit-width floating-point numbers.

[0123] In the exponential part, FP4 and FP6 need to compare to obtain the maximum exponent of the dot product part, while FP8 needs to sort the dot product part. The exponent sum is obtained, and finally FP4 obtains 6 3-bit results, FP6 obtains 4 4-bit results, and FP8 obtains 3 6-bit results. In the exponent comparison part, FP4 and FP6 need to determine the maximum value of the exponent, while FP8 needs to sort.

[0124] Six 3-bit comparators are used in the comparison part. The input of each 3-bit comparator is two unsigned 3-bit numbers, and the output is used to judge whether input one is greater than input two and whether input one is equal to input two. Since the judgment that input one is less than input two can be derived from the results of greater than and equal to, the corresponding output is cancelled. Both FP4 and FP8 will use all 3-bit comparators, while FP6 will use 3 comparators. Both FP4 and FP6 divide the comparators into two levels, and the input of the second-level comparator is obtained by selecting the larger one of the output and input of the first-level comparator. FP8 is not graded and all comparators are used directly.

[0125] For FP4, the 6 inputs of the first stage are 6 3-bit exponent sums, and then the larger input is selected according to the comparator output to obtain 3 larger 3-bit exponent sums. These three exponent sums are input into the second-stage comparator for comparison. For FP6, only 3 comparators are used, and the highest bit of the 4 4-bit exponent sums is not compared. The input data is determined by judging whether the highest bit is 1. For FP8, the 6-bit exponent sum is cut into the upper 3 bits and the lower 3 bits, and the 3 upper 3 bits and the 3 lower 3 bits are compared in pairs, using exactly 6 comparators.

[0126] Example 5

[0127] This embodiment shows a specific implementation of using a shifting method to shift to the maximum exponent sum in mantissa shifting. Figure 2 shows the mantissa data path bit widths required for several different shifting methods under FP4 precision.

[0128] FIG2(a) shows a shifting method of the present invention, in which the mantissa of the dot product is shifted right, while the mantissa of C is shifted left or right depending on the difference with the maximum exponent sum. In the process, the shifting aligns the seven 8-bit mantissas, which are then accumulated in the conventional data path. In addition, C needs to be shifted left and added together with the carry input. FIG2(b) shows a shifting method for selecting the maximum exponent and full-size accumulation. In this case, there are two possibilities: the maximum exponent is the exponent of C, or a certain exponent. These two possibilities are marked with different colors in the figure. The large exponent range of BF16 requires a considerable data path width. FIG2(c) illustrates a shifting method for selecting the maximum exponent and compressed accumulation. Regarding the method of compressed accumulation, two possibilities are also included, one of which is to compress additional zeros between the mantissas. The required data path width is much higher than the method proposed in the present invention. Therefore, it is obvious that selecting the maximum exponent sum in FP4 mode is reasonable and effective. The situation is similar in FP6 mode.

[0129] In FP4 and FP6 modes, the calculation is done directly in the regular datapath without using the extra datapath. In order to correctly perform shift operations in the regular datapath, such as Figure 4As shown, 6 3-bit comparators are used in FP4 mode, and the 6 inputs of the first stage are 3-bit exponent sums. According to the comparator output, three larger values ​​are selected, and these three values ​​are input into the second-stage comparator in pairs, and the maximum exponent sum is selected through the output of the second-stage comparator. In FP6 mode, only 3 comparators are used, and 2 comparators are used in the first stage to process 4 3-bit inputs. The highest bit of the 4 4-bit exponent sums is not involved in the comparison, and it is determined whether the highest bit is 1. In the case where the highest bit is not all 0, if the highest bit is 1, the input of the comparator is the last 3 bits; if the highest bit is 0, the input of the comparator is 0. In the case where all the highest bits are 0, the last 3 digits are used normally. After the two comparators obtain two larger values, the comparison is completed in the second-stage comparator.

[0130] Embodiment 6:

[0131] This embodiment shows a correctly rounded multiple low-bit-width floating-point fusion dot product addition calculation device based on mantissa data path analysis running in FP8 mode.

[0132] The computing device in FP8 mode has the same structure as that in Embodiment 1, and both data paths in the mantissa data path module 112 are used. The special design of FP8 mode ensures that effective mantissa processing and correct rounding can be achieved, thereby improving computing performance and accuracy.

[0133] The FP8 mode has two precisions: FP8-E4M3 and FP8-E5M2. They are all in the E5M3 format during calculation. The mantissa multiplication result is 8 bits. The mantissa of C is also 8 bits, so they are processed uniformly during analysis. For normalized numbers, one of the first two digits of the mantissa must be 1. Figure 3 As shown, for D 12 The analysis assumes that the second bit of M1 is 1 and the first bit of M2 is also 1. When calculating M2, M3, and M4, a 2-bit carry will occur. Therefore, when D 12 >1+9+2, M2, M3, and M4 are placed in the extra module. Similarly, when D 12 ≤12 and D 23 >11, M3 and M4 are placed in the extra module; when D 12 ≤12, D 23 ≤11 and D 34 When >10, M4 is put into the extra module.

[0134] For denormalized numbers, the formula used is V = (-1) S ×2 E-bias ×(m y-1 .m y-2…m0). By detecting the leading zero of the denormalized number, determine how many zeros are in front of the denormalized number mantissa multiplication result compared with the normal multiplication result, and record it as Subd after sorting. i . Constraint D in the case of normalized numbers ij >N, changes to D in case of denormalized numbers ij >N+Subd i -Subd j , by modifying the constraints, the extra computational overhead for denormalized numbers is avoided.

[0135] like Figure 4 As shown in the figure, in FP8 mode, all comparators are used to split the 6-bit exponent sum into high 3 bits and low 3 bits. The 3 high 3 bits and 3 low 3 bits are compared in pairs, using exactly 6 comparators. For the 3 exponent sums, the high 3 bits and the low 3 bits are sorted at the same time. When the high 3 bits are equal, the sorting result of the low 3 bits is used.

[0136] Finally, it should be noted that the above examples are only specific embodiments of the present invention. Obviously, the present invention is not limited to the above examples. All variations that can be directly derived or associated with the contents disclosed by a person skilled in the art are considered to be within the protection scope of the present invention.

Claims

1. A device for calculating dot product addition by fusing multiple low-bit-width floating-point numbers, characterized in that: include: An input segmentation and detection module, whose input is connected to an external memory, and whose output is connected to a sign processing module, an exponent processing module, a mantissa processing operation module and an anomaly detection module, is used to segment the input data and detect the segmented data; A sign processing module, whose output is connected to the mantissa processing operation module, the mantissa data path module, the normal output module and the output module, for performing an XOR calculation on the sign bit; An exponential processing module, whose output is connected to the exponential adjustment module and the mantissa data path module, is used to calculate the shift amount required when shifting the order and the reference exponent required when adjusting the exponent; A mantissa processing operation module, whose output is connected to the mantissa data path module, is used for performing denormalized number processing, mantissa multiplication and mantissa complement operation, and outputting the mantissa in the form of complement; A mantissa data path module, whose output is connected to the leading zero detection module and the mantissa normalization module, is used for performing mantissa shift, mantissa compression and mantissa addition to obtain an intermediate result of the mantissa; A leading zero detection module, whose output is connected to a mantissa normalization module, is used to detect how many bits before the most significant bit in the mantissa intermediate result are 0; A mantissa normalization module, whose output is connected to the rounding module, is used to shift the mantissa according to the number of leading zeros and normalize it to the result before rounding; A rounding module, whose output is connected to the normal output module and the abnormal output module, is used to select the final rounding result according to the rounding rule, and at the same time determine whether there is a rounding carry, and output the relevant result; An exponent adjustment module, whose output is connected to the normal output module, is used to perform calculations according to the base exponent when shifting the order, the number of leading zeros, and whether there is a carry when rounding, so as to obtain an adjusted exponent; The abnormality detection module, whose output is connected to the abnormality output module, is used to determine whether the operation is abnormal based on the received input segmentation and the detection result output by the detection module, and output an abnormality flag; A common output module, whose output is connected to the output module, is used to generate a common calculated output according to the sign processing result, the mantissa output and the exponent adjustment result; An abnormal output module, whose output is connected to the output module, is used to generate abnormal output according to the abnormal flag in the abnormal detection; The output module, whose output is connected to the external memory, is used to generate the final output according to the normal output and abnormal output results.

2. The multiple low-bit-width floating-point number fusion dot product addition computing device according to claim 1, characterized in that: The computing device performs fusion dot product addition calculation based on the following formula: Among them (A i ) i∈[0,N-1] and (B i ) i∈[0,N-1] Use low-width floating-point numbers, C uses a floating-point number in BF16 format, and N represents the length of the dot product; the operation represented by this formula is recorded as FDPNA; The computing device supports the calculation of floating point numbers in three formats: FP8, FP6 and FP4, and performs operations in the form of FDP3A, FDP4A and FDP6A respectively; wherein FP8 includes two formats: E4M3 and E5M2; FP6 includes two formats: E2M3 and E3M2; FP4 is E2M1 format; E is the exponent bit, and M is the mantissa bit.

3. The multiple low-bit-width floating-point number fusion dot product addition computing device as claimed in claim 2, characterized in that: The input segmentation and detection module segments the input data and detects the segmented data, including: The input data is 64 bits, where A i &B i Use 48 bits, C uses 16 bits, and according to the floating point type datatype used, split the input data into each floating point number float_i, and split float_i into sign, exponent and mantissa; Check whether the exponent and mantissa of the split data are all 0 or all 1 to determine whether it is a denormalized number, NaN, Infinity or zero; The sign bit is output to the sign processing module, the exponent bit is output to the exponent processing module, the non-normalized number detection result and the mantissa bit are output to the mantissa processing operation module, and the detection result is output to the anomaly detection module. The symbol processing module performs XOR calculation on the sign bit, A i , B i Perform an XOR operation with C and perform mantissa data calculation based on the sign of C.

4. The multiple low-bit-width floating-point number fusion dot product addition computing device as claimed in claim 3, characterized in that: The exponential processing module includes an exponential summing unit, an exponential comparing and sorting unit, and an exponential differencing unit; Among them, when the exponential summation unit performs exponential summation, for A i and B i Direct calculation of the exponential For the exponent of C, calculate it according to the floating point type Note as bias dif , then calculate E C -bias dif , so that the exponent of C is consistent with the exponent sum, where bias represents the bias of the floating point number; The index comparison and sorting unit obtains the maximum value through comparison for FP4 and FP6; in FP8 mode, E C -bias dif Compare with the sum of the three indices to determine the position and obtain the final sorting result; The exponent difference unit calculates the difference between the maximum exponent sums for FP4 and FP6, and the difference between sorted adjacent exponents for FP8.

5. The multiple low-bit-width floating-point number fusion dot product addition computing device as claimed in claim 4, characterized in that: The mantissa processing operation module includes a denormalized number processing unit, a mantissa multiplication unit and a mantissa complement operation unit; The denormalized number processing unit is based on the modified denormalized number formula V=(-1) S ×2 E-bias ×(m y-1 .m y-2 …m0) is used for calculation, and y represents the mantissa width of the floating point number; The mantissa multiplication unit uses four 4-bit multipliers and two 2-bit multipliers to perform mantissa multiplication operations, wherein FP8 uses three 4-bit multipliers, FP6 uses four 4-bit multipliers, and FP4 uses all multipliers; The mantissa complement operation unit performs a complement operation on the mantissa according to the sign after the mantissa multiplication is completed, and performs shift filling using 0 or 1 according to the sign in a subsequent shift operation.

6. The multiple low-bit-width floating-point number fusion dot product addition computing device as claimed in claim 5, characterized in that: The mantissa data path module includes a mantissa shift unit, a mantissa compression unit and a mantissa addition unit, and has two sets of parallel modules, the conventional module is used for FP4, FP6 and FP8 operations, and the additional module is only used for FP8 operations; Among them, the mantissa shift unit uses a barrel shifter to perform shift alignment; The mantissa compression unit uses a 4-2 compressor and a 3-2 compressor to compress the mantissa into sum and carry; The mantissa addition unit adds sum and carry and determines whether the final result requires a complement operation based on the sign.

7. The multiple low-bit-width floating-point number fusion dot product addition computing device according to claim 6, characterized in that: The mantissa normalization module performs conventional rounding based on the following formula, and performs additional expansion on the formula in FP8 mode: M R =z0z -1 z -2 …z -(L-1) G&M B =M R RS M O =M R or(M R +1)according GRS Among them, z0 represents the most significant bit, and L represents the mantissa width of the output; For BF16, L is 7, and there are three extra bits for the reserved bit (G), the rounding bit (R), and the sticky bit (S). The sticky bit indicates whether there is a 1 in the subsequent bit; the final rounded result M O From M R and M R +1 is selected according to GRS rules; The mantissa normalization module normalizes the result to M B form; The rounding module calculates M R and M R +1, and select M according to GRS rules O , and at the same time determine whether there is rounding carry, detect whether there is an inexact exception, and finally output the relevant results.

8. A method for calculating low-bit-width floating-point fusion dot product addition, based on the computing device according to any one of claims 1 to 7, characterized in that: The shifting method for FP4 and FP6 precision is optimized, and the calculation method of shifting to the maximum exponent sum is used for mantissa shift calculation.

9. The low-bit-width floating-point number fusion dot product addition calculation method according to claim 8, characterized in that: For FP8, an additional mantissa data path is added in addition to the regular mantissa data path to perform mantissa allocation to ensure correct rounding. The specific method is as follows: Adding additional shift, compression and addition units in the mantissa data path module to form an additional data path in parallel with the regular data path; Sort the exponents and assign mantissas according to the difference between adjacent exponents. Divide the four mantissas into two data paths. The mantissa with the largest exponent is placed in the regular data path, and the remaining mantissas are placed according to the difference between adjacent exponents. When sorting the exponents, the mantissa corresponding to the i-th sorted exponent is represented as M. i , and calculate D ij , which represents the difference between the i-th and j-th sorted indices, and relative to M i M that needs to be shifted j The number of i When D is 0, the index is considered to be the minimum; ij The number greater than the leading zero plus M R When the number of R is added to the number of carries, the subsequent mantissa bits are placed in the extra data path; When the exponent difference in the additional data path exceeds a preset value, a sticky bit is added in the additional data path to retain the overflowed bit; When performing rounding calculations, the rounding formula is expanded as follows: M R =z0z -1 z -2 …z -(L-1) G&M B =M R RSC N O =M R or(M R +1)according GRSC Among them, C is the sign consistency bit, which is used to indicate whether the sign of S is consistent with M O The symbol matches.

10. The low-bit-width floating-point number fusion dot product addition calculation method according to claim 8 or 9, characterized in that: The comparison and sorting methods for low-width floating-point exponents are as follows: Using different inputs and different classification operations, 6 3-bit comparators are used to implement comparison or sorting operations on three low-width floating-point numbers. The input of each 3-bit comparator is two unsigned 3-bit numbers, and the output is used to determine whether input one is greater than input two and whether input one is equal to input two; FP4 and FP8 use all 3-bit comparators, and FP6 uses 3 comparators; FP4 and FP6 both divide the comparators into two levels, and the input of the second-level comparator is obtained by selecting the larger one of the output and input of the first-level comparator; FP8 is not classified and all comparators are used directly.

Images