Hardware accelerator for floating point neural network and application method thereof

Abstract

The invention discloses a hardware accelerator for a floating point neural network and an application method thereof, the hardware accelerator comprises a weight buffer, an activation value buffer, a partial sum buffer and a two-dimensional space array formed by a PE group, the PE group comprises an FP32 floating point accumulator and a plurality of processing units PE sharing the FP32 floating point accumulator, the processing unit PE comprises a dot product unit, a low-precision accumulator, a WinFloat decoder, a data distributor, a first-in first-out queue FIFO and a floating-point number register, and the index bit width and the index bias of the low-precision accumulator can be configured to be used for processing data of different window types. The invention aims to provide the hardware accelerator for the floating-point neural network, which can process data of different window types and is small in chip area overhead, aiming at hierarchical accumulation of the floating-point neural network, and a corresponding application method is provided aiming at the hardware accelerator so as to realize optimization of a window floating-point format and window selection.

Description

Technical Field

[0001] The present invention relates to the technical field of hardware accelerators for integrated circuits, and particularly relates to a hardware accelerator for a floating-point neural network and an application method thereof. Background Art

[0002] In recent years, deep neural networks (DNNs) have achieved great development, and at the same time, they have brought huge storage and computing requirements. To reduce storage and computing overheads, a variety of model compression strategies have been proposed. Among them, vector scaling is a very promising solution. Vector scaling quantizes tensors at the vector granularity, converting a set of floating-point data into a set of signed integers and a shared exponent (i.e., block floating-point), thereby converting complex floating-point operations (alignment shift, normalization processing) into integer (INT) operations within the vector. The data type of int within the vector is of short bit width, thus greatly reducing storage and computing overheads, and therefore has been widely studied.

[0003] However, there are still frequent FP32 accumulations between vectors, which limits the further reduction of energy consumption. The latest work sums the vector products grouped by exponent magnitude. For example, BFP (Block Floating-Point) is a numerical representation method for deep neural networks (DNNs), aiming to improve computing efficiency and reduce hardware complexity by optimizing data types. A large number of studies have defined their own data types based on BFP, such as Flexpoint, BSFP, Flint, and MSFP. Their differences lie in the exponent and mantissa bit widths. For example, the exponent of MSFP is fixed at 8 bits, and the mantissa can be statically configured as 2 bits, 3 bits, or 4 bits. The exponent and mantissa bit widths of Flint can be dynamically configured during operation. However, even for the data types of BFP, FP32 (32-bit single-precision floating-point) accumulations are still required when summing between vectors, which limits the further reduction of energy consumption and area. The accumulator of the FAST architecture (FAST accumulator) is an accelerator based on the traditional BFP format. Its mantissa bit width can be configured as 2 bits or 4 bits according to different layers and supports stochastic rounding. The accumulator of the Bucket architecture (Bucket accumulator) is based on the MSFP data format. Before FP32 accumulation, the addends are grouped according to the exponent, so that the exponent order difference within the group is smaller, thereby reducing the shifter and adder, but the overhead of the in-group registers and grouping logic increases cannot be ignored. As Figure 1As shown, the accumulator of the Bucket architecture (Bucket accumulator) adds a level of local accumulator (INT-Acc) in front of the floating-point accumulator (FP-Acc) in the traditional FAST architecture. The operands are grouped according to the exponent, and those with similar exponents are grouped together, thus reducing the alignment shift overhead within the group. However, the floating-point accumulator is still required between groups, and a large number of registers are added to save the data within the group and the grouping logic. Figure 1 In it, IDot represents the dot product unit. The comparison of the area and energy consumption of the FAST architecture and the accumulator of the Bucket architecture is as Figure 2 shown. It can be seen that the Bucket architecture has a larger area but lower energy consumption.

[0004] Figure 3 Figure is a schematic diagram of the single-precision floating-point data format of the IEEE-754 standard in the prior art. Each element of the single-precision floating-point data format of the IEEE-754 standard consists of 1 sign bit, 8 exponent bits, and 23 mantissa bits, with a large representation range (2 8 ) and high data precision (1 / 2 23 ), while bringing a huge hardware overhead. Vector scaling groups tensors by vectors, and a shared scaling factor is extracted for each group. The format between groups is a short-bitwidth INT (integer) format. MSFP is one of the typical representatives. Its scaling factor is 8 bits and is a power of 2, thus simplifying the hardware design. For the main computational load of deep neural networks (DNNs) being matrix multiplication, its computational process is shown as follows:

[0005] C M×N = A M×K · W K×N ,

[0006] where A M×K is the input matrix, W K×N is the weight matrix, C M×N is the output matrix, M and K are the number of rows and columns of the input matrix, N is the number of columns of the output matrix. After the input matrix A M×K , the weight matrix W K×N and the output matrix C M×N are grouped by the common dimension g, the following can be obtained: and c mn , and there is:

[0007]

[0008] Among them, m and n are the number of rows and columns of the input matrix respectively, and k is the number of columns of the output matrix. To maintain the model accuracy, the group size cannot be too large. The common dimension g is usually 16 (g = 16), and the accumulation between groups needs to maintain the FP32 accuracy. Although the in-group is a simple INT dot product, the frequent FP32 accumulation between groups (MN(K - 1) / g) will result in a large power consumption overhead, making it an urgent problem to be solved. Summary of the Invention

[0009] The technical problem to be solved by the present invention: Aiming at the above problems of the prior art, a hardware accelerator for a floating-point neural network and its application method are provided. The present invention aims to provide a hardware accelerator for a floating-point neural network that can process data of different window types and has a small chip area overhead for hierarchical accumulation of floating-point neural networks, and provide a corresponding application method for the hardware accelerator to optimize the window floating-point format and window selection.

[0010] To solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0011] A hardware accelerator for a floating-point neural network, comprising a weight buffer, an activation value buffer, a partial sum buffer, and a two-dimensional spatial array composed of a PE group. The PE group includes an FP32 floating-point accumulator and multiple processing units PE that share the FP32 floating-point accumulator. The processing unit PE includes a dot product unit, a low-precision accumulator, a WinFloat decoder, a data distributor, a first-in-first-out queue FIFO, and a floating-point register. The dot product unit is used to perform a dot product operation on the mantissa m of the input activation value and weight. The output of the dot product unit generates two outputs through the first WinFloat decoder: one output serves as the first input of the first data distributor, and the other output serves as the first input of the low-precision accumulator. The output of the low-precision accumulator serves as the input of the second WinFloat decoder, and the output of the second WinFloat decoder serves as the second input of the first data distributor. The input partial sum forms two outputs after entering the third WinFloat decoder and serves as the first input of the second data distributor and the third data distributor respectively. The second input of the second data distributor is the output of the second WinFloat decoder, and the second input of the third data distributor is the result output by the floating-point register. The output of the second data distributor serves as the second input of the low-precision accumulator. The sum of the exponents e of the input activation value and weight serves as the third input of the low-precision accumulator. The outputs of the first data distributor and the third data distributor are concatenated and serve as the input of the first-in-first-out queue FIFO. The first-in-first-out queue FIFO, the FP32 floating-point accumulator, and the floating-point register are connected in sequence to output the result through the floating-point register. The exponent bit width and exponent bias of the low-precision accumulator are configurable to process data of different window types.

[0012] Optionally, the WinFloat decoder includes two NOT gates, one AND gate, one multiplexer, and two adders. After the exponent of the addend passes through the first NOT gate, it is added to the values of the window left interval and the window right interval through two adders respectively. The result of adding the value of the window right interval passes through the second NOT gate and then enters the AND gate with the result of adding the value of the window left interval. The output of the AND gate serves as the control signal of the multiplexer. If the control signal is 0, the addend is sent to the low-precision accumulator. If the control signal is 1, the addend is sent to the FP32 floating-point accumulator.

[0013] Optionally, the low-precision accumulator includes an exponent difference calculation module, a mantissa exchange module, two alignment shifters, an adder, and a normalization module. The exponent difference calculation module is used to calculate the exponent order difference of the two input exponents e a and e b . The mantissa exchange module is used to exchange the two input mantissas m a and mb Perform mantissa swapping to obtain a larger mantissa. Two cascaded alignment shifters are used to perform alignment shifting on the larger mantissa with an exponent bit width of 1 to 3, and bypass the second-stage alignment shifter when the exponent bit width of the larger mantissa is 4 to obtain a smaller aligned mantissa. Then, sum the smaller aligned mantissa and the larger mantissa through an adder. The normalization module is used to normalize the exponent difference, the smaller aligned mantissa, and the sum result of the larger mantissa and output the result.

[0014] In addition, the present invention also provides an application method for the aforementioned hardware accelerator for floating-point neural networks, including the following steps: using the hardware accelerator for floating-point neural networks to process the accumulation calculation of floating-point data inside and outside the window, and selecting a suitable window according to different data distributions through a preset window search algorithm to cover as much data as possible with a window as small as possible.

[0015] Optionally, when using the hardware accelerator for floating-point neural networks to process the accumulation calculation of floating-point data inside and outside the window, the data format of the floating-point data is the window floating-point format shown in the following formula:

[0016] Sign×2 Exponent-Bias ×1.Mantissa,

[0017] In the above formula, Sign is the sign bit of the floating-point data, Exponent is the exponent of the floating-point data, Bias is the exponent bias of the floating-point data, and Mantissa is the mantissa of the floating-point data.

[0018] Optionally, when using the hardware accelerator for floating-point neural networks to process the accumulation calculation of floating-point data inside and outside the window, the bit width of the mantissa of the floating-point data is the same as that of the 32-bit floating-point number FP32.

[0019] Optionally, before using the hardware accelerator for floating-point neural networks to process the accumulation calculation of floating-point data inside and outside the window, it further includes reading the configuration values of the exponent bit width and exponent bias of the floating-point data to determine the data representation range of the floating-point data according to the configuration values of the exponent bit width and exponent bias, and the data representation range of the floating-point data is compatible with the 32-bit floating-point number FP32.

[0020] Optionally, the selecting a suitable window according to different data distributions through a preset window search algorithm includes:

[0021] S1, extracting the exponent AExp of the activation value matrix and the exponent WExp of the weight matrix for the floating-point neural network;

[0022] S2, Multiply each row of the exponent AExp of the activation value matrix by the corresponding elements in each column of the exponent WExp of the weight matrix to obtain the element-wise product result AWEleExp;

[0023] S3, Initialize the exponent width ExpW, the bias variable FBias, and the temporary bias variable BiasT. The initial value of the exponent width ExpW is 2, and the initial values of the bias variable FBias and the temporary bias variable BiasT are both the value with the highest frequency in the element-wise product result AWEleExp;

[0024] S4, Calculate the number of elements EleInWin in the interval window according to the exponent width ExpW and the bias variable FBias;

[0025] S5, Increment the bias variable FBias by 1 and calculate the number of elements EleInWinT in the current interval;

[0026] S6, Determine whether the number of elements EleInWin in the interval window is less than the number of elements EleInWinT in the current interval. If it holds, assign the number of elements EleInWinT in the current interval to the number of elements EleInWin in the interval window and jump to step S5; otherwise, jump to step S7;

[0027] S7, Decrement the bias variable FBias by 1 and calculate the number of elements EleInWinT in the current interval;

[0028] S8, Determine whether the number of elements EleInWin in the interval window is less than the number of elements EleInWinT in the current interval. If it holds, assign the number of elements EleInWinT in the current interval to the number of elements EleInWin in the interval window and jump to step S7; otherwise, jump to step S9;

[0029] S9, Determine whether the result obtained by dividing the number of elements EleInWin in the interval window by the number of elements EleInWinT in the current interval is less than the preset threshold T. If it does not hold, increment the exponent width ExpW by 1, assign the bias variable FBias to the temporary bias variable BiasT, and jump to step S4; otherwise, jump to step S10;

[0030] S10, Use the exponent width ExpW as the exponent width Exponent of the floating-point data and the bias variable FBias as the exponent bias Bias of the floating-point data to determine the selected window size and position. The exponent width ExpW is used to determine the window size, and the bias variable FBias is used to determine the window position. Encode the activation values and weights of the floating-point neural network into the window floating-point format for subsequent calculations.

[0031] Optionally, when calculating the number of elements EleInWin in the interval window according to the exponent bit width ExpW and the bias variable FBias in step S4, the calculation function expression of the corresponding interval window is as follows:

[0032] Win = [FBias - 2 ExpW-1 - 1, FBias + 2 ExpW-1 - 1],

[0033] where Win is the interval window corresponding to the number of elements EleInWin, FBias is the bias variable, and ExpW is the exponent bit width.

[0034] Optionally, the function expression for calculating the number of elements EleInWinT in the current interval in step S5 is as follows:

[0035] EleInWinT = ((AWEleExp >= (FBias - 2 ExpW-1 - 1)) & (AWEleExp <= (FBias + 2 ExpW-1 - 1))).sum(),

[0036] where AWEleExp is the per-element product result, FBias is the bias variable, ExpW is the exponent bit width, & is the logical AND operation, sum() is the summation function, and the right side of the above function expression is used to calculate the number of values in the per-element product result AWEleExp that are between FBias - 2 ExoW-1 - 1 and FBias + 2 ExpW-1 - 1 as WleInWinT.

[0037] Compared with the prior art, the present invention mainly has the following advantages: The hardware accelerator for floating-point neural networks of the present invention can be configured with the exponent bit width and exponent bias of a low-precision accumulator to process data of different window types. A low-precision accumulator with a relatively narrow representation range is used to process dense normal values, and an FP32 floating-point accumulator is reserved to process long-tailed outliers. Therefore, it can achieve floating-point neural network hardware acceleration of a precision-aware accumulator, and process data inside and outside the window with the smallest hardware overhead (including area and power consumption), having the advantages of being able to process data of different window types and having a small chip area overhead. Description of the Drawings

[0038] Figure 1 It is a schematic structural diagram of an accumulator of the Bucket architecture in the prior art.

[0039] Figure 2 It is a schematic diagram of the area and energy consumption comparison between the FAST architecture and the accumulator of the Bucket architecture in the prior art.

[0040] Figure 3 It is a schematic diagram of the single-precision floating-point data format of the IEEE-754 standard in the prior art.

[0041] Figure 4 It is a schematic diagram of the hardware accelerator structure in the embodiment of the present invention.

[0042] Figure 5 It is a schematic diagram of the structure of the PE group in the embodiment of the present invention.

[0043] Figure 6 It is a schematic diagram of the data distribution of activation values, weight values, and product values in a floating-point neural network.

[0044] Figure 7 It is a schematic diagram of the structure of the WinFloat decoder in the embodiment of the present invention.

[0045] Figure 8 It is a schematic diagram of the structure of the low-precision accumulator in the embodiment of the present invention.

[0046] Figure 9 It is a schematic diagram of the window floating-point format with different configurations in the embodiment of the present invention, where (a) is the mantissa and (b) is the exponent.

[0047] Figure 10 It is a schematic diagram of the accumulation process with different strategies in the embodiment of the present invention.

[0048] Figure 11 It is a schematic diagram of the basic process of the window search algorithm in the embodiment of the present invention.

[0049] Figure 12 It is the configuration of WinFloat in BERT and its corresponding FP-Acc activation rate in the embodiment of the present invention.

[0050] Figure 13 It is a schematic diagram of the activation analysis results of the main components in the embodiment of the present invention.

[0051] Figure 14 It is a schematic diagram of the energy consumption comparison of different architectures in the embodiment of the present invention.

[0052] Figure 15 It is a schematic diagram of the performance comparison of different FP-Acc sharings and the PE area at the optimal FP-Acc sharing in the embodiment of the present invention, where (a) is the normalized performance of different FP-Acc sharings and (b) is the PE area at the optimal FP-Acc sharing.

[0053] Legend: 1. Weight buffer; 2. Activation value buffer; 3. Partial sum buffer; 4. PE group; 41. FP32 floating-point accumulator; 42. Processing unit PE; 421. Dot product unit; 422. Low-precision accumulator; 423. WinFloat decoder; 424. Data distributor; 425. First-in-first-out queue FIFO; 426. Floating-point register. Detailed implementation

[0054] To enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions of the present invention will be further described in detail below with reference to the accompanying drawings in the embodiments of the present invention.

[0055] Such as Figure 4 And Figure 5As shown in the figure, the hardware accelerator for the floating-point neural network in this embodiment includes a weight buffer 1, an activation value buffer 2, a partial sum buffer 3, and a two-dimensional space array composed of a PE group 4. The PE group 4 includes an FP32 floating-point accumulator 41 and multiple processing units PE42 that share the FP32 floating-point accumulator 41. The processing unit PE42 includes a dot product unit 421, a low-precision accumulator 422, a WinFloat decoder 423, a data distributor 424, a first-in-first-out queue FIFO425, and a floating-point register 426. The dot product unit 421 is used to perform a dot product operation on the mantissa m of the input activation value and weight. The output of the dot product unit 421 generates two outputs through the first WinFloat decoder 423: one output serves as the first input of the first data distributor 424, and the other output serves as the first input of the low-precision accumulator 422. The output of the low-precision accumulator 422 serves as the input of the second WinFloat decoder 423, and the output of the second WinFloat decoder 423 serves as the second input of the first data distributor 424. After the input partial sum enters the third WinFloat decoder 423, it forms two outputs and serves as the first inputs of the second data distributor 424 and the third data distributor 424 respectively. The second input of the second data distributor 424 is the output of the second WinFloat decoder 423, and the second input of the third data distributor 424 is the result output by the floating-point register 426. The output of the second data distributor 424 serves as the second input of the low-precision accumulator 422, and the sum of the exponents e of the input activation value and weight serves as the third input of the low-precision accumulator 422. The outputs of the first data distributor 424 and the third data distributor 424 are concatenated and serve as the input of the first-in-first-out queue FIFO425. The first-in-first-out queue FIFO425, the FP32 floating-point accumulator 41, and the floating-point register 426 are connected in sequence to output the result through the floating-point register 426. The exponent bit width and exponent bias of the low-precision accumulator 422 are configurable to process data of different window types. Figure 6 It is a schematic diagram of the data distribution of the activation value, weight, and product value of the floating-point neural network in the prior art. According to Figure 6It can be seen that the weights and activation values in the floating-point neural network follow a Laplace-like distribution, and the element-wise product terms that need to be accumulated in the load of the floating-point neural network also exhibit a Laplace-like distribution. Although the dynamic range is very wide, the data distribution is very concentrated (long tail). For floating-point addition, a large dynamic range requires a large shifter for alignment and then a large adder for summation, resulting in a large hardware overhead. Therefore, in this embodiment, the hardware accelerator for the floating-point neural network can be configured with the exponent width and exponent bias of the low-precision accumulator 422 to process data of different window types. A low-precision accumulator with a relatively narrow representation range is used to process dense normal values, and the FP32 floating-point accumulator 41 is reserved to process long-tail outliers, so as to achieve hardware acceleration of the floating-point neural network with a precision-aware accumulator and process data inside and outside the window with the smallest hardware overhead (including area and power consumption).

[0056] To distribute data between the low-precision accumulator 422 and the high-precision accumulator (FP32 floating-point accumulator 41), we designed a hardware-friendly WinFloat decoder 423, as Figure 7 shown, the WinFloat decoder 423 in this embodiment includes two NOT gates, one AND gate, one multiplexer, and two adders. After the exponent of the addend passes through the first NOT gate, it is added to the values of the left window interval and the right window interval through two adders respectively. And the result of adding the value of the right window interval passes through the second NOT gate and then enters the AND gate with the result of adding the value of the left window interval. The output of the AND gate serves as the control signal of the multiplexer. If the control signal is 0, the addend is sent to the low-precision accumulator 422, and if the control signal is 1, the addend is sent to the FP32 floating-point accumulator 41. The WinFloat decoder 423 in this embodiment only requires NOT gates, AND gates, multiplexers, and two adders. The inputs are the left and right intervals of the window and the exponent. If the exponent of the addend is within the window, the low-precision accumulator 422 is selected, otherwise the FP32 floating-point accumulator 41 is selected.

[0057] The low-precision accumulator 422 is used for summing data within the window. Compared with the standard FP32 floating-point accumulator, they have a similar structure, but their exponent order difference is smaller, so the shifter and adder are smaller. As Figure 8 shown, the low-precision accumulator 422 in this embodiment includes an exponent difference calculation module, a mantissa exchange module, two alignment shifters, one adder, and one normalization module. The exponent difference calculation module is used to calculate the exponent order difference for the two input exponents e a and e b , and the mantissa exchange module is used to exchange the two input mantissas m a and m bPerform mantissa swapping to obtain a larger mantissa. Two alignment shifters are cascaded to perform alignment shifting on the larger mantissa with an exponent bit width of 1 to 3, and bypass the second-stage alignment shifter when the exponent bit width of the larger mantissa is 4 to obtain a smaller aligned mantissa. Then, sum the smaller aligned mantissa and the larger mantissa through an adder. The normalization module is used to normalize the exponent difference, the smaller aligned mantissa, and the sum result of the larger mantissa and output the result.

[0058] In addition, this embodiment also provides an application method for the aforementioned hardware accelerator for floating-point neural networks, including the following steps: using the hardware accelerator for floating-point neural networks to process the accumulation calculation of floating-point data inside and outside the window, and selecting a suitable window according to different data distributions through a preset window search algorithm to cover as much data as possible with a window as small as possible.

[0059] In this embodiment, when using the hardware accelerator for floating-point neural networks to process the accumulation calculation of floating-point data inside and outside the window, the data format of the floating-point data is the window floating-point format (WinFloat format) shown in the following formula:

[0060] Sign×2 Exponent-Bias ×1.Mantissa,

[0061] In the above formula, Sign is the sign bit of the floating-point data, Exponent is the exponent of the floating-point data, Bias is the exponent bias of the floating-point data, and Mantissa is the mantissa of the floating-point data. The exponent bit width and exponent bias of the low-precision accumulator 422 can be configured to process data of different window types, corresponding to Exponent and Mantissa being configurable. Figure 9 This is a schematic diagram of the window floating-point format with different configurations in this embodiment, where (a) is the mantissa and (b) is the exponent. Figure 9In ExBy, Exponent has x bits and Bias is y. For example, E8B127 means Exponent has 8 bits and Bias is 127. Therefore, E8B127 is FP32 with an exponent range of [-126, 127]; E4B7 and E3B3 are similar to the IEEE-754 standard, but with a narrower exponent range, [-6, 7] and [-2, 3] respectively; the exponent range of E4B2 is [-1, 12]. Therefore, the window floating-point format (WinFloat format) can adjust its data representation range by configuring the exponent bit width and exponent bias, and is also compatible with the standard FP32 format. In this embodiment, when the hardware accelerator for the floating-point neural network is used to process the accumulation calculation of floating-point data inside and outside the window, the bit width of the mantissa of the floating-point data is the same as that of the 32-bit floating-point number FP32. Therefore, the window floating-point format (WinFloat format) has the same precision as the floating-point number FP. In this embodiment, before using the hardware accelerator for the floating-point neural network to process the accumulation calculation of floating-point data inside and outside the window, it also includes reading the configuration values of the exponent bit width and exponent bias of the floating-point data to determine the data representation range of the floating-point data according to the configuration values of the exponent bit width and exponent bias, and the data representation range of the floating-point data is compatible with the 32-bit floating-point number FP32.

[0062] The following uses an example of calculating to illustrate the advantages of the WinFloat data format, where a i , w i are both in the window floating-point format (WinFloat format). Figure 10 is a schematic diagram of the accumulation process of different strategies in the embodiment of the present invention. Among them, (a) is an example of accumulation, where elements are added pairwise in order, and 5 - 1 = 4 times of FP32 accumulation are required. (b) is the FAST accumulation strategy, that is, the accumulation strategy adopted by the accumulator of the FAST architecture; (c) is the Bucket accumulation strategy, that is, the accumulation strategy adopted by the accumulator of the Bucket architecture. The product terms are divided into 3 groups according to the exponent size (the more data, the wider the distribution range, the more groups). 3 registers are needed to save the intermediate results within the group, and 2 times of FP32 accumulation are needed between groups. (d) is the window-based accumulation strategy (WinAcc), that is, the accumulation strategy in the method of this embodiment. WinAcc divides the data into two parts, inside and outside the window, with a minimum window according to the data distribution. 3 low-precision accumulations are performed inside the window, and only 1 register is needed to save the intermediate result, and then 2 times of FP32 accumulation are performed outside the window. Compared with the example of accumulation (a) as the baseline, WinAcc requires fewer times of FP32 accumulation. Compared with the Bucket accumulation strategy (c), WinAcc requires fewer intermediate registers.

[0063] The window search algorithm is used for the size and position of the encoding window. As Figure 11 shown, in this embodiment, the appropriate window is selected according to different data distributions through the preset window search algorithm, including:

[0064] S1. Extract the exponent AExp of the activation value matrix and the exponent WExp of the weight matrix for the floating-point neural network;

[0065] S2. Multiply the corresponding elements of each row of the exponent AExp of the activation value matrix and each column of the exponent WExp of the weight matrix to obtain the element-by-element product result AWEleExp;

[0066] S3. Initialize the exponent bit width ExpW, the bias variable FBias, and the temporary bias variable BiasT. The initial value of the exponent bit width ExpW is 2, and the initial values of the bias variable FBias and the temporary bias variable BiasT are both the value with the highest frequency in the element-by-element product result AWEleExp;

[0067] S4. Calculate the number of elements EleInWin in the interval window according to the exponent bit width ExpW and the bias variable FBias;

[0068] S5. Increment the bias variable FBias by 1 and calculate the number of elements EleInWinT in the current interval;

[0069] S6. Determine whether the number of elements EleInWin in the interval window is less than the number of elements EleInWinT in the current interval. If it holds, assign the number of elements EleInWinT in the current interval to the number of elements EleInWin in the interval window and jump to step S5; otherwise, jump to step S7;

[0070] S7. Decrement the bias variable FBias by 1 and calculate the number of elements EleInWinT in the current interval;

[0071] S8. Determine whether the number of elements EleInWin in the interval window is less than the number of elements EleInWinT in the current interval. If it holds, assign the number of elements EleInWinT in the current interval to the number of elements EleInWin in the interval window and jump to step S7; otherwise, jump to step S9;

[0072] S9. Determine whether the result obtained by dividing the number of elements EleInWin in the interval window by the number of elements EleInWinT in the current interval is less than the preset threshold T. If it does not hold, increment the exponent bit width ExpW by 1, assign the bias variable FBias to the temporary bias variable BiasT, and jump to step S4; otherwise, jump to step S10;

[0073] S10. Use the exponent width ExpW as the exponent width Exponent of the floating-point data and the bias variable FBias as the exponent bias Bias of the floating-point data to determine the selected window size and position, where the exponent width ExpW is used to determine the window size, the bias variable FBias is used to determine the window position, and encode the activation values and weights of the floating-point neural network into the window floating-point format for subsequent calculations.

[0074] In step S4 of this embodiment, when calculating the number of elements EleInWin in the interval window according to the exponent width ExpW and the bias variable FBias, the calculation function expression of the corresponding interval window is:

[0075] Win = [FBias - 2 ExoW-1 - 1, FBias + 2 ExpW-1 - 1],

[0076] where Win is the interval window corresponding to the number of elements EleInWin, FBias is the bias variable, and ExpW is the exponent width.

[0077] In step S5 of this embodiment, the function expression for calculating the number of elements EleInWinT in the current interval is:

[0078] EleInWinT = ((AWEleExp >= (FBias - 2 ExpW-1 - 1)) & (AWEleExp <= (FBias + 2 ExpW-1 - 1))).sum(),

[0079] where AWEleExp is the element-wise product result, FBias is the bias variable, ExpW is the exponent width, & is the logical AND operation, sum() is the summation function, and the right side of the above function expression is used to calculate the number of elements in the element-wise product result AWEleExp whose values are between FBias - 2 ExpW-1 - 1 and FBias + 2 ExpW-1 - 1 as WleInWinT.

[0080] To verify the hardware accelerator (WinAcc) for floating-point neural networks in this embodiment, WinAcc was compared and evaluated with two state-of-the-art (SOTA) techniques, namely the FAST accumulator and the Bucket accumulator, in this embodiment. For consistency, the implementation of the FAST accumulator was simplified in this embodiment, omitting variable precision and stochastic rounding techniques. To further evaluate WinAcc, seven comparison architectures were adopted: WA-3, WA-4, WA-3-NoA, WA-4-NoA, WA-3-NoS, WA-4-NoS, and WA-NoS. Among them, "-3" and "-4" represent the exponent bitwidth of LC-Acc (low-precision accumulator 422), "-NoA" means not adopting the AutoWin algorithm (window search algorithm), and "-NoS" means not sharing FP-Acc (FP32 floating-point accumulator 41). For the hardware, all architectures were implemented in Verilog in this embodiment, and a clock frequency of 1 GHz was achieved under the 12-nm CMOS process. A static timing analysis tool was used to obtain the power consumption at the module level. Additionally, we also implemented a cycle-accurate simulator. Specifically, in this embodiment, the Conv2D and Linear operators of PyTorch were extended to track the activation counts of different computing components, and the energy overhead was obtained by combining the power consumption information at the module level. The experiments focused on seven typical DNN models, which covered CNN-based models and Transformer-based models, specifically including: ResNet-34, ResNet-50, VGG16, MobileNetV2, ViT (Vision Transformer), BERT, and Llama-2, as shown in Table 1.

[0081] Table 1: Applications and Datasets of Models

[0082]

[0083] To evaluate the reduction of FP32 activations by the window search algorithm, experimental measurements were conducted in this embodiment on the layer-by-layer WinFloat configurations and their corresponding FP-Acc activation rates under each network model in Table 1. Due to space limitations, Figure 12 only the exponent configurations of each layer of BERT and the corresponding FP-Acc activation percentages are shown. For layers with larger data, such as layers 1 and 2, the exponent bitwidth configuration is 4, and the FP32 activation ratio is low; for layers with smaller data, such as layers 3 and 4, the exponent bitwidth configuration is 3, and the FP-Acc activation ratio is higher. The experimental results show that for different workloads, the window search algorithm can find the optimal configuration of the window floating-point format (WinFloat format), thus effectively reducing the energy overhead.

[0084] In this embodiment, the activation ratios of the main computing components of three accumulators, namely FAST, Bucket, and WinAcc, are recorded on the simulator for 7 models, including IDot, LP-Acc, FP-Acc, and FP-Reg. To better demonstrate the advantages of the hardware accelerator (WinAcc) for floating-point neural networks in this embodiment, WA-3, WA-4, WA-3-NoA, and WA-4-NoA are also included in the experiment, as Figure 13 shown. The FAST accumulator frequently activates FP32, the Bucket accumulator has a very low usage rate of FP32, but the hardware overhead of LC-Acc is relatively large. The FP-Acc of the hardware accelerator (WinAcc) for floating-point neural networks in this embodiment is significantly lower than that of the FAST accumulator under all loads, and lower than that of the Bucket accumulator under models with a larger dynamic range, such as ResNet34 and ResNet50. WA-3 and WA-4 have the same hardware overhead as WA-3-NoA and WA-4-NoA, but the usage rate of FP32 by WA-3 and WA-4 is significantly lower than that of WA-3-NoA and WA-4-NoA, thanks to the AutoWin algorithm (window search algorithm).

[0085] Figure 14 shows the energy consumption of each design under different models. The FAST accumulator has the highest energy consumption because of its frequent use of the FP32 accumulator. Although the usage rate of the Bucket accumulator for FP32 is low, the energy consumption of its LC-Acc is relatively high, so its overall energy consumption is also high. Although WA-3 / 4 and WA-3 / 4-NoA have the same hardware overhead, WA-3 / 4 combines the AutoWin algorithm (window search algorithm), significantly reducing the usage rate of FP-Acc, and its energy consumption is also lower. For example, the energy consumption of Win-3 is 8.4 - 36.84% lower than that of WA-3-NoA. The hardware accelerator (WinAcc) for floating-point neural networks in this embodiment has the lowest energy consumption because it achieves the best balance between the activation rate of FP-Acc and the hardware overhead. Its energy consumption is 37.42 - 68.87% lower than that of the FAST accumulator and 13.58 - 33.95% lower than that of the Bucket.

[0086] Figure 15 In (a) of shows the impact of the number of PEs shared by 1 FP32 accumulator and the depth of the LP accumulator queue on performance. The more PEs are shared, the smaller the area, but the more frequent the FP32 blockage. This can be alleviated by increasing the depth of the LP accumulator queue, but it also increases the area. When the number of shared PEs is greater than 4, it is difficult to obtain an ideal performance even by increasing a large number of queues; when the number of shared PEs is 2, only a small amount of queue overhead is required to obtain a very low performance loss. Figure 15In (b), the areas between different designs are compared when the number of shared PEs is 2. WA-3-NoS adds a level of LC-Acc accumulator compared to the FAST accumulator. The accumulator of WA-4-NoS supports an exponent bitwidth larger than that of WA-3, and has a larger shifter and adder. WA-NoS further adds selection and bypass logic, making the exponent configurable, and its area increases by 15.5% compared to FAST. By adding a small amount of FIFO to share FP-Acc, the area is reduced by 33.3%. The final area of the hardware accelerator (WinAcc) for the floating-point neural network in this embodiment is reduced by 22.9% compared to FAST and by 9.5% compared to the Bucket accumulator.

[0087] It can be seen that the hardware accelerator (WinAcc) for the floating-point neural network in this embodiment utilizes a narrow-range low-overhead accumulator LC-Acc (low-precision accumulator 422) and FP-Acc (FP32 floating-point accumulator 41), and an intermediate data representation format WinFloat, and can effectively process the computational load of BFP, significantly improving energy efficiency and reducing chip area. The main insight is that the element-wise product terms in the floating-point neural network follow a Laplace-like distribution, enabling most data to be represented by a narrower dynamic bitwidth format, while using FP-Acc (FP32 floating-point accumulator 41) to process the long-tail data. The experimental results show that the hardware accelerator (WinAcc) for the floating-point neural network in this embodiment is significantly superior to other architectures in terms of energy consumption and area, while maintaining extremely high model performance.

[0088] The above description is only the preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions within the concept of the present invention belong to the protection scope of the present invention. It should be noted that for those of ordinary skill in the art, several improvements and refinements made without departing from the principle of the present invention should also be regarded as within the protection scope of the present invention.

Claims

1. A hardware accelerator for a floating-point neural network, characterized in that, It includes a weight buffer (1), an activation value buffer (2), a partial sum buffer (3), and a two-dimensional space array composed of PE groups (4). The PE group (4) includes an FP32 floating-point accumulator (41) and multiple processing units PE (42) that share the use of the FP32 floating-point accumulator (41). The processing unit PE (42) includes a dot product unit (421), a low-precision accumulator (422), a WinFloat decoder (423), a data distributor (424), a first-in first-out queue FIFO (425), and a floating-point register (426). The dot product unit (421) is used to perform a dot product operation on the mantissa m of the input activation value and weight. The output of the dot product unit (421) generates two outputs through the first WinFloat decoder (423): one output is used as the first input of the first data distributor (424), and the other output is used as the first input of the low-precision accumulator (422). The output of the low-precision accumulator (422) is used as the input of the second WinFloat decoder (423), and the output of the second WinFloat decoder (423) is used as the second input of the first data distributor (424). After the input partial sum enters the third WinFloat decoder (423), it forms two outputs and is respectively used as the first inputs of the second data distributor (424) and the third data distributor (424). The second input of the second data distributor (424) is the output of the second WinFloat decoder (423), and the second input of the third data distributor (424) is the result output by the floating-point register (426). The output of the second data distributor (424) is used as the second input of the low-precision accumulator (422), and the sum of the exponents e of the input activation value and weight is used as the third input of the low-precision accumulator (422). The outputs of the first data distributor (424) and the third data distributor (424) are concatenated and used as the input of the first-in first-out queue FIFO (425). The first-in first-out queue FIFO (425), the FP32 floating-point accumulator (41), and the floating-point register (426) are connected in sequence to output the result through the floating-point register (426). The exponent bit width and exponent bias of the low-precision accumulator (422) are configurable to process data of different window types.

2. The hardware accelerator for a floating-point neural network according to claim 1, wherein The WinFloat decoder (423) includes two NOT gates, one AND gate, one multiplexer, and two adders. After the exponent of the addend passes through the first NOT gate, it is added to the values of the window left interval and the window right interval through the two adders respectively. And the result of adding the value of the window right interval passes through the second NOT gate and then enters the AND gate with the result of adding the value of the window left interval. And the output of the AND gate is used as the control signal of the multiplexer. If the control signal is 0, the addend is sent to the low-precision accumulator (422), and if the control signal is 1, the addend is sent to the FP32 floating-point accumulator (41).

3. The hardware accelerator for a floating-point neural network according to claim 1, wherein The low-precision accumulator (422) includes an exponent difference calculation module, a mantissa swapping module, two alignment shifters, an adder, and a normalization module. The exponent difference calculation module is used to calculate the exponent order difference for the two input exponents e a and e b . The mantissa swapping module is used to swap the mantissas of the two input mantissas m a and m b according to the exponent order difference to obtain a larger mantissa. The two alignment shifters are cascaded to implement the alignment shift of the larger mantissa with an exponent bit width of 1 to 3, and bypass the second-stage alignment shifter when the exponent bit width of the larger mantissa is 4 to obtain the aligned smaller mantissa. The adder is used to sum the aligned smaller mantissa and the larger mantissa. The normalization module is used to normalize the exponent order difference, the aligned smaller mantissa, and the sum result of the larger mantissa and output the result.

4. An application method of the hardware accelerator for a floating-point neural network according to any one of claims 1 to 3, characterized in that, It includes the following steps: using a hardware accelerator for a floating-point neural network to process the accumulation calculation of floating-point data inside and outside the window, and selecting a suitable window according to different data distributions through a preset window search algorithm to cover as much data as possible with a window as small as possible.

5. The application method of the hardware accelerator for floating-point neural networks according to claim 4, characterized in that When using the hardware accelerator for a floating-point neural network to process the accumulation calculation of floating-point data inside and outside the window, the data format of the floating-point data is the window floating-point format shown by the following formula: Sign×2 Exponent-Bias ×1.Mantissa, In the above formula, Sign is the sign bit of the floating-point data, Exponent is the exponent of the floating-point data, Bias is the exponent bias of the floating-point data, and Mantissa is the mantissa of the floating-point data.

6. The application method of the hardware accelerator for a floating-point neural network according to claim 5, characterized in that, When using the hardware accelerator for a floating-point neural network to process the accumulation calculation of floating-point data inside and outside the window, the bit width of the mantissa of the floating-point data is the same as that of the 32-bit floating-point number FP32.

7. The application method of the hardware accelerator for a floating-point neural network according to claim 5, characterized in that Before using the hardware accelerator for a floating-point neural network to process the accumulation calculation of floating-point data inside and outside the window, it also includes reading the configuration values of the exponent bit width and exponent bias of the floating-point data to determine the data representation range of the floating-point data according to the configuration values of the exponent bit width and exponent bias, and the data representation range of the floating-point data is compatible with the 32-bit floating-point number FP32.

8. The application method of the hardware accelerator for the floating-point neural network according to claim 5, characterized in that The selection of a suitable window according to different data distributions through the preset window search algorithm includes: S1, extracting the exponent AExp of the activation value matrix and the exponent WExp of the weight matrix for the floating-point neural network; S2, multiplying the corresponding elements of each row of the exponent AExp of the activation value matrix and each column of the exponent WExp of the weight matrix to obtain the element-by-element product result AWEleExp; S3, initializing the exponent bit width ExpW, the bias variable FBias, and the temporary bias variable BiasT, and the initial value of the exponent bit width ExpW is 2, and the initial values of the bias variable FBias and the temporary bias variable BiasT are both the value with the highest frequency in the element-by-element product result AWEleExp; S4, calculating the number of elements EleInWin in the interval window according to the exponent bit width ExpW and the bias variable FBias; S5, adding 1 to the bias variable FBias and calculating the number of elements EleInWinT in the current interval; S6, determining whether the number of elements EleInWin in the interval window is less than the number of elements EleInWinT in the current interval. If it holds, assign the number of elements EleInWinT in the current interval to the number of elements EleInWin in the interval window and jump to step S5; otherwise, jump to step S7; S7, subtracting 1 from the bias variable FBias and calculating the number of elements EleInWinT in the current interval; S8, determining whether the number of elements EleInWin in the interval window is less than the number of elements EleInWinT in the current interval. If it holds, assign the number of elements EleInWinT in the current interval to the number of elements EleInWin in the interval window and jump to step S7; otherwise, jump to step S9; S9. Determine whether the result obtained by dividing the number of elements EleInWin in the interval window by the number of elements EleInWinT in the current interval is less than the preset threshold T. If not, increment the exponent width ExpW by 1, assign the bias variable FBias to the temporary bias variable BiasT, and jump to step S4; otherwise, jump to step S10. S10. Use the exponent width ExpW as the exponent width Exponent of the floating-point data and the bias variable FBias as the exponent bias Bias of the floating-point data to determine the selected window size and position. The exponent width ExpW is used to determine the window size, and the bias variable FBias is used to determine the window position. Encode the activation values and weights of the floating-point neural network into the window floating-point format according to the selected window size and position for subsequent calculations.

9. The application method of the hardware accelerator for a floating-point neural network according to claim 8, characterized in that, When calculating the number of elements EleInWin in the interval window according to the exponent width ExpW and the bias variable FBias in step S4, the calculation function expression of the corresponding interval window is: Win = [FBias - 2 ExpW-1 -1, FBias + 2 ExpW-1 -1], where Win is the interval window corresponding to the number of elements EleInWin, FBias is the bias variable, and ExpW is the exponent width.

10. The application method of the hardware accelerator for a floating-point neural network according to claim 8, wherein The function expression for calculating the number of elements EleInWinT in the current interval in step S5 is: EleInWinT = ((AWEleExp >= (FBias - 2 ExpW-1 - 1)) & (AWEleExp <= (FBias + 2 ExpW-1 -1))).sum(), Among them, AWEleExp is the element-wise product result, FBias is the bias variable, ExpW is the exponent bit width, & is the AND logical operation, sum() is the summation function. The right side of the above function expression is used to calculate the number of elements in AWEleExp, the element-wise product result, whose values are between FBias - 2 ExpW-1 - 1 and FBias + 2 ExpW-1 - 1 as EleInWinT.

Images