Error adaptive approximate multipliers for high energy efficient self-attention mechanism computation

Abstract

The application provides an error adaptive approximate multiplier for high energy efficiency self-attention mechanism calculation, and proposes an approximate multiplier which adaptively adjusts calculation error according to numerical size, first, an approximate partial product generator with negative calculation error and an approximate 4:2 compressor with positive calculation error are designed, the errors of the two are mutually compensated, the overall error can be effectively reduced, so that more approximate bits can be tolerated, and an approximate multiplier circuit with smaller area and lower power consumption is obtained, second, a power_gating control circuit is added to the approximate compressor in the first level compressor array of the Wallace tree, so that the calculation error can be adaptively adjusted according to the calculation value, and the application can realize more aggressive and lower power consumption multiplier circuit design.

Description

Technical Field

[0001] This invention relates to an error-adaptive approximate multiplier for computation of high-efficiency self-attention mechanisms. Background Technology

[0002] Transformer-based models have achieved superior accuracy compared to traditional Convolutional Neural Networks (CNNs) in many AI tasks, including natural language processing and computer vision. This superior accuracy is attributed to the self-attention mechanism, which allows the Transformer to have a global rather than a local receptive field. However, the global self-attention mechanism is approximately 100 times more computationally expensive than CNNs, and due to the different type of computation, existing CNN accelerators cannot process it efficiently. This has spurred an urgent need to design dedicated Transformer processors.

[0003] In global self-attention mechanisms, redundant content in human language or images generates a large number of naturally occurring weakly related tokens (WR-Tokens). After normalization by the softmax function, the attention calculation result is zero or close to zero. This leads to problems such as redundant computation, excessive energy consumption, and low hardware utilization, making energy-efficient self-attention computation challenging.

[0004] Approximate computation is a flexible computational method applicable to multiple application scenarios, especially in situations with a certain degree of fault tolerance, such as in the fields of deep learning, image processing, and scientific simulation (Reference: Liu W, Lombardi F, Shulte MA retrospective and prospective view of approximate computing[pointof view[J]. Proceedings of the IEEE,2020,108(3):394-399). Approximate computation can improve computational efficiency and accelerate model inference and experimental processes through a moderate loss of accuracy, while small errors may not seriously affect the acceptability of the results.

[0005] For big data analytics, IoT devices, and embedded systems, approximate computation helps achieve higher energy efficiency and longer device lifespan with limited resources. In these cases, moderate approximation can accelerate data processing and decision-making while preserving the overall trend of the data.

[0006] On the other hand, the rapid development of various Transformer models, such as the generative pre-trained models Transformer2 (GPT-2), Visual Transformer (ViT), and Swin-Transformer, has become one of the most important advancements in the field of deep learning. The self-attention mechanism plays a crucial role in the enormous success of Transformer models. The computational principle of the self-attention mechanism can be briefly described as follows: First, the input sequence is divided into a series of tokens, and these tokens are linearly transformed to obtain query (Q), key (K), and value (V) vectors. Then, Q is multiplied by the transpose of K to generate an attention score matrix, which measures the degree of correlation between different tokens. Next, the scores in each row of the score matrix are normalized using the softmax function, scaling the score exponent to a probability (P), representing the correlation of a specific token with all other tokens. Finally, the probability is quantized and multiplied by V to obtain the output. Each output is a weighted sum of all input tokens, where tokens with stronger correlation (SR-Tokens) have larger weights.

[0007] The global attention mechanism contains a large number of weakly correlated tokens (WR-Tokens). WR-Tokens have small attention scores, and after normalization by the softmax function, these small attention scores are exponentially reduced to near-zero probability, drastically weakening the contribution of WR-Tokens to accuracy. However, these small scores account for the majority of computational energy consumption, limiting the energy efficiency of the attention block. According to statistics from the paper "Wang Y, Qin Y, Deng D, et al. An energy-efficient transformer processor exploiting dynamic weak relevances in global attention[J].IEEE Journal of Solid-State Circuits,2022,58(1):227-242", weakly correlated tokens in the Transformer model contribute only 6.3% to accuracy but consume 93.1% of the computational energy. The analysis results of the literature "Wang Y, Qin Y, Deng D, et al. An energy-efficient transformer processor exploiting dynamic weak relevances in global attention[J].IEEE Journal of Solid-State Circuits,2022,58(1):227-242" show that: 1. A large number of weakly correlated tokens have high computational fault tolerance; only a small number of strongly correlated tokens need to be calculated precisely to ensure the accuracy of the transformer model. Therefore, an approximate calculation design method can be introduced to approximate the attention scores corresponding to weakly correlated tokens, thereby improving chip energy efficiency. 2. After the scores are normalized by softmax, they will scale exponentially. That is to say, the contribution of scores is not uniformly distributed, and the influence of smaller scores on the results will decay exponentially. Therefore, the smaller the scores, the higher the tolerance for error. This allows the introduction of a power-gating mechanism into the design of the approximate multiplier, enabling the multiplier's computational error to decrease as scores increase. On one hand, this reduces the proportion of the top K scores, allowing more attention scores to be calculated using the approximate multiplier; on the other hand, it allows more approximate compressors to be disabled in calculations with smaller scores, further reducing the power consumption of the approximate multiplier.

[0008] Most existing approximate multiplier designs involve separate partial product generators and compressors, leaving considerable room for hardware efficiency optimization. Furthermore, the errors in existing approximate multipliers are mostly uniformly distributed and cannot adapt to the error characteristics of self-attention mechanism computations. Summary of the Invention

[0009] Purpose of the invention: The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing an error-adaptive approximate multiplier for high-efficiency self-attention mechanism calculation, including an accurate partial product generator, an approximate partial product generator with positive calculation error, an accurate tree compressor module, an approximate tree compressor module with negative calculation error, and a power gating module.

[0010] Given a multiplier B and a multiplicand A being multiplied, the precise partial product generator is used to generate the high-order segment PP of the partial product. i,j Its logical expression is:

[0011]

[0012] Among them, a j a represents the j-th digit of the multiplicand A. j-1 b represents the (j-1)th digit of the multiplicand A; 2i-1 b represents the (2i-1)th digit of the multiplier B. 2i b represents the 2ith digit of the multiplier B. 2i+1 This represents the (2i+1)th digit of the multiplier B; This represents the XOR logical operation; The horizontal line in the text indicates a negation logical operation;

[0013] The approximate partial product generator is used to generate the low-order segment of the partial product, and its logical expression is as follows:

[0014]

[0015] The encoding error of the approximate partial product generator is negative:

[0016] PPG_A-PPG_E<0 (3)

[0017] Where PPG_A represents the encoded value of the approximate partial product generator, and PPG_E represents the encoded value of the exact partial product generator.

[0018] The precise tree-shaped compressor module is a precise 4:2 compressor, and its logical expression is:

[0019]

[0020]

[0021]

[0022] Where a1, a2, a3, and a4 are the four summing input signals of the precise 4:2 compressor, C in is the carry input signal of the precise 4:2 compressor, Carry is the carry output signal of the precise 4:2 compressor, Cout is the carry overflow signal of the precise 4:2 compressor, and Sum is the summation output signal of the precise 4:2 compressor.

[0023] The approximate tree compressor module includes an approximate 4:2 compressor with positive error in the lower segment. The logical expression of the approximate 4:2 compressor is as follows:

[0024] Sum = a1 + a2 + a3 + a4 (7)

[0025]

[0026] The calculation error of the approximate 4:2 compressor is positive:

[0027] PPA_A-PPA_E>0 (9)

[0028] Where PPA_A represents the calculated value of the approximate 4:2 compressor, and PPA_E represents the calculated value of the precise 4:2 compressor.

[0029] The power gating module generates a control code power_gating[gating_num-1:0], where gating_num represents the number of bits in the control code.

[0030] The power gating module includes a symbol detector and cascaded logic gates;

[0031] The sign detector is used to determine the sign of the multiplicand. If the multiplicand is positive, it performs gating_num OR operations using cascaded OR gates, starting from the most significant bit of the multiplicand, where gating_num is a natural number. If the multiplicand is negative, it performs gating_num AND operations using cascaded AND gates, starting from the most significant bit of the multiplicand, to obtain a gating_num bit control code. This code is used to control whether the gating_num approximate 4:2 compressors in the approximate tree compressor module, starting from the least significant bit, are turned off.

[0032] The power gating module itself has a power gating transistor controlled by a sign detection bit. If the multiplicand is positive, the cascaded OR gate is turned off; if the multiplicand is negative, the cascaded AND gate is turned off.

[0033] This invention proposes an approximate multiplier that adaptively adjusts computational error based on numerical values. First, it designs an approximate partial product generator with negative computational error and an approximate 4:2 compressor with positive computational error. The errors of these two components compensate for each other, effectively reducing the overall error and allowing for the tolerance of more approximate bits, resulting in a smaller area and lower power consumption approximate multiplier circuit. Second, a power-gating control circuit is added to the approximate compressor of the first-stage compressor array of the Wallace tree, enabling the computational error to adaptively adjust according to the calculated value: the error decreases when calculating large values, sacrificing some energy efficiency for higher accuracy; the error increases when calculating small values, sacrificing some accuracy for higher energy efficiency. Applying the proposed approximate multiplier to the computation of the self-attention mechanism in the Transformer model effectively reduces the computational energy consumption of WR-Tokens. The solution proposed in this invention is a novel and effective approach.

[0034] The present invention has the following beneficial effects:

[0035] 1. In an approximate multiplier, the calculation error of the partial product generator is positive, while the calculation error of the partial product summation circuit is negative. The two calculation errors compensate for each other, which can realize a more aggressive and lower power consumption multiplier circuit design.

[0036] 2. A power gating mechanism is introduced: the smaller the product, the more compressors are powered off, resulting in greater computational error and higher energy efficiency; conversely, the larger the product, the more compressors are powered on, resulting in smaller computational error. This adaptive error characteristic can be applied to matrix operations before softmax in neural network models, significantly improving energy efficiency. For example, in the calculation of attention, the computational error decreases parabolically with increasing score, which can reduce the proportion of top K, improve chip computational efficiency, and effectively reduce power consumption. Attached Figure Description

[0037] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0038] Figure 1 This is a structural diagram of the approximate multiplier provided by the present invention.

[0039] Figure 2 It is a point plot of an error-adaptive approximate multiplier.

[0040] Figure 3 This is a circuit diagram of the approximate partial product generator proposed in this invention.

[0041] Figure 4 This is a circuit diagram of an approximate 4-2 compressor.

[0042] Figure 5 This is the circuit diagram of the power gating module.

[0043] Figure 6a This is a schematic diagram of the error distribution curve of an approximate multiplier.

[0044] Figure 6b This is a schematic diagram of the error distribution curve of an approximate multiplier.

[0045] Figure 6c This is a schematic diagram of the error distribution curve of an approximate multiplier.

[0046] Figure 7 This is the error scan diagram of the error adaptive approximation multiplier. Detailed Implementation

[0047] like Figure 1 As shown, the approximate multiplier proposed in this invention includes an accurate partial product generator, an approximate partial product generator with positive computational error, an accurate tree compressor module, an approximate tree compressor module with negative computational error, and a power gating module.

[0048] Given a multiplier B and a multiplicand A being multiplied, an exact partial product generator is used to generate the high-order segments of the partial product. Its logical expression is as follows:

[0049]

[0050] The truth table for the exact partial product generator is shown in Table 1:

[0051] Table 1

[0052]

[0053]

[0054] The approximate partial product generator proposed in this invention is used to generate the low-order segment of the partial product, and its logical expression is as follows:

[0055]

[0056] The truth table for the approximate partial product generator is shown in Table 2.

[0057] Table 2

[0058]

[0059] Comparing equations (1) and (2), it can be seen that the logic expression of the approximate partial product generator is simpler than that of the exact partial product generator, and its circuit area and power consumption are smaller. Comparing Tables 1 and 2, it can be seen that the proportion of logic "0" in the approximate partial product generator is higher than that in the exact partial product generator. This characteristic ensures that the encoding error of the approximate partial product generator is negative, that is:

[0060] PPG_A-PPG_E<0 (3)

[0061] Where PPG_A represents the encoded value of the approximate partial product generator, and PPG_E represents the encoded value of the exact partial product generator.

[0062] The precise tree-structured compressor module can consist of various compressors with no precision loss, such as precise 4:2 compressors, precise 5:2 compressors, and precise 3:2 compressors. The logical expression for a precise 4:2 compressor is as follows:

[0063]

[0064]

[0065]

[0066] The truth tables for the precise 4:2 compressor are shown in Tables 3 and 4 (Table 3 is the truth table for the Sum term of the precise 4:2 compressor, and Table 4 is the truth table for the Carry term of the precise 4:2 compressor):

[0067] Table 3

[0068]

[0069] Table 4

[0070]

[0071] The approximate tree compressor module proposed in this invention includes a precise 4:2 compressor for the high-order segment and an approximate 4:2 compressor with positive error for the low-order segment. The logical expression of the approximate 4:2 compressor proposed in this invention is as follows:

[0072] Sum = a1 + a2 + a3 + a4 (7)

[0073]

[0074] The truth tables for the approximate 4:2 compressor are shown in Tables 5 and 6 (Table 5 is the truth table for the Sum term of the approximate 4:2 compressor, and Table 6 is the truth table for the Carry term of the approximate 4:2 compressor):

[0075] Table 5

[0076]

[0077] Table 6

[0078]

[0079] Comparing equations (4), (5), and (6) with equations (7) and (8), it can be found that the logic expression of the approximate 4:2 compressor is simpler than that of the precise 4:2 compressor, and its hardware overhead and power consumption are lower. Comparing Tables 5 and 6, it can be found that the proportion of logic "1"s in the approximate 4:2 compressor is higher than that in the precise 4:2 compressor. This characteristic ensures that the calculation error of the 4:2 compressor is positive, that is:

[0080] PPA_A-PPA_E>0 (9)

[0081] Where PPA_A represents the calculated value of the approximate 4:2 compressor, and PPA_E represents the calculated value of the precise 4:2 compressor.

[0082] This invention proposes to simultaneously apply an approximate partial product generator with opposite error directions and an approximate 4:2 compressor in the design of an approximate multiplier. The two parts of the error compensate for each other, thereby reducing the overall error of the multiplier.

[0083] Furthermore, to ensure that the calculation error decreases parabolically with increasing score during attention calculation, thereby reducing the proportion of tokens requiring precise calculation, improving chip computational efficiency, and effectively reducing power consumption, a power gating module is introduced in the approximate multiplier design proposed in this invention. The power gating module generates a control code, denoted as power_gating[gating_num-1:0], where gating_num represents the number of bits in the control code. The power gating module includes a sign detector and cascaded logic gates. The sign detector determines the sign of the multiplicand. If the multiplicand is positive, it performs `gating_num` OR operations using cascaded OR gates, starting from the most significant bit of the multiplicand; if the multiplicand is negative, it performs `gating_num` AND operations using cascaded AND gates, starting from the most significant bit of the multiplicand. Thus, the control code for the gating_num bits can be obtained. power_gating[0] controls whether the least significant bit of the approximate 4:2 compressor in the first-stage approximate compressor array is turned off. If power_gating[0] is 0, the compressor is turned off to save power. The control code for the remaining bits works in the same way. In addition, the power gating module itself also has a power gating transistor controlled by the sign detection bit. If the multiplicand is positive, the cascaded AND gate is turned off; if the multiplicand is negative, the cascaded OR gate is turned off.

[0084] In one embodiment of the present invention, a 16-bit signed approximate multiplier is implemented for example... Figure 2 As shown, the circuit of the approximate partial product generator proposed in this invention is as follows: Figure 3 As shown, the current digit a of the multiplicand j Connect the input terminal of the first inverter, and connect the output terminal of the first inverter to the first input terminal of the first AND gate and the first input terminal of the second AND gate. The current bit b of the multiplier 2i Connect the input of the second inverter, and connect the output of the second inverter to the third input of the first AND gate. The high-order bit b of the multiplicand... 2i+1 Connect the second input of the first AND gate and the second input of the second AND gate, the low-order bit b of the multiplicand 2i-1 Connect the input terminal of the third inverter to the input terminal of the third inverter. Connect the output terminal of the third inverter to the third input terminal of the second AND gate. Connect the output terminal of the first AND gate to the first input terminal of the first OR gate. Connect the output terminal of the second AND gate to the second input terminal of the first OR gate. Output the partially multiplied signal PP from the output terminal of the first OR gate. i,j .

[0085] The circuit of the approximate 4-2 compressor proposed in this invention is as follows: Figure 4 As shown, the first summation input signal a1 is connected to the first input terminal of the first OR gate, the input terminal of the first inverter, and the first input terminal of the first AND gate. The output terminal of the first inverter is connected to the first input terminal of the third AND gate. The second summation input signal a2 is connected to the second input terminals of the first OR gate, the second input terminals of the first AND gate, and the second input terminals of the third AND gate. The third summation input signal a3 is connected to the first input terminals of the second OR gate, the first input terminals of the second AND gate, and the third input terminals of the third AND gate. The fourth summation input signal a4 is connected to the second input terminals of the second OR gate and the second AND gate. The output terminal of the first OR gate is connected to the first input terminal of the third OR gate, and the output terminal of the second OR gate is connected to the second input terminal of the third OR gate. The output terminal of the third OR gate outputs the summation output signal sum. The output terminal of the first AND gate is connected to the first input terminal of the fourth OR gate, the output terminal of the second AND gate is connected to the second input terminal of the fourth OR gate, and the output terminal of the third AND gate is connected to the third input terminal of the fourth OR gate. The output terminal of the fourth OR gate outputs the carry output signal carry.

[0086] One embodiment of the power gate code generation circuit proposed in this invention is as follows: Figure 5 As shown, this embodiment generates a 6-bit power gating code, namely code0 to code5.

[0087] The circuit itself also has power gate transistors; MP0 and MN0 form a pair of power gate transistors, and MP1 and MN1 form another pair of power gate transistors. 15This is the sign bit of the multiplier. If the sign bit is "1" (negative), MP1 and MN1 will be turned on, and the control code generated by the AND gate array will be output through the selector. If the sign bit is "0" (positive), MP0 and MN0 will be turned on, and the control code generated by the OR gate array will be output through the selector.

[0088] To visually demonstrate the effectiveness of the error compensation technique proposed in this invention, three 16-bit approximate multipliers were implemented for comparison. The first multiplier only uses the proposed approximate partial product generator in the low-order segment's partial product generator; the rest of the circuitry performs exact calculations. The second multiplier only uses the proposed approximate 4:2 compressor in the low-order segment's compressor; the rest of the circuitry performs exact calculations. The third multiplier uses both the proposed approximate partial product generator and the approximate 4:2 compressor in the low-order segment. The error distribution of the three approximate multipliers is as follows: Figure 6a , Figure 6b , Figure 6c As shown, it can be seen that as the number of approximate bits increases, the absolute value of the calculation error of the three multipliers all increase. However, the error of the third multiplier is significantly smaller after mutual compensation by the approximate partial product generator and the approximate compressor.

[0089] To visually demonstrate the effect of the power gating module proposed in this invention on adjusting computational errors, a scan of the multiplier and multiplicand within the range of [-128, 127] was performed, and the resulting three-dimensional absolute error distribution is shown below. Figure 7 As shown, the closer the multiplicand is to 0, the greater the calculation error; conversely, the larger the absolute value of the multiplicand, the smaller the calculation error. Since in the self-attention mechanism calculation stage of the Transformer model, the weaker the correlation of the label, the smaller the calculated attention score, and the higher the fault tolerance of the calculation process. The error characteristics exhibited by the power-gated approximate multiplier proposed in this invention are highly consistent with the fault tolerance characteristics of the self-attention mechanism calculation. Applying this multiplier to the self-attention mechanism calculation of the Transformer model can effectively reduce energy consumption while maintaining reasonable accuracy.

[0090] This invention provides an error-adaptive approximate multiplier for high-efficiency self-attention mechanism computation. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. An error-adaptive approximate multiplier for high-efficiency self-attention mechanism computation, characterized in that, It includes an accurate partial product generator, an approximate partial product generator with positive computational error, an accurate tree compressor module, an approximate tree compressor module with negative computational error, and a power gating module; Given a multiplier B and a multiplicand A being multiplied, the precise partial product generator is used to generate the high-order segment of the partial product, and its logical expression is as follows: Among them, a j a represents the j-th digit of the multiplicand A. j-1 b represents the (j-1)th digit of the multiplicand A; 2i-1 b represents the (2i-1)th digit of the multiplier B. 2i b represents the 2ith digit of the multiplier B. 2i+1 This represents the (2i+1)th digit of the multiplier B; This represents the XOR logical operation; The approximate partial product generator is used to generate the low-order segment of the partial product, and its logical expression is as follows:

2. The error-adaptive approximate multiplier for high-efficiency self-attention mechanism computation according to claim 1, characterized in that, The encoding error of the approximate partial product generator is negative: PPG_A-PPG_E<0 (3) Where PPG_A represents the encoded value of the approximate partial product generator, and PPG_E represents the encoded value of the exact partial product generator.

3. The error-adaptive approximate multiplier for high-efficiency self-attention mechanism computation according to claim 2, characterized in that, The precise tree-shaped compressor module is a precise 4:2 compressor, and its logical expression is: Where a1, a2, a3, and a4 are the four summing input signals of the precise 4:2 compressor, C in is the carry input signal of the precise 4:2 compressor, Carry is the carry output signal of the precise 4:2 compressor, Cout is the carry overflow signal of the precise 4:2 compressor, and Sum is the summation output signal of the precise 4:2 compressor.

4. The error-adaptive approximate multiplier for high-efficiency self-attention mechanism computation according to claim 3, characterized in that, The approximate tree compressor module includes an approximate 4:2 compressor with positive error in the lower segment. The logical expression of the approximate 4:2 compressor is as follows: Sum = a1 + a2 + a3 + a4 (7) 5. The error-adaptive approximate multiplier for high-efficiency self-attention mechanism computation according to claim 4, characterized in that, The calculation error of the approximate 4:2 compressor is positive: PPA_A-PPA_E>0 (9) Where PPA_A represents the calculated value of the approximate 4:2 compressor, and PPA_E represents the calculated value of the precise 4:2 compressor.

6. The error-adaptive approximate multiplier for high-efficiency self-attention mechanism computation according to claim 5, characterized in that, The power gating module generates a control code power_gating[gating_num-1:0], where gating_num represents the number of bits in the control code.

7. The error-adaptive approximate multiplier for high-efficiency self-attention mechanism computation according to claim 6, characterized in that, The power gating module includes a symbol detector and cascaded logic gates; The sign detector is used to determine the sign of the multiplicand. If the multiplicand is positive, it performs gating_num OR operations using cascaded OR gates, starting from the most significant bit of the multiplicand, where gating_num is a natural number. If the multiplicand is negative, it performs gating_num AND operations using cascaded AND gates, starting from the most significant bit of the multiplicand, to obtain a gating_num bit control code. This code is used to control whether the gating_num approximate 4:2 compressors in the approximate tree compressor module, starting from the least significant bit, are turned off.

8. The error-adaptive approximate multiplier for high-efficiency self-attention mechanism computation according to claim 7, characterized in that, The power gating module itself has a power gating transistor controlled by a sign detection bit. If the multiplicand is positive, the cascaded OR gate is turned off; if the multiplicand is negative, the cascaded AND gate is turned off.

Images