A high-parallelism Softmax software and hardware co-design method

Abstract

The application discloses a high-parallel Softmax software and hardware collaborative design method, which improves the Softmax algorithm by introducing distribution-based and estimation methods and Log2 quantization, and realizes less area overhead, faster running speed and lower energy consumption. The method comprises the following steps: a statistical stage: processing the entire input vector to generate a histogram distribution diagram and the maximum value of the input sequence; an estimation stage: using a fixed-length histogram to calculate the exponential approximation and required by the Softmax; a normalization stage: performing the simplified exponential, division calculation element by element, and outputting the Softmax result after Log2 quantization; and a hardware implementation: for the case that the input sequence length is greater than the hardware data parallel processing length, the statistical unit is run multiple times to obtain the maximum value and distribution information of the complete sequence, then the estimation unit is executed once, and finally the normalization unit is run multiple times to complete the normalization processing on all elements.

Description

Technical Field

[0001] This invention belongs to the field of nonlinear operational circuit design, specifically involving a high-parallelism Softmax (a proprietary operator name, soft maximum value operator) hardware and software co-design method. Background Technology

[0002] Large language models (LLMs) are increasingly becoming a driving force behind significant advancements in artificial intelligence. However, their unique self-attention mechanism—consisting of matrix multiplication and softmax computation—leads to difficulties in accelerating inference. Softmax, with its complex nonlinear operations and low parallelism, severely limits the efficiency of LLMs in processing long text inputs. Nevertheless, due to the nonlinear nature of softmax and the tolerance of LLMs for softmax result deviations, approximating the softmax computation process or result to improve the hardware computation efficiency of softmax is attracting increasing attention from researchers.

[0003] However, existing Softmax hardware and software designs still have shortcomings: 1) Current works mostly use approximate calculation methods such as polynomial fitting and Taylor expansion to simplify the exponent calculation or division calculation in the Softmax operator, but these methods still require nonlinear operations on all elements and do not address the low parallelism of the Softmax operator, thus limiting the effect of reducing latency and energy consumption; 2) Some works reduce computational complexity and enhance parallelism by modifying the Softmax operator itself, such as replacing the exponent with a base-e exponent with a base-2 exponent to simplify the calculation, or omitting the process of finding the maximum value and the sum of the exponents and replacing these two values ​​with constant values ​​learned during training, but these works usually require retraining the model to maintain accuracy, thus limiting their practical applicability.

[0004] To address the aforementioned issues and ensure the efficient operation and superior performance of the Softmax operator, a high-parallelism Softmax hardware-software co-design method is proposed. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention aims to provide a high-parallelism Softmax hardware and software co-design method. By introducing a distribution-based sum estimation method and Log2 quantization (logarithmic quantization with base 2) to improve the Softmax algorithm, it avoids exponential and division calculations for all input elements, enabling Softmax hardware to process input data in high parallelism, achieving less area overhead, faster running speed, and lower energy consumption.

[0006] The technical problem solved by this invention can be achieved through the following specific technical solutions:

[0007] The Softmax algorithm includes a statistical phase, an estimation phase, and a normalization phase. The Softmax hardware includes a statistical unit, an estimation unit, and a normalization unit. The high-parallelism Softmax hardware-software co-design method includes the following steps:

[0008] 1) Statistical stage: Process the entire input vector to generate a histogram and the maximum value of the input sequence;

[0009] 2) Estimation stage: Calculate the exponential approximation required for Softmax using a fixed-length histogram;

[0010] 3) Normalization stage: The simplified exponentiation and division are performed element by element, and the Softmax result after Log2 quantization is output;

[0011] 4) Hardware implementation: For cases where the length of the input sequence is greater than the length of the hardware data parallel processing, the statistical unit is run multiple times to obtain the maximum value and distribution information of the complete sequence, then the sum estimation unit is executed once, and finally the normalization unit is run multiple times to complete the normalization processing of all elements.

[0012] Furthermore, the Softmax function is defined as follows:

[0013] ,

[0014] in, The input sequence for Softmax is... It is an exponential function. The maximum value in the input sequence. For exponents and sums.

[0015] Furthermore, in the statistical phase, the input sequence of the Softmax function is quantized into an N-bit representation, restricting the input values ​​to... In the number of cases, the exponent and calculation process of Softmax are written in a way based on statistical distribution; first, the number of input elements falling on each statistical point is counted, the exponent is calculated for fixed statistical points and then multiplied by the statistical count, and finally the sum is performed.

[0016] Furthermore, in the estimation stage, histogram statistics are performed on the input by truncating the dynamic range and increasing the sampling width, and then the exponent is calculated at each interval to approximately estimate the exponent sum; a compensation estimate is introduced. The estimated index and It is an approximate histogram estimate of the exponent and Compensation estimate The sum of these values, since the normalization stage requires the output of the Log2 quantized result, the output in the sum estimation stage needs to be:

[0017] .

[0018] Furthermore, in the normalization stage, the calculation result of Softmax is quantized using Log2 before being output. Log2 quantization is used to mathematically simplify the exponent and division calculations for each element. Then, the value approximation method is used to simplify the exponent and division operations of the normalization operation into shift and addition / subtraction operations.

[0019] Furthermore, the statistical unit performs histogram analysis on the input sequence and determines the relative maximum value between two values ​​in the sequence. This is achieved by executing the Hist Unit and Top2Max Unit in parallel, respectively; the Top2Max Unit organizes the Top2Comp units in a tree structure, and each Top2Comp unit selects two relatively large values ​​from four inputs.

[0020] Furthermore, the summation and estimation unit only needs to perform simplified statistical analysis on the number of input sequences falling within the i-th interval. Perform simple constant multiplication, then sum by addition, and finally sum by... The result can be obtained by multiplying the negative exponents; the Log2 Unit is used to perform the operation. function.

[0021] Furthermore, the normalization unit consists of multiple parallel Log2Quant units, each including a shifter, a subtractor, and an adder.

[0022] Compared with the prior art, the present invention has the following advantages:

[0023] (1) The high parallelism Softmax hardware and software co-design method of the present invention improves the estimation method of exponent sum by utilizing the nonlinear characteristics of the operator, and introduces Log2 quantization to simplify the element-wise exponent and division calculation, effectively reducing the computational amount and computational complexity of the Softmax algorithm and improving the parallelism of operation; the Softmax hardware implemented by this design method has a significant improvement in area, delay and energy consumption compared with the traditional precise Softmax circuit and the most advanced approximate Softmax design.

[0024] (2) The present invention improves the Softmax algorithm into three operation stages. In the statistical stage, the maximum value and distribution information of the input sequence are obtained. In the estimation stage, the approximate exponent and result are calculated. In the normalization stage, the simplified exponent and division are calculated element by element and the Softmax result after Log2 quantization is output.

[0025] (3) The present invention decouples the inherent dependencies of the sequential operations of the Softmax hardware and captures the relationship between the Softmax input sequence and the exponent sum. The statistical module and the maximum value module of the statistical unit are used in parallel, thereby realizing a high parallelism hardware architecture. The normalization unit simplifies the division process by introducing Log2 quantization, and realizes high parallelism normalization with lower hardware cost. While ensuring that the application accuracy is met, compared with the exact Softmax circuit and the state-of-the-art approximate Softmax design, the present invention achieves less area overhead, faster running speed and lower energy consumption. Attached Figure Description

[0026] Figure 1 This is a flowchart of the algorithm of the present invention;

[0027] Figure 2 This is a distribution diagram of the operator input sequence involved in the present invention;

[0028] Figure 3 This is a hardware implementation framework diagram of the present invention;

[0029] Figure 4 This is a comparison chart of the hardware circuit delay performance of this invention with the delay performance of other technical circuits;

[0030] Figure 5 This is a comparison chart of the power consumption indicators of the hardware circuit of this invention with those of other technical circuits. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0032] like Figure 1 As shown, this invention provides a high-parallelism Softmax hardware and software co-design method. By introducing a distribution-based sum estimation method and Log2 quantization, the Softmax algorithm is improved, avoiding exponential and division calculations for all input elements. This enables the Softmax hardware to process input data in high parallelism, achieving less area overhead, faster running speed, and lower energy consumption.

[0033] The Softmax algorithm includes a statistical phase, an estimation phase, and a normalization phase. The Softmax hardware includes a statistical unit, an estimation unit, and a normalization unit. The high-parallelism Softmax hardware-software co-design method includes the following steps:

[0034] (a) Statistical phase: Process the entire input vector to generate a histogram and the maximum value of the input sequence.

[0035] The Softmax function is defined as follows:

[0036] ,

[0037] in, The input sequence for Softmax is... It is an exponential function. The maximum value in the input sequence. For exponents and sums.

[0038] In quantized LLM (Large Language Model), the input sequence of the Softmax function is quantized into an N-bit representation, limiting the possible input values ​​to a certain range regardless of the quantization method or factor. The number of values ​​is within the range. Therefore, the exponent and calculation process of Softmax can be written based on statistical distribution:

[0039] ,

[0040] ,

[0041] in, yes Falling at each statistical point The above formula means that we can first count the number of input elements falling at each statistical point, calculate the exponent for these fixed statistical points, multiply it by the statistical count, and finally sum them up. Since the number of statistical points is limited, the number of exponents calculated in this formula is also limited, eliminating the need to calculate the exponent for each input value, effectively reducing the computational load in scenarios with long input sequences.

[0042] Considering the nonlinear nature of the exponential function, which means that elements far from the maximum value have the least impact on the overall sum, this invention proposes to approximate the exponential sum by truncating the dynamic range and increasing the sampling width, performing histogram statistics on the input, and then calculating the exponent at each interval.

[0043] ,

[0044] in, It represents the number of intervals in the histogram. It is the statistical count of the input sequence falling on the i-th interval. This is the representative value of the interval (used to replace other numbers in the interval in exponentiation). , , These are all hyperparameters that can be adjusted by the user.

[0045] (ii) Estimation stage: Calculate the exponential approximation required for Softmax using a histogram of fixed length.

[0046] like Figure 2 As shown, the distribution of a large number of inputs deviates significantly from the maximum value and can be well estimated by distribution-based methods. However, a few values ​​near the maximum value still have a significant impact on the exponent and the sum. Using interval distributions cannot accurately estimate the influence of these values. Therefore, to accurately estimate the influence of these values ​​near the maximum value, this invention introduces a compensation estimation method. This estimate takes into account the contribution of the maximum value itself. and the second largest value The impact of this is that if a histogram interval containing the maximum value has more than two values, these nearby values ​​will be replaced with the second largest value; therefore, it is also necessary to determine the number of values ​​within the maximum value interval. Then calculate as follows: :

[0047] ,

[0048] To ensure accurate estimation, the interval containing the maximum value needs to be... Set to 0 (after processing) Record That is, there is no need to consider interval pairs containing the maximum value. The impact. And The index i in the calculation no longer starts from 0, because regardless of In which range? It is always 0. Ultimately, the estimated exponent and... yes and The sum of . Since the normalization stage requires the output of the Log2 quantized result, the output of the sum estimation stage needs to be:

[0049] .

[0050] (iii) Normalization stage: The simplified exponent and division are calculated element by element and the Softmax result after Log2 quantization is output.

[0051] Log2 quantization modifies the quantization process from linear scaling to exponential scaling. Quantizer The definition of is:

[0052] ,

[0053] in, For the quantized numerical sequence, To quantize bit width, For symbolic functions, This is a truncation function. This indicates that rounding is performed. Used to find the maximum value of a sequence. Since the output of Softmax is always in... Therefore, when applying Log2 quantization to the Softmax result, the quantizer can be simplified to:

[0054] ,

[0055] Since truncation and rounding functions are relatively easy to implement in hardware, they are the focus of this study. accomplish:

[0056] ,

[0057] As can be seen, the original division has been transformed into addition under the effect of Log2 quantization. Furthermore, the original... It is also transformed into constant multiplication. Combined with a distribution-based sum estimation algorithm, the improved Softmax algorithm of this invention completely eliminates element-wise exponentiation and division calculations.

[0058] To further simplify the hardware implementation of constant multiplication, one can use... Approximation:

[0059] ,

[0060] Therefore, the normalization stage can be implemented through simple shifting, addition, and subtraction. This controllable hardware overhead makes it feasible to deploy a large number of normalization units in parallel.

[0061] (iv) Hardware implementation: For cases where the length of the input sequence is greater than the length of the hardware data parallel processing, the statistical unit is run multiple times to obtain the maximum value and distribution information of the complete sequence, then the sum estimation unit is executed once, and finally the normalization unit is run multiple times to complete the normalization processing of all elements.

[0062] like Figure 3 The diagram shows the Softmax hardware architecture, which includes three units corresponding to the algorithm stages. Regarding the hyperparameters used in the estimation algorithm, based on multiple experiments, a recommended selection is given here: maximum and minimum boundaries are ±6, interval width W=2, and the interval value represents the lower quarter point of the interval. This will lead to the number of intervals. Besides the six intervals within the ±6 boundary, values ​​greater than +6 belong to... Values ​​less than -6 belong to For intervals ,Pick .

[0063] The Histogram Unit performs histogram analysis on the input sequence, and the Top2Max Unit identifies the two largest values ​​in the sequence. The Top2Max Unit is organized in a tree structure by comparison units (Top2Comp Units), each of which selects two relatively larger values ​​from four inputs.

[0064] The estimation unit is the most complex module, but it is activated only once for each input sequence, so its overhead is negligible for long sequences. The pre-processing unit... Distinguish the larger values ​​in the middle and smaller value Find the interval containing the maximum value to get the number of values ​​within the maximum value interval. and the interval that will contain the maximum value Simplified statistics obtained by setting it to 0 And calculate auxiliary parameters To provide approximate histogram estimates of the exponent and Computing unit ( Used for histogram estimation and unit ( ) and compensation estimation unit ( Achieve final approximate estimation and The calculation. Because the interval represents the value. It was pre-selected. The implementation can be further simplified to the following formula, without worrying about precision overflow caused by exponent calculation:

[0065] ;

[0066] Therefore, in hardware implementation, it is only necessary to count the simplified number of input sequences falling on the i-th interval. Perform simple constant multiplication (essentially shifting, addition, and subtraction), then sum by addition, and finally combine with the maximum value of the input sequence. Multiplying by the negative exponent yields the result and the estimation result. The logarithmic calculation unit (Log2 Unit) is used to perform... function.

[0067] The normalization unit consists of multiple parallel log2quant units. The design of the log2quant unit is simple, containing only shifters, subtractors, and adders. This limited hardware overhead allows users to deploy a large number of units. As the length of the input sequence increases, the advantages of this low-power, low-latency method are further amplified.

[0068] In summary, steps (i) and (ii) of this invention are based on a distribution-based estimation method. Utilizing the nonlinear nature of exponential calculation and the tolerance for deviations in the exponential sum, the influence of elements far from the maximum value on the exponential sum is estimated solely through distribution statistics. Error compensation is then used to reflect the influence of elements close to the maximum value on the exponential sum, significantly reducing the computational load required for calculating the exponential sum. Step (iii) involves Log2 quantization output. The Softmax calculation result is then output after Log2 quantization. Log2 quantization mathematically simplifies the exponent and division calculations for each element. Furthermore, value approximation is used to simplify the normalization operation into shifting and addition / subtraction operations. Step (iv) is the hardware implementation. The basic architecture of the Softmax circuit consists of three units: a statistical unit, an estimation unit, and a normalization unit. The statistical unit uses the distribution statistics module and the maximum value module in parallel, while the normalization unit arranges a large number of simplified normalization modules. When the length of the input sequence is greater than the length of the hardware data parallel processing, the statistical unit is run multiple times to obtain the maximum value and distribution information of the complete sequence. Then, the estimation unit is executed once, and finally, the normalization unit is run multiple times to complete the normalization processing of all elements.

[0069] The functions and effects of this invention are further illustrated and demonstrated through the following simulation experiments:

[0070] (a) Simulation conditions

[0071] To evaluate the improved Softmax algorithm of this invention, extensive accuracy verification experiments were conducted on multiple models and datasets. Specifically, the datasets include RTE (from the non-patented literature Wang A, Singh A, Michael J, et al. GLUE: A multi-task benchmark and analysis platform for natural language understanding[J]. arXiv preprint arXiv:1804.07461, 2018.), PIQA (from the non-patented literature Bisk Y, Zellers R, Gao J, et al. Piqa: Reasoning about physical commonsense in natural language[C] / / Proceedings of the AAAI conference on artificial intelligence. 2020, 34(05): 7432-7439.), COPA (from the non-patented literature Wang A, Pruksachatkun Y, Nangia N, et al. Superglue: A stickier benchmark for general-purpose language understanding systems[J]. Advances in neural information processing systems, 2019, 32.), and BoolQ (from the non-patented literature Clark C, Lee K, Chang MW, et al. Boolq: Exploring the surprising difficulty of naturalyes / no questions[J]. arXiv preprint arXiv:1905.10044, 2019.), HellaSwag (from non-patent literature Zellers R, Holtzman A, Bisk Y, et al. Hellaswag: Can a machine really finish your sentence?[J]. arXiv preprint arXiv:1905.07830, 2019.), and Winogrande (from non-patent literature Sakaguchi K, Bras RL, Bhagavatula C, et al.).Winogrande: Anadversarial winograd schema challenge at scale[J]. Communications of the ACM, 2021, 64(9): 99-106.). The test models included OPT-125M (from the non-patented literature Zhang S, Roller S, Goyal N, et al. Opt: Open pre-trained transformer language models[J]. arXivpreprint arXiv:2205.01068, 2022.), GPT2 (from the non-patented literature Radford A, Wu J, ChildR, et al. Language models are unsupervised multitask learners[J]. OpenAIblog, 2019, 1(8): 9.), and Llama2-7B (from the non-patented literature Touvron H, Lavril T, IzacardG, et al. Llama: Open and efficient foundation language models[J]. arXivpreprint arXiv:2302.13971, 2023.) and their inference performance. The benchmark was an INT8 quantization model with full-precision FP32Softmax. Then, we replaced the FP32 Softmax with the FP16 Softmax version, the state-of-the-art approximation design SOLE Softmax (from the non-patent literature Wang W, Zhou S, Sun W, et al. SOLE: Hardware-Software Co-design of Softmax and LayerNorm for EfficientTransformer Inference[C] / / 2023 IEEE / ACM International Conference on ComputerAided Design (ICCAD). IEEE, 2023: 1-9.), and our optimization algorithm.

[0072] To evaluate hardware efficiency, the hardware proposed in this invention was compared with precise FP16 Softmax hardware (an industrial-precise 16-bit floating-point unit from the Synopsys DesignWare library) and SOLE Softmax hardware. All designs were implemented in Verilog HDL and synthesized and evaluated using Synopsys Design Compiler and UMC's 40nm library under consistent constraints. For fair latency comparisons, divider units were deployed in parallel during the normalization stages of both FP16 Softmax and SOLE, similar to the parallel Log2Smax unit of this application. Hardware performance was evaluated with data parallel processing lengths of 16 and 32, representing different hardware constraints.

[0073] (II) Simulation Results

[0074] (1) Application accuracy test and comparison

[0075] The table below shows the accuracy results of the proposed method on various LLM models and tasks. Clearly, the proposed algorithm achieves performance very close to the baseline: the average accuracy of GPT2 and Llama2-7B decreased by 0.65% and 0.34%, respectively. It even brings a 0.07% accuracy improvement to OPT-125M. It can be considered that the impact of the proposed algorithm is comparable to FP16 Softmax and SOLE.

[0076]

[0077] These consistent results across multiple models demonstrate the general and accurate representation of the proposed algorithm for Softmax results. Furthermore, it operates without additional retraining or fine-tuning during model inference, eliminating costly training overhead and further enhancing its practicality.

[0078] (2) Hardware implementation and comparison

[0079] 1) Area Expenses

[0080] The table below details the area requirements for each design when the hardware data parallel processing length (denoted by P) is 16 and 32. When P=16, our design achieves a 12.28% area reduction compared to SOLE and an 80.73% area reduction compared to FP16 Softmax. When P=32, these area savings further increase to 16.67% and 83.94%, respectively.

[0081] This advantage stems from the estimation unit, which is independent of the hardware vector size. Therefore, compared to other designs, scaling the hardware of this invention to achieve higher parallelism requires fewer additional hardware components.

[0082]

[0083] 2) Running time and energy consumption

[0084] To evaluate the performance of the design of this invention under long text input sequences, the input sequence lengths were adjusted to 32, 64, 128, and 256. The runtime and power consumption of all designs were measured, and all results were standardized against the performance of the hardware of this invention, such as... Figure 4 and Figure 5 As shown.

[0085] For hardware data parallel processing lengths P of 16 and 32, the experimental results show a consistent trend: as the input sequence length increases, the design's advantages in speed and energy consumption become more significant. For P=32, the design of this invention reduces latency by 9.90%-44.75% and 69.78%-81.48% compared to SOLE and FP16 Softmax, respectively. With an input sequence length of 256, the design of this invention at P=32 achieves a 17.36% energy saving compared to SOLE and a 79.63% energy saving compared to FP16 Softmax. This improvement can be attributed to the simplified statistical unit and normalized unit.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A high-parallelism Softmax hardware-software co-design method, characterized in that, The Softmax algorithm includes a statistical phase, an estimation phase, and a normalization phase. The Softmax hardware includes a statistical unit, an estimation unit, and a normalization unit. The Softmax hardware-software co-design method includes the following steps: 1) Statistical stage: Process the entire input vector to generate a histogram and the maximum value of the input sequence; 2) Estimation stage: Calculate the exponential approximation required for Softmax using a fixed-length histogram; 3) Normalization stage: The simplified exponentiation and division are performed element by element, and the Softmax result after Log2 quantization is output; 4) Hardware implementation: For cases where the length of the input sequence is greater than the length of the hardware data parallel processing, the statistical unit is run multiple times to obtain the maximum value and distribution information of the complete sequence, then the sum estimation unit is executed once, and finally the normalization unit is run multiple times to complete the normalization processing of all elements.

2. The high-parallelism Softmax hardware-software co-design method according to claim 1, characterized in that, The Softmax function is defined as follows: ， in, The input sequence for Softmax is... It is an exponential function. The maximum value in the input sequence. For exponents and sums.

3. The high-parallelism Softmax hardware-software co-design method according to claim 2, characterized in that, During the statistical phase, the input sequence to the Softmax function is quantized into an N-bit representation, restricting the input values ​​to a specific range. In the number of cases, the exponent and calculation process of Softmax are written in a way based on statistical distribution; first, the number of input elements falling on each statistical point is counted, the exponent is calculated for fixed statistical points and then multiplied by the statistical count, and finally the sum is performed.

4. The high-parallelism Softmax hardware-software co-design method according to claim 2, characterized in that, In the estimation phase, histogram statistics are performed on the input by truncating the dynamic range and increasing the sampling width, and then the exponent is calculated at each interval to approximately estimate the exponent sum; a compensation estimate is introduced. The estimated index and It is an approximate histogram estimate of the exponent and Compensation estimate The sum of these values, since the normalization stage requires the output of the Log2 quantized result, the output in the sum estimation stage needs to be: 。 5. The high-parallelism Softmax hardware-software co-design method according to claim 2, characterized in that, During the normalization stage, the calculation result of Softmax is quantized using Log2 before being output. Log2 quantization simplifies the exponent and division calculations for each element mathematically. Furthermore, the value approximation method is used to simplify the exponent and division operations of the normalization operation into shift and addition / subtraction operations.

6. The high-parallelism Softmax hardware-software co-design method according to claim 1, characterized in that, The statistical unit performs histogram analysis on the input sequence and determines the relative maximum value between two values ​​in the sequence. This is achieved by executing the Hist Unit and Top2Max Unit in parallel, respectively; the Top2Max Unit organizes the Top2Comp units in a tree structure, and each Top2Comp unit selects two relatively large values ​​from four inputs.

7. The high-parallelism Softmax hardware-software co-design method according to claim 1, characterized in that, The sum and estimation unit only needs to count the number of times the input sequence falls on the i-th interval. Perform simple constant multiplication, then sum by addition, and finally sum by... The result can be obtained by multiplying the negative exponents; the Log2 Unit is used to perform the operation. function.

8. The high-parallelism Softmax hardware-software co-design method according to claim 1, characterized in that, The normalization unit consists of multiple parallel Log2Quant units, each of which includes a shifter, a subtractor, and an adder.

Images