Large language model softmax function hardware acceleration circuit and method

Abstract

The invention discloses a large language model softmax function hardware acceleration circuit and method, and belongs to the field of neural network hardware acceleration of super-large scale integrated circuits. An input sequence is divided into a plurality of data blocks which are processed in parallel, and three-stage pipeline division is adopted, so that average single calculation delay is shortened to G clock cycles, the calculation parallelism is improved, the calculation speed of a softmax function is improved, and the reasoning delay is reduced; and a sparse mask strategy of sparse threshold comparison is introduced, the sparsity of data is fully utilized, the problems of high calculation complexity and high calculation delay of the softmax function and the data access bottleneck of the softmax function are solved, the calculation cost is remarkably reduced, and the calculation efficiency is improved. In addition, a softmax function hardware circuit adaptive to software optimization is constructed, calculation delay and memory access pressure are reduced through the hardware circuit, and data processing efficiency is improved.

Description

Technical Field

[0001] This invention relates to a hardware acceleration circuit and method for the softmax function of a large language model, belonging to the field of neural network hardware acceleration of very large-scale integrated circuits. Background Technology

[0002] In recent years, Large Language Models (LLMs), represented by the Transformer architecture, have made groundbreaking progress, with their parameter count and model size exploding, enabling them to process and generate more complex natural language content. The core idea of ​​LLMs is to use deep learning models with massive amounts of data and parameters to master the statistical patterns of language through self-supervised learning, thereby gaining the ability to understand and generate human language. This relies on the mathematical properties of the softmax function to transform the output of the LLM into an interpretable probability distribution. The softmax function is based on the properties of exponential functions, amplifying the differences between different inputs through exponential operations. It cleverly transforms the raw scores (logits) of the model output into a probability distribution between 0 and 1, with a sum of 1, clearly reflecting the prediction probability of each category.

[0003] However, with the continuous development of deep learning models, especially in scenarios involving large-scale data and building complex models, the inherent defects of the softmax function have gradually become apparent. Its calculation process involves exponential and normalization operations, both of which have high computational complexity and require significant computational resources and time. During the inference phase, this high complexity directly leads to significant latency, severely impacting the model's real-time response capabilities. For applications with stringent timeliness requirements, such as real-time speech recognition and autonomous driving decision-making, this problem has become a key bottleneck restricting its development.

[0004] In addition, during model operation, massive amounts of data need to be frequently transferred and exchanged between memory and computing units, which not only consumes a lot of time but also consumes huge amounts of energy resources. Such frequent memory access operations greatly limit the running efficiency of large language models, causing the models to face many challenges in practical deployment and application, such as difficulty in running efficiently on resource-constrained devices or performance mismatches in large-scale parallel computing.

[0005] Existing technologies include several optimization schemes for the softmax function. For example, Wang M proposed an efficient, low-complexity softmax hardware architecture in his paper "A high-speed and low-complexity architecture for softmax function in deep learning[C] / / 2018 IEEE Asia Pacific Conference on Circuits and Systems (APCCAS).IEEE, 2018: 223-226." This architecture significantly reduces computational complexity and resource consumption through mathematical transformations, linear fitting, and hardware optimization techniques, achieving high throughput and energy efficiency in a 28nm process. However, the approximation method used in this approach leads to certain computational errors, failing to meet the requirements of high-precision scenarios. Furthermore, this method is not suitable for situations with large input ranges or floating-point inputs. JR Stevens, in "Softermax: Hardware / software co-design of an efficient softmax for transformers[C] / / 2021 58th ACM / IEEE Design Automation Conference (DAC). IEEE, 2021: In “469-474,” an efficient softmax optimization method was proposed. Through cardinality replacement, low-precision computation, and online normalization, significant energy consumption and area optimization were achieved in Transformer inference with minimal accuracy loss. However, this scheme only targets the optimization of the Transformer inference stage and has poor generality. The accuracy range is limited and it has strong hardware dependence. In “An energy-efficient transformer processor exploiting dynamicweak relevances in global attention[J]. IEEE Journal of Solid-State Circuits,2022, 58(1): 227-242,” Wang Y proposed a method based on dynamic weak correlation. By dynamically identifying and skipping redundant operations in attention computation, the energy efficiency of Transformer processing is greatly improved with almost no loss of accuracy. However, this method introduces additional hardware complexity and control overhead, has high scheduling strategy complexity, and poor scalability. Furthermore, the detection logic of this method is simple and lacks universality.

[0006] Faced with the dual challenges of high computational latency of the softmax function and large memory access bottlenecks of large language models, the industry urgently needs innovative technical solutions to address these issues and promote the efficient application of deep learning models in a wider range of fields. Summary of the Invention

[0007] To optimize the above problems, improve the computational parallelism of the softmax function, fully utilize the sparsity of the input data of the softmax function, and ultimately reduce computational latency and memory access pressure, this invention provides a hardware acceleration circuit and method for the softmax function of a large language model, thereby improving the parallelism during computation.

[0008] The first objective of this invention is to provide a hardware acceleration circuit for the softmax function of a large language model. This hardware acceleration circuit includes a sparse processing unit (SPU), a preprocessing unit (PRPU), a cache management unit (BMU), and a post-processing unit (POPU). The SPU is used for sparse detection and execution of sparse operations. The PRPU is used for calculating the block sum. The BMU is used for caching and exchanging intermediate data and reducing the block sum. The POPU is used to calculate the final function output and the sparse mask based on the cached data from the previous level. The softmax function is calculated using this hardware circuit.

[0009] The Sparse Processing Unit (SPU) receives the maximum value of the previous round group from the Cache Management Unit (BMU) every clock cycle. At the same time, set the current maximum value of the wheel group. The sparse index sp_idx and the maximum index max_idx are sent to the cache management unit (BMU) for storage. The number of representative groups.

[0010] Each clock cycle, the Preprocessing Unit (PRPU) receives the PRPU enable signal (prpu_en) and the PRPU selection signal (prpu_sel) from the Sparse Processing Unit (SPU), and receives the maximum value of the previous round group from the Buffer Management Unit (BMU). Compared with the previous round of block division At the same time, the current block and Send to the Cache Management Unit (BMU) register.

[0011] The post-processing unit (POPU) receives the final sparse index o_sp_idx, the final maximum index o_max_idx, and the global maximum value among the currently processed elements from the cache management unit (BMU). and intermediate variables And based on the input sequence data group The final output result is calculated. and sparse mask .

[0012] The buffer processing unit (BMU) registers the integer part of the current group's maximum value generated by the sparse processing unit (SPU). sparse index sp_idx and maximum value index max_idx, as well as the blocks generated by the preprocessing unit PRPU and .

[0013] The Sparse Processing Unit (SPU) and the Preprocessing Unit (PRPU) run concurrently. The SPU is responsible for executing sparsity checks and corresponding sparsity operations in the algorithm; the PRPU is responsible for calculating the block sum corresponding to each input data in the B data blocks. The B preprocessing units (PRPUs) simultaneously calculate the corresponding blocks and This continues until the G sets of input sequences have been iterated through; where , represents the number of input data sets, and G represents the number of input data sets in a single data block of the block; , represents the number of data blocks, and B represents the number of data blocks to be divided; The post-processing unit POPU calculates the corresponding function output value and sparse mask based on the final sparse index o_sp_idx and the final maximum index o_max_idx output by the buffer processing unit BMU.

[0014] Furthermore, the sparse processing unit (SPU) includes a maximum value calculation module (GMC), an indexing and preprocessing unit control module (IPC), a first subtractor (SUB1), and a second subtractor (SUB2). The maximum value calculation module GMC calculates the current maximum value by accepting the input sequence data group and inputs it into the first subtractor SUB1. The first subtractor SUB1 obtains the maximum value of the previous round group from the cache management unit BMU and performs the subtraction operation. It then takes the absolute value of the operation result to obtain the current round group maximum value and the maximum value comparison signal max_comp. This absolute value is then transmitted to the second subtractor SUB2, which transmits the current round group maximum value to the cache management unit BMU and the maximum value comparison signal max_comp to the index and preprocessing unit control module IPC. The second subtractor SUB2 subtracts the absolute value output by the first subtractor SUB1 from a preset sparse threshold to obtain the difference judgment signal diff, and transmits it to the index and preprocessing unit control module IPC. The index and preprocessing unit control module IPC generates the preprocessing unit enable signal prpu_en, the preprocessing unit selection signal prpu_sel, the sparse index sp_idx, and the maximum value index max_idx based on the difference judgment signal diff and the maximum value comparison signal max_comp, and transmits them to the cache management unit BMU for temporary storage.

[0015] The maximum value calculation module GMC calculates the maximum value based on the input sequence data group. The maximum value within the group was calculated. Then the maximum value will be compared with Find the difference and take the absolute value of the difference. The first subtractor SUB1 performs a subtraction operation to determine the integer part of the current group's maximum value. And the maximum value comparison signal max_comp; if If positive, then the integer part of the maximum value in the current group. Set as Otherwise, retain the original value, i.e. ; if If the value is negative, the maximum value comparison signal max_comp is set to 1; otherwise, the maximum value comparison signal max_comp is set to 0.

[0016] Then, the absolute value is processed by the second subtractor SUB2. With a pre-set sparsity threshold If you do subtraction, If the result is positive, it indicates that the sparsity condition has been met, and the difference judgment signal diff is set to 1; otherwise, it is set to 0.

[0017] The index and preprocessing unit control module IPC generates the preprocessing unit enable signal prpu_en, the preprocessing unit selection signal prpu_sel, the sparse index sp_idx, and the maximum index max_idx based on the difference judgment signal diff and the maximum value comparison signal max_comp, and transmits them to the cache management unit BMU for temporary storage.

[0018] When the difference judgment signal diff=1 and the maximum value comparison signal max_comp=0, the sparse index sp_idx is set to the current group number. Otherwise, the original value is retained; when the maximum value comparison signal max_comp=0, the maximum value index max_idx is set to the current group number. Otherwise, keep the original value.

[0019] In addition, when the difference judgment signal diff=1 and the maximum value comparison signal max_comp=1, the preprocessing unit enable signal prpu_en will be set to 0, and otherwise set to 1; When the difference judgment signal diff=1 and the maximum value comparison signal max_comp=0, the preprocessing unit selection signal prpu_sel will be set to 0; otherwise, it will be set to 1.

[0020] Furthermore, the preprocessing unit PRPU is used to implement the preprocessing process of the algorithm, including the first function calculation module SPIE1, adder ADDER, subtractor SUB, multiplexer MUX, and shifter SHIFT. The minuend input of the subtractor SUB receives the result generated by the sparse processing unit SPU. The subtraction input receives the input from the buffer processing unit (BMU). The shifter SHIFT receives input from the subtractor SUB output and external input. The output of the subtractor SUB is: As the shift bit width; As the number of bits to be shifted. The input port of the first function calculation module SPIE1 receives externally input... The signal, whose output serves as both the output of the adder ADDER and the input of the multiplexer MUX; the output of the adder ADDER serves as another input of the multiplexer MUX; the selection signal of the multiplexer MUX is the preprocessing unit selection signal prpu_sel output from the sparse processing unit SPU, and the output of the multiplexer MUX... As the output of the pre-processing unit PRPU; In the preprocessing unit PRPU, the difference is first calculated by the subtractor SUB. ,in From the cache processing unit (BMU), From the sparse processing unit (SPU); then according to For the difference Perform a shift operation, in which It is the sum of the denominator of the previous round of block calculations from the cache processing unit (BMU). The first function calculation module SPIE1 is used for approximate calculation. In addition, the preprocessing unit (PRPU) also calculates... , as candidate output blocks.

[0021] Furthermore, the multiplexer in the preprocessing unit (PRPU) adjusts the output value according to the preprocessing unit selection signal prpu_sel from the sparse processing unit (SPU). The selection is performed; if the preprocessing unit selection signal prpu_sel is 1, then the output value is... Set as Otherwise, output the value. Set directly to ; then It is sent to the cache processing unit (BMU) for caching.

[0022] Furthermore, the cache processing unit (BMU) includes an adder tree module (ADDTREE), a leader detector (LOD), a PRMAX buffer, a PRSUM buffer, a first multiplexer (MUX1), a second multiplexer (MUX2), a third multiplexer (MUX3), an IDX buffer, a POMAX buffer, and a POSUM buffer.

[0023] The output port of the first multiplexer MUX1 is connected to the input port of the IDX buffer, and the output port of the IDX buffer is connected to the 0 input port of the first multiplexer MUX1. Simultaneously, the 1 input port of the first multiplexer MUX1 is connected to the externally input sparse index sp_idx and maximum value index max_idx. The output port of the IDX buffer also serves as the output port of the buffer management unit (BMU), connecting to the outside. The input port of the PRMAX buffer is connected to the externally input... The output ports are connected to the external output port of the module and the input port 1 of the second multiplexer MUX2, respectively; the output port of the second multiplexer MUX2 is connected to the input port of the POMAX buffer, and the output port of the POMAX buffer is connected to the input port 0 of the second multiplexer MUX2 and the external output port of the module, respectively; the input port of the PRSUM buffer is connected to the external input block and signal. The output ports are connected to the external output port and input port 1 of the third multiplexer MUX3, respectively; the output port of the third multiplexer MUX3 is connected to the input port of the POSUM buffer; the output port of the POSUM buffer is connected to input port 0 of the third multiplexer MUX3 and input port of the adder tree module ADDTREE; the output port of the adder tree module ADDTREE is connected to the input port of the leader detector LOD; the leader detector LOD outputs intermediate variables. As the output of the preprocessing unit PRPU.

[0024] The PRMAX and PRSUM buffers are used to cache the real-time group maximum values ​​from the sparse processing unit (SPU). And the blocks from the preprocessing unit PRPU and .

[0025] The update signal serves as the strobe signal for the first multiplexer MUX1, the second multiplexer MUX2, and the third multiplexer MUX3, and is simultaneously connected to these multiplexers. It is used to indicate that the data iteration calculation is complete. When all G data points in each data block have been traversed, the update signal is pulled high for one clock cycle and remains low for the remaining clock cycles.

[0026] In the next clock cycle, the cache processing unit (BMU) will use the cached value as the value from the previous cycle. , The data is sent to the sparse processing unit (SPU) and the preprocessing unit (PRPU) for iterative computation.

[0027] When the update signal goes high, the first multiplexer MUX1, the second multiplexer MUX2, and the third multiplexer MUX3 will cause the subsequent buffers to cache the final maximum value of the group. And the final block and ,in Cached in the POMAX buffer Cached in the POSUM buffer. This is easy to understand. It is the global maximum value. The data cached in the POSUM buffer is then reduced using an addition tree to obtain the final denominator. Subsequently, intermediate variables in the CPSS algorithm are obtained through a lead detector LOD calculation. ,Right now .

[0028] When the update signal is low, the sparse index sp_idx and the maximum index max_idx from the sparse processing unit (SPU) are latched in the POIDX buffer. When the update signal is high, the index signals are finally updated and latched, and then transmitted to the post-processing unit (POPU). Since the update signal simultaneously controls the first multiplexer MUX1, the second multiplexer MUX2, and the third multiplexer MUX3 to separate the input sequence data, while the IDX buffer, POMAX buffer, POSUM buffer, adder tree module ADDTREE, and leader detector LOD in the buffer processing unit (BMU) buffer the data of the previous sequence, the PRMAX buffer and PRSUM buffer can buffer the data of the next sequence online. Therefore, the buffer processing unit (BMU) enables the entire system to process the input sequence in a pipelined manner.

[0029] Furthermore, the post-processing unit POPU includes a second function calculation module SPIE2, a third function calculation module SPIE3, a result control module RC, a multiplexer MUX, a first subtractor SUB1, a second subtractor SUB2, and a third subtractor SUB3.

[0030] The two input ports of the first subtractor SUB1 receive respectively Signals and Signal, and implement The output port is connected to the minuend input port of the second subtractor SUB2; the subtrahend input port of the second subtractor SUB2 receives the sparse threshold. The input ports of the result control module RC receive the output of the second subtractor SUB2, as well as the sparse index sp_idx and the maximum index max_idx, respectively. Its output port is connected to the strobe signal port of the multiplexer MUX. The input port of the second function calculation module SPIE2 receives intermediate variables from the output of the preprocessing unit PRPU. Its output port is connected to the subtrahend input port of the third subtractor SUB3. The minuend input port of the third subtractor SUB3 is connected to the output port of the first subtractor SUB1, and its output port is connected to the input port of the third function calculation module SPIE3; the output port of the third function calculation module SPIE3 is connected to the 0 input port of the multiplexer MUX, the 1 input port of the multiplexer MUX is connected to the 0 signal, and its output port serves as the output port of the entire post-processing unit POPU, connected to the outside.

[0031] The second function calculation module, SPIE2, is used for approximate calculations in the softmax optimization method. intermediate variables The results come from the upper-level cache processing unit (BMU). The calculation results of the second function calculation module (SPIE2) are intermediate variables required for subsequent calculations by the post-processing unit (POPU).

[0032] The post-processing unit (POPU) calculates the sparse mask through some simple operations. It then performs corresponding calculations on the input values. First, it uses a subtractor to obtain the intermediate value. Then a subtractor is used for comparison. and sparsity threshold The size relationship between them; if Greater than This indicates that the current data being processed has reached the sparsity condition, and the variables... Set to 1, otherwise It is 0.

[0033] The result control module RC is used to perform the final sparsity judgment on the data and select the output data; for an input sequence value If its group number j satisfies Or satisfy Then sparse operation is required. At this time, the RC output strobe signal is 1, and the output result of the post-processing unit POPU is... , ; If the index j of the input data is between the sparse index sp_idx and the maximum index max_idx, no sparse operation is performed, and the RC output strobe signal is 1. When calculating the output, the intermediate value is first obtained. The final output result is then obtained through the third function calculation module SPIE3. , .

[0034] The second objective of this invention is to provide a hardware acceleration method for the softmax function of a large language model. This hardware acceleration method is implemented based on the aforementioned hardware acceleration circuit. This method improves computational parallelism by dividing the input sequence into several data blocks that can be processed in parallel. At the same time, it introduces a sparse masking strategy based on sparse threshold comparison to make full use of the sparsity of the data. This method is based on an optimized softmax function, the expression of which is:

[0035] By employing a base substitution method, the base e is replaced with a more hardware-friendly base of 2. Furthermore, an iterative method is used to calculate the denominator. This represents the i-th input value in the input sequence. This represents intermediate quantities in the iterative calculation of the denominator. This represents the global maximum value; additionally, it replaces the maximum value among the currently processed elements with the integer part of the maximum value among the currently processed elements. To ensure that the relevant calculations are easier to implement in hardware; This method introduces configurable parallel computation, dividing the input sequence into B data blocks, each containing G input data points. The G input data points in each block are processed in parallel. One data point from each of the B data blocks is processed per clock cycle, meaning B data blocks are processed in parallel per clock cycle. A total of G clock cycles are required to complete the processing of all data. In the softmax function software optimization method, the sum of the denominators for each block is calculated according to the above expression, and finally, the sums are accumulated and reduced to obtain the complete denominator.

[0036] Furthermore, this method introduces sparsity judgment; the proposed hardware acceleration method can be divided into a preprocessing process and a postprocessing process. The preprocessing process completes the block division of the input sequence and... The calculation of sparse index sp_idx and maximum index max_idx; the post-processing process is to calculate the final function output based on the above results and the input sequence values, while performing sparse operations.

[0037] During preprocessing, the number of data sets to be processed is first calculated (the number of sets is denoted as...). The maximum value in ) and the previous round (the number of groups is denoted as The absolute value of the difference between the maximum values ​​is calculated, and the result is compared with the set sparsity threshold. Compare the results; if the calculated result is greater than the sparsity threshold... If the difference is 1, then set the difference judgment signal diff to 1; otherwise, set it to 0. Simultaneously compare the currently processed data groups (the number of groups is denoted as...). The maximum value in ) and the previous round (the number of groups is denoted as The relationship between the maximum values ​​is represented by the maximum value comparison signal max_comp; if the maximum value in the current data group is less than the maximum value in the previous round, the maximum value comparison signal max_comp is set to 1, otherwise it is set to 0. Other uses The integer part representing the maximum value of the current group is represented by sp_idx for sparse index and max_idx for maximum index. When the difference judgment signal diff=1 and the maximum value comparison signal max_comp=0, the sparse index sp_idx is set to the current group number. Otherwise, the sparse index sp_idx remains unchanged; when the maximum value comparison signal max_comp=0, the maximum value index max_idx is set to 0. Otherwise, the maximum value index max_idx will remain unchanged.

[0038] When the difference judgment signal diff=0, the sparse operation is not performed, and the accumulation is normal. When the difference judgment signal diff=1 and the maximum value comparison signal max_comp=0, it indicates that the maximum value in the currently processed data group is much larger than the already processed data value, and the accumulated sum data of each data block is... It is set to the power of 2 of the current input calculated value of each block; In addition, the sparse index sp_idx and the maximum index max_idx are recorded for sparsity determination during post-processing.

[0039] In the post-processing, for the number of groups Input data smaller than the sparse index sp_idx undergoes sparsification, resulting in an output of 0 (sparse mask). Set to zero; For the number of groups Input data that is greater than the sparse index sp_idx and less than the maximum index max_idx will not undergo sparsification. For the number of groups If the input data is greater than the maximum index max_idx, then according to The value of is used to determine whether to perform sparsification; where This represents a sparse selection signal. Represents the input sequence values. Represents a Boolean function; if sparse selection signal If the value is 1, sparsification is required; otherwise, it is not. The aforementioned software optimization method for the softmax function enables highly parallel computation that supports pipelined technology.

[0040] Compared with the prior art, the beneficial effects of this invention are: This invention provides a hardware acceleration circuit and method for the softmax function of a large language model. The circuit supports configurable computational parallelism and sparse masking. By introducing a sparse masking strategy based on sparse threshold comparison, it fully utilizes the sparsity of the data, solving the problems of high computational complexity, high computational latency, and memory access bottlenecks in the softmax function of large language models. This significantly reduces the computational cost of the softmax function and improves computational efficiency. Furthermore, by dividing the input sequence into several data blocks that can be processed in parallel and employing a three-stage pipeline partitioning, the average single-process latency is shortened to G clock cycles, increasing computational parallelism and greatly improving the computational speed of the softmax function, reducing computational latency and memory access pressure, and improving data processing efficiency. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a schematic diagram of the overall architecture and computational pipeline of the hardware acceleration circuit for the softmax function of a large language model provided in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the sparse processing unit (SPU) in a hardware acceleration circuit for the softmax function of a large language model, provided in Embodiment 1 of the present invention. Figure 3 This is a schematic diagram of the preprocessing unit (PRPU) in a hardware acceleration circuit for the softmax function of a large language model provided in Embodiment 1 of the present invention. Figure 4 This is a schematic diagram of the cache management unit (BMU) in a hardware acceleration circuit for the softmax function of a large language model, provided in Embodiment 1 of the present invention. Figure 5This is a schematic diagram of the post-processing unit (POPU) in a hardware acceleration circuit for the softmax function of a large language model provided in Embodiment 1 of the present invention. Figure 6 The flowchart illustrates a hardware acceleration method for the softmax function of a large language model, as provided in Embodiment 2 of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0044] Example 1 This embodiment provides a hardware acceleration circuit for the softmax function of a large language model, the structure of which is as follows: Figure 1 As shown, it includes a sparse processing unit (SPU), a preprocessing unit (PRPU), a cache management unit (BMU), and a post-processing unit (POPU). The SPU and PRPU are both interconnected with the BMU, and the output of the SPU is also connected to the PRPU. The output of the BMU is connected to the POPU. The Sparse Processing Unit (SPU) is used to perform sparse judgment and execute sparse operations; the Preprocessing Unit (PRPU) is used to calculate the block sum; the Cache Management Unit (BMU) is used for caching and exchanging intermediate data and reducing the block sum; the Postprocessing Unit (POPU) is used to calculate the final function output and sparse mask based on the cached data from the upper level. The Sparse Processing Unit (SPU) receives the maximum value of the previous round group from the Cache Management Unit (BMU) every clock cycle. At the same time, set the maximum value of the current group. The sparse index sp_idx and the maximum value index max_idx are sent to the cache management unit (BMU) for storage.

[0045] Each clock cycle, the Preprocessing Unit (PRPU) receives the PRPU enable signal (prpu_en) and the PRPU selection signal (prpu_sel) from the Sparse Processing Unit (SPU), and receives the maximum value of the previous round group from the Buffer Management Unit (BMU). Compared with the previous round of block division At the same time, the current block and Send to the Cache Management Unit (BMU) register.

[0046] The post-processing unit (POPU) receives the final sparse index o_sp_idx, the final maximum index o_max_idx, and the maximum value m among the currently processed elements from the cache management unit (BMU). N and intermediate variables And based on the input sequence value The final output value is calculated. and sparse mask .

[0047] The cache processing unit (BMU) registers the sparse processing unit (SPU) generated by the cache processing unit (BMU). sparse index sp_idx and maximum value index max_idx, as well as the blocks generated by the preprocessing unit PRPU and .

[0048] The Sparse Processing Unit (SPU) and the Preprocessing Unit (PRPU) run concurrently. The SPU is responsible for executing sparsity checks and corresponding sparsity operations in the algorithm; the PRPU is responsible for calculating the block sum corresponding to each input data in the B data blocks. The B preprocessing units (PRPUs) simultaneously calculate the corresponding blocks and This continues until the G sets of input sequences have been iterated through; where... , represents the number of input data sets, and G represents the number of input data sets in a single data block of the block; , represents the number of data blocks, and B represents the number of data blocks to be divided; The post-processing unit POPU calculates the corresponding function output value and sparse mask based on the final sparse index o_sp_idx and the final maximum index o_max_idx output by the buffer processing unit BMU.

[0049] The structure of the sparse processing unit (SPU) is as follows: Figure 2 As shown, it includes a maximum value calculation module GMC, an index and preprocessing unit control module IPC, a first subtractor SUB1, and a second subtractor SUB2; The maximum value calculation module GMC calculates the current maximum value by accepting the input sequence data group and inputs it into the first subtractor SUB1. The first subtractor SUB1 obtains the maximum value of the previous round group from the cache management unit BMU and performs the subtraction operation, and takes the absolute value of the operation result to obtain the current round group maximum value and the maximum value comparison signal max_comp. Then, the absolute value is transmitted to the second subtractor SUB2, which transmits the current round group maximum value to the cache management unit BMU and the maximum value comparison signal max_comp to the index and preprocessing unit control module IPC. The second subtractor SUB2 subtracts the absolute value output by the first subtractor SUB1 from the preset sparse threshold to obtain the difference judgment signal diff, and transmits it to the index and preprocessing unit control module IPC. The index and preprocessing unit control module IPC generates the preprocessing unit enable signal prpu_en, the preprocessing unit selection signal prpu_sel, the sparse index sp_idx, and the maximum value index max_idx based on the difference judgment signal diff and the maximum value comparison signal max_comp, and transmits them to the cache management unit BMU for temporary storage.

[0050] The maximum value calculation module GMC calculates the maximum value based on the input sequence data group. The maximum value within the group was calculated. Then the maximum value will be compared with Find the difference and take the absolute value of the difference. The first subtractor SUB1 performs a subtraction operation to determine the integer part of the current group's maximum value. And the maximum value comparison signal max_comp; if If it is positive, then Set as Otherwise, retain the original value, i.e. ; if If the value is negative, the maximum value comparison signal max_comp is set to 1; otherwise, the maximum value comparison signal max_comp is set to 0.

[0051] Then, the absolute value is processed by the second subtractor SUB2. With a pre-set sparsity threshold If you do subtraction, If the result is positive, it indicates that the sparsity condition has been met, and the difference judgment signal diff is set to 1; otherwise, it is set to 0.

[0052] The index and preprocessing unit control module IPC generates the preprocessing unit enable signal prpu_en, the preprocessing unit selection signal prpu_sel, the sparse index sp_idx, and the maximum index max_idx based on the difference judgment signal diff and the maximum value comparison signal max_comp, and transmits them to the cache management unit BMU for temporary storage.

[0053] When the difference judgment signal diff=1 and the maximum value comparison signal max_comp=0, the sparse index sp_idx is set to the current group number. Otherwise, the original value is retained; when the maximum value comparison signal max_comp=0, the maximum value index max_idx is set to the current group number. Otherwise, keep the original value.

[0054] In addition, when the difference judgment signal diff=1 and the maximum value comparison signal max_comp=1, the preprocessing unit enable signal prpu_en will be set to 0, and otherwise set to 1; When the difference judgment signal diff=1 and the maximum value comparison signal max_comp=0, the preprocessing unit selection signal prpu_sel will be set to 0; otherwise, it will be set to 1.

[0055] The preprocessing unit (PRPU) is used to implement the preprocessing process of the algorithm, and its structure is as follows: Figure 3 As shown, it includes a first function calculation module SPIE1, an adder ADDER, a subtractor SUB, a multiplexer MUX, and a shifter SHIFT; The minuend input of the subtractor SUB receives the input from the sparse processing unit SPU. The subtraction input receives the buffer processing unit (BMU) from... The shifter SHIFT receives the previous block sum from the subtractor SUB output and the external input. The output of the subtractor SUB is: As the shift bit width; As the number of bits to be shifted. The input port of the first function calculation module SPIE1 receives externally input... The signal, whose output serves as both the output of the adder ADDER and the input of the multiplexer MUX; the output of the adder ADDER serves as another input of the multiplexer MUX; the selection signal of the multiplexer MUX is the preprocessing unit selection signal prpu_sel output from the sparse processing unit SPU, and the output of the multiplexer MUX... As the output of the pre-processing unit PRPU; In the preprocessing unit PRPU, the difference is first calculated by the subtractor SUB. ,in From the cache processing unit (BMU), From the sparse processing unit (SPU); then according to For the difference Perform a shift operation, in which It is the previous round's block sum from the cache of the BMU (Browser Memory Unit). The first function computation module SPIE1 is used for approximate computation. In addition, the preprocessing unit (PRPU) also calculates... , as candidate output blocks.

[0056] The multiplexer in the preprocessing unit (PRPU) selects the output value based on the preprocessing unit selection signal prpu_sel from the sparse processing unit (SPU). The selection is performed; if the preprocessing unit selection signal prpu_sel is 1, then the output value is... Set as Otherwise, output the value. Set directly to ; then It is sent to the cache processing unit (BMU) for caching.

[0057] The structure of the buffer processing unit (BMU) is as follows: Figure 4 As shown, it includes an adder tree module ADDTREE, a leader detector LOD, a PRMAX buffer, a PRSUM buffer, a first multiplexer MUX1, a second multiplexer MUX2, a third multiplexer MUX3, an IDX buffer, a POMAX buffer, and a POSUM buffer.

[0058] The output port of the first multiplexer MUX1 is connected to the input port of the IDX buffer, and the output port of the IDX buffer is connected to the 0 input port of the first multiplexer MUX1. Simultaneously, the 1 input port of the first multiplexer MUX1 is connected to the externally input sparse index sp_idx and maximum value index max_idx. The output port of the IDX buffer also serves as the output port of the buffer management unit (BMU), connecting to the outside. The input port of the PRMAX buffer is connected to the integer part of the current group's maximum value passed in from the outside. The output ports are connected to the external output port of the module and the input port 1 of the second multiplexer MUX2, respectively; the output port of the second multiplexer MUX2 is connected to the input port of the POMAX buffer, and the output port of the POMAX buffer is connected to the input port 0 of the second multiplexer MUX2 and the external output port of the module, respectively; the input port of the PRSUM buffer is connected to the external input block and signal. The output ports are connected to the external output port and input port 1 of the third multiplexer MUX3, respectively; the output port of the third multiplexer MUX3 is connected to the input port of the POSUM buffer; the output port of the POSUM buffer is connected to input port 0 of the third multiplexer MUX3 and input port of the adder tree module ADDTREE; the output port of the adder tree module ADDTREE is connected to the input port of the leader detector LOD; the leader detector LOD outputs intermediate variables. As the output of the preprocessing unit PRPU.

[0059] The PRMAX and PRSUM buffers are used to cache the real-time group maximum values ​​from the sparse processing unit (SPU). And a portion from the pre-processing unit PRPU and .

[0060] The update signal serves as the strobe signal for the first multiplexer MUX1, the second multiplexer MUX2, and the third multiplexer MUX3, and is simultaneously connected to these multiplexers. It is used to indicate that the data iteration calculation is complete. When all G data points in each data block have been traversed, the update signal is pulled high for one clock cycle and remains low for the remaining clock cycles.

[0061] In the next clock cycle, the cache processing unit (BMU) will use the cached value as... , The data is sent to the sparse processing unit (SPU) and the preprocessing unit (PRPU) for iterative computation.

[0062] When the update signal goes high, the first multiplexer MUX1, the second multiplexer MUX2, and the third multiplexer MUX3 will cause the subsequent buffers to cache the final maximum value of the group. And the final block and ,in Cached in the POMAX buffer Cached in the POSUM buffer. This is easy to understand. It is the global maximum value. The data cached in the POSUM buffer is then reduced using an addition tree to obtain the final denominator. Subsequently, intermediate variables in the CPSS algorithm are obtained through a lead detector LOD calculation. ,Right now .

[0063] When the update signal is low, the sparse index sp_idx and the maximum index max_idx from the sparse processing unit (SPU) are latched in the POIDX buffer. When the update signal is high, the index signals are finally updated and latched, and then transmitted to the post-processing unit (POPU). Since the update signal simultaneously controls the first multiplexer MUX1, the second multiplexer MUX2, and the third multiplexer MUX3 to separate the input sequence data, while the IDX buffer, POMAX buffer, POSUM buffer, adder tree module ADDTREE, and leader detector LOD in the buffer processing unit (BMU) buffer the data of the previous sequence, the PRMAX buffer and PRSUM buffer can buffer the data of the next sequence online. Therefore, the buffer processing unit (BMU) enables the entire system to process the input sequence in a pipelined manner.

[0064] The structure of the post-processing unit POPU is as follows: Figure 5 As shown, it includes a second function calculation module SPIE2, a third function calculation module SPIE3, a result control module RC, a multiplexer MUX, a first subtractor SUB1, a second subtractor SUB2, and a third subtractor SUB3.

[0065] The two input ports of the first subtractor SUB1 receive respectively Signals and Signal, and implement The output port is connected to the minuend input port of the second subtractor SUB2; the subtrahend input port of the second subtractor SUB2 receives the sparse threshold. The input ports of the result control module RC receive the output of the second subtractor SUB2, as well as the sparse index sp_idx and the maximum index max_idx, respectively. Its output port is connected to the strobe signal port of the multiplexer MUX. The input port of the second function calculation module SPIE2 receives intermediate variables from the output of the preprocessing unit PRPU. Its output port is connected to the subtrahend input port of the third subtractor SUB3. The minuend input port of the third subtractor SUB3 is connected to the output port of the first subtractor SUB1, and its output port is connected to the input port of the third function calculation module SPIE3; the output port of the third function calculation module SPIE3 is connected to the 0 input port of the multiplexer MUX, the 1 input port of the multiplexer MUX is connected to the 0 signal, and its output port serves as the output port of the entire post-processing unit POPU, connected to the outside.

[0066] The second function calculation module, SPIE2, is used for approximate calculations in the CPSS algorithm. intermediate variables The results come from the upper-level cache processing unit (BMU). The calculation results of the second function calculation module (SPIE2) are intermediate variables required for subsequent calculations by the post-processing unit (POPU).

[0067] The post-processing unit (POPU) calculates the sparse mask through some simple operations. It then performs corresponding calculations on the input values. First, it uses a subtractor to obtain the intermediate value. Then a subtractor is used for comparison. and sparsity threshold The size relationship between them; if Greater than This indicates that the current data being processed has reached the sparsity condition, and the variables... Set to 1, otherwise It is 0.

[0068] The result control module (RC) is used to perform the final sparsity assessment and gating of the output data; for a given input data... If its group number j satisfies Or satisfy Then sparse operation is required. At this time, the RC output strobe signal is 1, and the output result of the post-processing unit POPU is... , ; If the index j of the input data is between the sparse index sp_idx and the maximum index max_idx, no sparse operation is performed, and the RC output strobe signal is 1. When calculating the output, the intermediate value is first obtained. The final output result is then obtained through the third function calculation module SPIE3. , .

[0069] Example 2 This embodiment provides a hardware acceleration method for the softmax function of a large language model. The method is based on the hardware acceleration circuit for the softmax function of a large language model described in Embodiment 1. The method improves the computational parallelism by dividing the input sequence into several data blocks that can be processed in parallel. At the same time, a sparse masking strategy based on sparse threshold comparison is introduced to make full use of the sparsity of the data. This method is based on an optimized softmax function, the expression of which is:

[0070] This method employs a base substitution approach, replacing the base 'e' with a more hardware-friendly base of 2. Additionally, iterative calculations are used to compute the denominator. This represents the i-th input value in the input sequence. This represents intermediate quantities in the iterative calculation of the denominator. This represents the global maximum value; additionally, it replaces the maximum value among the currently processed elements with the integer part of the maximum value among the currently processed elements. To ensure that the relevant calculations are easier to implement in hardware; This method also introduces configurable parallel computation, dividing the input sequence into B data blocks, each containing G input data points. The G input data points in each block are processed in parallel. One data point from each of the B data blocks is processed per clock cycle, with B data blocks processed in parallel per clock cycle. A total of G clock cycles are required to complete the processing of all data. In this method, the data in each block is used to calculate the sum of the denominators for each block according to the above expression, and finally, the sums are accumulated and reduced to obtain the complete denominator.

[0071] This method introduces sparsity judgment based on the above. The proposed hardware acceleration method for the softmax function of the large language model can be divided into a preprocessing process and a postprocessing process. The preprocessing process completes the block division and... The calculation of sparse index sp_idx and maximum index max_idx; the post-processing process is to calculate the final function output based on the above results and the input sequence values, while performing sparse operations.

[0072] The specific flow of the preprocessing and postprocessing processes in this method is as follows: Figure 6 As shown, the entire input sequence is divided into B data blocks, and each data block contains G data points. The parallelism of the entire computation process is B, meaning that B data points are processed simultaneously at each step.

[0073] In the preprocessing stage, the main tasks include processing the sparse index sp_idx, the maximum value index max_idx, and dividing the data into blocks. The calculation takes a total of G clock cycles.

[0074] During the preprocessing stage, the first clock cycle first processes the first set of input data. Processing and calculation are performed to obtain... The values ​​of diff and max_comp are used to calculate the sum of the first round of block divisions. And the sparse index sp_idx and the maximum index max_idx at this time. The next clock cycle will process the second set of input data. The process is performed to calculate the second round of block division. Then update the sparse index sp_idx and the maximum index max_idx. Repeat this process for G clock cycles to calculate the final block sum. and global maximum value Blocking and The reduction and accumulation yields the denominator of the final softmax function. .

[0075] After the preprocessing stage, the algorithm proceeds to the postprocessing stage. Similar to the preprocessing stage, the postprocessing stage has a parallelism of B, processing B data points simultaneously at a time. The postprocessing stage then outputs the final function result. and sparse mask The calculation takes a total of G clock cycles.

[0076] In the first clock cycle, the first set of input data is processed first. The data is processed to calculate the sparse selection signal corresponding to the first set of input data. And further calculate the final function output result corresponding to the first set of input data. and sparse mask In the second clock cycle, the above calculation operation will be performed on the second group of input data to complete the calculation of all input sequence data, thus completing the calculation process of the softmax function software optimization algorithm.

[0077] During preprocessing, the number of data sets to be processed is first calculated (the number of sets is denoted as...). The maximum value in ) and the previous round (the number of groups is denoted as The absolute value of the difference between the maximum values ​​is calculated, and the result is compared with the set sparsity threshold. Compare the results; if the calculated result is greater than the sparsity threshold... If the difference is 1, then set the difference judgment signal diff to 1; otherwise, set it to 0. Simultaneously compare the currently processed data groups (the number of groups is denoted as...). The maximum value in ) and the previous round (the number of groups is denoted as The relationship between the maximum values ​​is represented by the maximum value comparison signal max_comp; if the maximum value in the current data group is less than the maximum value in the previous round, the maximum value comparison signal max_comp is set to 1, otherwise it is set to 0. Other uses The integer part representing the maximum value of the current group is represented by sp_idx for sparse index and max_idx for maximum index. When the difference judgment signal diff=1 and the maximum value comparison signal max_comp=0, the sparse index sp_idx is set to the current group number. Otherwise, the sparse index sp_idx remains unchanged; when the maximum value comparison signal max_comp=0, the maximum value index max_idx is set to 0. Otherwise, the maximum value index max_idx will remain unchanged.

[0078] When the difference judgment signal diff=0, the sparse operation is not performed, and the accumulation is normal. When the difference judgment signal diff=1 and max_comp=0, it indicates that the maximum value in the currently processed data group is much larger than the already processed data value, and the corresponding blocks and data of each data block are different. It is set to the power of 2 of the current input calculated value of each block; In addition, the sparse index sp_idx and the maximum index max_idx are recorded for sparsity determination during post-processing.

[0079] In the post-processing, for the number of groups Input data smaller than the sparse index sp_idx undergoes sparsification, resulting in an output of 0 (sparse mask). Set to zero; For the number of groups Input data that is greater than the sparse index sp_idx and less than the maximum index max_idx will not undergo sparsification. For the number of groups If the input data is greater than the maximum index max_idx, then according to The value of is used to determine whether to perform sparsification; where This represents a sparse selection signal. Represents the input sequence values. Represents a Boolean function; if If the value is 1, then sparsification needs to be performed; otherwise, it is not necessary.

[0080] Some steps in the embodiments of the present invention can be implemented using software, and the corresponding software program can be stored in a readable storage medium, such as an optical disc or a hard disk.

[0081] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A hardware acceleration circuit for the softmax function of a large language model, characterized in that, The hardware acceleration circuit includes: a sparse processing unit (SPU), a preprocessing unit (PRPU), a cache management unit (BMU), and a postprocessing unit (POPU). The input to the sparse processing unit SPU is an input sequence data group, and the output from the cache management unit BMU is judged for sparsity and the corresponding sparse operation is performed according to the preset sparse threshold. The results of the operation are transmitted to the preprocessing unit PRPU and the cache management unit BMU respectively. The preprocessing unit (PRPU) accepts the input sequence data group, combines the outputs from the sparse processing unit (SPU) and the cache management unit (BMU), and transmits the processed result to the cache management unit (BMU). The cache management unit (BMU) is connected to the sparse processing unit (SPU), the preprocessing unit (PRPU), and the postprocessing unit (POPU). The post-processing unit POPU accepts the input sequence data group and the output of the buffer management unit BMU, and outputs the final result.

2. The hardware acceleration circuit according to claim 1, characterized in that, The sparse processing unit (SPU) includes a maximum value calculation module (GMC), an indexing and preprocessing unit control module (IPC), a first subtractor (SUB1), and a second subtractor (SUB2). The maximum value calculation module GMC calculates the current maximum value by accepting the input sequence data group and inputs it into the first subtractor SUB1. The first subtractor SUB1 obtains the maximum value of the previous round group from the cache management unit BMU and performs a subtraction operation, and takes the absolute value of the operation result to obtain the current round group maximum value and the maximum value comparison signal max_comp. Then, the absolute value is transmitted to the second subtractor SUB2, which transmits the current round group maximum value to the cache management unit BMU and the maximum value comparison signal max_comp to the index and preprocessing unit control module IPC. The second subtractor SUB2 subtracts the absolute value output by the first subtractor SUB1 from the preset sparse threshold to obtain the difference judgment signal diff, and transmits it to the index and preprocessing unit control module IPC. The index and preprocessing unit control module IPC generates the preprocessing unit enable signal prpu_en, the preprocessing unit selection signal prpu_sel, the sparse index sp_idx, and the maximum value index max_idx based on the difference judgment signal diff and the maximum value comparison signal max_comp, and transmits them to the cache management unit BMU for temporary storage.

3. The hardware acceleration circuit according to claim 2, characterized in that, The preprocessing unit PRPU is used to implement the preprocessing process of the algorithm, including the first function calculation module SPIE1, adder ADDER, subtractor SUB, multiplexer MUX and shifter SHIFT. The subtractor's minuend input receives the output of the sparse processing unit (SPU), and the subtrahend input receives the output of the buffer processing unit (BMU). The subtractor's output is transmitted to the shifter (SHIFT) as the shift width, and the BMU's output is transmitted to the shifter (SHIFT) as the shifted number for shifting operations. The first function computation module (SPIE1) receives and processes the input sequence data group. Its output and the output of the shifter (SHIFT) are simultaneously connected to the adder (ADDER). The adder's output is connected to the D1 input port of the multiplexer (MUX), and the first function computation module (SPIE1)'s output is connected to the D0 input port of the multiplexer (MUX). The multiplexer (MUX) selects the output based on the preprocessing unit selection signal (prpu_sel) output by the sparse processing unit (SPU), and transmits the output result as the output of the preprocessing unit (PRPU) to the buffer management unit (BMU).

4. The hardware acceleration circuit according to claim 3, characterized in that, The cache processing unit (BMU) includes an adder tree module (ADDTREE), a leader detector (LOD), a PRMAX buffer, a PRSUM buffer, a first multiplexer (MUX1), a second multiplexer (MUX2), a third multiplexer (MUX3), an IDX buffer, a POMAX buffer, and a POSUM buffer. The first multiplexer MUX1 receives the sparse index sp_idx and maximum index max_idx from the sparse processing unit SPU at its D1 input port. Its output is connected to the IDX buffer. The output of the IDX buffer is connected to the D0 input port of the first multiplexer MUX1 on one side and to the post-processing unit POPU on the other. The PRMAX buffer receives the output from the sparse processing unit SPU, and its output is connected to both the sparse processing unit SPU and the D1 input port of the second multiplexer MUX2. The output of the second multiplexer MUX2 is connected to the POMAX buffer, and the output of the POMAX buffer is connected to the second multiplexer MUX2 on one side. The D0 input port of selector MUX2 is connected to the post-processing unit POPU on the other side; the POSUM buffer receives the output of the pre-processing unit PRPU, and its output is connected to the pre-processing unit PRPU and the D1 input port of the third multiplexer MUX3 respectively. The output of the third multiplexer MUX3 is connected to the POSUM buffer, and the output of the POSUM buffer is connected to the D0 input port of the third multiplexer MUX3 on one side and to the adder tree module ADDTREE on the other side; the output of the adder tree module ADDTREE is connected to the leader detector LOD, and the output of the leader detector LOD is connected to the post-processing unit POPU; Among them, the strobe signals of the first multiplexer MUX1, the second multiplexer MUX2, and the third multiplexer MUX3 are update signals.

5. The hardware acceleration circuit according to claim 4, characterized in that, The post-processing unit POPU includes a second function calculation module SPIE2, a third function calculation module SPIE3, a result control module RC, a multiplexer MUX, a first subtractor SUB1, a second subtractor SUB2, and a third subtractor SUB3. The first subtractor SUB1 simultaneously accepts two input sequence data groups and the output of the buffer processing unit BMU. Its output is connected to the minuend of the second subtractor SUB2 and the third subtractor SUB3. The sparse threshold is used as the subtrahend input to the subtrahend end of the second subtractor SUB2. The output of the second subtractor SUB2 is connected to the result control module RC. The result control module RC also accepts the sparse index sp_idx and the maximum value index max_idx output by the buffer processing unit BMU. The output of the result control module RC is connected as a strobe signal to the selection input port of the multiplexer MUX. The second function calculation module SPIE2 accepts the output of the buffer processing unit BMU. Its output is connected to the subtrahend end of the third subtractor SUB3. The output of the third subtractor SUB3 is connected to the third function calculation module SPIE3. The output of the third function calculation module SPIE3 is connected to the D0 input port of the multiplexer MUX. The input of the D1 input port of the multiplexer MUX is a 0 signal. The output of the multiplexer MUX is used as the output of the post-processing unit POPU.

6. A hardware acceleration method for the softmax function of a large language model, characterized in that, The hardware acceleration method is implemented based on the hardware acceleration circuit described in any one of claims 1-5; The hardware acceleration method is as follows: by dividing the input sequence into several data blocks that are processed in parallel, and introducing a sparse masking strategy for sparse threshold comparison to achieve software optimization of the softmax function; The software optimization method includes a preprocessing process and a postprocessing process. In the preprocessing process, the block sum, sparse index, and maximum value index are calculated based on the input data. In the postprocessing process, the final function output is calculated based on the sparse index, maximum value index, and input data calculated in the preprocessing process, while performing sparse operations. The softmax function expression upon which the software optimization method is based is: in, Represents the first in the input sequence data set One input value, This represents intermediate quantities in the iterative calculation of the denominator. Represents the global maximum value. The integer part representing the maximum value among the currently processed elements.

7. The hardware acceleration method according to claim 6, characterized in that, The input sequence is divided into B data blocks, each containing G input data. One data block is processed in each clock cycle. B data blocks are processed in parallel in each clock cycle. It takes a total of G clock cycles to complete the processing of all data. The input data in each data block is used to calculate the sum of the denominators of each data block, and finally the complete denominator is obtained by summation and reduction.

8. The hardware acceleration method according to claim 7, characterized in that, The preprocessing process includes: The current maximum value is calculated by the maximum value calculation module GMC, and then the difference between it and the previous round maximum value is taken and the absolute value is obtained. The current round maximum value and the maximum value comparison signal max_comp are obtained by judgment. If the difference between the current maximum value and the maximum value of the previous round is positive, the current maximum value is set to the current maximum value; otherwise, the previous maximum value is retained. If the difference between the current maximum value and the maximum value of the previous round is negative, then the maximum value comparison signal max_comp is set to 1; otherwise, the maximum value comparison signal max_comp is set to 0. By subtracting the absolute value from the sparsity threshold, a difference judgment signal is obtained based on the result, and it is determined whether the sparsity condition has been met. If the result of the subtraction operation between the absolute value and the sparse threshold is positive, it indicates that the sparsity condition has been met, and the difference judgment signal diff is set to 1; otherwise, it is set to 0. Then, based on the difference judgment signal diff and the maximum value comparison signal max_comp, the preprocessing unit enable signal prpu_en, the preprocessing unit selection signal prpu_sel, the sparse index sp_idx, and the maximum value index max_idx are generated; When the difference judgment signal diff=1 and the maximum value comparison signal max_comp=0, the sparse index sp_idx is set to the current group number and the maximum value index max_idx is set to the current group number; otherwise, the sparse index sp_idx and the maximum value index max_idx remain unchanged. In addition, the preprocessing unit selection signal prpu_sel will be set to 0, and otherwise set to 1. When the difference judgment signal diff=1 and the maximum value comparison signal max_comp=1, the preprocessing unit enable signal prpu_en will be set to 0; otherwise, it will be set to 1. When the difference judgment signal diff=1 and the maximum value comparison signal max_comp=0, the preprocessing unit selection signal prpu_sel will be set to 0; otherwise, it will be set to 1. When the difference judgment signal diff=0, the sparse operation is not performed, and the accumulation is normal. Finally, the preprocessing unit enable signal prpu_en, the preprocessing unit selection signal prpu_sel, the sparse index sp_idx, and the maximum value index max_idx are transmitted to the cache management unit (BMU) for temporary storage.

9. The hardware acceleration method according to claim 8, characterized in that, The post-processing process includes: The final output result is calculated by the function calculation unit, and the sparse selection signal is obtained by the subtractor. Then, the RC control module selects the sparse judgment and output data according to the result. If the current number of groups is less than the sparse index sp_idx, the RC output strobe signal of the result control module is 1, then the sparse operation is performed, and the corresponding output result is 0; If the current number of groups is between the sparse index sp_idx and the maximum index max_idx, then no sparse operation is performed; If the current number of groups is greater than the maximum index max_idx, then the sparsity operation is determined based on the sparsity selection signal. If the sparsity selection signal is 1, then the sparsity operation is performed; otherwise, it is not required.

Images