Softmax module with reconfigurable assembly line
By designing a pipeline-reconfigurable Softmax module and dynamically adjusting the processing mode, the problem of resource utilization and performance bottlenecks in tasks of different input scales is solved, and efficient Softmax calculation is achieved.
Patent Information
- Application Number
- CN202510490186.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-18
- Publication Date
- 2025-08-15
AI Technical Summary
When facing tasks of different input scales, existing Softmax hardware accelerators have problems such as insufficient hardware resource utilization or performance bottlenecks and poor usage flexibility.
Design a pipeline-reconstructible Softmax module. By comparing modules, exponential-accumulative-logarithmic modules and exponential modules, the coarse-grained or fine-grained pipeline modes are dynamically switched according to the size of the data to be processed, and multiple sets of data are processed in parallel to avoid numerical overflow and division operations.
It improves hardware utilization and computing efficiency, reduces unnecessary memory access power consumption and path delay, adapts to task requirements of different scales, and has good flexibility and adaptability.
Smart Images
Figure CN120492033A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of integrated circuits, and in particular to a pipeline reconfigurable Softmax module. Background Art
[0002] The Softmax function is a crucial function in artificial intelligence (AI). Its computational efficiency has always been a key bottleneck in AI, particularly in neural network performance. For example, AI implementations based on the Transformer architecture require extensive Softmax calculations. Implementing Softmax calculations using software can be inefficient and slow, hindering the performance of AI models.
[0003] By using hardware circuits to perform Softmax function calculations, the computational efficiency and speed of the Softmax function can be improved. Such hardware circuits can also be called Softmax hardware accelerators. The principle of the Softmax hardware accelerator is to design specific circuits to perform the sub-function calculations in the Softmax function separately. However, the Softmax function is a multi-input function, and its input scale generally varies with the tasks to be processed by the artificial intelligence model. For example, in natural language processing (NLP) tasks, the attention head of a large language model may involve thousands of dimensions of matrix operations, while in lightweight models or edge computing scenarios, it may only involve dozens or even a few dimensions of operations. That is, when facing different tasks, the input scale of the Softmax function may vary in a wide range from a few to thousands.
[0004] Current Softmax hardware accelerators are generally designed for tasks with a specific input scale. The hardware resource allocation of such Softmax hardware accelerators is fixed. When used in tasks with smaller input scales, they are prone to insufficient hardware resource utilization. When used in tasks with larger input scales, they are prone to hardware resource performance bottlenecks, that is, they have poor flexibility. Summary of the Invention
[0005] In view of the technical problems of poor flexibility in the current Softmax hardware accelerator technology, the present invention aims to provide a pipeline-reconfigurable Softmax module.
[0006] An embodiment of the present invention includes a pipeline reconfigurable Softmax module, wherein the pipeline reconfigurable Softmax module is used to perform Softmax processing on acquired data to be processed, wherein the data to be processed includes an activation value, and the pipeline reconfigurable Softmax module includes:
[0007] A comparison module, configured to determine a global maximum value from the data to be processed in a first working cycle;
[0008] an exponential-accumulation-logarithmic module, configured to perform exponential operation, accumulation operation, and logarithmic operation on the activation value and the global maximum value in a second working period to obtain a second temporary value;
[0009] an exponential module, configured to perform an exponential operation on the second temporarily stored value in a third working cycle to obtain a Softmax processing result of the activation value;
[0010] The comparison module, the exponential-accumulation-logarithm module and the exponential module determine whether to operate in a coarse-grained pipeline mode or a fine-grained pipeline mode according to the size of the data to be processed.
[0011] Furthermore, determining whether to operate in a coarse-grained pipeline mode or a fine-grained pipeline mode according to the size of the data to be processed includes:
[0012] When the size of the data to be processed is greater than the data width threshold, selecting to operate in the coarse-grained pipeline mode;
[0013] When the size of the data to be processed is less than or equal to the data width threshold, the fine-grained pipeline mode is selected.
[0014] Furthermore, the pipeline reconfigurable Softmax module also includes:
[0015] A storage module is used to obtain the data to be processed.
[0016] Furthermore, the storage module is used to obtain a row of elements in the attention map as the data to be processed;
[0017] The first working cycle is any working cycle, the first working cycle, the second working cycle and the third working cycle are sequentially arranged and adjacent working cycles, and the data to be processed is updated as the first working cycle changes.
[0018] Furthermore, the pipeline reconfigurable Softmax module further includes a first register group and a second register group;
[0019] When operating in the coarse-grained pipeline mode, the comparison module transmits intermediate processing results to the exponential-accumulation-logarithm module entirely through the first register group, and the exponential-accumulation-logarithm module transmits intermediate processing results to the exponential module entirely through the second register group;
[0020] When operating in the fine-grained pipeline mode, the comparison module and the exponential-accumulation-logarithm module directly transmit intermediate processing results, and the exponential-accumulation-logarithm module and the exponential module directly transmit intermediate processing results.
[0021] Furthermore, the comparison module includes:
[0022] a comparison unit, configured to obtain data from the storage module and perform data comparison to determine a maximum value of the data;
[0023] an updating unit, configured to update the data stored in the first register group according to a processing result of the comparing unit;
[0024] a first data selecting unit, configured to send the global maximum value or the activation value obtained by the comparing unit as a first temporary value to the first register group;
[0025] The first register unit is used to store the processing result of the comparison unit and send it to the exponential-accumulation-logarithm module.
[0026] Furthermore, the exponential-accumulation-logarithmic module includes:
[0027] a second data selection unit, configured to obtain the first temporary value from the first register group or the data to be processed from the storage module, and send the data to the exponential-accumulation-logarithm operation unit;
[0028] a third data selection unit, configured to obtain the first temporary value from the first register group or obtain the processing result of the comparison unit from the first register unit, and send the result to the exponential-accumulation-logarithm operation unit;
[0029] a fourth data selecting unit, configured to obtain the first temporary value from the first register group or obtain the processing result of the comparing unit from the first register unit, and send the result to the second temporary value calculating unit;
[0030] an exponential-accumulation-logarithmic operation unit, configured to sequentially perform exponential, accumulation, and logarithmic operations on the data sent by the second data selection unit and the third data selection unit;
[0031] a second temporary value calculation unit, configured to process the data sent by the fourth data selection unit to obtain the second temporary value;
[0032] The second register unit is used to store the processing result of the exponential-accumulation-logarithm operation unit and send it to the exponential module.
[0033] Furthermore, the index module includes:
[0034] a fifth data selecting unit, configured to obtain the processing result of the exponential-accumulation-logarithm operation unit from the second register unit or obtain the data to be processed from the storage module, and send the result to the exponential value calculation unit;
[0035] an exponential value calculation unit, configured to obtain the second temporary value from the second register group, and process the second temporary value and the data sent by the fifth data selection unit to obtain an exponential value;
[0036] An exponential operation unit is used to perform an exponential operation on the exponential value to obtain a Softmax processing result of the activation value.
[0037] Furthermore, when the pipeline reconfigurable Softmax module operates in the coarse-grained pipeline mode:
[0038] The comparison unit is configured to compare the data to be processed through a plurality of clock cycles, wherein in any clock cycle, a portion of data whose size does not exceed a data width threshold is obtained from the data to be processed, a local maximum value corresponding to the portion of data is obtained, and the local maximum value is sent to the update unit;
[0039] The updating unit is configured to compare, within any clock cycle, the local maximum value generated by the comparing unit in the same clock cycle with the first temporary value stored in the first register group, and update the first register group using the larger one as a new first temporary value; the first temporary value after being updated over all clock cycles is the global maximum value;
[0040] The second data selection unit is used to obtain the data to be processed from the storage module and send it to the exponential-accumulation-logarithm operation unit;
[0041] The third data selection unit is used to obtain the first temporary value from the first register group and send it to the exponential-accumulation-logarithm operation unit;
[0042] The fourth data selection unit is used to obtain the first temporary value from the first register group and send it to the second temporary value calculation unit;
[0043] The fifth data selection unit is used to obtain the data to be processed from the storage module and send it to the index value calculation unit.
[0044] Furthermore, when the pipeline reconfigurable Softmax module operates in the fine-grained pipeline mode:
[0045] The comparison unit is used to determine a global maximum value from the data to be processed, and send the global maximum value to the first register unit;
[0046] The first data selection unit is used to send the activation value as a first temporary value to the first register group;
[0047] The second data selection unit is used to obtain the first temporary value from the first register group and send it to the exponential-accumulation-logarithm operation unit;
[0048] The third data selection unit is used to obtain the processing result of the comparison unit from the first register unit and send it to the exponential-accumulation-logarithm operation unit;
[0049] The fourth data selection unit is used to obtain the processing result of the comparison unit from the first register unit and send it to the second temporary value calculation unit;
[0050] The fifth data selection unit is used to obtain the processing result of the exponential-accumulation-logarithm operation unit from the second register unit and send it to the exponential value calculation unit.
[0051] The beneficial effects of the present invention are as follows: the pipeline-reconfigurable Softmax module in the embodiment has the advantages of avoiding numerical overflow and eliminating the need for division operations, and realizes the selection of coarse-grained pipeline mode and fine-grained pipeline mode by sensing the size Ma of the data to be processed, thereby realizing pipeline reconfiguration; wherein, when Ma is large, the coarse-grained pipeline mode is executed, which can realize parallel pipeline processing of multiple groups of data to be processed, thereby improving the hardware utilization of each module and the computational efficiency of the Softmax function; when Ma is small, the fine-grained pipeline mode is executed, which can reduce unnecessary memory access power consumption and path delay; therefore, the pipeline-reconfigurable Softmax module has good flexibility and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] Figure 1 Schematic diagram of the structure of the pipeline reconfigurable Softmax module in the embodiment;
[0053] Figure 2 Schematic diagram of the working sequence of the pipeline reconfigurable Softmax module in the embodiment;
[0054] Figure 3 Schematic diagram of the working principle of the pipeline reconfigurable Softmax module in the coarse-grained pipeline mode in the embodiment;
[0055] Figure 4 Schematic diagram of the working principle of the pipeline reconfigurable Softmax module in the fine-grained pipeline mode in the embodiment. DETAILED DESCRIPTION
[0056] For a set of data x1, x2...x i ...x Ma If you want to perform a specific test on one of the data x i Perform Softmax calculation, then this specific data x i It is called the activation value, and the calculation formula of Softmax is:
[0057]
[0058] Since formula (1) contains exponential operations, when the exponent is large, the intermediate result will become very large. Therefore, if we directly calculate according to formula (1), it is easy to cause numerical overflow. Formula (1) can be transformed into
[0059] (2)
[0060] In formula (2), x max It is x1, x2...x i ...x Ma The maximum value among them. Using formula (2) to calculate Softmax, the numerical overflow problem can be alleviated because the exponent becomes smaller.
[0061] Although formula (2) solves the problem of numerical stability, if the Softmax module is designed according to formula (2), then such a Softmax module needs to perform three key steps in sequence: (1) Find the maximum value x max (2) Calculate the exponential and accumulation; (3) Perform the division normalization operation. This serial computing mode results in a time complexity as high as O(3n), requiring multiple data traversals. When implemented in hardware, it consumes a large amount of computing resources and memory bandwidth, which is particularly inefficient for resource-constrained edge computing devices.
[0062] However, if a block-based approach is used, large-scale Softmax calculations are broken down into multiple sub-blocks. While this approach can alleviate memory pressure, due to the strict dependencies between the computational steps, true pipeline parallelism cannot be achieved. Especially when processing dynamically changing input sizes, a fixed block-based strategy can lead to two extreme situations: for small inputs, block-based calculations actually add additional control overhead; for very large inputs, data dependencies between blocks can cause serious pipeline stalls.
[0063] Based on the problems of scale change in the Softmax calculation process and the low efficiency caused by the need for serial processing, in this embodiment, the Softmax calculation formula is designed based on formula (1):
[0064] (3)
[0065] And design the Softmax module based on formula (3).
[0066] Formula (3) is a set of data x1, x2...x i ...x Ma Processing is performed to calculate any activation value x i Softmax calculation result Softmax(x i ).
[0067] In formula (3), the exponential term is reduced by the maximum value x max It can prevent numerical overflow caused by excessively large exponential terms, and there is no need to perform division operations, thereby greatly reducing the complexity of hardware design. However, the mathematical implementation of formula (3) itself still has data dependency problems, that is, the calculation of some steps requires the calculation results of the previous step, and each step needs to traverse all x values, which increases both the calculation delay and the number of memory accesses. If the size of Ma is not distinguished and a unified calculation is performed, it will have a negative impact. For example, after the hardware structure is designed, for data to be processed with a smaller Ma, the delay and power consumption caused by multiple memory accesses are unnecessary.
[0068] In order to implement the Softmax calculation based on formula (3) and alleviate the problems of unnecessary memory access delay and power consumption, in this embodiment, a Ma-aware pipeline reconfigurable Softmax module is designed.
[0069] In this embodiment, the structure of the pipeline reconfigurable Softmax module designed based on formula (3) is as follows: Figure 1 As shown. Figure 1 ,The pipeline reconfigurable Softmax module includes hardware modules such as storage module, ,comparison module, exponential-accumulation-logarithmic module and exponential module, ,which can be manufactured by integrated circuit technology.
[0070] In this embodiment, the storage module is used to store or obtain data to be processed from the outside. A set of data to be processed may include Ma data, which can be specifically represented as x1, x2...x i ...x Ma , which includes the activation value x i. Wherein Ma=1-8192. Specifically, a group of data to be processed may be data that have a certain relationship with each other in the processing of the artificial intelligence model. For example, a group of data to be processed may be a row of data in the attention map obtained by processing the artificial intelligence model of the Transformer structure. When an attention map is stored in the storage module, the storage module stores multiple rows of data in the attention map, that is, multiple groups of data to be processed. In this embodiment, unless otherwise specified, when "data to be processed" is mentioned, it may refer to a specific group of data to be processed x1, x2...x i ...x Ma .
[0071] In this embodiment, the comparison module, the exponential-accumulation-logarithmic module and the exponential module form a pipeline working mode. For example, for a set of data to be processed x1, x2...x i ...x Ma , the comparison module, the exponential-accumulation-logarithmic module and the exponential module perform corresponding processing in the first working cycle, the second working cycle and the third working cycle respectively, so as to process the data x1, x2...x i ...x Ma Perform Softmax calculation.
[0072] Specifically, the comparison module, in the first working cycle, compares the data to be processed x1, x2, ..., x i ...x Ma Determine the global maximum x max After the first working cycle ends, the second working cycle begins. In the second working cycle, the exponential-accumulation-logarithmic module is activated according to the activation value x. i and the global maximum x max Perform exponential operation, accumulation operation and logarithmic operation to obtain the second temporary value tmp reg2 After the second working cycle ends, the third working cycle begins. During the third working cycle, the index module calculates the value of the second temporary storage value tmp reg2 Perform exponential operation to obtain the activation value x i Softmax processing result Softmax(x i ), thus completing the activation value x i Softmax calculation. Since the activation value x i You can process data x1, x2...x i ...x Ma Therefore, the Softmax module can complete the Softmax calculation of all data.
[0073] In this embodiment, the data width of the interface between the storage module and the comparison module in the Softmax module can be fixed. In this embodiment, the size of this fixed value is called the data width threshold, which can be 32. In this way, the comparison module and other modules can read data with a number of bits equal to the data width threshold from the storage module in each clock cycle, that is, read a group of 32 data in each clock cycle. Therefore, for a group of to-be-processed data x1, x2, ..., x stored in the storage module, i ...x Ma , the comparison module takes clock cycles, i.e. clock cycles to read it completely, where is the ceiling function.
[0074] In this embodiment, the Figure 2 The working cycle of each module in the Softmax module is set as shown below. The length of each working cycle is equal to clock cycles, so the length of the working cycle can be dynamically adjusted according to the length Ma of a group of data to be processed (specifically equivalent to the number of elements in a row in the attention map).
[0075] Reference Figure 2 , assuming that the comparison module reads a set of to-be-processed data corresponding to the first row of elements in the attention map from the storage module in a specific working cycle, namely the first working cycle, and processes it. After the processing is completed, the comparison module sends the processing result to the exponential-accumulation-logarithm module, and enters the second working cycle. The exponential-accumulation-logarithm module processes the data from the comparison module (corresponding to the first row of elements in the attention map). After the processing is completed, the exponential-accumulation-logarithm module sends the processing result to the exponential module, and enters the third working cycle. The exponential module processes the data from the exponential-accumulation-logarithm module (corresponding to the first row of elements in the attention map), thereby forming a pipeline processing method.
[0076] Reference Figure 2 Since the comparison module has completed the processing of the set of data to be processed corresponding to the first row of elements in the attention map after the first working cycle, the comparison module can read a new set of data to be processed (such as the second row of elements in the attention map) from the storage module for processing in the second working cycle. That is, for the comparison module, the second working cycle is equivalent to its "first working cycle", that is, the "first working cycle" has changed over time, and the "data to be processed" to be processed by the comparison module has also changed accordingly.
[0077] pass Figure 2The pipeline working mode shown can improve the utilization of each hardware module in the Softmax module and improve the computational efficiency of the Softmax function.
[0078] In this embodiment, refer to Figure 1 The comparison module is provided with a first data selection unit, the exponential-accumulation-logarithm module is provided with a second data selection unit, a third data selection unit and a fourth data selection unit, and the exponential module is provided with a fifth data selection unit. These data selection units have two output terminals, 0 and 1, respectively, and can respectively select the data received at their respective input terminals to be output from the output terminal 0 or the output terminal 1, thereby selecting the data flow direction within each module and between modules, so that the specific connection structure of the pipeline between the comparison module, the exponential-accumulation-logarithm module and the exponential module changes. Therefore, the pipeline formed by the comparison module, the exponential-accumulation-logarithm module and the exponential module is reconfigurable.
[0079] In this embodiment, the storage module can sense the size of the length Ma of a single group of data to be processed and determine the relative size of Ma and the data width threshold (32); if the size Ma of the single group of data to be processed is greater than the data width threshold (32), then the coarse-grained pipeline mode (CGP) is selected; if the size Ma of the single group of data to be processed is less than or equal to the data width threshold (32), then the fine-grained pipeline mode (FGP) is selected.
[0080] Specifically, if the coarse-grained pipeline mode CGP is selected, the storage module or other components can send control instructions to each data selection unit, so that each data selection unit selects to output data from its own 1 output terminal, so that the data in the Softmax module flows to Figure 1 As shown by the solid arrow in Figure 3 As shown; if the fine-grained pipeline mode FGP is selected, the storage module or other components can send control instructions to each data selection unit, so that each data selection unit selects to output data from its own 0 output terminal, so that the data flow in the Softmax module is as follows Figure 1 As shown by the dotted arrow in Figure 4 shown.
[0081] 1. Coarse-grained pipeline mode CGP
[0082] In this embodiment, the coarse-grained pipeline mode (CGP) is implemented for Ma > 32. In this case, since the size of a single group of data to be processed, Ma, is greater than the data width threshold (32), the data to be processed needs to be divided into multiple groups of 32. The comparison module reads one group in each clock cycle, thereby completing the reading in multiple clock cycles.
[0083] Reference Figure 3 In the coarse-grained pipeline mode CGP, the comparison module, exponential-accumulation-logarithmic module, exponential module and its internal units work according to the following process:
[0084] S1. The comparison unit processes data x1, x2, ... x through multiple clock cycles. i ...x Ma Compare, where in any clock cycle, from the data to be processed x1, x2...x i ...x Ma Obtaining partial data whose size does not exceed the data width threshold, obtaining the local maximum value corresponding to the partial data, and sending the local maximum value to the update unit;
[0085] S2. The update unit compares the local maximum value generated by the comparison unit in the same clock cycle with the first temporary value stored in the first register group in any clock cycle, and updates the first register group with the larger one as the new first temporary value; after all clock cycles, the first temporary value tmp is updated. reg1 is the global maximum x max ;
[0086] Specifically, the data to be processed with a length of Ma can be divided into Group partial data, wherein the first group partial data includes x1, x2...x 32 etc. 32 data, the second group of data includes x 33 、x2……x 64 and so on for 32 data... (the length of the last group of partial data may be less than 32); the comparison unit traverses each group of partial data, and the comparison unit can compare the local maximum value in this group of partial data from a group of partial data (no more than 32 data) within one clock cycle. For example, assuming that the comparison unit obtains the local maximum value of the first group of partial data as x2, the comparison unit sends the local maximum value x2 to the update unit, and the update unit compares the local maximum value x2 with the first temporary value tmp stored in the first register group through the first data selection unit. reg1 (The initial value can be 0) and compare it with the first temporary value tmp stored in the first register group reg1 Updated to x2 and tmp reg1The larger one between the two, then the comparison unit obtains the local maximum value from the second set of partial data (assuming x 63 ), then the comparison unit will take this local maximum value x 63 Send it to the update unit, which uses the first data selection unit to convert this local maximum value x 63 The first temporary value tmp stored in the first register group reg1 Compare and store the first temporary value tmp in the first register group reg1 Update to x 63 with tmp reg1 The larger one between ... until the pending data x1, x2 ... x is completed i ...x Ma After processing all the data in the first register group, the first temporary value tmp stored in the first register group reg1 That is the data to be processed x1, x2...x i ...x Ma The global maximum x in max ;
[0087] S3. The second data selection unit obtains the data to be processed x1, x2...x from the storage module i ...x Ma , sent to the exponential-accumulation-logarithm operation unit;
[0088] S4. The third data selection unit obtains the first temporary value tmp from the first register group. reg1 =x max , sent to the exponential-accumulation-logarithm operation unit;
[0089] S5 exponential - accumulation - logarithmic operation unit according to the data sent by the second data selection unit and the third data selection unit, sequentially perform exponential, accumulation and logarithmic operations;
[0090] Specifically, the exponential-accumulation-logarithm operation unit obtains x1, x2...x from the second data selection unit and the third data selection unit. i ...x Ma and x max etc., perform exponential, accumulation, and logarithmic operations in sequence:
[0091] Exponential operation: tmp i =exp(x i -x max )
[0092] Accumulation operation:
[0093] Logarithm operation: tmp ln =ln(tmp sum )
[0094] The exponential-accumulation-logarithm operation unit will perform exponential, accumulation and logarithmic operations on the result tmp ln Sending to the second temporary value calculation unit;
[0095] S6. The fourth data selection unit obtains the first temporary value tmp from the first register group. reg1 =x max , sent to the second temporary value calculation unit;
[0096] S7. The second temporary value calculation unit processes the data sent by the fourth data selection unit to obtain a second temporary value;
[0097] Specifically, the second temporary value calculation unit obtains tmp from the exponential-accumulation-logarithm operation unit. ln , obtain x from the fourth data selection unit max , calculated according to the following formula:
[0098] tmp reg2 =x max +tmp ln
[0099] Thus the second temporary value tmp is calculated reg2 The second temporary value calculation unit calculates the second temporary value tmp reg2 Send to the second register group for storage;
[0100] S8. The fifth data selection unit obtains the data to be processed x1, x2...x from the storage module i ...x Ma (or just get the activation value x i ), sent to the index value calculation unit;
[0101] S9. The exponential value calculation unit obtains the second temporary value tmp from the second register group. reg2 , according to the second temporary value tmp reg2 The data x sent by the fifth data selection unit i Perform processing to obtain index value;
[0102] Specifically, the exponent value calculation unit obtains the second temporary value tmp from the second register group. reg2 =x max +tmp ln , obtain the activation value x from the fifth data selection unit i , calculated according to the following formula:
[0103] x i ′=x i -tmp reg2
[0104] Thus, the exponential value x is calculated i ′, the exponential value calculation unit converts the exponential value x i 'Sent to the exponential operation unit;
[0105] S10. Exponentiation operation unit for exponential value x i 'Perform exponential operation to obtain the Softmax processing result of the activation value;
[0106] Specifically, the exponential operation unit performs calculations according to the following formula:
[0107] Softmax(x i )=exp(x i ′)
[0108] Thus the activation value x is calculated i Softmax processing result Softmax(x i ).
[0109] According to the process of S1-S10 and Figure 3 As shown in the data flow, when working in the coarse-grained pipeline mode CGP, the comparison module passes x completely to the exponential-accumulation-logarithm module through the first register group. max The exponential-accumulation-logarithm module passes the intermediate processing results to the exponential module completely through the second register group. ln Wait for the intermediate processing results.
[0110] According to the process of S1-S10 and Figure 3 As shown in the data flow, when working in the coarse-grained pipeline mode CGP, the data to be processed x1, x2...x i ...x Ma Divide into multiple groups and enter the comparison module cycle by cycle. Such a pipeline structure can achieve Figure 2 The processing timing shown, that is, in the same working cycle (for example, the third working cycle), the comparison module, the exponential-accumulation-logarithmic module and the exponential module respectively process different data to be processed (for example, data from different rows of the attention map). In this way, although there is data dependency between the three modules for the processing of the same set of data to be processed, the pipeline processing of different groups of data to be processed (between rows of the attention map) is realized, which can enhance the hardware utilization of each module and improve the computational efficiency of the Softmax function.
[0111] 2. Fine-grained pipeline mode FGP
[0112] In this embodiment, the fine-grained pipeline mode FGP is implemented for Ma ≤ 32. In this case, since the size of a single group of data to be processed Ma ≤ the data width threshold (32), the comparison module can read and process all the data to be processed in one clock cycle.
[0113] Reference Figure 4 In the fine-grained pipeline mode FGP, the comparison module, exponential-accumulation-logarithmic module, exponential module and its internal units work according to the following process:
[0114] P1. The comparison unit compares the data to be processed x1, x2...x i ...x Ma Determine the global maximum x max , the global maximum value x max Send to the first register unit D1;
[0115] P2. The first data selection unit will activate the value x i As the first temporary value tmp reg1 , sent to the first register group;
[0116] P3. The second data selection unit obtains the first temporary value tmp from the first register group. reg1 =x i , sent to the exponential-accumulation-logarithm operation unit;
[0117] P4. The third data selection unit obtains the processing result of the comparison unit from the first register unit D1, that is, the global maximum value x max , sent to the exponential-accumulation-logarithm operation unit;
[0118] P5 exponential - accumulation - logarithmic operation unit according to the data sent by the second data selection unit and the third data selection unit, followed by exponential, accumulation and logarithmic operations;
[0119] Specifically, the exponential-accumulation-logarithm operation unit obtains x1, x2...x from the second data selection unit and the third data selection unit. i ...x Ma and x max etc., perform exponential, accumulation, and logarithmic operations in sequence:
[0120] Exponential operation: tmp i =exp(x i -x max )
[0121] Accumulation operation:
[0122] Logarithm operation: tmp ln=ln(tmp sum )
[0123] The exponential-accumulation-logarithm operation unit will perform exponential, accumulation and logarithmic operations on the result tmp ln Send to the second register unit D2 for storage;
[0124] P6. The fourth data selection unit obtains the processing result of the comparison unit, i.e., the global maximum value x, from the first register unit D1. max , sent to the second temporary value calculation unit;
[0125] P7. The second temporary value calculation unit processes the data sent by the fourth data selection unit to obtain a second temporary value;
[0126] Specifically, the second temporary value calculation unit obtains x from the fourth data selection unit. max , get tmp from the first register group reg1 =x i , calculated according to the following formula:
[0127] tmp reg2 =x i -x max
[0128] Thus the second temporary value tmp is calculated reg2 The second temporary value calculation unit calculates the second temporary value tmp reg2 Send to the second register group for storage;
[0129] P8. The fifth data selection unit obtains the processing result of the exponential-accumulation-logarithm operation unit from the second register unit D2, that is, tmp ln , sent to the exponential value calculation unit;
[0130] P9. The exponential value calculation unit obtains the second temporary value tmp from the second register group. reg2 , according to the second temporary value tmp reg2 The data tmp sent by the fifth data selection unit ln Perform processing to obtain index value;
[0131] Specifically, the exponent value calculation unit obtains the second temporary value tmp from the second register group. reg2 =x i -x max , get tmp from the second register unit D2 ln , calculated according to the following formula:
[0132] x i ′=tmp reg2 -tmp ln
[0133] Thus, the exponential value x is calculated i ′, the exponential value calculation unit converts the exponential value x i 'Sent to the exponential operation unit;
[0134] P10. Exponentiation unit for exponential value x i 'Perform exponential operation to obtain the Softmax processing result of the activation value;
[0135] Specifically, the exponential operation unit performs calculations according to the following formula:
[0136] Softmax(x i )=exp(x i ′)
[0137] Thus the activation value x is calculated i Softmax processing result Softmax(x i ).
[0138] According to the process of P1-P10 and Figure 4 As shown in the data flow, when working in the fine-grained pipeline mode FGP, the comparison module also directly passes x to the exponential-accumulation-logarithm module through the first register unit D1 set by it. max The exponential-accumulation-logarithm module also directly transmits tmp to the exponential module through the second register unit D2 set by it. ln Wait for the intermediate processing results.
[0139] According to the process of P1-P10 and Figure 4 As can be seen from the data flow shown, when operating in fine-grained pipeline mode FGP, the working cycles of the comparison module, exponential-accumulation-logarithmic module, and exponential module, namely the first working cycle, the second working cycle, and the third working cycle, each require only one clock cycle. In other words, under the fine-grained pipeline mode FGP, the calculation time of the Softmax function only requires 3 clock cycles, which can reduce the path delay caused by data dependence; the fine-grained pipeline mode FGP fully reuses the first register group and the second register group without the need for additional storage access, which not only reduces memory access power consumption but also avoids additional storage or computing resource overhead.
[0140] The pipeline reconfigurable Softmax module in this embodiment implements the Softmax calculation based on formula (3), which has the advantages of avoiding numerical overflow and not requiring division operations. In addition, the coarse-grained pipeline mode and the fine-grained pipeline mode are selected by sensing the size Ma of the data to be processed, thereby realizing pipeline reconfiguration. When Ma is large, the coarse-grained pipeline mode is executed, which can realize parallel pipeline processing of multiple groups of data to be processed (between rows of the attention map), thereby improving the hardware utilization of each module and the calculation efficiency of the Softmax function. When Ma is small, the fine-grained pipeline mode is executed, which can reduce unnecessary memory access power consumption and path delay.
[0141] Since the size of Ma reflects the scale of the data to be processed (generally the attention map), the pipeline-reconfigurable Softmax module can realize two pipeline modes, which can perform Softmax function calculations on attention maps of different sizes, and adapt to the specific needs of different attention maps while optimizing performance. Therefore, the pipeline-reconfigurable Softmax module has good flexibility and adaptability.
[0142] Moreover, in CGP mode, the maximum value search, exponential accumulation and normalization stages are used to process different rows of data in parallel, and in FGP mode, the intermediate results are passed directly through registers to avoid memory access delays, thereby eliminating the serial dependency bottleneck of traditional Softmax through inter-row parallelism and register pass-through; in FGP mode, the register group stores xi and xi-xmax separately, and in CGP mode, the register group serves as a data transfer station for the inter-row pipeline, so that the same group of registers can be reused in different modes as maximum value storage, intermediate result cache or output buffer, thereby realizing dynamic reuse of hardware resources and improving hardware utilization.
[0143] It should be noted that, unless otherwise specified, when a feature is referred to as being "fixed" or "connected" to another feature, it may be directly fixed or connected to the other feature, or it may be indirectly fixed or connected to the other feature. In addition, the descriptions of up, down, left, right, etc. used in this disclosure are only relative to the relative positional relationships of the components of the present disclosure in the accompanying drawings. The singular forms of "a", "" and "the" used in this disclosure are also intended to include the plural forms, unless the context clearly indicates otherwise. In addition, unless otherwise defined, all technical and scientific terms used in this embodiment have the same meaning as those generally understood by those skilled in the art. The terms used in the specification of this embodiment are only for describing specific embodiments and are not intended to limit the invention. The term "and / or" used in this embodiment includes any combination of one or more related listed items.
[0144] It should be understood that, although the present disclosure may adopt the term first, second, third etc. to describe various elements, these elements should not be limited to these terms.These terms are only used to distinguish the elements of the same type from each other.For example, without departing from the scope of the present disclosure, the first element may also be referred to as the second element, and similarly, the second element may also be referred to as the first element.The use of any and all examples or exemplary language ("for example", "such as" etc.) provided by the present embodiment is only intended to better illustrate embodiments of the present invention, and unless otherwise required, the scope of the present invention will not be limited.
[0145] It should be appreciated that embodiments of the present invention can be implemented or practiced by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The methods can be implemented in a computer program using standard programming techniques - including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner - according to the methods and figures described in the specific embodiments. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system. However, if desired, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. In addition, the program can be run on a programmed application-specific integrated circuit for this purpose.
[0146] In addition, the operations of the process described in this embodiment may be performed in any suitable order, unless otherwise indicated in this embodiment or otherwise clearly contradicted by the context. The process described in this embodiment (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as a code (e.g., executable instructions, one or more computer programs, or one or more applications) executed on one or more processors, by hardware or a combination thereof. A computer program includes a plurality of instructions that may be executed by one or more processors.
[0147] Furthermore, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented as machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, RAM, ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or portions thereof, can be transmitted over a wired or wireless network. When such media includes instructions or programs that implement the above steps in conjunction with a microprocessor or other data processor, the invention of this embodiment includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention also includes the computer itself.
[0148] The computer program can be applied to input data to perform the functions of the present embodiment, thereby converting the input data to generate output data that is stored in a non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the converted data represents a physical and tangible object, including a specific visual depiction of the physical and tangible object produced on the display.
[0149] The above are merely preferred embodiments of the present invention. The present invention is not limited to the aforementioned embodiments. As long as the technical effects of the present invention are achieved by the same means, any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention. Within the scope of protection of the present invention, various modifications and variations of the technical solutions and / or implementation methods may be made.
Claims
1. A pipeline reconfigurable Softmax module, characterized in that: The pipeline reconfigurable Softmax module is used to perform Softmax processing on the acquired data to be processed, where the data to be processed includes an activation value. The pipeline reconfigurable Softmax module includes: A comparison module, configured to determine a global maximum value from the data to be processed in a first working cycle; an exponential-accumulation-logarithmic module, configured to perform exponential operation, accumulation operation, and logarithmic operation on the activation value and the global maximum value in a second working period to obtain a second temporary value; an exponential module, configured to perform an exponential operation on the second temporarily stored value in a third working cycle to obtain a Softmax processing result of the activation value; The comparison module, the exponential-accumulation-logarithm module and the exponential module determine whether to operate in a coarse-grained pipeline mode or a fine-grained pipeline mode according to the size of the data to be processed.
2. The pipeline reconfigurable Softmax module according to claim 1, characterized in that The determining whether to operate in a coarse-grained pipeline mode or a fine-grained pipeline mode according to the size of the data to be processed includes: When the size of the data to be processed is greater than the data width threshold, selecting to operate in the coarse-grained pipeline mode; When the size of the data to be processed is less than or equal to the data width threshold, the fine-grained pipeline mode is selected.
3. The pipeline reconfigurable Softmax module according to claim 1 or 2, characterized in that: The pipeline reconfigurable Softmax module also includes: A storage module is used to obtain the data to be processed.
4. The pipeline reconfigurable Softmax module according to claim 3, characterized in that: The storage module is used to obtain a row of elements in the attention map as the data to be processed; The first working cycle is any working cycle, the first working cycle, the second working cycle and the third working cycle are sequentially arranged and adjacent working cycles, and the data to be processed is updated as the first working cycle changes.
5. The pipeline reconfigurable Softmax module according to claim 3, characterized in that: The pipeline reconfigurable Softmax module further includes a first register group and a second register group; When operating in the coarse-grained pipeline mode, the comparison module transmits intermediate processing results to the exponential-accumulation-logarithm module entirely through the first register group, and the exponential-accumulation-logarithm module transmits intermediate processing results to the exponential module entirely through the second register group; When operating in the fine-grained pipeline mode, the comparison module and the exponential-accumulation-logarithm module directly transmit intermediate processing results, and the exponential-accumulation-logarithm module and the exponential module directly transmit intermediate processing results.
6. The pipeline reconfigurable Softmax module according to claim 5, characterized in that The comparison module includes: a comparison unit, configured to obtain data from the storage module, perform data comparison and determine the maximum value of the data; an updating unit, configured to update the data stored in the first register group according to a processing result of the comparing unit; a first data selecting unit, configured to send the global maximum value or the activation value obtained by the comparing unit as a first temporary value to the first register group; The first register unit is used to store the processing result of the comparison unit and send it to the exponential-accumulation-logarithm module.
7. The pipeline reconfigurable Softmax module according to claim 6, characterized in that: The exponential-accumulation-logarithmic module includes: a second data selection unit, configured to obtain the first temporary value from the first register group or the data to be processed from the storage module, and send the data to the exponential-accumulation-logarithm operation unit; a third data selection unit, configured to obtain the first temporary value from the first register group or obtain the processing result of the comparison unit from the first register unit, and send the result to the exponential-accumulation-logarithm operation unit; a fourth data selecting unit, configured to obtain the first temporary value from the first register group or obtain the processing result of the comparing unit from the first register unit, and send the result to the second temporary value calculating unit; an exponential-accumulation-logarithmic operation unit, configured to sequentially perform exponential, accumulation, and logarithmic operations on the data sent by the second data selection unit and the third data selection unit; a second temporary value calculation unit, configured to process the data sent by the fourth data selection unit to obtain the second temporary value; The second register unit is used to store the processing result of the exponential-accumulation-logarithm operation unit and send it to the exponential module.
8. The pipeline reconfigurable Softmax module according to claim 7, characterized in that: The index module includes: a fifth data selecting unit, configured to obtain the processing result of the exponential-accumulation-logarithm operation unit from the second register unit or obtain the data to be processed from the storage module, and send the result to the exponential value calculation unit; an exponential value calculation unit, configured to obtain the second temporary value from the second register group, and process the second temporary value and the data sent by the fifth data selection unit to obtain an exponential value; An exponential operation unit is used to perform an exponential operation on the exponential value to obtain a Softmax processing result of the activation value.
9. The pipeline reconfigurable Softmax module according to claim 8, characterized in that: When the pipeline reconfigurable Softmax module works in the coarse-grained pipeline mode: The comparison unit is configured to compare the data to be processed through a plurality of clock cycles, wherein in any clock cycle, a portion of data whose size does not exceed a data width threshold is obtained from the data to be processed, a local maximum value corresponding to the portion of data is obtained, and the local maximum value is sent to the update unit; The updating unit is configured to compare the local maximum value generated by the comparing unit in the same clock cycle with the first temporary value stored in the first register group within any clock cycle, and use the larger one as a new first temporary value to update the first register group; The first temporarily stored value after being updated through all clock cycles is the global maximum value; The second data selection unit is used to obtain the data to be processed from the storage module and send it to the exponential-accumulation-logarithm operation unit; The third data selection unit is used to obtain the first temporary value from the first register group and send it to the exponential-accumulation-logarithm operation unit; The fourth data selection unit is used to obtain the first temporary value from the first register group and send it to the second temporary value calculation unit; The fifth data selection unit is used to obtain the data to be processed from the storage module and send it to the index value calculation unit.
10. The pipeline reconfigurable Softmax module according to claim 8, characterized in that: When the pipelined reconfigurable Softmax module works in the fine-grained pipeline mode: The comparison unit is used to determine a global maximum value from the data to be processed, and send the global maximum value to the first register unit; The first data selection unit is used to send the activation value as a first temporary value to the first register group; The second data selection unit is used to obtain the first temporary value from the first register group and send it to the exponential-accumulation-logarithm operation unit; The third data selection unit is used to obtain the processing result of the comparison unit from the first register unit and send it to the exponential-accumulation-logarithm operation unit; The fourth data selection unit is used to obtain the processing result of the comparison unit from the first register unit and send it to the second temporary value calculation unit; The fifth data selection unit is used to obtain the processing result of the exponential-accumulation-logarithm operation unit from the second register unit and send it to the exponential value calculation unit.