Implementation circuit, method and application of S-type activation function based on ASIC (Application Specific Integrated Circuit)

Abstract

The invention discloses an implementation circuit of an S-type activation function based on an ASIC (Application Specific Integrated Circuit). The implementation circuit comprises an input analysis module, an address generation module, a lookup table storage module and a multiplication and addition calculation module, the input analysis module is used for receiving input data in a floating-point number format and separating sign bits, index bits and mantissa bits; the address generation module is used for converting input data into a fixed point number and generating a lookup table address; the lookup table storage module is used for storing values after segmentation quantization of the S-type function and outputting corresponding function values; the multiply-add calculation module completes multiply-add operation to obtain an output value of the S-type activation function; and the multiply-add calculation module is used for outputting an activation function result to a subsequent calculation unit. The invention further provides an S-type activation function implementation method using the circuit, and the S-type activation function implementation method has a wide application prospect.

Description

Technical Field

[0001] This invention belongs to the field of AI algorithm and hardware implementation technology, and relates to an ASIC-based sigmoid activation function implementation circuit, method and application. Background Technology

[0002] Currently, AI algorithm models are in a period of rapid development. Traditional activation functions (such as ReLU) are widely used due to their hardware friendliness. However, emerging models, especially LLM and Deepseek represented by Transformer, extensively use sigmoid activation functions, including SiLU (Swish) (defined as SiLU(x) = x·σ(x), where σ(x) is the Sigmoid function), due to their significant advantages in model accuracy. However, the hardware implementation of sigmoid activation functions, including SiLU, faces bottlenecks.

[0003] Hardware mapping of nonlinear modules such as activation functions has always been a challenge. Implementing sigmoid, SiLU, and other sigmoid activation functions on ASICs mainly faces the following problems:

[0004] 1. Exponentiation / division operations have high computational complexity, requiring simultaneous calculation of the Sigmoid function (involving exponentiation, division, and multiplication), far exceeding the comparison and selection operations of ReLU. Implementing this through polynomial fitting or table lookup naturally leads to accuracy loss.

[0005] 2. Resource consumption: Generally, processing polynomial fitting requires multiple stages of multipliers and adders. For example, the implementation of Sigmoid requires two multipliers and one adder. Conventional ASIC processing of Sigmoid consumes a lot of multiply-adder resources, resulting in increased area and power consumption.

[0006] 3. Fixed-point number bit width limitation: In order to reduce resource consumption, 16-bit floating-point numbers (fp16) are often used, which will lead to the accumulation of activation value truncation error and affect the accuracy of the model.

[0007] 4. Pipeline delay: Piecewise fitting of Sigmoid requires at least 6 clock cycles, which may become a computational bottleneck. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention aims to provide an ASIC-based implementation circuit, method, and application of sigmoid activation functions. In practical applications, solving the hardware implementation challenge of nonlinear activation functions when deploying large AI models on ASICs can accelerate the computation process of large model algorithms. For example, sigmoid activation functions such as SiLU typically involve multiplication, division, and exponentiation operations. Exponentiation and division operations are complex and require more space, increasing chip area and power consumption. Furthermore, complex computation paths increase latency, thus reducing the operating frequency of processing units. If nonlinear activation functions can minimize cache resource usage while maintaining accuracy, it will help improve overall hardware performance. This invention employs a lookup table plus computation method to effectively balance cache and computation resources, solving the problems of high computational latency in non-lookup table methods and high cache consumption in lookup table methods.

[0009] The technical solutions of this invention include the following:

[0010] An ASIC-based implementation circuit for a sigmoid activation function, the circuit comprising: an input parsing module, an address generation module, a lookup table storage module, and a multiply-accumulate calculation module;

[0011] The input parsing module is connected to the external data bus of the ASIC chip and is used to receive floating-point format input data and separate the sign bit, exponent bit and mantissa bit.

[0012] The input terminal of the address generation module is connected to the output terminal of the input parsing module, and is used to convert the input data into a fixed-point number and generate a lookup table address;

[0013] The address input terminal of the lookup table storage module is connected to the output terminal of the address generation module, and is used to store the value of the sigmoid function after piecewise quantization and output the corresponding function value;

[0014] The multiply-accumulate calculation module includes two input terminals: a first input terminal connected to the output terminal of the input parsing module for receiving input data, and a second input terminal connected to the output terminal of the lookup table storage module for receiving function values; the multiply-accumulate calculation module performs multiply-accumulate operations to obtain the output value of the sigmoid activation function;

[0015] The output of the multiply-accumulate calculation module is connected to the data path of the ASIC chip and is used to output the activation function result to the subsequent calculation unit.

[0016] In one specific implementation, the lookup table storage module only stores the positive half-axis data of the input data, and is connected to the control signal of the address generation module through symmetry operation;

[0017] And / or,

[0018] The lookup table is stored in the bram and / or lram of the lookup table storage module.

[0019] The present invention also provides a method for implementing an S-type activation function based on the above circuit, the method comprising:

[0020] Step 1: The input parsing module receives the input floating-point data, parses it into sign bit, exponent bit and mantissa bit, and performs range judgment on the input;

[0021] Step 2: The address generation module converts the floating-point data into a fixed-point number and generates the lookup table address through truncation and rounding logic;

[0022] Step 3: The lookup table storage module calls the address of the lookup table, outputs the function value corresponding to the intermediate function obtained by the S-type activation function after translation and / or transformation in the coordinate system, and rewrites the S-type activation function;

[0023] Step 4: The multiply-accumulate module calls the input data and intermediate function values, performs the calculation according to the rewritten S-shaped activation function, and outputs the result to the subsequent data path.

[0024] The sigmoid activation functions include the SiLU function, the Sigmoid function, and the Tanh function.

[0025] In one specific implementation, when the input floating-point number is less than the lower limit of the lookup table's range, the address generation module will typically saturate to the minimum effective address; when the input floating-point number is greater than the upper limit of the lookup table's range, the address generation module will typically saturate to the maximum effective address.

[0026] In one specific implementation, the minimum effective address is 0, and the maximum effective address is 2^N - 1.

[0027] In step two, when converting floating-point data to fixed-point numbers, the non-linear representation of the floating-point number is mapped to the linear address space of the lookup table, and the integer part is used to generate the address.

[0028] Based on the range and number of entries in the lookup table, calculate the position of the floating-point number in the limited space represented by each address interval in the lookup table, and obtain the fixed-point number corresponding to the floating-point number;

[0029] Extract the high M bits of the fixed-point number as the intermediate process address value;

[0030] Round the least significant bit region of the fixed-point number to obtain the final N-bit lookup table address;

[0031] In one specific implementation, the least significant bit region discarded in the fixed-point number is checked, and the size of the region is determined with a threshold of 0.5. The intermediate process address value is incremented by 1 or truncated to obtain the final N-bit lookup table address.

[0032] The intermediate function is segmented and quantized on the positive half axis, into five segments X∈(0,1),(1,2),(2,4),(4,8),(8,16), with each segment having an exponent bit width and a fraction bit width.

[0033] And / or,

[0034] The exponent is represented using 4 bits, and the fraction is represented using 8 bits;

[0035] And / or,

[0036] The lookup table size is smaller than the capacity of a single BRAM block.

[0037] This invention also provides the application of the above-described circuit or method in accelerating the operation of sigmoid activation functions in large language models.

[0038] The present invention also provides a hardware system for implementing the above method, the hardware system comprising: a memory and a processor; the memory storing a computer program, which, when executed by the processor, implements the above method.

[0039] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0040] In one specific embodiment of the present invention, the SiLU function expression SiLU=x × Sigmoid(x) is rewritten as SiLU=x×(F(X)+0.5); the intermediate function F(X) is segmented and quantized on the positive half axis, divided into five segments X∈(0,1),(1,2),(2,4),(4,8),(8,16), and each segment is set with an exponent bit width and a fraction bit width respectively;

[0041] Where F(X) is the function obtained by shifting the Sigmoid(x) function downwards by 0.5 units in the coordinate system, F(X) = Sigmoid(x) - 0.5.

[0042] In one specific embodiment of the present invention, the calculation of the SiLU function supports a multiply-add structure of (A×B)+(C×D), and x and F(X), 0.5 and x are used as multiplication pairs in the calculation when SiLU is calculated.

[0043] In the specific implementation of this invention, the FMA (Fused Multiply-Add) single multiply-add unit used performs multiplication and addition operations simultaneously with a single instruction, and the mathematical expression is result = (a × b) + c; the DFMA (Dual FMA) double multiply-add unit refers to the design that integrates two independent FMA units at the same time.

[0044] The beneficial effects of this invention include: By implementing the sigmoid activation function in ASIC circuits using a combination of lookup tables and multiply-accumulate methods, this significantly reduces on-chip memory usage and computational latency compared to traditional polynomial fitting or full lookup table implementations. Specifically, after employing piecewise quantization and symmetry optimization, even in FP16 data format, the storage capacity required for the lookup table is reduced by more than 90% compared to traditional lookup table methods; simultaneously, the number of clock cycles for SiLU function computation is reduced from 7 to 3, a 57% reduction in time consumption. Therefore, this invention achieves dual optimization of storage resources and computational efficiency while ensuring the accuracy of activation function computation, effectively improving the overall performance of ASIC accelerators in large-scale AI model inference. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This represents the SiLU function curve.

[0047] Figure 2 This represents the curve of the Sigmoid function.

[0048] Figure 3 To represent the curve of the Sigmoid function after shifting it 0.5 units along the y-axis, the lookup table method only needs to represent the positive half-axis portion.

[0049] Figure 4 This indicates the position of the SiLU function computation module within the overall computational architecture.

[0050] Figure 5 This diagram illustrates the FMA module architecture that supports the calculation of (A×B)+C.

[0051] Figure 6 This diagram illustrates the DFMA module architecture when supporting (A×B)+(C×D) calculations.

[0052] Figure 7This represents the block diagram of the FFN components in the large LLM model.

[0053] Figure 8 This represents the hyperbolic tangent function tanh curve.

[0054] Figure 9 This represents the hyperbolic tangent function tanh curve, which retains only the positive half-axis due to symmetry. Detailed Implementation

[0055] The present invention will be further described in detail below with reference to the specific embodiments and accompanying drawings. Except for the contents specifically mentioned below, the processes, conditions, and experimental methods for implementing the present invention are all common knowledge and general knowledge in the art, and the present invention does not have any particular limitations.

[0056] The hardware implementation of nonlinear activation functions is one of the key bottlenecks in accelerating large-scale AI models using ASICs. This problem needs to be addressed when designing AI large-scale model accelerators based on ASIC chips. Considering computational speed and hardware resources, nonlinear activation functions are often implemented using lookup tables, which essentially discretize the input values. However, lookup tables inevitably introduce two major drawbacks: firstly, limited resolution leads to accuracy loss; secondly, compensating for accuracy loss requires more sampling points, resulting in higher RAM usage. A balance between these two factors is difficult to achieve. This invention aims to reduce RAM usage on ASIC chips while maintaining accuracy as much as possible.

[0057] This invention provides a method for implementing sigmoid activation functions, including SiLU, Sigmoid, and Tanh, using an ASIC-based circuit. The function values ​​of these sigmoid activation functions approach the saturation boundary as the input x→±∞, exhibiting monotonic increasing and continuous differentiability. Since hardware implementation of nonlinear functions generally requires table lookup, the number of points in different function segments can be appropriately increased or decreased to achieve more accurate representation. The Tanh function inherently possesses parity-even symmetry, and the SiLU / Sigmoid functions exhibit parity-even symmetry after image translation. Hardware implementation can reduce resource consumption by storing only the positive half-axis data. Furthermore, it facilitates the use of piecewise approximation to simplify calculations and reduce hardware implementation complexity.

[0058] The graph of the SiLU function is as follows: Figure 1 As shown.

[0059] The graph of the Sigmoid function is as follows: Figure 2 As shown, the graph of the function after shifting by 0.5 units is as follows: Figure 3 As shown.

[0060] like Figure 4As shown, the data flow and module connection structure of the SiLU activation function in this invention include the following:

[0061] The output of NAP0 first enters BUFFER0, and then is sent to MULT / ADD-ARRAY:MLP to perform matrix multiplication and addition operations.

[0062] The output of NAP1 is split into two paths: one path enters the SiLU activation function module, undergoes nonlinear activation processing, and is then written back to BUFFER1; the other path is directly written to BUFFER1 as parallel input for subsequent data.

[0063] The data from BUFFER1 is further input into MULT / ADD-ARRAY:MLP, where it is used together with the data from BUFFER0 for matrix calculations.

[0064] In a specific implementation, the subsequent operations of output NAP0 and output NAP1 can be performed in parallel.

[0065] MULT / ADD-ARRAY: The calculation results of MLP are written to BUFFER2, and then output from BUFFER2 to NAP2 to complete the subsequent data processing.

[0066] In one specific implementation, the AI ​​large model is accelerated by ASIC. The data format is FP16. The SiLU function is completely mapped to a lookup table, which requires 16 bits × 2^16 = 128 KByte of bram to fit. In one specific implementation, the size of a whole bram inside the ASIC is 8 KByte, and a total of 16 brams are required.

[0067] The process of mapping the value represented by the FP16 floating-point number to the limited address space of the lookup table is roughly as follows: input value range judgment, floating-point to fixed-point conversion, truncation / rounding logic calculation, and output address to lookup table.

[0068] Specifically,

[0069] Procedure 1. Parse the FP16 data bits: S (15): Sign bit (1 bit); E (14:10): Exponent bits (5 bits), biased at 15; M (9:0): Mantissa bits (10 bits). The sign bit S, exponent E, and mantissa M are separated. S is ignored in subsequent processing because the lookup table (LUT) address is essentially unsigned. The key is processing E and M.

[0070] Step 2. Input data range determination: Compare E / M to determine if fp16 falls within the valid input range [Min, Max] of the LUT design. Handle out-of-range cases (such as NaN, Infinity, underflow, overflow, etc.). If fp16 < Min, the multiplexer MUX will typically saturate to the minimum valid address (usually 0). If fp16 > Max, the multiplexer MUX will typically saturate to the maximum valid address (usually 2^N - 1).

[0071] Step 3. Floating-point to fixed-point conversion: This maps the non-linear representation of the floating-point number to the linear address space of the LUT (each address represents an equally spaced point in the input domain), and its integer part is used to generate the address. The floating-point number fp16 is converted to a higher-precision fixed-point number. The design of the LUT determines the resolution of the input domain. If the LUT has 2^N entries covering the range [A, B], then each address interval represents (B - A) / (2^N - 1). The position of fp16 in this linear space needs to be calculated.

[0072] Step 4. Process the higher-precision fixed-point number obtained in Step 3 and extract the high-order bits containing the target address information. Directly take the high M bits of the fixed-point number as the intermediate process address value, represented as Temp_Addr[M-1:0].

[0073] Step 5. Round the address value obtained in Step 4 to reduce quantization error and obtain the final N-bit address Addr[N-1:0]. Check the least significant bit (LSB) region of Temp_Addr that was discarded to determine whether to add 1 or truncate it. The rounded N-bit value is the final LUT address Addr[N-1:0]. Connect the output line (pin) of the circuit module that generates the address to the address port of the LUT (SRAM).

[0074] In one specific implementation, the SiLU function is calculated as SiLU = x × Sigmoid(x), where the Sigmoid(x) function has the following characteristics: 1. It exhibits parity-even symmetry after translation along the y-axis; 2. The slope of the Sigmoid(x) function curve varies depending on the value of x, taking a value close to -1 in the x < -5 interval and a value close to 1 in the x > 5 interval. Shifting the Sigmoid(x) function downwards by 0.5 in the coordinate system yields F(X) = Sigmoid(x) - 0.5; thus, SiLU = x × (F(X) + 0.5) = x × F(X) + 0.5 × x. The F(X) function curve is based on parity-even symmetry at (0,0). Due to this symmetry, only the data of the positive x-axis can be stored, reducing RAM resource usage by 50%.

[0075] Figure 5 This is the FMA module and flowchart in this invention, wherein,

[0076] 1) Operand preprocessing: Perform sign bit processing, exponent bit processing, and mantissa bit extension on inputs a, b, and c.

[0077] 2) Generation of partial products of multiplication: The complex multiplication is decomposed into a set of simpler partial products. For the multiplication of two n-digit numbers, n partial products need to be generated. Each partial product is generated by logical operations on the corresponding bits of the multiplicand and multiplier.

[0078] 3) Wallace / Dadda Compressed Tree - 3:2 (3 inputs, 2 outputs) / 4:2 (4 inputs, 2 outputs) compressor array, 2 outputs for sum / carry: Receives partial products; multiplies n numbers to generate n partial products. The number of partial products is gradually reduced through 3:2 or 4:2 compressors. The two compressed numbers (sum and carry) are then summed using an adder to obtain the product.

[0079] 4) Calculate the exponent difference ΔE: Compare the exponent difference ΔE between input A×B and operand C. AB -E C .

[0080] 5) Mantissa C Alignment Shifter: Shifts the mantissa of C to the right by ΔE to align it (aligns the smaller exponent to the larger exponent).

[0081] 6) The three-operand CSA adder takes Sum_AB (sum), Carry_AB (carry), and the shifted mantissa C (C_aligned) as inputs. It adds these three inputs to produce two numbers: a sum vector and a carry vector. The sum of these two vectors equals the sum of the three inputs. Its outputs are Sum_CSA and Carry_CSA, satisfying: Sum_CSA + (Carry_CSA << 1) = Sum_AB + Carry_AB + C_aligned.

[0082] 7) Carry Propagation Adder (CPA) - Kogge-Stone Structure (This structure is faster, but has a larger area and higher power consumption): The Sum_CSA and Carry_CSA vectors output by CSA are added together to obtain the final result S (without normalization).

[0083] 8) Leading Zero Detection (LZA): Predicts the number of leading zeros in the result by analyzing the bits of the operand.

[0084] 9) Mantissa Normalization Shifter: Based on the shift amount output by LZA, a left shifter is used to shift the addition result to the left. Simultaneously, the exponent value is subtracted by the shift amount to ensure the final result's mantissa is in standard format.

[0085] 10) Rounding logic: According to the IEEE 754 standard, whether to add 1 to the mantissa is determined by the guard bit (the least significant bit of the result G), the rounding bit (the most significant bit to be shifted out R), the sticky bit S (all shifted-out bits are logically ORed), and the least significant bit (LSB) of the result.

[0086] 11) Output: Generates the final floating-point operation result, including the sign, exponent, and mantissa.

[0087] like Figure 6 The diagram shows the DFMA module and flowchart in this invention.

[0088] a. Input data preprocessing: The input terminal contains four floating-point inputs: input a, input b, input c, and input d; the sign bit, exponent bit, and mantissa bit are separated and preprocessed.

[0089] b. Floating-point multiplication operation tree:

[0090] Inputs a and b enter floating-point multiplier 1, and inputs c and d enter floating-point multiplier 2. Partial products are generated using Booth-Wallace multiplication trees.

[0091] c. Exponent Comparison and Shift Control: When calculating a×b + c×d, the two products a×b and c×d may differ in magnitude. To add them correctly, the mantissa of the smaller product must be shifted to the right so that its exponent is aligned with the exponent of the larger product. Output ΔE / select signal (which is the smaller product).

[0092] d. Partial product compression:

[0093] Multiplication is decomposed into multiple computational steps. Partial product compression quickly reduces the large number of partial products generated after multiplication into a small number of values ​​(usually two values: a sum vector and a carry vector), preparing for the final summation. If no compression is performed and all partial products are added directly using a single adder, the carry needs to be passed from the least significant bit to the most significant bit, increasing path delay. DFMA has two independent multiplication operations, a×b and c×d, so two separate partial product compression modules are required.

[0094] e. Multiplier 1 outputs Sum1 and Carry1; multiplier 2 outputs Sum2 and Carry2.

[0095] f. The mantissa alignment shifter shifts the mantissa bits according to the shift signal provided by the exponent comparison and shift control module. It shifts the mantissa of the smaller product to the right by ΔE bits, aligning its decimal point with that of the larger product. The shifter simultaneously generates a sticky bit S, which, along with the shifted mantissa, is sent to the subsequent adder.

[0096] g. Four-operand adder (CSA structure): The four input operands are: the high-order mantissa of the larger product (e.g., Sum_ab), the carry-in of the larger product (e.g., Carry_ab), the high-order mantissa of the smaller product after shifting (e.g., the shifted Sum_cd), and the carry-in of the smaller product after shifting (e.g., the shifted Carry_cd). Two outputs are generated (Sum_final, Carry_final). The carry chain is merged after addition compression (3 inputs → 2 outputs).

[0097] h. Normalization / Rounding: The internal data bit width supported by multipliers and adders is higher than the data bit width of the computation. For example, when calculating input data FP16, the multiplier supports 16-bit × 16-bit, and the adder supports 16-bit × 16-bit, plus carry. The computation result output by the adder may have issues with unnormalization / excessive precision. Unnormalization means that the output computation result has a large bit width, and its most significant bit 1 may not be in its proper position. Therefore, the mantissa and exponent need to be adjusted to meet the mantissa representation format of the Normalized Number in the IEEE 754 standard, truncating the high-precision mantissa to the target precision, and adjusting the value according to the rounding rules.

[0098] i. Output: Output the final floating-point operation result.

[0099] Example 1: Five-segment table lookup quantification

[0100] The quantization interval is adjusted based on the curvature of the Sigmoid function curve. FP16 data format: (-1) sign × 2 exponent -15 ×(1+fraction / 1024); Within the (x,y) coordinate system, segmented quantization is performed, resulting in the following 5 segments. Each segment with a fraction value of (0,1024) requires 10 bits ([9:0]) to represent.

[0101] 1. When x takes the value (0,1), the exponent value range is (0,14), which requires 4 bits to represent ([3:0]);

[0102] 2. When x takes the value (1,2), exponent takes the value 15;

[0103] 3. When x takes values in the range (2, 4), the exponent takes the value 16;

[0104] 4. When x takes values in the range (4, 8), the exponent takes the value 17;

[0105] 5. When x takes values in the range (8, 16), the exponent takes the value 18;

[0106] 6. When x >= 16, the F(X) curve approaches 0.5, and this value can be directly taken.

[0107] Therefore, the exponent is represented using 4 bits [3:0]. The bit width of the fraction value can be selected according to the quantization scheme. In one embodiment, [9:2] (8 bits) is selected, and the lower 2 bits are discarded. The sigmoid function is quantized in 5 segments and requires a total of 2^4 × 2^8 + 4 × 2^8 = 5120 bits. Since the curvature of the function curve changes the most when x takes values in the range (0, 5), the quantization accuracy can be appropriately increased, or the user can determine the fraction bit width according to the actual usage situation. Even if all 5 segments are represented with full precision, a total of 16K + 4K = 20K bits are required. In one embodiment, the capacity of a single ASIC block bram is 8K bytes (64K bits). If it is less than the capacity of a single ASIC block bram of 8K bytes (64K bits), the storage of the lookup table can be solved within one bram. Even when the number of sampling points of the lookup table is small (less than 32K bits, which is half of the capacity of a single bram), it can be stored in Lram (lut ram), releasing the bram resources.

[0108] From the above, it can be obtained that SiLU = x × (F(X) + 0.5) = x × F(X) + 0.5 × x. The value of F(X) is obtained through the lookup table method, and calculating x × F(X) + 0.5 × x (0.5 needs to be represented as a floating point number) can obtain the value of the SiLU activation function. The last step of calculating x × F(X) + 0.5 × x is a multiply-add operation, and the function value can be obtained by using the dedicated floating point multiply-add resources on the ASIC chip, completing the conversion of the exponentiation and division operations of the SiLU function to the multiply-add operation.

[0109] Embodiment 2

[0110] Calculate and add two segments of lookup table quantization; on the basis of segmented quantization, the RAM resource consumption can be further reduced by making appropriate adjustments to the above segmented quantization.

[0111] 1. When x takes values in the range (0, 3), that is, when 0 < x < 3, use the 3rd-order Taylor expansion ([ Call the multiply-add器 to calculate. The Taylor expansion of the sigmoid function:

[0112] At x = 0 ,

[0113] ,

[0114] 1 ,

[0115] ,

[0116] Substituting into the Taylor formula, we get

[0117] ;

[0118] Expansion of the SiLU function

[0119] ;

[0120] The coefficient of the x^2 term in this formula is too large when -3 < x < 3: resulting in overfitting when x > 0 and underfitting when x < 0; the coefficient of the x^4 term is too small: resulting in a rapid decay of the output when x > 2 and x < -2. Optimize The mean square error MSE in [-3, 3].

[0121] Let the polynomial be ,

[0122]

[0123] [[ID=)]] By numerical integration and matrix solution, we get ;

[0124]

[0125] The function value calculation can directly call FMA, but the calculation requires 3 - 4 clock cycles.

[0126] 2. When x takes values in (3, 4), the exponent takes the value 16;

[0127] 3. When x takes values in (4, 8), the exponent takes the value 17;

[0128] 4. When x takes values in (8, +∞), and when x > 8, the F(X) curve approaches 0.5, and this value can be directly taken.

[0129] The exponent is represented by 4 bits [3:0]. The bit width of the fraction value is selected according to the quantization scheme. In one embodiment, [9:2] (8 bits) is selected, and the lower 2 bits are discarded. The sigmoid function requires a total of 2^2 × 2^8 = 1024 bits for 2-segment quantization. Using full-precision representation requires 8 Kbit in total. In one embodiment, the capacity of a single ASIC bram is 8 Kbyte (64 Kbit). Even without using Bram, only registers can meet the storage requirements. Compared with the 5-segment look-up table quantization, additional computing resources (FMA or multipliers and adders) are required for the value range of -3 < x < 3.

[0130] Compared with the look-up table method, the present invention effectively reduces the usage of on-chip cache ram in ASIC. SiLU = x × Sigmoid(x), with the floating-point data format FP16. Representing the SiLU function by the pure look-up table method requires 16 bit × 2^16 = 128 KByte, while after optimizing the look-up table in the present invention, even in full-precision representation, only (2 K + 512) Byte of ram is now required, and the storage resources have decreased by 98% compared with the traditional look-up table method. Compared with the fitting method, calculating Sigmoid(x) requires 6 clock cycles, and it also requires 1 to 2 clock cycles to calculate x × Sigmoid(x), for a total of 7 to 8 clock cycles. For the present invention to calculate SiLU, the look-up table method for Sigmoid(x) requires 1 clock cycle, and SiLU = x × (F(X) + 0.5) = x × F(X) + 0.5 × x. Calculating the multiply-add using FMA (calculating a × b + c) only requires 2 clock cycles, with a total of 3 clock cycles spent. Without considering the actual implemented clock frequency differences, the calculation time is reduced from 7 clock cycles to 3 clock cycles, a 57% reduction in time consumption.

[0131] If DFMA (dual multiply-add unit, calculating a × b + c × d) is used, the value of x × F(X) + 0.5 × x can be calculated in 1 clock cycle, and a total of 2 clock cycles are consumed to complete the SiLU function calculation. With the same clock frequency, the time consumption is reduced by 71% (from 7 cycles to 2 cycles). However, the area of DFMA is 60% larger than that of FMA. Which computing resources to use and the distribution of computing resources need to be specifically analyzed in combination with the ASIC design.

[0132] If a further optimized implementation of Bram resources is adopted, with segmented calculation of SiLU(x) and floating-point data format FP16, the storage resource reduction is more significant compared to the traditional lookup table method. Storage resources are reduced by 80% compared to Scheme 1. The segmented Sigmoid(x) lookup table method requires one clock cycle, while SiLU = x × (F(X) + 0.5) = x × F(X) + 0.5 × x, calculating multiplication and addition using FMA (calculating a × b + c), only requires two clock cycles. Considering...

[0133] FMA computation requires a maximum of 4 clock cycles. Without considering the actual clock frequency differences in implementation, the computation time is reduced from 7 clock cycles to 4 clock cycles, a reduction of 42.8% in time consumption.

[0134] In one specific implementation, SiLU calculation and matrix operations can be performed simultaneously, which requires additional multiply-accumulator resources. If a conflict occurs, from a software perspective, instruction scheduling and time-sharing can be used. From a hardware perspective, a selector can be used to choose whether to use the multiply-accumulator reserved for matrix operations to calculate SiLU or to use an additional or dedicated SiLU multiply-accumulator.

[0135] The LLM large model uses ASIC acceleration. The on-chip storage / computing resources of the ASIC can fully leverage the advantages of hardware parallelism to perform matrix operations, effectively shortening the token generation time of the LLM large model. Taking the FFN functional module in the LLM large model deployed on the ASIC as an example, the FFN composition structure is as follows: Figure 7 As shown.

[0136] As shown in the FFN structure, the result of the Gate matrix multiplication calculation is output and then passed through the SiLU function module. SiLU can be calculated along the path, specifically by using a lookup table. The calculation formula is SiLU = x × Sigmoid(x), where Sigmoid(x) is shifted along the y-axis to obtain F(X) = Sigmoid(x) - 0.5; SiLU = x × (F(X) + 0.5) = x × F(X) + 0.5 × x. The DFMA module on the ASIC chip supports calculations of A × B, A × B + C, or A × B + C × D (A / B / C / D are input values). The newly constructed function x × F(X) + 0.5 × x by SiLU is precisely supported by DFMA, where x is the input value and the value of F(X) is obtained through a lookup table. SiLU is executed after the matrix calculation is completed. The SiLU calculation does not occur simultaneously with the matrix calculation, so it can share DFMA calculation resources. This invention achieves SiLU function implementation without increasing ASIC computing resources, fully releasing ASIC computing power and effectively improving ASIC design performance. Taking the calculation of A × B + C × D as an example, other ASICs need to call FMA or build a multiply-accumulator using underlying computing resources, while DFMA can directly calculate the result. From a performance perspective, under ideal conditions, using FMA requires 2 clock cycles to obtain the result. In one embodiment, the actual frequency of the DFMA multiply-accumulator is 800MHz, and the actual frequency of FMA is 1.2GHz. DFMA, however, only consumes 1 clock cycle to obtain the result of (A × B + C × D), resulting in a performance improvement of 33%.

[0137] Furthermore, this invention significantly optimizes on-chip ASIC cache RAM resources. The full-precision lookup table required for the SiLU function (for calculating the Sigmoid function) requires 128 KBytes (16 bits × 2^16 / 8), and the conversion between fp16 floating-point numbers and decimal numbers is (-1). sign × 2 exponent-15 × (1+fraction / 1024), sign bit 1 bit, exponent bit 5 bits, fraction 10 bits. After quantization, the precision loss fraction is 8 bits, requiring 32KByte. This invention uses segmented quantization, employing a five-segment lookup table quantization method. Full-precision representation requires a total RAM resource of 16Kbit + 4Kbit = 2.5KByte. After fraction quantization with 8 bits for precision loss, a total RAM resource of 5Kbit (0.625KByte) is required, reducing storage resources by more than 90%. If two-segment lookup table quantization is selected, full-precision representation requires a total RAM resource of 1KByte, reducing RAM storage resources by 99%.

[0138] A comparison test was conducted between the SiLU activation function implemented using Python's PyTorch library and the hardware-based SiLU function of this invention to measure the relative error. 32768 random numbers in fp16 format were generated and used as input values ​​for both the software and hardware SiLU functions. The software SiLU calculation result was used as the true value, and the hardware SiLU calculation result as the test value. The output values ​​were compared one by one to calculate the relative error. The average relative error is 0.05%.

[0139] Example 3

[0140] The high-precision model RegNetY uses the SiLU function (y=x+SiLU(Conv(x))) at the residual connection merging point; compared to the traditional ResNet y = x + ReLU(Conv(x)), SiLU replaces the traditional ReLU.

[0141] In the original ResNet, the residual branch is usually Conv->BN->ReLU, and then added back to the main path at the end. However, RegNetY places SiLU after convolution and before addition, so the activation function can be applied to the result of the entire residual branch Conv(x), which is equivalent to making the nonlinear transformation occur before feature fusion.

[0142] Conv(x), the value calculated by convolution, serves as the input variable for the SiLU function. The convolution calculation is relatively time-consuming and is not included in the statistics. y = x + SiLU(Conv(x)) adds one addition operation compared to SiLU(x) in FFN, increasing the computation time by one clock cycle. Compared to traditional lookup tables, this invention reduces the computation time from 8 clock cycles to 5 clock cycles, a reduction of 37.5%. Storage resource optimization is referenced in FFN.

[0143] This project uses PyTorch 2.1 + TensorRT 8.6, FP16 data format, to implement ImageNet validation set image classification on a GPU. The inference latency per image is 8.2ms, and the TOP-1 accuracy (the proportion of the class with the highest predicted probability matches the actual label, calculated using the formula...) is... The accuracy rate was 80.2%, and the testing process strictly followed the ImageNet standard evaluation protocol. The same model was deployed on ASIC hardware to implement image classification on the ImageNet validation set, with a clock frequency of 1GHz, bath_size=64, inference latency of 3.96ms per image, and a TOP-1 accuracy of 79.7%, representing a 0.5% decrease in accuracy, which is within an acceptable range.

[0144] Example 4

[0145] The YOLO models are mainly YOLOv5, v7, v8, v9, and v10. The SiLU activation function has completely replaced LeakyReLU and has become the default activation function in its backbone and neck networks.

[0146] The SiLU function is used in the model in the following locations.

[0147] Backbone: In the convolutional module, each convolutional layer is followed by a SiLU activation function to extract primary feature values ​​(such as edges and textures).

[0148] Feature Pyramid (Neck): In feature fusion modules such as SPP / SPPF and PANE, SiLU is used to enhance the semantic expressive power of multi-scale features.

[0149] Detection Head: Following the convolutional layers of the classification and regression branches, SiLU improves the accuracy of bounding box prediction and its class probability.

[0150] Taking the YOLOv8 model as an example, this paper analyzes the advantages of hardware implementation of the SiLU function based on typical scenarios such as urban autonomous driving (BDD100K), industrial defect detection (DAGM), and security monitoring (COCOval2017 dataset).

[0151] COCO (Common Objects in Context) is the most widely used benchmark dataset in the field of object detection. The dataset contains 5000 images covering 80 categories and offering rich scene representation. The YOLOv8 model was used to process the same data using both pure CPU and CPU+ASIC (the same server with an ASIC accelerator card). In real-time streaming processing (batch size=1), the pure CPU server achieved 8.2 FPS and 118.5 ms latency, while adding the ASIC accelerator card resulted in 94 FPS and 11.2 ms latency, representing a more than 10-fold improvement. The accuracy mAP@0.5 was 0.95. The average AP value calculated across 10 IoU thresholds was 44.9% for the pure CPU solution and 44.7% for the CPU+ASIC solution, which are quite close and can be considered as having no accuracy loss. In batch processing (batch size=64), the acceleration effect was even more significant. The pure CPU throughput was only 22 fps, while the CPU+ASIC acceleration reached 321.1 fps, a 14.6-fold improvement.

[0152] For the industrial defect detection scenario (based on the DAGM2007 dataset), due to the limited resources of ARM processors commonly used in industrial equipment in practical applications, the lightweight version YOLOv8n was chosen as the model. DAGM2007 contains 10 different categories of textured background defects, with images being 512x512 grayscale images. The training set consists of composite images of defect-free and defective images, while the test set consists of complex composite images that more closely resemble real-world scenes. The CPU solution uses an ARM Cortex-A72 processor, and the CPU+ASIC solution adds an ASIC acceleration board. The ASIC hardware acceleration maintains the same accuracy as the software (mAP@0.5=89.1%), with no accuracy loss. GPU throughput is 4.2fps; ASIC throughput is 52fps; the performance improvement is over 12 times.

[0153] The urban autonomous driving scenario (based on the BDD100K dataset) requires the simultaneous detection of multiple categories, scales, and forms of targets (e.g., vehicles, pedestrians, traffic lights). Processing high-resolution images places high demands on the system's throughput, latency, and energy consumption. The BDD100K dataset contains 100,000 high-definition (1280x720) video clips, labeled with 10 target classes, covering diverse driving scenarios such as daytime, nighttime, rain / snow, urban areas, and highways. To handle the extremely complex visual scene, the extra-large version of the YOLOv8x model was selected. This model calls the SiLU function frequently, allowing for thorough testing of ASIC acceleration performance. The CPU solution used was an RTX4090 consumer-grade GPU, and the CPU+ASIC solution used an ASIC acceleration board. The accuracy metric mAP@0.5:0.95 was essentially consistent with the GPU, at 47.8%, with no loss in accuracy. The GPU throughput was 87fps with a latency of 11.5ms; the ASIC throughput was 237fps with a latency of 4.5ms, demonstrating a significant performance improvement.

[0154] Example 5

[0155] The tanh function is implemented using a piecewise linear approximation with a lookup table. The lookup table does not directly store the tanh function value; instead, it stores the offset dx, y-value, and slope k corresponding to the starting point of each interval. The quantization interval is adjusted according to the curvature of the tanh function curve. FP16 data format: (-1) sign × 2 exponent-15 ×(1+fraction / 1024); When x ≥ 9.0, tanh(x)≈1.0, directly return 1.0; When x is very small (e.g., x<2^{-6}), tanh(x)≈x, directly return x. 2^(-6)≤x<9, in the (x,y) coordinate system, segmented quantization is performed, divided into the following 5 segments. The mantissa value is assigned according to the degree of nonlinearity of each segment function, which can further reduce storage resource usage.

[0156] The exponent exp ranges from 9 to 18 (10 values ​​in total), corresponding to a normal range of x-axis values ​​from approximately 0.015625 to 16. As can be seen from the tanh function graph, tanh(x) has its largest second derivative near x=1.0, which is also the most curved part of the curve and the most difficult to approximate with a linear function. Using more points in this segment can effectively improve accuracy with minimal increase in resources. The mantissa bit width (M) is selected based on the rate of change of the exp value, as follows:

[0157] 1. x takes values ​​(0, 0.5), exp values ​​9 to 13: the function curve changes rapidly, take M=6 bits, each exp value has 64 sub-intervals.

[0158] 2. x takes values ​​of (0.5, 2.0), exp values ​​of 14 to 15: the function curve changes the fastest, take M=7 bits, each exp value has 128 sub-intervals.

[0159] 3. x takes the value (2,4), exp value 16 (actual exponent exp-15=1): the function curve changes rapidly, take M=4 bits, there are 16 sub-intervals.

[0160] 4. x takes the value (4,8), exp value 17 (actual exponent exp-15=2): the function curve changes slowly, take M=2 positions, and there are 4 sub-intervals.

[0161] 5. x takes values ​​(8, 16), exp value 18 (actual exponent exp-15=3): the function curve changes very slowly, take M=2 positions, and there are 4 sub-intervals.

[0162] exp9-13: 5 values ​​× 64 subintervals = 320 subintervals

[0163] exp14-15: 2 values ​​× 128 subintervals = 256 subintervals

[0164] exp16: 1 value × 16 subintervals = 16 subintervals

[0165] exp17: 1 value × 4 subintervals = 4 subintervals

[0166] exp18: 1 value × 4 subintervals = 4 subintervals

[0167] Total number of subintervals: 320 + 256 + 16 + 4 + 4 = 600 subintervals

[0168] Each sub-interval stores three FP16 values ​​(base_x, base_y, base_k). FP16 data is 16 bits, so each item occupies 48 bits (6 bytes).

[0169] Total number of bits: 600 sub-intervals × 48 bits = 28800 bits; Total number of bytes: 28800 bits / 8 = 3600 bytes. In one embodiment, the minimum size of a single BRAM block in the ASIC is 1KB, requiring 4 BRAM blocks to store the tanh function lookup table.

[0170] The following is a description of (base_x, base_y, base_k):

[0171] 1. `base_x` is the reference point input value, representing the x-coordinate of the starting point of the current sub-interval, which is also the lower limit or minimum value of the sub-interval. Its core function is to calculate the offset. For any input `x` falling within this sub-interval, `dx = x - base_x` can determine how far the value of `x` is from the reference point of this sub-interval. The offset `dx` is used for subsequent linear correction. `dx = x - base_x` is calculated using an FP16 floating-point subtractor.

[0172] 2. base_y: Base point output value, representing the function value at the base point base_x, base_y = tanh(base_x). base_y is the baseline or zero point for linear approximation, representing the y-value at the point base_x. base_y is an FP16 floating-point constant and must be pre-calculated and stored in the ASIC.

[0173] 3. `base_k`: Baseline slope / derivative, representing the first derivative (i.e., slope) of the function at the baseline point `base_x`. For the tanh function, its derivative is d(tanh(u)) / du = 1 - tanh²(u). Therefore, `base_k = 1 - base_y²`. `base_k` provides the slope for a linear approximation. The function is approximated using the tangent line to the tanh function at `base_x`. According to the basic idea of ​​calculus, within a very small interval, the function value can be highly approximated by its tangent line: f(x) ≈ f(base_x) + f'(base_x) × (x - base_x). `base_k × dx` is a correction term based on the rate of change of the function itself, correcting `base_y` to the target value. `base_k` is an FP16 floating-point constant and must be pre-calculated and stored in the ASIC. Calculating `base_k × dx` requires an FP16 floating-point multiplier.

[0174] For any point x, first determine the interval in which the value of x lies, and then use the corresponding tangent equation y = base_y + base_k ×(x - base_x) to calculate the approximate value of the tanh function.

[0175] A comparative test was conducted using the tanh activation function implemented in Python's PyTorch library and the hardware implementation of the tanh function from this invention to measure the relative error. 32768 random numbers in fp16 format were generated and used as input values ​​for both the software and hardware implementations of the tanh function. The software calculation result was used as the Golden value, and the hardware calculation result as the test value. The output values ​​were compared one by one to calculate the relative error. The average relative error is 0.00697%.

[0176] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0177] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0178] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0179] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0180] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0181] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

[0182] The scope of protection of this invention is not limited to the above embodiments. Any variations and advantages that can be conceived by those skilled in the art without departing from the spirit and scope of this invention are included in this invention and are protected by the appended claims.

Claims

1. A circuit for implementing a sigmoid activation function based on ASIC, characterized in that, The circuit includes: an input parsing module, an address generation module, a lookup table storage module, and a multiplication-accumulation calculation module; The input parsing module is connected to the external data bus of the ASIC chip and is used to receive floating-point format input data and separate the sign bit, exponent bit and mantissa bit. The input terminal of the address generation module is connected to the output terminal of the input parsing module, and is used to convert the input data into a fixed-point number and generate a lookup table address; The address input terminal of the lookup table storage module is connected to the output terminal of the address generation module, and is used to store the value of the sigmoid function after piecewise quantization and output the corresponding function value; The multiply-accumulate calculation module includes two input terminals: a first input terminal connected to the output terminal of the input parsing module for receiving input data, and a second input terminal connected to the output terminal of the lookup table storage module for receiving function values; the multiply-accumulate calculation module performs multiply-accumulate operations to obtain the output value of the sigmoid activation function; The output of the multiply-accumulate calculation module is connected to the data path of the ASIC chip and is used to output the activation function result to the subsequent calculation unit.

2. The circuit as described in claim 1, characterized in that, The lookup table storage module stores only the positive half-axis data of the input data and is connected to the control signal of the address generation module through symmetry operation; And / or, The lookup table is stored in the bram and / or lram of the lookup table storage module.

3. A method for implementing an S-type activation function based on the circuit described in claim 1 or 2, characterized in that, The implementation method includes: Step 1: The input parsing module receives the input floating-point data, parses it into sign bit, exponent bit and mantissa bit, and performs range judgment on the input; Step 2: The address generation module converts the floating-point data into a fixed-point number and generates the lookup table address through truncation and rounding logic; Step 3: The lookup table storage module calls the address of the lookup table, outputs the function value corresponding to the intermediate function obtained by the S-type activation function after translation and / or transformation in the coordinate system, and rewrites the S-type activation function; Step 4: The multiply-accumulate module calls the input data and intermediate function values, performs the calculation according to the rewritten S-shaped activation function, and outputs the result to the subsequent data path. The sigmoid activation functions include the SiLU function, the Sigmoid function, and the Tanh function.

4. The method as described in claim 3, characterized in that, In step two, when converting floating-point data to fixed-point numbers, the non-linear representation of the floating-point number is mapped to the linear address space of the lookup table, and the integer part is used to generate the address.

5. The method as described in claim 4, characterized in that, Based on the range and number of entries in the lookup table, calculate the position of the floating-point number in the limited space represented by each address interval in the lookup table, and obtain the fixed-point number corresponding to the floating-point number; Extract the high M bits of the fixed-point number as the intermediate process address value; The least significant bit region of the fixed-point number is rounded to obtain the final N-bit lookup table address.

6. The method as described in claim 5, characterized in that, Check the least significant bit region that was discarded in the fixed-point number, determine the size of the region with a threshold of 0.5, add 1 to the intermediate process address value or truncate it to obtain the final N-bit lookup table address.

7. The method as described in claim 3, characterized in that, In step three, the intermediate function is segmented and quantized on the positive half axis, into five segments X∈(0,1),(1,2),(2,4),(4,8),(8,16), with each segment having an exponent bit width and a fraction bit width. And / or, The exponent is represented using 4 bits, and the fraction is represented using 8 bits; And / or, The lookup table size is smaller than the capacity of a single BRAM block.

8. The application of the ASIC circuit as described in claim 1 or 2, or the method as described in any one of claims 3-7, in accelerating the operation of the sigmoid activation function in a large language model.

9. A hardware system for implementing the method as described in any one of claims 3-7, characterized in that, The hardware system includes a memory and a processor; the memory stores a computer program that, when executed by the processor, implements the method as described in any one of claims 3-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 3-7.

Images