FPGA circuit and implementation method for realizing large language model SiLU function calculation

Abstract

The application discloses an FPGA circuit for realizing SiLU function calculation of a large language model, and the FPGA circuit comprises input modules, secEXP modules, reciprocal modules, multiplication modules and output modules which are electrically connected in sequence; the input module is configured to receive a hidden layer data input value v of the large language model; the secEXP module is configured to perform secEXP exponential calculation on the hidden layer data input value v to obtain a corresponding exponential value ev; the reciprocal module is configured to calculate the reciprocal of the sum value 1+ev to obtain an inverse operation value inv; the multiplication module is configured to multiply the inverse operation value and the original hidden layer data input value v, and reassign the product result to v; and the output module is configured to receive the reassignment value v and output it. The application further discloses an FPGA implementation method for SiLU function calculation of a large language model, and has wide application scenarios.

Description

Technical Field

[0001] This invention belongs to the field of computer and artificial intelligence technology, and relates to an FPGA circuit and implementation method for calculating SiLU functions of a large language model. Background Technology

[0002] In large language models such as Llama, the main functions of SiLU include: enabling gradients to propagate more smoothly and improving training stability; preserving negative information and enhancing the model's expressive power; and providing better computational efficiency and convergence speed, making it more suitable for Transformer architectures compared to ReLU or GELU.

[0003] However, due to the massive number of parameters in large language models, current technologies typically deploy them in high-performance CPU and GPU hardware systems (such as x86 CPUs paired with NVIDIA A100 accelerator cards). Deploying them on FPGAs is limited by hardware resources such as lookup tables and memory; therefore, there are very few cases in the industry where SiLU function computations are directly deployed on FPGAs. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the present invention aims to provide an FPGA circuit and implementation method for calculating SiLU functions in large language models. This method avoids the time and bandwidth overhead of transferring data between the CPU and the CPU before and after calculating SiLU operators, thus improving performance. This method uses fewer lookup tables and BRAM resources, employing only simple arithmetic units such as shifting and multiplication. It consumes fewer FPGA resources, has higher performance, and is easier to deploy on FPGAs. Therefore, it can be used with fewer resources and is not dependent on specific FPGA IP libraries, thus adapting to various FPGA platforms and achieving lightweight SiLU function calculation.

[0005] This invention provides an FPGA circuit for implementing SiLU function calculation, comprising: an input module, a secEXP module, a reciprocal module, a multiplication module, and an output module; the input module, the secEXP module, the reciprocal module, the multiplication module, and the output module are electrically connected to each other;

[0006] In the FPGA circuit of this invention, the input module first obtains the input value, which generally comes from the buffer data of the hidden layer. Then, the secEXP method of this invention is used to calculate the value ev of secEXP(-v). Next, the reciprocal module calculates the reciprocal value inv of 1+ev. Then, the input value v and inv are multiplied to obtain the result v. Finally, v is output to complete the operation of the SiLU operator.

[0007] In one specific implementation, the input module, the secEXP module, the reciprocal module, the multiplication module, and the output module can be connected via registers and / or an on-chip bus;

[0008] The input module is configured to receive hidden layer data from a large language model, use it as input value, and assign it to v.

[0009] The secEXP module is configured to perform secEXP exponent calculation on the input value to obtain the exponent value, and then assign the value to ev;

[0010] The reciprocal module is configured to obtain the reciprocal of the sum 1+ev, obtain the inverse operation value, and assign it to inv;

[0011] The multiplication module is configured to multiply the reciprocal and the original hidden layer data input value v, and reassign the product result to v.

[0012] The output module is configured to receive the reassigned value of v and output it.

[0013] The secEXP module may further include: a CORDIC iteration unit, a lookup table unit, and a scaling control unit;

[0014] The CORDIC iterative unit is used to gradually approximate the value of the exponential function through shift and addition / subtraction operations under table lookup control;

[0015] The lookup table unit is used to store a preset hyperbolic function parameter tanhLUT lookup table;

[0016] The scaling control unit is used to scale out-of-range input values ​​to fit CORDIC calculations and to perform restoration processing before outputting the results.

[0017] The secEXP module selects different CORDIC paths or returns an approximate constant value based on different numerical ranges of the input value v through one or more logic judgment circuits.

[0018] This invention also provides an FPGA implementation method for a large language model SiLU operator, the implementation method utilizing the aforementioned FPGA circuit, comprising:

[0019] Step 1: Assign the hidden layer data from the large language model as input value to v;

[0020] Step 2: Calculate the value of secEXP(-v) using the secEXP method and assign it to ev;

[0021] Step 3: Calculate the reciprocal of (1+ev) in the reciprocal module and then assign it to inv;

[0022] Step 4: Calculate the product of v in Step 1 and inv in Step 3 in the multiplication module, and reassign the value to v.

[0023] Step 5: Return the reassigned output value v.

[0024] Step two, the secEXP method includes the following steps:

[0025] Step 2.1. Assign initial values ​​to the return value and the temporary variable; in one specific implementation, the initial value of both the return value and the temporary variable is 0;

[0026] Step 2.2. Determine the range based on the different input radian values ​​ang, and directly calculate or output a constant value using the cordic method, or scale the input radian value before calculating using the cordic method and then restoring the original scale;

[0027] Step 2.3. Return the output values ​​obtained by different calculation methods respectively.

[0028] In step 2.2, when judging the magnitude of the input radian value ang, one or more judgments are required before the final output.

[0029] In one specific implementation, the steps of the secEXP method include:

[0030] Step 2.2.1. Determine whether the input radian value falls within the range of (-1.05, -0.02]. If it falls within this range, calculate cordicExp(ang) using the cordic method; if it does not fall within the above range, proceed directly to the next determination.

[0031] Step 2.2.2. Determine whether the input radian value is within the range of (-0.02, 0.0]. If it is within this range, return a constant value of 1.0. If it is not within the range, divide the input radian value by 16 and then calculate cordicExp(ang). Assign the calculation result to the return value and a temporary variable. Then perform 16 multiplication operations on the value of the temporary variable and return the result after iteration.

[0032] Step 2.2.3. Output the return value ret obtained in step 2.2.1 or step 2.2.2.

[0033] The present invention also provides the above-described FPGA circuit, or the above-described implementation method, for the application of SiLU functions in a large language model on an FPGA.

[0034] The present invention also provides a hardware system comprising: a memory and a processor; the memory stores a computer program, which, when executed by the processor, implements the above-described method.

[0035] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0036] The beneficial effects of this invention include: overcoming the resource limitations of deploying large language models on FPGAs, implementing the SiLU operator's operational logic with fewer LUTs and BRAM resources. Through optimization and improvement of the traditional CORDIC exp algorithm, the input value range is expanded, and accuracy is improved, making it possible to implement large language models on an FPGA platform. In one specific embodiment, compared with the traditional lookup table method, the BRAM resource usage is reduced from 128KB to 128B, approximately a 1000-fold reduction; within certain input value ranges, such as 0 to 1, the average error is reduced by 1000 times; and when resource usage and error are significantly optimized compared to existing technologies, the LUT usage is comparable to the traditional method. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart of the method for calculating the SiLU function on an FPGA according to the present invention.

[0039] Figure 2 This is a flowchart of the secEXP calculation method of the present invention.

[0040] Figure 3 This is a schematic diagram of the process for calculating Exp(ang) using the cordic method in this invention.

[0041] Figure 4 This is a schematic diagram of the tanhLUT lookup table value used in this invention.

[0042] Figure 5 This is a schematic diagram showing the test cases of different input value ranges performed on c-model using the method of the present invention in an embodiment of the invention, and the comparison with the running results of the c standard library (math.h). Detailed Implementation

[0043] The present invention will be further described in detail below with reference to the specific embodiments and accompanying drawings. Except for the contents specifically mentioned below, the processes, conditions, and experimental methods for implementing the present invention are all common knowledge and general knowledge in the art, and the present invention does not have any particular limitations.

[0044] This invention provides an FPGA circuit for calculating the SiLU function of a large language model. The flowchart of the method for calculating the SiLU operator using this FPGA circuit is shown below. Figure 1 As shown. First, the hidden layer data from the large language model is assigned as input to `v`. Second, the value of `secEXP(-v)` is calculated using the `secEXP` method in this invention and assigned to `ev`, representing the exponent value of the calculation result. Third, the reciprocal of `(1+ev)` is calculated in the reciprocal module and assigned to `inv`, representing the result of the inverse operation. Fourth, the product of `v` and `inv` is calculated in the multiplication module and reassigned to `v`. Fifth, the reassigned output value `v` is returned.

[0045] In large language models (such as Llama), the SiLU operator is formulated as silu(x) = x * σ(x), where σ(x) is the sigmoid function. The formula for the sigmoid function is... This formula uses the natural exponential function exp, which is an exponential function with the real number e (e≈2.71828) as its base. The traditional CORDIC method for calculating the exp function only allows input values ​​within the range of approximately [-1.11, 1.11]. In contrast, the input range of the exp function in SiLU is generally [-10, 10], and in extreme cases, it can reach [-20, 20]. This invention extends the input range of the traditional CORDIC method to meet the computational requirements of the SiLU operator in large language models.

[0046] The secEXP method in this invention also optimizes the accuracy of the CORDIC algorithm. The traditional CORDIC algorithm calculates values ​​in the [-1.11, 0] region, but its accuracy is low at both ends (i.e., the end close to -1.11 and the end close to 0), which cannot meet the accuracy requirements of the SiLU operator in large language models. This invention modifies these two ends to meet the accuracy requirements of the SiLU operator for the exp function.

[0047] In one specific implementation, for the correction at the -1.11 end, the input value is processed in multiple blocks, and then the CORDIC method is applied to calculate exp(x) step by step. Finally, the result is exponentially operated on (e.g., squared) to approximate a more accurate value. For the correction at the 0 end, based on the characteristics of the exponential function, when the input value is very close to 0, the output of exp(x) changes very little and is stable, approximately 1. Therefore, when the input value is in a very small range (e.g., (-0.02, 0.0)), the output is directly set to 1.0.

[0048] In one specific implementation, the judgment and corresponding processing for different input value ranges are shown in Table 1 below:

[0049] Table 1

[0050] scope deal with (-∞，-1.05] cordicExp(ang / 16) (-1.05，-0.02] cordicExp(ang) (-0.02，0.0] 1.0 (0，∞) cordicExp(ang / 16)

[0051] The secEXP method used in the second step above is a simplified extension of the traditional CORDIC method. Figure 2 This is the algorithm flowchart for this method. First, initial values ​​are assigned to the return value ret (urn) and the temporary variable temp (orary). Then, it checks whether the input radian value ang is within the range (-1.05, -0.02]. If it is, the traditional cordic method is used to calculate and the ret value is returned. If not, the next check is performed to determine whether the value of ang is within the range (-0.02, 0.0). If it is, the value 1.0 is returned directly. If not, ang is divided by 16 and the EXP value is calculated using the traditional cordic method. The result is assigned to ret and temp. ret performs 16 multiplications on temp, which is equivalent to obtaining the 16th power of temp. Finally, the ret value obtained from the above calculations is returned.

[0052] The value of 16 here is the optimal value after experimental comparison, but other values ​​can be selected according to the actual situation.

[0053] The cordic method used in this invention is as follows: Figure 3 As shown, the cordic method includes the following steps: initializing the shift value to prepare for iterative calculation; determining the size of the input value and selecting different calculation paths; increasing the shift value until the set number of iterations is reached; and returning the calculation result x+y as the final output value.

[0054] In one specific implementation, the initial value of the shift value sh (shift) is first assigned to 1. Then, a loop body that needs to iterate n times begins. This loop body first checks whether the current ang value is greater than 0. If it is, it proceeds according to... Figure 3The calculation is performed on the branch containing the box on the right; otherwise, the calculation is performed directly on the branch in the box below. During the calculation, the lookup table tanhLUT will be accessed; the values ​​of this table are shown below. Figure 4 The table in the table. After completing the above calculation, the value of sh becomes sh+1. If the loop has not reached n times, it returns to start the next loop. If the loop has reached n times, it returns the value of (x+y).

[0055] Figure 4 The table in the table is a lookup table for the values ​​of tanhLUT[i]. Each index corresponds to the value of the inverse hyperbolic tangent function for a certain angle. For example, the angle corresponds to 1 / 2, 1 / 4, 1 / 8, etc., arctanh(1 / 2) = 0.5493061443, arctanh(1 / 4) = 0.2554128119, etc.

[0056] More specifically, in the cordic method, the calculation of the loop body includes:

[0057] 1) If ang > 0, perform the following calculation steps:

[0058] Subtract the value of the lookup table tanhLUT[i] from ang: ang -= tanhLUT[i];

[0059] The temporary variable tmpx is assigned the current value x.

[0060] Update the value of x, calculate x += y >> sh, which means shifting y left by sh positions and then subtracting the value;

[0061] Update the y value, calculate y += tmpx >> sh, which means shifting tmpx left by sh bits and then subtracting this value;

[0062] or,

[0063] If ang <= 0, perform the following calculation steps:

[0064] Add the value of tanhLUT[i] to ang: ang += tanhLUT[i];

[0065] The temporary variable tmpx is assigned the current value x.

[0066] Update the value of x by calculating x-=y>>sh, which means shifting y to the right by sh bits and then adding the value.

[0067] Update the y value by calculating y - = tmpx >> sh, which means shifting tmpx right by sh bits and adding the value; and / or,

[0068] 2) After completing 1), increment the shift value sh by 1, and repeat 1) until the preset number of loops is reached.

[0069] When implementing large-scale language models (such as the Llama2 model), it is necessary to perform efficient computation and storage of a large number of parameters. Especially during the model inference process, the computation of the SiLU activation function is a critical performance bottleneck. Traditional implementation methods usually rely on high-performance GPUs or TPUs, while in certain specific scenarios, such as edge computing, real-time inference, or power-constrained devices, using FPGAs to implement large model inference has greater advantages.

[0070] As shown Figure 5 in a specific embodiment of the present invention, the comparison test results between the SiLU computation implemented by the method of the present invention and the standard SiLU theoretical value show that the accuracy error meets the requirements; Figure 5 , csilu is the approximate SiLU value computed by the method of the present invention; silu is the theoretical standard SiLU function value; delta is the absolute error between the two; rel_delta is the relative error.

[0071] In the FPGA implementation scheme of the SiLU activation function proposed by the present invention, the adopted secEXP method can effectively solve the resource bottleneck and efficiency bottleneck problems of current large language models during inference deployment.

[0072] Taking large language models such as Llama2 as an example, the output of their hidden layers requires frequent calls to the SiLU activation function. This function internally includes exponential function computation exp(–x) and division computation (1+exp(–x))⁻¹. If a traditional lookup table method or floating-point function computation is adopted, it will seriously occupy LUT and BRAM resources on the FPGA.

[0073] The secEXP method proposed by the present invention combines segmentation processing with simplified CORDIC iteration, and can complete exponential approximation using only basic shift, addition / subtraction and multiplication operations. In a specific interval (such as –1.05<x≤–0.02), the original CORDIC value is directly used for computation; while in the small value interval (such as x≤–1.05), the input is scaled and restored through 16 multiplications to avoid the overhead of large-value lookup tables. Compared with lookup table implementations, this method significantly saves BRAM resources and facilitates deployment on low-resource FPGAs.

[0074] The Llama2 model needs to maintain high throughput and low-latency response during the inference process. Although the traditional implementation on GPU has powerful performance, it has adaptation problems in low-power scenarios (such as edge AI or embedded deployment).

[0075] Through the hardware-implemented secEXP path, the present invention dynamically selects the optimal computation strategy (such as returning 1.0, direct CORDIC, scaled CORDIC) under different input ranges, which can avoid redundant latency caused by a unified path.

[0076] The implementation scheme of this invention does not depend on any specific vendor's IP core, and only uses basic shift operations, lookup table control and a finite number of multiplications, thus having a high degree of structural versatility.

[0077] This simple structure and clear computation path facilitate the transfer of models from data centers to terminal devices, while also ensuring compatibility with various low- to mid-range FPGAs and possessing strong general adaptability.

[0078] In edge computing, embedded intelligence, and other scenarios, nonlinear activation functions such as SiLU, if not handled properly, can become computational bottlenecks and sources of power consumption. The secEXP method proposed in this invention significantly reduces the power required for a single computation by avoiding floating-point calculations and complex function modules.

[0079] Meanwhile, due to reduced BRAM usage and controllable DSP consumption, the overall hardware resource utilization efficiency is higher, and lower-cost mid-range FPGAs can be selected to complete the core calculations of large models, reducing the deployment hardware cost.

[0080] Therefore, the FPGA implementation scheme of SiLU function based on the secEXP method proposed in this invention takes into account the computational accuracy, computational efficiency, resource control, platform universality and deployment scalability during the model inference process. It provides a practical and engineering-valued solution for the FPGA deployment of large language models and has good prospects for promotion.

[0081] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0082] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0083] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0084] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0085] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0086] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

[0087] The scope of protection of this invention is not limited to the above embodiments. Any variations and advantages that can be conceived by those skilled in the art without departing from the spirit and scope of this invention are included in this invention and are protected by the appended claims.

Claims

1. An FPGA circuit for implementing SiLU function calculation of a large language model, characterized in that, The FPGA circuit includes an input module, a secEXP module, a reciprocal module, a multiplication module, and an output module that are connected in sequence. The input module is configured to receive input values ​​v from the hidden layer data of a large language model. The secEXP module is configured to perform secEXP exponent calculation on the hidden layer data input value v to obtain the corresponding exponent value ev. The reciprocal module is configured to calculate the reciprocal of the sum 1+ev, and obtain the inverse operation value inv; The multiplication module is configured to multiply the inverse operation value and the original hidden layer data input value v, and reassign the product result to v. The output module is configured to receive the reassigned value of v and output it.

2. The FPGA circuit as described in claim 1, characterized in that, The input module, the secEXP module, the reciprocal module, the multiplication module, and the output module are connected via registers and / or on-chip bus. And / or, The FPGA circuit is configured to support SiLU function computation tasks that process one or more input values ​​in parallel.

3. The FPGA circuit as described in claim 1, characterized in that, The secEXP module further includes: a CORDIC iteration unit, a lookup table unit, and a scaling control unit; The CORDIC iterative unit is used to gradually approximate the value of the exponential function through shift and addition / subtraction operations under table lookup control; The lookup table unit is used to store a preset hyperbolic function parameter tanhLUT lookup table; The scaling control unit is used to scale out-of-range input values ​​to fit CORDIC calculations and to perform restoration processing before outputting the results.

4. The FPGA circuit as described in claim 1, characterized in that, The secEXP module selects different CORDIC paths or returns an approximate constant value based on different numerical ranges of the input value v through one or more logic judgment circuits.

5. An FPGA implementation method for SiLU function computation of a large language model, characterized in that, The implementation method utilizes the FPGA circuit as described in any one of claims 1-4, and includes the following steps: Step 1: Assign the hidden layer data from the large language model as input value to v; Step 2: Calculate the value of secEXP(-v) using the secEXP method and assign it to ev; Step 3: Calculate the reciprocal of (1+ev) in the reciprocal module and then assign it to inv; Step 4: Calculate the product of v and inv in the multiplication module and reassign it to v; Step 5: Return the reassigned output value v.

6. The implementation method as described in claim 5, characterized in that, Step two further includes: Step 2.

1. Assign initial values ​​to the return value and temporary variables; Step 2.

2. Determine the range based on the different input radian values ​​ang, and directly calculate or output a constant value using the cordic method, or scale the input radian value before calculating using the cordic method and then restoring the original scale; Step 2.

3. Return the output values ​​obtained by different calculation methods respectively.

7. The implementation method as described in claim 6, characterized in that, In step 2.2, after performing one or more size judgments on the input radian value, the output obtained by the cordic calculation is output. And / or, Step 2.2 further includes: 2.2.

1. Determine whether the input radian value falls within the range of (-1.05, -0.02]. If it falls within this range, calculate cordicExp(ang) using the cordic method; if it does not fall within the above range, proceed directly to the next judgment. 2.2.

2. Determine if the input radian value is within the range of (-0.02, 0.0]. If it is within this range, return a constant value of 1.

0. If it is not within the range, divide the input radian value by 16 and then calculate cordicExp(ang). Assign the result to the return value and a temporary variable. Then perform 16 multiplication operations on the value of the temporary variable and return the result after iteration. 2.2.

3. Output the return value ret obtained in step 2.2.1 or step 2.2.

2.

8. The FPGA circuit as described in any one of claims 1-4, or the implementation method as described in any one of claims 5-7, implements the SiLU function in the large language model in an FPGA application.

9. A hardware system, characterized in that, The hardware system includes a memory and a processor; the memory stores a computer program that, when executed by the processor, implements the method as described in any one of claims 5-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 5-7.

Images