Non-linear function calculation method for large language model

Abstract

The invention provides a large language model-oriented nonlinear function calculation method. The method comprises the following steps of: obtaining initial approximate parameter configuration; inputting the initial approximate parameter configuration into a quantitative configuration model to obtain target optimization parameter configuration; the quantitative configuration model is used for performing function approximation calculation according to the initial approximation parameter configuration to obtain optimized parameter configuration; function approximation calculation is carried out according to all the basic nonlinear functions after unified configuration processing based on optimization parameter configuration, and activation function initial configuration is obtained; replacing the basic nonlinear function in the large language model with the initial configuration of the activation function and the basic nonlinear function of all approximation calculations to obtain target optimization parameter configuration; and selecting a function module according to the target optimization parameter configuration, performing module configuration on the function module by using the target optimization parameter configuration, and calculating a nonlinear function by using the function module. By means of the method, the problems that a nonlinear function is long in calculation period and high in hardware resource overhead are solved.

Description

Technical Field

[0001] The present application relates to the technical field of nonlinear function calculation, and in particular to a nonlinear function calculation method for a large language model. Background Art

[0002] Large language models are widely used in natural language processing, text generation, code writing, conversation and chat, etc. The currently commonly used large language models contain a large number of nonlinear operations. Nonlinear operations appear in the calculation of activation functions, Softmax functions and normalization layers, which greatly affect the performance of the model. The activation function enables the neural network to handle complex tasks by introducing nonlinear characteristics. At the same time, the complex calculation of nonlinear functions brings great difficulties to hardware implementation.

[0003] In order to meet the growing computational needs of nonlinear functions, more and more algorithms are being applied to the computation of nonlinear functions. The SRT (Sweeney-Robertson-Tocher) algorithm is widely used in the division operation in digital circuit design. It usually uses a lookup table of the dividend and divisor to determine each quotient. The piecewise polynomial approximation algorithm based on the lookup table can approximate the calculation of various nonlinear functions such as the reciprocal and the logarithm with base 2.

[0004] Because large language models use a lookup table approach that requires looking up the parameters of each power in the nonlinear function, it is impossible to optimize the power and other parameters of the nonlinear function, resulting in high computational complexity and huge consumption of hardware resources. In addition, the above methods often only focus on basic nonlinear functions, such as square roots and logarithms with a base of 2; for the calculation of complex activation functions in large language models, it is still necessary to use a combination of basic nonlinear functions to perform calculations through iterations, which requires multiple calculation cycles to complete the calculation, resulting in a long calculation cycle. Summary of the invention

[0005] The present application provides a nonlinear function calculation method for a large language model to solve the problem of long nonlinear function calculation cycle in the prior art.

[0006] The method comprises:

[0007] Obtaining an initial approximate parameter configuration; the initial approximate parameter configuration corresponds to a number of basic nonlinear functions in a large language model;

[0008] The initial approximate parameter configuration is input into a quantization configuration model to obtain a target optimization parameter configuration; the quantization configuration model is used to perform function approximation calculation according to the initial approximate parameter configuration to obtain an optimization parameter configuration, and the optimization parameter configuration conforms to the approximation of all the basic nonlinear functions; function approximation calculation is performed according to all the basic nonlinear functions after unified configuration processing based on the optimization parameter configuration to obtain an initial configuration of the activation function; the initial configuration of the activation function and all the approximation calculation basic nonlinear functions after the unified configuration processing are used to replace the nonlinear functions in the large language model to obtain the target optimization parameter configuration that meets the error loss threshold;

[0009] A corresponding function module is selected according to the target optimization parameter configuration, the function module is configured using the target optimization parameter configuration, and the nonlinear function of the large language model is calculated using the function module.

[0010] Preferably, the initial approximation parameter configuration includes several basic nonlinear functions to be approximated, an activation function to be approximated, a segmented maximum absolute error, a maximum precision loss of a large language model, a maximum number of segments of the basic nonlinear function, and a maximum quantization bit width of polynomial coefficients.

[0011] Preferably, the function approximation calculation includes:

[0012] Selecting the basic nonlinear function to be approximated;

[0013] Initializing the basic nonlinear function to be approximated to obtain an initialized nonlinear function;

[0014] Calculating a function segmentation error according to the initialized nonlinear function, and iteratively calculating the function segmentation error until the function segmentation error is less than the segmentation maximum absolute error, thereby obtaining a first optimization parameter configuration;

[0015] A parameter configuration that satisfies the approximate calculation of all the basic nonlinear functions is selected to obtain the optimized parameter configuration.

[0016] Preferably, the step of initializing the basic nonlinear function to be approximated comprises:

[0017] The highest power of the polynomial of the basic nonlinear function to be approximated, the number of basic nonlinear function segments, and the quantization bit width of the polynomial coefficients are set to 1;

[0018] The number of activation function segments of the activation function to be approximated is set to 1.

[0019] Preferably, the step of iteratively calculating the function segmentation error until the function segmentation error is less than the segmentation maximum absolute error comprises:

[0020] Determine whether the function segmentation error is less than the segmentation maximum absolute error;

[0021] If yes, storing the initial approximate parameter configuration as the first optimized parameter configuration;

[0022] If not, the basic nonlinear function to be approximated is configured and adjusted to obtain an iterative nonlinear function, and a function segmentation error is calculated according to the iterative nonlinear function until the function segmentation error is less than the segmentation maximum absolute error.

[0023] Preferably, the configuration adjustment includes:

[0024] Determine whether the quantization bit width of the polynomial coefficient is less than the maximum quantization bit width of the polynomial coefficient;

[0025] If yes, then the quantization bit width of the polynomial coefficients of the basic nonlinear function to be approximated is increased by 1, and the function segmentation error is recalculated;

[0026] If not, determining whether the number of segments of the basic nonlinear function is less than the maximum number of segments of the basic nonlinear function;

[0027] If yes, then the power exponent of the number of basic nonlinear function segments of the basic nonlinear function to be approximated is increased by 1, and the function segmentation error is recalculated;

[0028] If not, the highest power of the polynomial of the basic nonlinear function to be approximated is increased by 1, and the function segmentation error is recalculated.

[0029] Preferably, the step of selecting a parameter configuration that meets the approximate calculation of all the basic nonlinear functions includes:

[0030] Determining whether the first optimization parameter configuration conforms to the approximate calculation of all the basic nonlinear functions;

[0031] If yes, then the optimization parameter configuration is obtained;

[0032] If not, the basic nonlinear function to be approximated is reselected, and approximate iterative calculation is performed until the first optimization parameter configuration meets the approximate calculation of all the basic nonlinear functions.

[0033] Preferably, the unified configuration process includes:

[0034] Using the optimization parameter configuration, uniformly configure the polynomial coefficient quantization bit width and the highest power of the polynomial of all the basic nonlinear functions to obtain a plurality of uniformly configured functions;

[0035] Perform function calculation according to all the unified configuration functions to obtain the initial configuration of the activation function;

[0036] Using the initial configuration of the activation function and a number of basic nonlinear functions of approximation calculations that have completed unified configuration processing to replace all the basic nonlinear functions in the large language model, and calculating the loss error of the large language model after function replacement;

[0037] Determining whether the loss error is less than the maximum accuracy loss of the large language model;

[0038] If yes, then the target optimization parameter configuration is obtained;

[0039] If not, the number of activation function segments of the initial configuration of the activation function is increased by 1, and the loss iteration of the large language model is performed until the loss error is less than the maximum accuracy loss of the large language model.

[0040] Preferably, the function module includes a pre-processing module, a polynomial calculation module and a post-processing module;

[0041] The preprocessing module includes a plurality of preprocessing units, each of which is used to preprocess different basic nonlinear functions and activation functions;

[0042] The polynomial calculation module is used to perform polynomial calculation according to the processing result of the preprocessing module;

[0043] The post-processing module includes a plurality of post-processing units, each of which is used to post-process different basic nonlinear functions and activation functions to obtain function output results.

[0044] Preferably, the preprocessing includes decomposing the single-precision input data corresponding to the basic nonlinear function into an exponential part and a mantissa part according to the operation code, performing exponential part calculation, obtaining polynomial coefficients, and processing mantissa part extraction and parameter calculation;

[0045] The polynomial calculation includes performing polynomial approximate calculation and leading zero detection according to the polynomial result output by the preprocessing module;

[0046] The post-processing includes merging and normalizing the polynomial results outputted from the polynomial calculation.

[0047] As can be seen from the above content, the present application provides a nonlinear function calculation method for a large language model, the method comprising obtaining an initial approximate parameter configuration; the initial approximate parameter configuration corresponds to several basic nonlinear functions in the large language model; the initial approximate parameter configuration is input into a quantization configuration model to obtain a target optimization parameter configuration; the quantization configuration model is used to perform function approximation calculation according to the initial approximate parameter configuration to obtain an optimization parameter configuration, and the optimization parameter configuration conforms to the approximation of all the basic nonlinear functions; function approximation calculation is performed according to all the basic nonlinear functions after unified configuration processing based on the optimization parameter configuration to obtain an initial configuration of the activation function; the nonlinear function in the large language model is replaced by the initial configuration of the activation function and the basic nonlinear function of all approximate calculations after unified configuration processing to obtain the target optimization parameter configuration that meets the error loss threshold; the corresponding function module is selected according to the target optimization parameter configuration, the function module is configured by the target optimization parameter configuration, and the nonlinear function of the large language model is calculated by the function module. The present application solves the problems of long activation function calculation cycle and large hardware resource overhead in the prior art through the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0049] Figure 1 A flowchart of a nonlinear function calculation method for a large language model for this application;

[0050] Figure 2 This is an example diagram of the function module in this application. DETAILED DESCRIPTION

[0051] In order to enable those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application.

[0052] In some technologies, a large language model cannot optimize parameters such as the power of a nonlinear function, resulting in high computational complexity and huge consumption of hardware resources. In order to solve this technical problem, the present application provides a nonlinear function calculation method and system for a large language model. The nonlinear function calculation method and system for a large language model are described below:

[0053] For example, in order to meet the growing computational requirements of nonlinear functions, more and more methods are applied to the realization of nonlinear functions. The SRT (Sweeney-Robertson-Tocher) algorithm is widely used in the division operation in digital circuit design. It usually uses the lookup table QDS (Quotient Digit Selection) of the dividend and the divisor to determine each quotient. Based on the principle that the tangent of a function at a certain point can approximate the function curve near the point, the Newton-Raphson method iteratively adjusts the intersection of the tangent and the x-axis to approximate the root of the equation, which can be used to calculate the reciprocal of the square root. The CORDIC (Coordinate Rotation Digital Computer) algorithm approximates the final rotation angle by rotating a series of small angles multiple times, and can obtain high-precision nonlinear function calculation results. However, the above methods often require iteration to obtain the final result, and the calculation cycle is long. The piecewise polynomial approximation algorithm based on the lookup table can approximate the calculation of various nonlinear functions such as the reciprocal and the exponential with base 2. However, its huge lookup table and high power bring huge hardware resource consumption. In addition, the above methods often only focus on basic nonlinear functions, such as square roots and exponentials with base 2. The calculation of complex activation functions in large language models still uses the basic nonlinear function combination calculation method, which requires multiple calculation cycles to complete the calculation.

[0054] Based on the above problems, the present application provides the following embodiments.

[0055] Embodiment 1:

[0056] Figure 1 The present invention provides a flowchart of a nonlinear function calculation method for a large language model.

[0057] See also Figure 1 It can be seen that this embodiment provides a nonlinear function calculation method for a large language model, and the method includes:

[0058] S100, obtaining an initial approximate parameter configuration; the initial approximate parameter configuration corresponds to a plurality of basic nonlinear functions in a large language model;

[0059] Specifically, in this embodiment, first, the parameter configuration of the polynomial approximation is obtained, and the basic nonlinear functions to be approximated f1, ..., fn , the activation function to be approximated Act, the maximum absolute error of the segment ε, the maximum accuracy loss of the large language model L, the maximum number of segments of the basic nonlinear function 2 λ , the maximum quantization bit width of the polynomial coefficients ρ. For each basic nonlinear function, search for its optimal parameter configuration in turn.

[0060] The basic nonlinear function to be approximated and the related parameters in the initial approximate parameter configuration provide a data basis for subsequent configuration quantization.

[0061] The method further comprises:

[0062] S200, inputting the initial approximate parameter configuration into a quantization configuration model to obtain a target optimization parameter configuration; the quantization configuration model is used to perform function approximation calculation according to the initial approximate parameter configuration to obtain an optimization parameter configuration, and the optimization parameter configuration conforms to the approximation of all the basic nonlinear functions; performing function approximation calculation according to all the basic nonlinear functions after unified configuration processing based on the optimization parameter configuration to obtain an initial configuration of the activation function; using the initial configuration of the activation function and all the basic nonlinear functions of the approximate calculation after the unified configuration processing to replace the nonlinear function in the large language model, to obtain the target optimization parameter configuration that meets the error loss threshold;

[0063] Specifically, in this embodiment, the quantization configuration model is mainly used for function approximation calculation processing and function configuration replacement processing, and thereby obtains the basic nonlinear function and activation function after configuration optimization. It can be understood that, Figure 1 The steps that perform calculation functions are all completed in the quantitative configuration model.

[0064] Wherein, the function approximation calculation includes:

[0065] S211, selecting the basic nonlinear function to be approximated;

[0066] Specifically, in this embodiment, before performing function approximation calculation, it is necessary to first select one basic nonlinear function to be approximated from a plurality of basic nonlinear functions to be approximated, and perform subsequent approximation calculation based on the selected basic nonlinear function to be approximated.

[0067] Exemplarily, in all the basic nonlinear functions f1, ..., f n Select a function, for example, select f1.

[0068] The function approximation calculation also includes:

[0069] S212, initializing the basic nonlinear function to be approximated to obtain an initialized nonlinear function;

[0070] Specifically, in this embodiment, before performing function approximation, the selected basic nonlinear function to be approximated and related parameters need to be initialized. It can be understood that the process of function approximation is to perform continuous function iterative calculations by using different limiting parameters.

[0071] Among them, step S212 includes setting the highest power of the polynomial of the basic nonlinear function to be approximated, the number of basic nonlinear function segments, and the quantization bit width of the polynomial coefficients to 1; and setting the number of activation function segments of the activation function to be approximated to 1. The initialization process can be understood as setting some parameters that need to be optimized and adjusted to the initial state first, that is, adjusting these parameters to 1.

[0072] Exemplarily, the initialization parameters are: the highest power of the polynomial γ=1, the number of activation function segments η=1, the number of nonlinear function segments 2k=1, k=0, and the quantization bit width of the polynomial coefficient τ=1.

[0073] The function approximation calculation also includes:

[0074] S213, calculating a function segmentation error according to the initialized nonlinear function and the initialized activation function, and iteratively calculating the function segmentation error until the function segmentation error is less than the segmentation maximum absolute error, thereby obtaining a first optimization parameter configuration;

[0075] Specifically, in this embodiment, the function segmentation error is calculated for the initialized nonlinear function and the initialized activation function, and continuous iterative calculations are performed based on this. During the iterative calculation process, the relevant parameters of the function are continuously adjusted until the function segmentation error is finally obtained to be smaller than the segmentation maximum absolute error. The parameter configuration at this time is the first optimization parameter configuration.

[0076] For example, the REMEZ algorithm is used to calculate the function to be approximated f at the highest polynomial power γ, the number of segments is 2 k The polynomial coefficients under the configuration are quantized according to the total quantization bit width τ. Since the highest power of the polynomial is γ, the number of polynomial coefficients is γ+1. For a fixed total quantization bit width, there are multiple polynomial coefficient quantization combinations. Therefore, the algorithm traverses all quantization combinations and selects the quantization coefficient coef with the smallest MAE for storage. The MAE under this configuration represents the err under the total quantization bit width.

[0077] The specific steps of iterative calculation in step S213 are as follows:

[0078] Determine whether the function segmentation error is less than the segmentation maximum absolute error;

[0079] If yes, storing the initial approximate parameter configuration as the first optimized parameter configuration;

[0080] If not, the basic nonlinear function to be approximated is configured and adjusted to obtain an iterative nonlinear function, and a function segmentation error is calculated according to the iterative nonlinear function until the function segmentation error is less than the segmentation maximum absolute error.

[0081] The core content of step S213 is the configuration adjustment, through which each iterative calculation is optimized, and thereby a parameter configuration that meets the error requirements is obtained.

[0082] The specific steps of the configuration adjustment include:

[0083] Determine whether the quantization bit width of the polynomial coefficient is less than the maximum quantization bit width of the polynomial coefficient;

[0084] If yes, then the quantization bit width of the polynomial coefficients of the basic nonlinear function to be approximated is increased by 1, and the function segmentation error is recalculated;

[0085] If not, determining whether the number of segments of the basic nonlinear function is less than the maximum number of segments of the basic nonlinear function;

[0086] If yes, then the power exponent of the number of basic nonlinear function segments of the basic nonlinear function to be approximated is increased by 1, and the function segmentation error is recalculated;

[0087] If not, the highest power of the polynomial of the basic nonlinear function to be approximated is increased by 1, and the function segmentation error is recalculated.

[0088] Specifically, in this embodiment, it is determined whether err meets the error requirement ε. If it meets the requirement, proceed to the next step. If it does not meet the requirement, modify the parameter configuration and perform the error test in step S213 again. Considering the complexity of hardware implementation, when it is not greater than the maximum number of segments of the basic nonlinear function and the maximum quantization bit width of the polynomial coefficient, first consider increasing the quantization bit width and then consider increasing the number of segments. When both the quantization bit width and the number of segments have reached the maximum value but still cannot meet the accuracy requirement, increase the highest power of the polynomial.

[0089] The function approximation calculation also includes:

[0090] S214, selecting a parameter configuration that meets the approximate calculation of all the basic nonlinear functions to obtain the optimized parameter configuration.

[0091] Furthermore, in some embodiments, Figure 1It can be seen that after obtaining the optimization parameter configuration that satisfies all basic nonlinear function approximate calculations, it also includes a step of further loss calculation using the optimization parameter configuration. The specific process can be understood as the unified configuration processing, function replacement processing, and calculation of error loss based on the large language model after function replacement processing.

[0092] The unified configuration process can be understood as Figure 1 The quantization bit widths and the highest powers of the polynomials of all basic nonlinear functions are unified, and the unified configuration process includes:

[0093] Using the optimization parameter configuration, uniformly configure the polynomial coefficient quantization bit width and the highest power of the polynomial of all the basic nonlinear functions to obtain a plurality of uniformly configured functions;

[0094] Perform function calculation according to all the unified configuration functions to obtain the initial configuration of the activation function;

[0095] Replacing all the basic nonlinear functions in the large language model with the initial configuration of the activation function and the basic nonlinear functions of the approximate calculation of the unified configuration, and calculating the loss error of the large language model after the function replacement;

[0096] Determining whether the loss error is less than the maximum accuracy loss of the large language model;

[0097] If yes, then the target optimization parameter configuration is obtained;

[0098] If not, the number of activation function segments of the initial configuration of the activation function is increased by 1, and the loss iteration of the large language model is performed until the loss error is less than the maximum accuracy loss of the large language model.

[0099] Specifically, in this embodiment, after finding the optimal parameter configuration of all basic nonlinear functions, some parameters are unified to improve hardware utilization: the same quantization configuration and polynomial highest power γ are used for all basic nonlinear functions, and the accuracy and hardware overhead are weighed again. Then, the activation function is calculated, and the highest power γ and quantization configuration of the activation function are the results of the unification of the basic nonlinear functions.

[0100] (1) Similarly, the REMEZ algorithm is used to calculate the polynomial coefficients of the activation function to be approximated, and then quantized according to the quantization configuration.

[0101] (2) After replacing all nonlinear functions including activation functions in the large language model LLM with the polynomial approximation calculation function obtained under the above configuration, the accuracy of the large model is tested and compared with the accuracy of the original model to obtain the error loss.

[0102] (3) Determine whether the error loss meets the conditions. If not, the accuracy can be improved by increasing the number of activation function segments. Increase the number of activation function segments and repeat steps (1)-(2) until the error loss conditions are met.

[0103] For single-precision floating-point numbers, their normalized numbers can be expressed as:

[0104] V=(-1) S ×1.M×2 E-127 ;

[0105] Where S is the sign bit, S is 0 for positive numbers, and S is 1 for negative numbers. M is the mantissa, and E is the exponent. The calculation of nonlinear functions can be divided into the calculation of the exponential part and the mantissa part. For example, the calculation result of the reciprocal can be obtained by the following formula:

[0106]

[0107] The calculation of the exponential part can be processed according to the algorithm logic of the nonlinear function. Since the input of the mantissa is in the interval [1, 2), the above-mentioned algorithm can be used to obtain the optimal parameter configuration of the polynomial approximation for approximate calculation.

[0108] Among them, the above algorithm is used to configure the parameters of the Llama and Falcon large language models. The basic nonlinear functions to be approximated include the reciprocal, the reciprocal square root, and the exponential with base 2. The activation functions to be approximated are SiLU and GELU. The relevant activation function calculation formula is as follows, where Φ(x) is the cumulative distribution function of the Gaussian distribution.

[0109]

[0110] GELU(x)=x·Φ(x)

[0111] The maximum absolute error of the segmentation ε = 0.1, the maximum precision loss of the large language model L = 0.01, the maximum number of segments of the basic nonlinear function is 128, and the maximum total quantization bit width of the polynomial coefficients is 32. The optimal configuration obtained according to the above algorithm is as follows:

[0112]

[0113] The method further comprises:

[0114] S300, selecting a corresponding function module according to the target optimization parameter configuration, configuring the function module using the target optimization parameter configuration, and calculating the nonlinear function of the large language model using the function module.

[0115] Specifically, in this embodiment, after obtaining the target optimization parameter configuration, the corresponding function module can be found according to the target optimization parameter configuration, and one function module can correspond to several basic nonlinear functions and an activation function, and each corresponding function in the function module is configured through the target optimization parameter configuration, and the corresponding nonlinear function in the large language model is calculated using the configured function module.

[0116] Among them, the preprocessing includes decomposing the single-precision input data of the corresponding basic nonlinear function into an exponential part and a mantissa part according to the operation code, performing exponential part calculation, obtaining polynomial coefficients, and processing mantissa part extraction and parameter calculation; the polynomial calculation includes performing polynomial approximate calculation and leading zero detection according to the polynomial result output by the preprocessing module; the post-processing includes merging and normalizing the polynomial results output by the polynomial calculation.

[0117] Embodiment 2:

[0118] Figure 2 This is an example diagram of the function module in this application.

[0119] See also Figure 2 It can be seen that Figure 2 Several modules for calculating nonlinear functions are described in , but it should be noted that Figure 2 The actual function in the example is only an example description of this embodiment, and this embodiment is not the only one that can be used. Figure 2 The actual function in is used for function calculation. The type of function can be adjusted according to different requirements.

[0120] A nonlinear computing device is designed for the Llama and Falcon large language models. The device uses the best parameter configuration obtained by the algorithm. The device can calculate basic nonlinear functions, including reciprocal, reciprocal square root, exponential with base 2, and activation functions SiLU and GELU. The piecewise polynomial approximation configuration uses the best configuration found in the algorithm implementation. (polynomial degree, quantized polynomial coefficients and number of segments).

[0121] The hardware architecture diagram is shown in the figure below. The pre-processing module decomposes the input in FP32 format into the exponential part and the mantissa part, calculates the exponential part, and extracts x for subsequent polynomial approximation calculation. Different nonlinear functions require different units, and the corresponding quantized coefficients are stored in the pre-processing unit, while the pre-processing unit is shared for different activations. The opcode is used to select the input variable for the next polynomial approximation calculation according to the target calculation function. A unified polynomial approximation operation is used to calculate the nonlinear function of the mantissa part, and the variable minus is used to control whether to calculate c0-c1x or c0+c1x. The post-processing module combines the calculation results of the mantissa part and the calculation results of the exponential part and normalizes them to the single-precision floating point data format.

[0122] (1) Pre-processing module

[0123] ①Exponential function preprocessing module

[0124] Initialization: For normalized numbers, add the integer 1 at the 24th bit. For denormalized numbers, do not process them and record them as frac_1. Shift according to the exponent value to get the fixed-point representation of the real value with the decimal point between the 23rd and 24th bits. real =E-127, if E real If it is a positive number, shift frac_1 left by E. real bit; if E real If it is a negative number, shift frac_1 right by -E real bit, and the result is recorded as frac_2. Then the integer part int is the 24th to 31st bits of frac_2 (frac_2[30:23]), and the fractional part frac is the lower 23 bits of frac_2 (frac_2[22:0]).

[0125] Exponential part norm_exp calculation: Calculation 2 x When using the function, according to the fp32 data format, the output result E-127 is the integer part of the input result. If the input is a positive number, norm_exp = 127 + int, and the fractional part frac is recorded as frac_3. If the input is a negative number, there is:

[0126] 2 -(int+frac) =2 -(int+1-1+frac) =2 -(int+1) 2 (1-frac) =2 -(int+1) 2 frac补 ;

[0127] Since the exponent bit of the fp32 data format is biased, norm_exp = 127-(int+1). In particular, if frac is 0, norm_exp = 127-int. The two's complement of the fractional part frac is recorded as frac_3.

[0128] minus: For the exponential function, minus is 0 and c0+c1x is calculated.

[0129] Polynomial coefficients: Use uniform c0, c1 coefficients for the exponential function.

[0130] Extraction of x: x is the lower 23 bits of frac_3 (frac_3[22:0]).

[0131] ② Reciprocal preprocessing module

[0132] Initialization: For normalized numbers, add an integer 1 to the 24th bit. For denormalized numbers, if the exponent bits E are all 0, but the highest bit of M (index is 22) is 1, the mantissa segment is shifted 1 bit to the left to get a 24-bit initialization result. If the exponent bits E are all 0 and the highest bit of M is 0, and the second highest bit (index is 21) is 1, the mantissa segment is shifted 2 bits to the left and the lower 24 bits are taken as the mantissa initialization result. In other cases, since the maximum representation value of the FP32 data format is (2-2 -23 )×2 127 , the result of the inverse calculation will exceed the maximum representation value of the FP32 data format, so it is calculated directly in the post-processing module.

[0133] Calculation of the exponent part norm_exp: If the exponent bits E are all 0 and the highest bit of M is 0 and the second highest bit is 1, norm_exp = 255-E; in other cases, norm_exp = 254-E.

[0134] minus: For the reciprocal function, minus is 1, and c0-c1x is calculated.

[0135] Polynomial coefficients: Use uniform c0, c1 coefficients for the reciprocal function.

[0136] Extraction of x: x is the lower 23 bits of the initialization result.

[0137] ③Square root inverse preprocessing module

[0138] Initialization: Add the integer 1 at the 24th bit to the normalized number, denoted as frac_1, and the actual exponent value E real=E-127; For denormalized numbers, first determine the number of zeros before the 1 in the highest bit of the 23-bit mantissa M, recorded as zero_num. Shift M left by zero_num+1, so that the 24th bit integer is 1, and the format becomes 1.M', recorded as frac_1, and the actual exponent value is E real =-126-(zero_num+1).

[0139] Exponential part norm_exp calculation and polynomial coefficient determination: Since the inverse square root is calculated, the real exponential value needs to be converted to an even number. If the exponent is an odd number, E even =(E real -1) / 2, use the second set of polynomial coefficients to look up the table; if the exponent is an even number, E even =E real / 2, use the first set of polynomial coefficients to look up the table. norm_exp = 127-E even .

[0140] minus: For the inverse square root function, minus is 1, and c0-c1x is calculated.

[0141] Extraction of x: x is the lower 23 bits of the initialization result.

[0142] ④Activation function preprocessing module

[0143] Use a unified activation function for different activation functions to improve hardware utilization.

[0144] Determination of polynomial coefficients and minus: Determine based on the exponent bit, sign bit and target activation function to be calculated of the FP32 input. Only the input with exponent between 124 and 128 is looked up in the table for polynomial approximation calculation. The rest is directly sent to the post-processing module for processing. For the GELU function, when the exponent bit is 128 and the sign bit is 1, the polynomial coefficients need to be selected based on the highest bit of the mantissa. The SiLU function has a total of 10 groups of polynomial coefficients, and the GELU function has a total of 11 groups of polynomial coefficients. Each group of coefficients corresponds to its own minus value.

[0145] Extraction of x: x is the mantissa segment of the FP32 input data.

[0146] (2) LZD module

[0147] It is used to detect leading zeros of the polynomial calculation result and determine the number of zeros before the first 1 of the result, which is recorded as zero_num_sum. zero_num_sum, the polynomial calculation result sum and the norm_exp parameter obtained by preprocessing will be sent to post-processing for post-processing.

[0148] (3) Post - processing module

[0149] ① Exponential function post - processing

[0150] Output is 0 for negative infinity; for positive infinity input, output is positive infinity; for nan input, output is nan; for positive number input, when the exponent part plus the bias is greater than 133, the result exceeds the maximum value that can be represented by the fp32 data format, and the output is positive infinity; for negative number input, when the exponent part plus the bias is greater than 134, the output result exceeds the minimum value that can be represented by the fp32 denormalized number, and 0 is directly output; if the exponent part plus the bias is less than 103, representing a very small number, 1 is directly output; for other cases, perform the following steps:

[0151] The sign bit of the output result is always 0, indicating that the input is always positive.

[0152] If the norm_exp obtained from pre - processing is less than 1, it proves to be a denormalized output. Shift sum right by 1 - norm_exp, denoted as sum_norm, and the exponent bit of the output is 0; for the remaining cases, if norm_exp < zero_num_sum + 1, it proves that it will still become a denormalized number after shifting, so shift sum left by norm_exp - 1, and the exponent bit of the output is 0; for the remaining cases, shift sum left by zero_num_sum bits, denoted as sum_norm, and the exponent bit of the output is norm_exp - zero_num_sum.

[0153] Extract bits 9 - 31 of sum_norm as the mantissa bits of the output result (sum_norm[30:8]).

[0154] ② Reciprocal post - processing

[0155] If the input is of infinite or nan type, directly output the results 0 and nan respectively; if the input is a denormalized number with an absolute value less than or equal to 2 -128 perform the following processing for other inputs:

[0156] The sign bit of the output result is the same as the input.

[0157] If norm_exp < zero_num_sum + 1, it proves that it will become a denormalized number after shifting, so shift sum left by norm_exp - 1, and the exponent bit of the output is 0; for the remaining cases, shift sum left by zero_num_sum bits, denoted as sum_norm, and the exponent bit of the output is norm_exp - zero_num_sum.

[0158] The 9th to 31st bits of sum_norm are truncated as the mantissa of the output result (sum_norm[30:8]).

[0159] ③ Post-processing of the inverse square root

[0160] When the input is ±0, the output is ±∞; when the input is a negative number, the output is -nan; when the input is nan, the output is nan; when the input is positive infinity, the output is 0; in other cases, perform the following steps:

[0161] The sign bit of the output result is always 0, indicating that the input is always a positive number. The exponent bit of the output result is norm_exp-zero_num_sum; shift sum left by zero_num_sum to get sum_norm, and intercept the 9th to 31st bits of sum_norm as the mantissa of the output result (sum_norm[30:8]).

[0162] ④Activation function post-processing

[0163] When the input is a positive number with an exponent greater than 128 or an exponent less than 4, the input value is output directly; when the input is a negative number with an exponent greater than 128, 0 is output directly; when the input exponent is less than 124 and not less than 4, half of the input value is output.

[0164] The remaining cases are handled as follows:

[0165] The sign bit of the output result is consistent with the input; the exponent bit of the output result is 128-zero_num_sum; sum is shifted left by zero_num_sum to obtain sum_norm, and the 9th to 31st bits of sum_norm are intercepted as the mantissa bits of the output result (sum_norm[30:8]).

[0166] This embodiment has the following advantages:

[0167] This embodiment searches for the optimal data configuration of the piecewise polynomial approximation, reduces the size of the lookup table as much as possible, reduces the power and number of segments of the piecewise polynomial approximation, and thus reduces the computational complexity of the nonlinear function and the consumption of hardware resources.

[0168] This embodiment takes into account the complexity of the activation function implementation, adopts the same power and quantization bit width configuration as the basic nonlinear function, improves the approximation accuracy by adjusting the number of segments, and further improves the hardware utilization.

[0169] The method and hardware architecture provided in this embodiment are implemented on a large language model. Compared with the traditional table-based piecewise polynomial approximation algorithm, at the same frequency, the area and power consumption are reduced by 68.33% and 74.99% respectively.

[0170] The above specific implementation methods further explain in detail the purpose, technical solutions and beneficial effects of the embodiments of the present application. It should be understood that the above are only specific implementation methods of the embodiments of the present application and are not used to limit the protection scope of the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the embodiments of the present application should be included in the protection scope of the embodiments of the present application.

Claims

1. A nonlinear function calculation method for a large language model, characterized in that: The method comprises: Obtaining an initial approximate parameter configuration; the initial approximate parameter configuration corresponds to a number of basic nonlinear functions in a large language model; The initial approximate parameter configuration is input into a quantization configuration model to obtain a target optimization parameter configuration; the quantization configuration model is used to perform function approximation calculation according to the initial approximate parameter configuration to obtain an optimization parameter configuration, and the optimization parameter configuration conforms to the approximation of all the basic nonlinear functions; function approximation calculation is performed according to all the basic nonlinear functions after unified configuration processing based on the optimization parameter configuration to obtain an initial configuration of the activation function; the initial configuration of the activation function and all the approximation calculation basic nonlinear functions after unified configuration processing are used to replace the nonlinear function in the large language model to obtain the target optimization parameter configuration that meets the error loss threshold; A corresponding function module is selected according to the target optimization parameter configuration, the function module is configured using the target optimization parameter configuration, and the nonlinear function of the large language model is calculated using the function module.

2. The nonlinear function calculation method for a large language model according to claim 1, characterized in that: The initial approximation parameter configuration includes several basic nonlinear functions to be approximated, an activation function to be approximated, a segmented maximum absolute error, a maximum precision loss of a large language model, a maximum number of segments of the basic nonlinear function, and a maximum quantization bit width of polynomial coefficients.

3. The nonlinear function calculation method for a large language model according to claim 2, characterized in that: The function approximation calculation includes: Selecting the basic nonlinear function to be approximated; Initializing the basic nonlinear function to be approximated to obtain an initialized nonlinear function; Calculating a function segmentation error according to the initialized nonlinear function, and iteratively calculating the function segmentation error until the function segmentation error is less than the segmentation maximum absolute error, thereby obtaining a first optimization parameter configuration; A parameter configuration that satisfies the approximate calculation of all the basic nonlinear functions is selected to obtain the optimized parameter configuration.

4. The nonlinear function calculation method for a large language model according to claim 3, characterized in that: The step of initializing the basic nonlinear function to be approximated comprises: The highest power of the polynomial of the basic nonlinear function to be approximated, the number of basic nonlinear function segments, and the quantization bit width of the polynomial coefficients are set to 1; The number of activation function segments of the activation function to be approximated is set to 1.

5. The nonlinear function calculation method for a large language model according to claim 4, characterized in that: The step of iteratively calculating the function segmentation error until the function segmentation error is less than the segmentation maximum absolute error comprises: Determine whether the function segmentation error is less than the segmentation maximum absolute error; If yes, storing the initial approximate parameter configuration as the first optimized parameter configuration; If not, the basic nonlinear function to be approximated is configured and adjusted to obtain an iterative nonlinear function, and a function segmentation error is calculated according to the iterative nonlinear function until the function segmentation error is less than the segmentation maximum absolute error.

6. The nonlinear function calculation method for a large language model according to claim 5, characterized in that: The configuration adjustment includes: Determine whether the quantization bit width of the polynomial coefficient is less than the maximum quantization bit width of the polynomial coefficient; If yes, then the quantization bit width of the polynomial coefficients of the basic nonlinear function to be approximated is increased by 1, and the function segmentation error is recalculated; If not, determining whether the number of segments of the basic nonlinear function is less than the maximum number of segments of the basic nonlinear function; If yes, then the power exponent of the number of basic nonlinear function segments of the basic nonlinear function to be approximated is increased by 1, and the function segmentation error is recalculated; If not, the highest power of the polynomial of the basic nonlinear function to be approximated is increased by 1, and the function segmentation error is recalculated.

7. The nonlinear function calculation method for a large language model according to claim 6, characterized in that: The step of selecting a parameter configuration that meets the approximate calculation of all the basic nonlinear functions comprises: Determining whether the first optimization parameter configuration conforms to the approximate calculation of all the basic nonlinear functions; If yes, then the optimization parameter configuration is obtained; If not, the basic nonlinear function to be approximated is reselected, and approximate iterative calculation is performed until the first optimization parameter configuration meets the approximate calculation of all the basic nonlinear functions.

8. The nonlinear function calculation method for a large language model according to claim 7, characterized in that: The unified configuration process includes: Using the optimization parameter configuration, uniformly configure the polynomial coefficient quantization bit width and the highest power of the polynomial of all the basic nonlinear functions to obtain a plurality of uniformly configured functions; Perform function calculation according to all the unified configuration functions to obtain the initial configuration of the activation function; Using the initial configuration of the activation function and a number of basic nonlinear functions of approximation calculations that have completed unified configuration processing to replace all nonlinear functions in the large language model, and calculating the loss error of the large language model after function replacement; Determining whether the loss error is less than the maximum accuracy loss of the large language model; If yes, then the target optimization parameter configuration is obtained; If not, the number of activation function segments of the initial configuration of the activation function is increased by 1, and the loss iteration of the large language model is performed until the loss error is less than the maximum accuracy loss of the large language model.

9. The nonlinear function calculation method for a large language model according to claim 1, characterized in that: The function module includes a pre-processing module, a polynomial calculation module and a post-processing module; The preprocessing module includes a plurality of preprocessing units, each of which is used to preprocess different basic nonlinear functions and activation functions; The polynomial calculation module is used to perform polynomial calculation according to the processing result of the preprocessing module; The post-processing module includes a plurality of post-processing units, each of which is used to post-process different basic nonlinear functions and activation functions to obtain function output results.

10. The nonlinear function calculation method for a large language model according to claim 9, characterized in that: The preprocessing includes decomposing the single-precision input data corresponding to the basic nonlinear function into an exponential part and a mantissa part according to the operation code, performing exponential part calculation, obtaining polynomial coefficients, and processing mantissa part extraction and parameter calculation; The polynomial calculation includes performing polynomial approximate calculation and leading zero detection according to the polynomial result output by the preprocessing module; The post-processing includes merging and normalizing the polynomial results outputted from the polynomial calculation.

Images