Activation function value or layer normalization function value computation method and apparatus

Abstract

A method and apparatus for calculating activation function values ​​or layer normalization function values ​​are disclosed. The method includes: obtaining independent variables, which are numerical values ​​to be calculated in the operation process of activation functions or layer normalization functions in a large language model based on a neural network; processing the independent variables based on dedicated hardware to obtain the operation results of a nonlinear operator, wherein the processing is implemented through a multi-stage pipeline, with each stage performing a part of the operation process and the output data of each stage serving as the input data for the next stage; and obtaining activation function values ​​or layer normalization function values ​​based on software configuration. The processing of the independent variables based on dedicated hardware includes: determining a decimal intermediate value and an integer intermediate value based on the independent variables; obtaining segmentation parameters corresponding to the decimal intermediate value; determining a first intermediate result based on the segmentation parameters corresponding to the decimal intermediate value; determining a second intermediate result based on the integer intermediate value; and obtaining the operation results of the nonlinear operator based on the first and second intermediate results.

Description

Technical Field

[0001] This disclosure relates to, but is not limited to, the field of integrated circuit technology, and in particular to a method and apparatus for calculating activation function values ​​or layer normalization function values. Background Technology

[0002] An activation function is a function added to artificial neural networks to help the network learn complex patterns in the data. To enhance the network's representational and learning capabilities, activation functions in neural networks are non-linear. Layer normalization functions adjust the distribution of input data by normalizing it along a specific dimension, giving it zero mean and unit variance.

[0003] Currently, there are two main methods for calculating activation functions or layer normalization functions: (1) Software computing scheme based on central processing unit (CPU) / graphics processing unit (GPU) has the advantage of being flexible and capable of computing different types, different precisions and different numbers of activation functions or layer normalization functions. Its disadvantage is that it is more time-consuming and less efficient than dedicated hardware computing. (2) Dedicated hardware computing scheme has the advantage of short computing time, but the disadvantage is that it is inflexible and not universal. Each activation function or layer normalization function and each data precision requires its own dedicated hardware circuit, which cannot support newly emerging activation function or layer normalization function algorithms, nor can it support a variable number of computing tasks. Summary of the Invention

[0004] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0005] This disclosure provides a method for calculating activation function values ​​or layer normalization function values, including: Obtain the independent variable, wherein the independent variable is the value to be calculated in the operation process of the activation function or layer normalization function in the large language model based on the neural network; The independent variable is processed using dedicated hardware to obtain the computational results of the nonlinear operator; The activation function value or layer normalization function value is obtained based on the software configuration; The process of processing the independent variable based on dedicated hardware includes: determining the decimal median value and the integer median value based on the independent variable, wherein the decimal median value is within a preset range, the preset range is divided into multiple interval segments, and each interval segment corresponds to a set of segment parameters; Obtain the segmentation parameters corresponding to the decimal midpoint; determine the first intermediate result based on the segmentation parameters corresponding to the decimal midpoint; determine the second intermediate result based on the integer midpoint. Based on the first intermediate result and the second intermediate result, the operation result of the nonlinear operator is obtained; The step of obtaining the activation function value or layer normalization function value based on software configuration includes: obtaining the activation function value or layer normalization function value according to the operation result of the nonlinear operator.

[0006] This disclosure also provides an activation function value or layer normalization function value calculation device, applied in a neural network processor. The neural network processor includes the nonlinear operator calculation device and a processor. The nonlinear operator calculation device implements the operation result of the nonlinear operator through a hardware arithmetic unit. The processor synthesizes the operation result of the nonlinear operator through a software algorithm to obtain the activation function value or layer normalization function value. The nonlinear operator calculation device includes an input unit, an output unit, and an arithmetic unit, wherein: The input unit is configured to receive the argument; The output unit is configured to output the calculation result; The computation unit is configured to perform the following computation process: determine the decimal median value and the integer median value based on the independent variable, wherein the decimal median value is within a preset interval range, the preset interval range is divided into multiple interval segments, and each interval segment corresponds to a set of segmentation parameters; obtain the segmentation parameters corresponding to the decimal median value; determine a first intermediate result based on the segmentation parameters corresponding to the decimal median value; determine a second intermediate result based on the integer median value; and obtain the computation result of the nonlinear operator based on the first intermediate result and the second intermediate result. The computing unit is a hardware computing unit, which includes multiple pipelines. Each pipeline implements at least part of the computing process, and the output data of each pipeline is used as the input data of the next pipeline.

[0007] The activation function value or layer normalization function value calculation method and apparatus provided in this disclosure determine a decimal intermediate value and an integer intermediate value based on the independent variable, wherein the decimal intermediate value is within a preset interval range, the preset interval range is divided into multiple interval segments, and each interval segment corresponds to a set of segmentation parameters; obtain the segmentation parameters corresponding to the decimal intermediate value; determine a first intermediate result based on the segmentation parameters corresponding to the decimal intermediate value; determine a second intermediate result based on the integer intermediate value; and obtain the calculation result based on the first intermediate result and the second intermediate result. This provides a dedicated hardware calculation method that can support multiple different nonlinear operators. Based on this method, hardware calculation circuits for different nonlinear operators can be implemented, thereby realizing the rapid calculation of nonlinear operators.

[0008] After reading and understanding the accompanying diagrams and detailed descriptions, the other aspects can be understood. Attached Figure Description

[0009] The accompanying drawings are used to provide an understanding of the technical solutions of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of this disclosure and do not constitute a limitation on the technical solutions of this disclosure.

[0010] Figure 1 A flowchart illustrating a nonlinear operator calculation method as an exemplary embodiment of this disclosure; Figure 2 This is a schematic diagram of the structure of a nonlinear operator computing device, which is an exemplary embodiment of the present disclosure. Figure 3 This is a schematic diagram of the structure of an electronic device as an exemplary embodiment of the present disclosure. Detailed Implementation

[0011] The embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. Unless otherwise specified, the embodiments and features described herein can be combined arbitrarily.

[0012] Unless otherwise defined, the technical or scientific terms used in the embodiments of this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in the embodiments of this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" indicate that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, but do not exclude other elements or objects.

[0013] Currently, the activation functions or layer normalization functions commonly used in neural networks or large language models include the following types: (1) Sigmoid: also called the Logistic function, is used for the output of hidden layer neurons. It can map a real number to the interval (0, 1) and can be used for binary classification. Sigmoid is a very common activation function. The expression of the function is as follows: .

[0014] (2) Tanh: The Tanh activation function is also called the hyperbolic tangent activation function. It is similar to the Sigmoid function, but the Tanh function compresses it to the interval between -1 and 1. The expression of the function is as follows: .

[0015] (3) ReLU, a piecewise linear function, compensates for the gradient vanishing problem of the sigmoid and tanh functions. The function expression is as follows: .

[0016] (4) Leaky ReLU, an improved ReLU function. To solve the gradient vanishing problem in the ReLU activation function, we use Leaky ReLU when x < 0. The function expression is as follows: ; in, It is a very small number, such as 0.1, 0.01, etc.

[0017] (5) GELU: An activation function based on the Gaussian error function. Compared to activation functions such as ReLU, GELU is smoother, which helps to improve the convergence speed and performance of the training process. The function expression is as follows: (x+0.044715 .

[0018] in, 0.044715 are two adjustment coefficients for the GELU function.

[0019] (6) SiLU: A smooth nonlinear activation function that combines the characteristics of Sigmoid and linearity. Its function expression is as follows: .

[0020] (7) Softmax: A commonly used activation function, mainly used to convert each element of the input tensor into a probability value between 0 and 1, and the sum of these probability values ​​is 1. The function expression is as follows: = .

[0021] (8) RMSNorm (Root Mean Square Layer Normalization): A normalization function, a variant of layer normalization, characterized by using only the root mean square for normalization, omitting the mean calculation step. The expression is as follows: .

[0022] It should be noted that the activation functions or layer normalization functions mentioned above are only examples, and the actual activation functions or layer normalization functions are not limited to these.

[0023] In related technologies, function values ​​are directly calculated using software methods or dedicated hardware circuits. However, software methods are time-consuming and inefficient, while dedicated hardware circuits are inflexible and not universally applicable. This disclosure addresses this issue by designing dedicated hardware to calculate the core operators (nonlinear operators) of the aforementioned functions. When calculating activation function or layer normalization function values, the calculation results of several core operators can be synthesized using software methods to obtain the final activation function or layer normalization function value. In calculating activation function or layer normalization function values, the calculation speed of this disclosure is close to that of dedicated hardware circuits, and it represents a 30- to 100-fold improvement compared to software methods. Simultaneously, the flexibility of this disclosure is close to that of software methods; it can support newly emerging activation function or layer normalization function algorithms without changing or adding dedicated hardware, and its hardware resource consumption is reduced by 20% to 30% compared to dedicated hardware circuits. The calculation method provided in the embodiments of this disclosure can be applied to programmable hardware-accelerated numerical calculations of activation functions and layer normalization functions in large language models.

[0024] like Figure 1 As shown, this disclosure provides a method for calculating nonlinear operators, including: Step 101: Obtain the independent variable and determine the decimal median value and the integer median value based on the independent variable. The decimal median value is within a preset range, which is divided into multiple interval segments, and each interval segment corresponds to a set of segment parameters. Step 102: Obtain the segmentation parameters corresponding to the decimal median value; determine the first intermediate result based on the segmentation parameters corresponding to the decimal median value; determine the second intermediate result based on the integer median value. Step 103: Obtain the calculation result based on the first intermediate result and the second intermediate result.

[0025] The nonlinear operator calculation method provided in this disclosure determines the intermediate value of a decimal and an intermediate value of an integer based on the independent variable. The intermediate value of the decimal is located within a preset interval range, which is divided into multiple interval segments. Each interval segment corresponds to a set of segmentation parameters. The segmentation parameters corresponding to the intermediate value of the decimal are determined. A first intermediate result is determined based on the segmentation parameters corresponding to the intermediate value of the decimal. A second intermediate result is determined based on the intermediate value of the integer. The calculation result is obtained based on the first intermediate result and the second intermediate result. This provides a dedicated hardware calculation method that can support multiple different nonlinear operators. Based on this method, hardware calculation circuits for different nonlinear operators can be implemented.

[0026] For example, a nonlinear operator may include one or more of the following operators: exponential (EXP) operator Inverse square root (RSQRT) operator Natural logarithm (LN) operator However, this disclosure does not impose any restrictions on this.

[0027] Various nonlinear operators can be used individually or in combination to compute all of the aforementioned activation functions or layer normalization functions.

[0028] In some exemplary embodiments, the method further includes: Check if the independent variable is an invalid or special value; When the independent variable is an invalid or special value, output the pre-stored calculation result corresponding to the invalid or special value.

[0029] In this embodiment of the disclosure, for different nonlinear operators, the corresponding invalid values ​​or special values ​​may be the same or different.

[0030] In this embodiment of the disclosure, invalid values ​​are mainly determined based on the domain of the independent variable of each operator. For example, for the RECIP operator, x=0 is an invalid value, and when x=zero (+ / -0), f(x) is infinite (+ / -∞); for the RSQRT operator, x≤0 is an invalid value, and when x=+0, f(x) is positive infinity (+∞); for the LN operator, x≤0 is an invalid value, and when x=+0, f(x) is negative infinity (-∞), and so on.

[0031] In this embodiment of the disclosure, special values ​​mainly include negative infinity (-∞), positive infinity (∞), and non-numerical (NaN). For example, for the EXP operator, when x = positive infinity (+∞), f(x) is positive infinity (+∞); when x = negative infinity (-∞), f(x) is +0. For the RECIP operator, when x = infinity (+ / -∞), f(x) is zero (+ / -0). For the RSQRT operator, when x = positive infinity (+∞), f(x) is +0. For the LN operator, when x = positive infinity (+∞), f(x) is positive infinity (+∞).

[0032] In some exemplary embodiments, in step 101, the number of independent variables obtained can be one or more. This disclosure does not limit the number of independent variables obtained.

[0033] In this embodiment of the disclosure, the input unit can receive the independent variable x. In addition to receiving the value of the independent variable x, the input unit can also receive one or more of the following parameters: the type of operator to be calculated, the coefficient table (the value of mathematical constants) to be used in the operation, the precision of the independent variable (the floating-point format and bit width of x), the precision of the dependent variable (the floating-point format and bit width of f(x)), the number of independent / dependent variables (how many x's in total or how many f(x's) need to be obtained), etc.

[0034] In this embodiment of the disclosure, the number of independent variables received by the input unit at one time can be multiple. That is, based on the nonlinear operator calculation method provided in this embodiment of the disclosure, a general-purpose vector operation circuit based on hardware can be realized.

[0035] When the input unit receives multiple independent variables at a time, the nonlinear operator type corresponding to each independent variable can be the same or different, the precision and / or format of each independent variable can be the same or different, and the precision and / or format of the dependent variable corresponding to each independent variable can be the same or different. After obtaining the calculation result, the precision and / or format of the obtained calculation result can be converted according to the precision and / or format of the dependent variable to be output.

[0036] In some exemplary implementations, the segmentation parameters corresponding to an interval segment may include a set of corresponding slopes and intercepts.

[0037] In this embodiment of the disclosure, the curve of the first intermediate result corresponding to the decimal median value in each interval can be regarded as a straight line segment. The curve of the first intermediate result corresponding to the decimal median value within the preset interval is a curve composed of multiple straight line segments. Each decimal median value and the first intermediate result corresponding to the decimal median value can be regarded as a point on a certain straight line segment. By pre-storing a set of corresponding slopes and intercepts for each interval, the first intermediate result corresponding to each decimal median value within the preset interval can be obtained by linear calculation.

[0038] In this embodiment of the disclosure, within a given interval, the average error between the first intermediate result determined based on the segmentation parameters corresponding to the decimal median value and the actual first intermediate result corresponding to the decimal median value is 0 or close to 0. That is, the calculated result will not be overall larger or smaller than the true value; rather, some values ​​will be larger and some smaller, thus avoiding accumulated errors and meeting the high precision requirements of neural networks. In this embodiment of the disclosure, the first intermediate result corresponding to the actual decimal median value can be set according to the type of nonlinear operator, and this disclosure does not impose any restrictions on this.

[0039] In some exemplary embodiments, the preset interval range may be pre-divided into sixteen interval segments; however, the preset interval range and the number of interval segments can be set as needed. This disclosure does not limit this.

[0040] In some embodiments described below, the preset interval range is set to [1, 2), and in other embodiments, the preset interval range is set to [0.25, 0.5). However, those skilled in the art will understand that the size of the preset interval range can be set according to actual needs. For example, the preset interval range can also be set to other ranges such as [0.5, 1).

[0041] In this embodiment of the disclosure, the preset interval range is divided into sixteen interval segments as an example. However, in actual use, the preset interval range can also be divided into other numbers of interval segments besides sixteen.

[0042] In some exemplary embodiments, determining the first intermediate result in step 102 based on the segmentation parameter corresponding to the decimal intermediate value may include: The first intermediate result is calculated using the following formula: =a +b; in, This is the first intermediate result. is the middle value of the decimal, a is the corresponding slope, and b is the corresponding intercept.

[0043] In some other exemplary embodiments, the segmentation parameter corresponding to an interval segment may include a first intermediate result corresponding to at least two decimal midpoints within the interval segment. For example, the segmentation parameter corresponding to an interval segment may include a first intermediate result corresponding to the midpoints of the first and last decimal midpoints of each interval segment. However, this disclosure does not limit this.

[0044] In some exemplary embodiments, in step 102, determining the first intermediate result based on the segmentation parameter corresponding to the decimal intermediate value includes: The first intermediate result is calculated using the following formula: ; in, and The midpoint between two decimal values ​​is pre-stored for each interval. The middle value of a decimal The corresponding first intermediate result, The middle value of a decimal The corresponding first intermediate result, The decimal median value determined based on the independent variable. The middle value of a decimal The corresponding first intermediate result.

[0045] In some exemplary embodiments, in step 103, the calculation result is obtained based on the first intermediate result and the second intermediate result, including: Calculate the product or sum of the first intermediate result and the second intermediate result, and use the product or sum of the first intermediate result and the second intermediate result as the result of the operation.

[0046] In this embodiment of the disclosure, the specific method of obtaining the calculation result based on the first intermediate result and the second intermediate result can be determined according to the type of nonlinear operator, and this disclosure does not impose any restrictions on this.

[0047] The following describes in detail the nonlinear operator calculation method of this disclosure using various nonlinear operators as examples. It should be noted that the nonlinear operator calculation method of this disclosure is not limited to the nonlinear operators listed below; that is, the nonlinear operator calculation method of this disclosure can also be applied to the calculation of any other type of nonlinear operator.

[0048] In some exemplary embodiments, the nonlinear operator is the EXP operator. .

[0049] Determining the decimal and integer median values ​​based on the independent variable includes: multiplying the independent variable by... e, obtain the first intermediate value, and divide the first intermediate value into a decimal intermediate value and an integer intermediate value. The sum of the integer intermediate value and the decimal intermediate value is equal to the first intermediate value.

[0050] In this embodiment, the range of the decimal median value can be within the interval [1, 2).

[0051] In some exemplary embodiments, the first intermediate value is a floating-point number format including a sign part, a mantissa part, and an exponent part. Dividing the first intermediate value into a decimal intermediate value and an integer intermediate value includes: The values ​​of the decimal and integer parts of the first intermediate value are determined based on the sign part, mantissa part, and exponent part of the first intermediate value; The values ​​of the decimal and integer parts of the first intermediate value are adjusted based on the size of the decimal part and the preset compensation value to obtain the decimal intermediate value and the integer intermediate value.

[0052] In this embodiment of the disclosure, the preset compensation value can be 1 or 2. Adjusting the values ​​of the decimal and integer parts of the first intermediate value based on the size of the decimal part and the preset compensation value includes: When the decimal part of the first intermediate value is between [-1, 0), take the preset compensation value of 2, add 2 to the decimal part of the first intermediate value, and subtract 2 from the integer part of the first intermediate value; When the decimal part of the first intermediate value is between [0, 1), the preset compensation value is taken as 1, the decimal part of the first intermediate value is increased by 1, and the integer part of the first intermediate value is decreased by 1.

[0053] For the EXP operator, suppose we let = ,but = Transform the independent variable x into and will Decompose into integer parts and decimal part ,but It can be represented as ,Right now Integer part It is an integer consisting of a sign bit and an integer part, with a fractional part. It is a floating-point number consisting of three parts: sign bit, exponent bit, and mantissa bit.

[0054] For example, for the EXP operator, taking x=5 as an example, the independent variable x is transformed into... , =5 ,when When represented in FP32 format, the accurate value is 7.213475, and its binary format is as follows: .

[0055] The first to the 23rd digits are the mantissa, the 24th to the 31st digits are the exponent, and the 32nd digit is the sign digit.

[0056] The exponent digit = " =129, and The actual value of the exponent is 2, therefore, take The high 2 bits of the mantissa "Combined The hidden bits of the mantissa are obtained as " ",Right now integer part = 7. At this time Remaining mantissas = " " decimal part The hidden bit of the mantissa is the 5th bit from the left, therefore The mantissa is its lower 23-5=18 bits, then padded with 5 zeros, that is, " ". The exponent is 129-5=124 = " " The actual value of the exponent is -3. Therefore... In FP32 format: .

[0057] The current decimal part It is a floating-point number with a value in the range [-1, 1). If the decimal part is used directly... As the decimal midpoint, determining the interval ID to which the decimal midpoint belongs becomes quite complicated. Therefore, in this embodiment of the disclosure, the decimal part... Convert to the middle decimal value in the range [1, 2). This allows for the rapid determination of the interval ID to which the median value of a decimal belongs.

[0058] To obtain the median decimal value within the range [1, 2) ,when When < 0, let = +2, = -2, then = ( / ) ( ) = = ; when When >0, let = +1, = -1, then = ( / ) ( ) = = .

[0059] Taking x=5 as an example again, in this example, >0, passed +1 gives the midpoint of the decimal values ​​in the interval [1, 2). In FP32 format, it is represented as: .

[0060] Based on the above adjustments to the mantissa, the exponent is compensated accordingly while ensuring the final value remains unchanged. Since the mantissa was adjusted to the range [1, 2) by adding 1, the exponent now needs to be subtracted by 1 to obtain a new exponent, denoted as the integer median value. = 7-1 = 6.

[0061] In some exemplary implementations, the decimal median value is determined. The interval segment includes: Determine the middle value of a decimal The high four bits of the mantissa in the floating-point format; The highest four digits of the mantissa are used as the decimal midpoint. The ID of the interval segment it is located in.

[0062] In this embodiment of the disclosure, for the EXP operator, the decimal intermediate value is used... Converting to a base-2 floating-point number in the interval [1, 2) (i.e., including the sign bit, exponent bit, and mantissa bit) achieves the conversion of decimal intermediate values. Quantification.

[0063] For example, the decimal midpoint within the interval [1, 2). The corresponding quantized floating-point numbers are shown in Table 1:

[0064] Table 1 Based on the middle value of decimals The high four bits of the mantissa, representing the "quantization value," are used to look up a coefficient table to obtain a specific set of coefficients (slope a and intercept b). For the EXP operator, let... ,Will Within the interval [1, 2) If the function curve can be viewed as a curve composed of multiple straight line segments, then the function within each interval... The curve can be approximated as: a +b.

[0065] For the EXP operator, an exemplary table of slope "a" coefficients is shown in Table 2, and an exemplary table of intercept "b" coefficients is shown in Table 3:

[0066] Table 2

[0067] Table 3 Taking x=5 as an example again, =1.2134752, which belongs to the interval [1.1875, 1.25) of ID=3 in Table 1 above. Therefore, according to Tables 2 and 3, the corresponding slope a=0x3fce7b54 = 1.6131387, and the corresponding intercept b=0x3eb9261f = 0.36161896 are obtained by calculating using the following formula. Value: a +b= 1.6131387 1.2134752 + 0.36161896 = 2.3191227.

[0068] In this embodiment, The middle value of a decimal. The actual corresponding first intermediate result is The first intermediate result, a, is determined based on the segmentation parameters corresponding to the decimal midpoint. +b, within a given interval, the mean error between the two is 0 or close to 0. This ensures that the calculated result within a given interval will not be overall too large or too small compared to the true value; instead, some values ​​will be larger and some smaller, thus avoiding cumulative error and meeting the high precision requirements of neural networks.

[0069] In some exemplary embodiments, for the EXP operator, determining the second intermediate result based on the integer median value includes: determining that the sign bit of the second intermediate result is 0, the mantissa bits are multiple 0s, and the actual value of the exponent bit is the integer median value. That is, calculating a power with base 2 and the integer median value as the exponent, and using the calculated power as the second intermediate result.

[0070] In this embodiment of the disclosure, the "normalized" exponent portion is... (i.e., the intermediate value of an integer) is converted into a base-2 floating-point number, denoted as . (That is, it also includes three parts: the sign bit, the exponent bit, and the mantissa bit). Specifically, let... Then when > 0, = {1'b0, +127, 23'b0}={1'b0, +8'd126, 23'b0};When When < 0, = {1'b0, +127, 23'b0}= {1'b0, +8'd125, 23'b0}.

[0071] Taking x=5 as an example again, = 6, then = {1'b0, 8'd6 + 8'd127, 23'b0} = 64, its FP32 format representation is: .

[0072] In some exemplary embodiments, for the EXP operator, the calculation result is obtained based on the first intermediate result and the second intermediate result, including: Calculate the product of the first intermediate result and the second intermediate result, and use the product of the first intermediate result and the second intermediate result as the result of the operation.

[0073] because , Then there is = .

[0074] Taking x=5 as an example again, = 2.3191227 64 = 148.42386. And... When the true value is represented using FP32, it can be accurate to 148.41316, with a relative error of -7.2. This meets the requirements for computational accuracy.

[0075] In some other exemplary embodiments, the nonlinear operator is the RECIP operator. .

[0076] Determining the decimal and integer median values ​​based on the independent variable includes: using the last digit of the independent variable as the decimal median value and the exponent of the independent variable as the integer median value. In this embodiment, the range of the decimal median value can be within the interval [1, 2).

[0077] For the RECIP operator, the independent variable is represented as a base-2 floating-point number and decomposed into three parts: sign bit, exponent bit, and mantissa bit.

[0078] Since the independent variable x can be expressed as ×(1+M× )× Where S is the sign bit (1 bit), M is the mantissa bit (23 bits), and E is the exponent bit (8 bits), therefore, we can let =1+M× , = E-127, therefore x= × × .

[0079] For example, for the RECIP operator, taking x=7 as an example, its FP32 format representation is as follows: .in, =" = 1.75, which is already within the interval [1, 2), therefore, no adjustment is needed and it can be directly used as the decimal median. Correspondingly, the exponent does not need to be adjusted and can be directly used as the integer median value, i.e. = 2.

[0080] In some exemplary implementations, the decimal median value is determined. The interval segment includes: Determine the middle value of a decimal The high four bits of the mantissa in the floating-point format; The highest four digits of the mantissa are used as the decimal midpoint. The ID of the interval segment it is located in.

[0081] In this embodiment of the disclosure, similar to the aforementioned EXP operator, the decimal intermediate value is used. Converting to a base-2 floating-point number in the interval [1, 2) (i.e., including the sign bit, exponent bit, and mantissa bit) achieves the conversion of decimal intermediate values. Quantization. In this embodiment, the decimal median value is within the interval [1, 2). The corresponding quantized floating-point numbers can be found in Table 1 above.

[0082] Based on the middle value of decimals The high four bits of the mantissa, representing the "quantization value," are used to look up a coefficient table to obtain a specific set of coefficients (slope a and intercept b). For the RECIP operator, since 1 / x = = × × ,make = ,Will Within the interval [1, 2) If the function curve can be viewed as a curve composed of multiple straight line segments, then the function within each interval... The curve can be approximated as: a +b.

[0083] For the RECIP operator, an exemplary table of slope "a" coefficients is shown in Table 4, and an exemplary table of intercept "b" coefficients is shown in Table 5:

[0084] Table 4

[0085] Table 5 Taking x=7 as an example again, since = 1.75. Referring to Table 1, 1.75 belongs to the interval [1.75, 1.8125) for ID=12 in Table 1. Therefore, according to Tables 4 and 5, the corresponding slope a = 0xbea181b3 = -0.31544265, and the corresponding intercept b = 0x3f8fcabc = 1.1233745. The following formula is used to calculate... Value: a +b= -0.31544265 1.75 + 1.1233745 = 0.57134985.

[0086] In this embodiment, The middle value of a decimal. The actual corresponding first intermediate result is The first intermediate result, a, is determined based on the segmentation parameters corresponding to the decimal midpoint. +b, within a given interval, the mean error between the two is 0 or close to 0. This ensures that the calculated result within a given interval will not be overall too large or too small compared to the true value; instead, some values ​​will be larger and some smaller, thus avoiding cumulative error and meeting the high precision requirements of neural networks.

[0087] In some exemplary implementations, for the RECIP operator, determining a second intermediate result based on an integer intermediate value includes: determining that the sign bit of the second intermediate result is the value of the sign bit of the independent variable, the mantissa bit is multiple 0 bits, and the actual value of the exponent bit is the opposite of the integer intermediate value.

[0088] That is, the value of the sign bit of the second intermediate result is set to the value of the sign bit of the independent variable, and the actual value of the exponent bit of the second intermediate result is set to the opposite of the integer intermediate value.

[0089] In this embodiment of the disclosure, the integer intermediate value Convert to a base-2 floating-point number, denoted as (That is, it also includes three parts: the sign bit, the exponent bit, and the mantissa bit). Specifically, let... = Then when x>0, In FP32 format: = {1'b0, +8'd127, 23'b0};When x<0, In FP32 format: ={1'b1, +8'd127, 23'b0}.

[0090] Taking x=7 as an example again, since x>0 and = 2, therefore, = {1'b0, 8'd127 - 8'd2, 23'b0} = 0.25, its FP32 format representation is: .

[0091] In some exemplary embodiments, for the RECIP operator, the computation result is obtained based on the first intermediate result and the second intermediate result, including: Calculate the product of the first intermediate result and the second intermediate result, and use the product of the first intermediate result and the second intermediate result as the result of the operation.

[0092] because , Then 1 / x = × .

[0093] Taking x=5 as an example again, 1 / x= 0.57134985 × 0.25 = 0.14283746. The true value of 1 / 7, when expressed in FP32, can be accurate to 0.14285715, with a relative error of -1.4 × 10⁻⁶. This meets the requirements for computational accuracy.

[0094] In some other exemplary embodiments, the nonlinear operator is the RSQRT operator. .

[0095] The process of determining the decimal and integer median values ​​based on the independent variable includes: taking the mantissa of the independent variable as the decimal median value, and dividing the exponent of the independent variable by 2 and rounding it down in the direction of decreasing absolute value to obtain the integer median value.

[0096] In this embodiment, the range of the decimal median value can be within the interval [1, 2).

[0097] Since the independent variable x can be expressed as ×(1+M× )× Where S is the sign bit (1 bit), M is the mantissa bit (23 bits), and E is the exponent bit (8 bits), therefore, we can let =1+M× , = E-127, therefore x= × × .

[0098] Since x must be greater than 0 for the RSQRT operator, x can be expressed as × So, 1 / = = × It can make = Then when When it is a positive odd number, It can be set to: = ; when When it is a negative odd number, It can be set to: = .

[0099] Combining the two cases above, the integer median value It can be set to That is, taking along the direction of decreasing absolute value. The integer part of / 2. Let the compensation value be... ,but It can be represented as ={1'b0, - +8'd127, 23'b0} ={1'b0, - +8'd127, 23'b0} .

[0100] In some exemplary implementations, for the RSQRT operator, based on the integer intermediate value Determine the second intermediate result, including: Calculations are based on base 2 and are expressed as the median of an integer. The opposite of the exponent is the power of the exponent. The calculated power is used as the third intermediate result (that is, the sign bit of the third intermediate result is 0, the mantissa bits are multiple 0s, and the actual value of the exponent is the intermediate value of the integer). (the opposite of the number) Calculate the third intermediate result and the preset first compensation coefficient. The product of the third intermediate result and the preset first compensation coefficient is used. The product of is used as a second intermediate result .

[0101] In some exemplary implementations, when the exponential part of the independent variable When it is a positive odd number, the first compensation coefficient for When the exponential part of the independent variable When it is a negative odd number, the first compensation coefficient for When the exponential part of the independent variable When it is an even number, the first compensation coefficient The value is 1.

[0102] For example, for the RSQRT operator, taking x=11 as an example, its FP32 format representation is as follows:

[0103] .

[0104] in, =" = 1.375, which is already within the interval [1, 2), therefore, no adjustment is needed and it can be directly used as the decimal midpoint. The actual value of the exponent is "3". We need to take the integer part of the result after dividing it by 2 and moving in the direction of decreasing absolute value, and use this as the midpoint of the integer. = = = 1. In this example, due to the exponent of x... ="3", that is For positive odd numbers, a first compensation coefficient needs to be set. = = When the actual value of the exponent of x is even, the first compensation coefficient can be considered... = 1.

[0105] In this embodiment of the disclosure, similar to the aforementioned EXP and RECIP operators, the RSQRT operator also uses the decimal intermediate value. Converting to a base-2 floating-point number in the interval [1, 2) (i.e., including the sign bit, exponent bit, and mantissa bit) achieves the conversion of decimal intermediate values. Quantization. In this embodiment, the decimal median value is within the interval [1, 2). The quantized floating-point numbers can be referred to in Table 1 above.

[0106] Based on the middle value of decimals The high four bits of the mantissa "quantization value" are used to look up the coefficient table to obtain a specific set of coefficients (slope a and intercept b). For the RSQRT operator, let... = ,Will Within the interval [1, 2) If the function curve can be viewed as a curve composed of multiple straight line segments, then the function within each interval... The curve can be approximated as: a +b.

[0107] For the RSQRT operator, an exemplary table of slope "a" coefficients is shown in Table 6, and an exemplary table of intercept "b" coefficients is shown in Table 7:

[0108] Table 6

[0109] Table 7 Taking x=11 as an example again, since = 1.375. Referring to Table 1, 1.375 belongs to the interval [1.375, 1.4375) for ID=6 in Table 1. Therefore, according to Tables 6 and 7, the corresponding slope a = 0xbe99774c = -0.2997383, and the corresponding intercept b = 0x3fa1e60c = 1.264833. The following formula is used to calculate... Value: a +b= -0.2997383 1.375 + 1.264833 = 0.85269284.

[0110] In this embodiment, The middle value of a decimal. The actual corresponding first intermediate result is The first intermediate result, a, is determined based on the segmentation parameters corresponding to the decimal midpoint. +b, within a given interval, the mean error between the two is 0 or close to 0. This ensures that the calculated result within a given interval will not be overall too large or too small compared to the true value; instead, some values ​​will be larger and some smaller, thus avoiding cumulative error and meeting the high precision requirements of neural networks.

[0111] because =1, = ,therefore, = {1'b0, 8'd127 – 8'd1, 23'b0} = 0.5 ( = 0.35355338. Where 0.5 is represented in FP32 format as:

[0112] .

[0113] In some exemplary embodiments, for the RSQRT operator, the computation result is obtained based on the first intermediate result and the second intermediate result, including: Calculate the product of the first intermediate result and the second intermediate result, and use the product of the first intermediate result and the second intermediate result as the result of the operation.

[0114] because = , = Then there is 1 / = = × = .

[0115] Taking x=11 as an example again, 1 / = 0.85269284 × 0.35355338 = 0.30147242. And... When the true value is represented in FP32 format, it can be accurate to 0.30151134, with a relative error of -1.3× This meets the requirements for computational accuracy.

[0116] In some other exemplary embodiments, the nonlinear operator is the LN operator. .

[0117] Among them, determining the median value of decimals and the median value of integers based on the independent variable includes: Divide the mantissa of the independent variable by the preset first multiplier to obtain the decimal middle value; Add the square root of the first multiplier to the exponent of the independent variable to obtain the integer median value.

[0118] For example, the decimal middle value is in the range of [0.25, 0.5), and the first multiplier is 4; however, this disclosure does not limit this.

[0119] Since the independent variable x can be expressed as ×(1+M× )× Where S is the sign bit (1 bit), M is the mantissa bit (23 bits), and E is the exponent bit (8 bits), therefore, we can let =1+M× , = E-127, therefore the independent variable x= × × .

[0120] Since the independent variable x must be greater than 0 for the LN operator, x can be expressed as: × ,in, The value is within the interval [1, 2). Because... ln The value is 0, if you directly select Using a decimal median value will result in a relatively large error in the synthesized curve. Therefore, this disclosure addresses this issue by... Divide by 4 (i.e., the first multiplier can be 4) to convert it into a decimal midpoint within the range [0.25, 0.5). Then, x = × = × × = × .

[0121] For example, for the LN operator, taking x=9.6 as an example, its FP32 format representation is as follows:

[0122] .

[0123] For the LN operator, the mantissa value needs to be adjusted to the range [0.25, 0.5). When x = 9.6, the actual value of the mantissa is "1.00110011001100110011010" = 1.2. Dividing this by 4 gives the value within the range [0.25, 0.5). In FP32 format, it is represented as:

[0124] .

[0125] In some exemplary implementations, the decimal median value is determined. The interval segment includes: Determine the middle value of a decimal The high four bits of the mantissa in the floating-point format; The highest four digits of the mantissa are used as the decimal midpoint. The ID of the interval segment it is located in.

[0126] In this embodiment of the disclosure, for the LN operator, the decimal median value is used... This converts the decimal part into a base-2 floating-point number (containing the sign bit, exponent, and mantissa) within the range [0.25, 0.5), thus achieving the conversion of the decimal intermediate value. Quantification.

[0127] For example, the decimal midpoint within the range [0.25, 0.5). The corresponding quantized floating-point numbers can be found in Table 8.

[0128]

[0129] Table 8 Based on the middle value of decimals The high four bits of the mantissa are used to look up the coefficient table to obtain a specific set of coefficients (slope a and intercept b). For the LN operator, lnx = ln( × )= ln +ln =ln + ln .make ln ,Will Within the interval [0.25, 0.5) If the function curve can be viewed as a curve composed of multiple straight line segments, then the function within each interval... The curve can be approximated as: a +b.

[0130] For the LN operator, an exemplary table of slope "a" coefficients is shown in Table 9, and an exemplary table of intercept "b" coefficients is shown in Table 10:

[0131] Table 9

[0132] Table 10 Taking x=9.6 as an example again, since = 0.3, which falls within the interval [0.29688, 0.3125) of ID=3 in Table 8 above. Therefore, according to Tables 9 and 10, the corresponding slope a=0x4052010d = 3.2813141, and the corresponding intercept b=0xc00c0dfb = -2.1883533. The following formula is used to calculate... Value: a +b= 3.2813141×0.3+(-2.1883533) = -1.2039591.

[0133] In this embodiment, The middle value of a decimal. The actual corresponding first intermediate result is ln The first intermediate result, a, is determined based on the segmentation parameters corresponding to the decimal midpoint. +b, within a given interval, the mean error between the two is 0 or close to 0. This ensures that the calculated result within a given interval will not be overall too large or too small compared to the true value; instead, some values ​​will be larger and some smaller, thus avoiding cumulative error and meeting the high precision requirements of neural networks.

[0134] When x = 9.6, the exponent is The actual value is "3", so we need to take the integer part after adding 2. As an intermediate value of integers = 3 + 2 = 5. (The integer midpoint is used for calculation.) Converted to floating-point numbers, it can be represented in FP32 format as follows:

[0135] .

[0136] In addition, a second compensation coefficient is added. Then ln2, lnx = ln + ln = + .

[0137] In some exemplary implementations, based on the integer intermediate value Determine the second intermediate result ,include: Calculate the median value of integers Second compensation coefficient The product of the integer median value Second compensation coefficient The product of is used as a second intermediate result .

[0138] For the LN operator, let = ln = Taking x=9.6 as an example again, = = 5 ln2 = 3.465736.

[0139] In some exemplary embodiments, based on the first intermediate result Second intermediate result The calculation results include: Calculate the first intermediate result With the second intermediate result The sum of the first intermediate results With the second intermediate result The sum is used as the result of the operation.

[0140] Taking x=9.6 as an example again, lnx= -1.2039591 + 3.465736 = 2.261777. The true value of ln9.6, expressed in FP32, is accurate to 2.261763, with a relative error of 6.2. This meets the requirements for computational accuracy.

[0141] like Figure 2 As shown in the figure, this disclosure also provides a nonlinear operator computing device, including an input unit, an output unit, and a computation unit, wherein: The input unit is configured to receive arguments; The output unit is configured to output the calculation result; The arithmetic unit is configured to perform the following arithmetic process: determine the decimal median value and the integer median value based on the independent variable, wherein the decimal median value is within a preset interval range, the preset interval range is divided into multiple interval segments, and each interval segment corresponds to a set of segment parameters; obtain the segment parameters corresponding to the decimal median value; determine the first intermediate result based on the segment parameters corresponding to the decimal median value; determine the second intermediate result based on the integer median value; and obtain the arithmetic result based on the first intermediate result and the second intermediate result.

[0142] For example, a nonlinear operator may include one or more of the following operators: exponential (EXP) operator Inverse square root (RSQRT) operator Natural logarithm (LN) operator However, this disclosure does not impose any restrictions on this.

[0143] In this embodiment of the disclosure, the calculation of all the aforementioned activation functions or layer normalization functions can be achieved through a variety of different operators or a combination of a variety of different operators.

[0144] In some exemplary embodiments, the computing unit includes a multi-stage pipeline, each stage of which performs at least part of the computing process, and the output data of each stage of the pipeline serves as the input data of the next stage of the pipeline.

[0145] In some exemplary embodiments, the computing unit is a six-stage pipeline structure, wherein: The first-level pipeline is configured to determine the decimal median and the integer median based on the independent variable; The second-level pipeline is configured to adjust the decimal midpoint to a preset range and adjust the integer midpoint based on the adjustment of the decimal midpoint; the preset range is divided into multiple intervals, each interval corresponding to a set of segment parameters; The third-level pipeline is configured to determine the segmentation parameters corresponding to the decimal median value; and to determine the second intermediate result based on the integer median value. The fourth-level pipeline is configured to determine the first intermediate result based on the segmentation parameters corresponding to the decimal midpoint; The fifth-level pipeline is configured to obtain the computation result based on the first intermediate result and the second intermediate result; The sixth-level pipeline is configured to perform precision conversion on the calculation results; when the independent variable is an invalid or special value, it outputs the pre-stored calculation result corresponding to the invalid or special value.

[0146] It should be noted that the six-stage pipeline structure of the computing unit in the exemplary embodiments of this disclosure is merely one engineering implementation scheme. In actual use, the number of pipeline stages and the functions configured for each stage can be set according to the actual chip's clock frequency, the IP library used, the process and physical design library, etc., and this disclosure does not impose any restrictions on this. When setting the number of pipeline stages and the functions configured for each stage, the processing delay of each pipeline stage should be kept roughly balanced as much as possible.

[0147] The following describes the nonlinear operator computing device of this disclosure embodiment, taking a six-stage pipelined structure of the computing unit as an example.

[0148] In this embodiment of the disclosure, the input unit is before the first stage pipeline, the output unit is after the last stage pipeline, and there are several stages of pipeline in between.

[0149] The input unit is responsible for providing the independent variable (the value of x), operator type (one of the four types of operators), coefficient table (the value of mathematical constants) required by the nonlinear operator, precision of the independent variable (floating-point format and bit width of x), precision of the dependent variable (floating-point format and bit width of f(x)), and the number of independent / dependent variables (how many x's in total or how many f(x's) need to be obtained).

[0150] The output unit is responsible for providing the calculated dependent variable (the value of f(x)). Depending on the configuration of the input unit, a batch may contain the calculation results of multiple dependent variables, and the accuracy of the results meets the requirements of the input configuration.

[0151] Each intermediate pipeline stage can adopt a similar structure. Each pipeline stage can include a computation subunit, a control subunit, and a storage subunit. The computation subunit is used to perform the specific calculations, the control subunit is used to control the calculation process, and the storage subunit is used to store the coefficient tables used in the calculation process and the calculation results of each pipeline stage. The output data of the previous pipeline stage (including the input unit) serves as the input data for the next pipeline stage (including the output unit).

[0152] The following uses four nonlinear operators (EXP operators) as examples. RECIP operator RSQRT operator LN operator Taking this as an example, all four nonlinear operators are processed by these six pipeline stages. Each pipeline stage performs a part of the functions in the operation of the four nonlinear operators. The specific functions performed by a particular pipeline stage for a particular operator may be different.

[0153] (I) The first-stage production line is the pre-processing stage I, responsible for: 1) Determine whether the independent variable x is a valid or invalid value. For example, for the RECIP operator, x=0 is an invalid value; for the RSQRT operator, x≤0 is an invalid value; for the LN operator, x≤0 is an invalid value, etc.

[0154] 2) Determine whether the independent variable is a special value, such as whether it is negative infinity (-∞), positive infinity (∞), or non-numerical (NaN).

[0155] 3) For the EXP operator, transform the independent variable x into... ,in, = ,but = And will Decompose into integer parts and decimal part ,but It can be represented as ,Right now Integer part It is an integer consisting of a sign bit and an integer part, with a fractional part. It is a floating-point number consisting of three parts: sign bit, exponent bit, and mantissa bit.

[0156] For example: taking x=5 as an example, when When represented in FP32 format, the accurate value is 7.213475, and its binary format is as follows: .

[0157] The exponent digit = " =129, and The actual value of the exponent is 2, therefore, take The high 2 bits of the mantissa "Combined The hidden bits of the mantissa are obtained as " ",Right now integer part = 7. At this time Remaining mantissas = " " decimal part The hidden bit of the mantissa is the 5th bit from the left, therefore The mantissa is its lower 23-5=18 bits, then padded with 5 zeros, that is, " ". The exponent is 129-5=124 = " " The actual value of the exponent is -3. Therefore... In FP32 format: .

[0158] 4) For RECIP, RSQRT and LN operators, the independent variable is represented as a base-2 floating-point number and decomposed into three parts: sign bit, exponent bit and mantissa bit.

[0159] For example, for the RECIP operator, taking x=7 as an example, its FP32 format representation is as follows: .

[0160] For the RSQRT operator, taking x=11 as an example, its FP32 format representation is as follows:

[0161] .

[0162] For the LN operator, taking x=9.6 as an example, its FP32 format representation is as follows:

[0163] .

[0164] (II) The second stage pipeline is the preprocessing stage II, which is responsible for making specific adjustments (or "normalization") to the mantissa and exponent bits of the output of the preprocessing stage I. The processing methods are different for different operators.

[0165] (1) For the EXP, RECIP, and RSQRT operators, adjust the value of the mantissa to the range [1, 2). For the LN operator, adjust the value of the mantissa to the range [1, 2). The value is adjusted to the range of [0.25, 0.5).

[0166] For the EXP operator, the current fractional part It is a floating-point number with a value in the range [-1, 1), and a pre-set compensation value for the decimal part. Adjustments need to be made. When When < 0, the compensation value is 2, let = +2; when When the value is greater than 0, the compensation value is 1. Let = +1, which converts it to the interval [1, 2). Taking x=5 as an example again, since... >0, passed +1 gives the midpoint of the decimal values ​​in the interval [1, 2). In FP32 format, it is represented as: .

[0167] For the RECIP operator, let =1+M× M is the value of the mantissa of the independent variable x, since (1+M× The value of ) is already within the range [1, 2), therefore, no adjustment is needed, and it can be directly used as the decimal median. Taking x=7 as an example, the actual last digit is "1.11000000000000000000000" = 1.75, which is already within the range [1, 2) and requires no adjustment. =1.75.

[0168] For the RSQRT operator, let =1+M× M is the value of the mantissa of the independent variable x. Since 1 + M × The value of ) is already within the range [1, 2), therefore, no adjustment is needed, and it can be directly used as the decimal median. Taking x=11 as an example, the actual last digit is "1.011000000000000000000000" = 1.375, which is already within the range [1, 2) and requires no adjustment. =1.375.

[0169] For the LN operator, let =1+M× , M is the value of the mantissa of the independent variable x. The value is adjusted to the range [0.25, 0.5). Taking x = 9.6 as an example, the actual value of the mantissa is "1.00110011001100110011010" = 1.2. By dividing it by 4, we can obtain the value within the range [0.25, 0.5). Represented in FP32 format as

[0170] .

[0171] (2) Based on the above adjustment of the mantissa, the exponent is compensated accordingly while ensuring that the value of the independent variable remains unchanged.

[0172] For the EXP operator, the exponent is adjusted using the aforementioned compensation value of 1 or 2 to obtain the integer intermediate value. When < 0, due to = +2, therefore = -2; when When >0, due to = +1, therefore, = -1. Taking x=5 as an example again, because the mantissa is adjusted to the range [1, 2) by adding 1, the exponent needs to be subtracted by 1 to obtain a new exponent, denoted as . = 7-1 = 6.

[0173] For the RECIP operator, the exponent does not need to be adjusted, i.e. Taking x=7 as an example again, = 2.

[0174] For the RSQRT operator, take the direction of decreasing absolute value. The integer part of / 2 is used as ,Right now = Taking x=11 as an example, its exponent... The actual value is "3", therefore = = = 1.

[0175] For the LN operator, the exponent of the independent variable x is... Add 2, as ,Right now = Taking x=9.6 as an example again, the exponent... The actual value is "3", therefore... = 3+2= 5.

[0176] (III) The third-level production line is the pre-processing level III, responsible for: (1) Quantize the mantissa of the preprocessed Level II output and convert it into a quantized base 2 floating-point number in the range of [1, 2) or [0.25, 0.5) (that is, it also includes the sign bit, exponent bit and mantissa bit).

[0177] For example, within the interval [1, 2), The corresponding quantized floating-point numbers are shown in Table 1 above.

[0178] For example, within the interval [0.25, 0.5), The corresponding quantized floating-point numbers are shown in Table 8 above.

[0179] (2) According to The high four digits of the mantissa are used to look up the coefficient table to obtain a specific set of coefficients (slope a and intercept b).

[0180] For the EXP operator, an exemplary table of slope "a" coefficients is shown in Table 2 above, and an exemplary table of intercept "b" coefficients is shown in Table 3 above. Taking x=5 as an example again... =1.2134752. Referring to Table 1, 1.2134752 belongs to the interval [1.1875~1.25] of ID=3 in Table 1. Therefore, according to Table 2 and Table 3, the corresponding slope a=0x3fce7b54 = 1.6131387 and the corresponding intercept b=0x3eb9261f = 0.36161896 are obtained.

[0181] For the RECIP operator, an exemplary table of slope "a" coefficients is shown in Table 4, and an exemplary table of intercept "b" coefficients is shown in Table 5. Taking x=7 as an example again, since... = 1.75. Referring to Table 1, 1.75 belongs to the interval [1.75~1.8125] of ID=12 in Table 1. Therefore, according to Table 4 and Table 5, the corresponding slope a=0xbea181b3 = -0.31544265 and the corresponding intercept b=0x3f8fcabc = 1.1233745 are obtained.

[0182] For the RSQRT operator, an exemplary table of slope "a" coefficients is shown in Table 6, and an exemplary table of intercept "b" coefficients is shown in Table 7. Taking x=11 as an example again, since... = 1.375. Referring to Table 1, 1.375 belongs to the interval [1.375~1.4375] of ID=6 in Table 1. Therefore, according to Table 6 and Table 7, the corresponding slope a=0xbe99774c = -0.2997383 and the corresponding intercept b=0x3fa1e60c = 1.264833 are obtained.

[0183] For the LN operator, an exemplary table of slope "a" coefficients is shown in Table 9, and an exemplary table of intercept "b" coefficients is shown in Table 10. Taking x=9.6 as an example again, since... = 0.3, which belongs to the interval [0.29688~0.3125) of ID=3 in Table 8 above. Therefore, according to Table 9 and Table 10, the corresponding slope a=0x4052010d = 3.2813141 and the corresponding intercept b=0xc00c0dfb = -2.1883533 are obtained.

[0184] (3) Convert the "normalized" exponent part of the preprocessing level II output into a floating-point number with base 2, denoted as y_exp (which also includes the sign bit, exponent bit and mantissa bit).

[0185] For the EXP operator, let Then when > 0, = {1'b0, +127, 23'b0}={1'b0, +8'd126, 23'b0};When When < 0, = {1'b0, +127, 23'b0}= {1'b0, +8'd125, 23'b0}. Taking x=5 as an example again, = 6, then ={1'b0, 8'd6 + 8'd127, 23'b0} = 64, its FP32 format representation is: .

[0186] For the RECIP operator, let = Then when x>0, In FP32 format: = {1'b0, +8'd127, 23'b0};When x<0, In FP32 format: ={1'b1, +8'd127, 23'b0}. Taking x=7 as an example again, since x>0 and = 2, therefore, = {1'b0, 8'd127 - 8'd2, 23'b0} = 0.25, its FP32 format representation is: .

[0187] For the RSQRT operator, let = ,but = ,in, The preset compensation coefficient is used when the exponential part of the independent variable x... When it is a positive odd number, for When the exponential part of the independent variable x When it is a negative odd number, for When the exponential part of the independent variable x When it is even, The value is 1. Taking x=11 as an example again, since... =1, = ( (which is a positive odd number), therefore = {1'b0,8'd127 – 8'd1, 23'b0} = 0.5 ( = 0.35355338. Where 0.5 is represented in FP32 format as:

[0188] .

[0189] For the LN operator = ,in, The preset compensation coefficient, ln Taking x=9.6 as an example again, = 5 ln2 = 3.465736.

[0190] (IV) The fourth-level pipeline is the core computing level, responsible for: (1) The first intermediate result is obtained by calculating according to the following formula. ( The value of ), where, To preprocess the quantized values ​​of the Level III output, , A specific set of coefficients for preprocessing the Level III output.

[0191] ( ) + .

[0192] For the EXP operator, taking x=5 as an example again, = 1.6131387 1.2134752 + 0.36161896 = 2.3191227.

[0193] For the RECIP operator, taking x=7 as an example again, = -0.31544265 1.75 + 1.1233745 = 0.57134985.

[0194] For the RSQRT operator, taking x=11 as an example again, = -0.2997383 1.375 + 1.264833 = 0.85269284.

[0195] For the LN operator, taking x=9.6 as an example again, = 3.2813141×0.3+(-2.1883533) = -1.2039591.

[0196] (2) For the floating-point number representing the reduced exponent part of the preprocessing level III output (denoted as ) Without any special treatment, it was directly transported to the next level.

[0197] (V) The fifth stage pipeline is the post-processing stage I, responsible for processing the output of the core computing stage. and The values ​​of the dependent variable f(x) are treated differently by different operators.

[0198] (1) For the EXP, RECIP and RSQRT operators, the combined formula is: .

[0199] For the EXP operator, taking x=5 as an example again, f(x) = 2.3191227 64 = 148.42386. And... When the true value is represented using FP32, it can be accurate to 148.41316, with a relative error of -7.2. This meets the requirements for computational accuracy.

[0200] For the RECIP operator, taking x=7 as an example again, f(x) = 0.57134985 × 0.25 = 0.14283746. The true value of 1 / 7, expressed in FP32, can be accurate to 0.14285715, with a relative error of -1.4 × 10⁻⁶. This meets the requirements for computational accuracy.

[0201] For the RSQRT operator, taking x=11 as an example again, f(x) = 0.85269284 × 0.35355338 = 0.30147242. And... When the true value is represented in FP32 format, it can be accurate to 0.30151134, with a relative error of -1.3× This meets the requirements for computational accuracy.

[0202] (2) For the LN operator, the synthesis formula is: .

[0203] For the LN operator, taking x=9.6 as an example again, f(x) = -1.2039591 + 3.465736 = 2.261777. The true value of ln9.6, expressed in FP32, can be accurate to 2.261763, with a relative error of 6.2. This meets the requirements for computational accuracy.

[0204] (VI) The sixth-level production line is post-processing level II, responsible for: (1) If the preprocessing level I determines that the independent variable x is invalid, then the corresponding special value is used as the calculation result of the dependent variable f(x) and converted into a format and bit width that meet the accuracy requirements provided by the input unit. For example, for the RECIP operator, when x = zero (+ / -0), f(x) is infinite (+ / -∞). For the RSQRT operator, when x = +0, f(x) is positive infinity (+∞). For the LN operator, when x = +0, f(x) is negative infinity (-∞), etc.

[0205] (2) If the preprocessing level I determines that the independent variable x is a special value, then the corresponding special value is used as the calculation result of the dependent variable f(x) and converted into a format and bit width that meet the precision requirements provided by the input unit. For example, for the EXP operator, when x = positive infinity (+∞), f(x) is positive infinity (+∞); when x = negative infinity (-∞), f(x) is +0. For the RECIP operator, when x = infinity (+ / -∞), f(x) is zero (+ / -0). For the RSQRT operator, when x = positive infinity (+∞), f(x) is +0. For the LN operator, when x = positive infinity (+∞), f(x) is positive infinity (+∞).

[0206] (3) If the preprocessing stage I determines that the independent variable x is a normal value, then the output of the postprocessing stage I is used as the calculation result of the dependent variable f(x) and converted into a format and bit width that meet the accuracy requirements provided by the input unit.

[0207] In this embodiment of the disclosure, when the first-stage pipeline determines that the independent variable x is an invalid or special value, the first-stage and intermediate pipelines can directly transmit the independent variable x to the last-stage pipeline in sequence, and the last-stage pipeline can then obtain the pre-set calculation result corresponding to the invalid or special value and output it.

[0208] In the above embodiments of this disclosure, the boundary division of the pipeline, the functional definition of each pipeline stage, and the definition and connection relationship of several components within each pipeline stage are all specifically defined and designed for the four nonlinear operators exemplified in this scheme. The dedicated hardware in traditional schemes is defined and designed for a specific type of activation function. Although there may be pipeline-style structures or modular designs, the abstract granularity, function, scale, operation steps, and input / output definitions of each component do not have a strict one-to-one mapping or analogy relationship.

[0209] In the embodiments of this disclosure, within a given interval, the average error between the first intermediate result determined based on the segmentation parameter corresponding to the decimal median value and the actual corresponding first intermediate result is 0 or close to 0. That is, the calculated result will not be overall larger or smaller than the true value; instead, some values ​​will be larger and some smaller, thereby avoiding accumulated errors and meeting the high precision requirements of neural networks.

[0210] In the embodiments disclosed above, the quantized floating-point number intervals for the EXP, RECIP, and RSQRT operators are defined as [1, 2), and for the LN operator, the quantized floating-point number intervals are defined as [0.25, 0.5). Compared to the [0, 1) interval, the [1, 2) and [0.25, 0.5) intervals can be directly determined based on the "normalized" decimal midpoints. The highest four digits of the last four digits are used to directly select the ID of the corresponding interval, without needing to... Comparing with the left and right boundaries of each segment interval makes hardware implementation easier. In addition, for the LN operator, the interval [0.25, 0.5) avoids intervals like [1, 2) or [0.5, 1) that contain boundary values ​​of "1", thus avoiding the problem of large errors in direct calculation near "1" by the LN operator.

[0211] like Figure 3 As shown, this disclosure also provides an electronic device, including: a processor and a nonlinear operator computing device as described in any of the above embodiments.

[0212] The electronic device of this disclosure embodiment realizes rapid calculation of nonlinear operators through the aforementioned nonlinear operator calculation device. When it is necessary to calculate a function containing any one or more of the aforementioned nonlinear operators, the calculation results of one or more nonlinear operators can be synthesized by a software algorithm in the processor to obtain the final function value to be calculated.

[0213] In this embodiment of the disclosure, the processor may be a central processing unit (CPU), a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a tensor processing unit (TPU), or other artificial intelligence (AI) chips.

[0214] For example, the electronic device may be a neural network processor; however, this disclosure is not limiting in this regard.

[0215] In this embodiment, the type of activation function or layer normalization function to be calculated, the precision of the input / output data, and the number of input / output data can be configured by software, thereby realizing a hardware-accelerated calculation method for various activation functions or layer normalization functions with variable precision and variable number. This embodiment significantly improves calculation speed and offers high flexibility, supporting emerging activation function or layer normalization function algorithms without requiring changes or additions to dedicated hardware. It can be applied to programmable hardware-accelerated numerical calculations of activation functions and layer normalization functions in large language models.

[0216] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0217] It should be noted that the above embodiments or implementation methods are merely exemplary and not restrictive. Therefore, this disclosure is not limited to the content specifically shown and described herein. Various modifications, substitutions, or omissions can be made to the form and details of the implementations without departing from the scope of this disclosure.

Claims

1. A method for calculating activation function values ​​or layer normalization function values, characterized in that, include: Obtain the independent variable, wherein the independent variable is the value to be calculated in the operation process of the activation function or layer normalization function in the large language model based on the neural network; The independent variable is processed using dedicated hardware to obtain the operation result of the nonlinear operator. The processing is implemented through a multi-stage pipeline, with each stage of the pipeline performing at least part of the operation process, and the output data of each stage of the pipeline serving as the input data of the next stage of the pipeline. The activation function value or layer normalization function value is obtained based on the software configuration; The process of processing the independent variable based on dedicated hardware includes: determining the decimal median value and the integer median value based on the independent variable, wherein the decimal median value is within a preset range, the preset range is divided into multiple interval segments, and each interval segment corresponds to a set of segment parameters; Obtain the segmentation parameters corresponding to the decimal midpoint; determine the first intermediate result based on the segmentation parameters corresponding to the decimal midpoint; determine the second intermediate result based on the integer midpoint. Based on the first intermediate result and the second intermediate result, the operation result of the nonlinear operator is obtained; The step of obtaining the activation function value or layer normalization function value based on software configuration includes: obtaining the activation function value or layer normalization function value according to the operation result of the nonlinear operator.

2. The method according to claim 1, characterized in that, The segmentation parameters corresponding to one of the interval segments include a set of corresponding slopes and intercepts, or, include a first intermediate result corresponding to at least two of the decimal intermediate values ​​within the interval segment; The step of determining the first intermediate result based on the segmentation parameters corresponding to the intermediate decimal value includes: The first intermediate result is calculated according to the following formula: ==a +b; or, ; in, This is the first intermediate result. Let be the middle value of the decimal, 'a' be the corresponding slope, and 'b' be the corresponding intercept. and The midpoint between two decimal values ​​is pre-stored for each interval. The middle value of a decimal The corresponding first intermediate result, The middle value of a decimal The corresponding first intermediate result.

3. The method according to claim 1, characterized in that, The method further includes: Detect whether the independent variable is an invalid or special value; When the independent variable is an invalid or special value, the pre-stored calculation result corresponding to the invalid or special value is output.

4. The method according to claim 1, characterized in that, The nonlinear operator is an exponential function with base e; The step of determining the decimal median value and the integer median value based on the independent variable includes: Multiply the independent variable by log2e to obtain the first intermediate value, wherein the first intermediate value is a floating-point number format including the sign part, the mantissa part and the exponent part; The size of the compensation value is determined based on the size of the decimal part of the first intermediate value; The fractional part of the first intermediate value is added to the compensation value to obtain the fractional intermediate value; the integer part of the first intermediate value is subtracted from the compensation value to obtain the integer intermediate value, wherein the fractional intermediate value is within the range of [1, 2). The step of determining the second intermediate result based on the integer intermediate value includes: determining that the sign bit of the second intermediate result is 0, the mantissa bit is multiple 0s, and the actual value of the exponent bit is the integer intermediate value.

5. The method according to claim 1, characterized in that, The nonlinear operator is a reciprocal function; The step of determining the decimal median value and the integer median value based on the independent variable includes: taking the mantissa of the independent variable as the decimal median value and taking the exponent of the independent variable as the integer median value, wherein the decimal median value is within the range of [1, 2). The step of determining the second intermediate result based on the integer intermediate value includes: determining that the sign bit of the second intermediate result is the value of the sign bit of the independent variable, the mantissa bit is multiple 0s, and the actual value of the exponent bit is the opposite of the integer intermediate value.

6. The method according to claim 1, characterized in that, The nonlinear operator is the inverse square root function; The step of determining the decimal intermediate value and the integer intermediate value based on the independent variable includes: taking the mantissa of the independent variable as the decimal intermediate value, dividing the exponent of the independent variable by 2 and rounding it in the direction of decreasing absolute value to obtain the integer intermediate value, wherein the decimal intermediate value is within the range of [1, 2). Determining the second intermediate result based on the integer intermediate value includes: The third intermediate result is determined to be a power of 2, with the opposite of the intermediate integer value as the exponent; Calculate the product of the third intermediate result and the preset first compensation coefficient, and use the product of the third intermediate result and the first compensation coefficient as the second intermediate result; when the exponent of the independent variable is a positive odd number, the first compensation coefficient is... When the exponent of the independent variable is a negative odd number, the first compensation coefficient is: When the exponent of the independent variable is even, the first compensation coefficient is 1.

7. The method according to any one of claims 4 to 6, characterized in that, The step of obtaining the operation result of the nonlinear operator based on the first intermediate result and the second intermediate result includes: Calculate the product of the first intermediate result and the second intermediate result, and use the product of the first intermediate result and the second intermediate result as the operation result of the nonlinear operator.

8. The method according to claim 1, characterized in that, The nonlinear operator is the natural logarithm function; The step of determining the decimal median value and the integer median value based on the independent variable includes: dividing the mantissa of the independent variable by a preset first multiplier to obtain the decimal median value, and adding the square root of the preset first multiplier to the exponent of the independent variable to obtain the integer median value, wherein the decimal median value is within the range of [0.25, 0.5). The step of determining the second intermediate result based on the integer intermediate value includes: calculating the product of the integer intermediate value and a preset second compensation coefficient, and using the product of the integer intermediate value and the second compensation coefficient as the second intermediate result; The step of obtaining the operation result of the nonlinear operator based on the first intermediate result and the second intermediate result includes: calculating the sum of the first intermediate result and the second intermediate result, and taking the sum of the first intermediate result and the second intermediate result as the operation result of the nonlinear operator.

9. The method according to claim 1, characterized in that, The step of obtaining the segmentation parameters corresponding to the decimal median value includes: Determine the high four bits of the mantissa portion of the floating-point format of the decimal midpoint; The value of the highest four digits of the mantissa is used as the interval ID of the middle value of the decimal. Obtain the segmentation parameter corresponding to the interval segment ID, and use it as the segmentation parameter corresponding to the decimal median value.

10. A device for calculating activation function values ​​or layer normalization function values, characterized in that, This technology is applied in neural network processors, which include a nonlinear operator computation device and a processor. The nonlinear operator computation device implements the computation results of the nonlinear operators through a hardware arithmetic unit, and the processor synthesizes the computation results of the nonlinear operators through a software algorithm to obtain activation function values ​​or layer normalization function values. The nonlinear operator computation device includes an input unit, an output unit, and a computation unit, wherein: The input unit is configured to receive the argument; The output unit is configured to output the calculation result; The computation unit is configured to perform the following computation process: determine the decimal median value and the integer median value based on the independent variable, wherein the decimal median value is within a preset interval range, the preset interval range is divided into multiple interval segments, and each interval segment corresponds to a set of segmentation parameters; obtain the segmentation parameters corresponding to the decimal median value; determine a first intermediate result based on the segmentation parameters corresponding to the decimal median value; determine a second intermediate result based on the integer median value; and obtain the computation result of the nonlinear operator based on the first intermediate result and the second intermediate result. The computing unit is a hardware computing unit, which includes multiple pipelines. Each pipeline implements at least part of the computing process, and the output data of each pipeline is used as the input data of the next pipeline.

Images