Self-adaptive layer normalization hardware acceleration system and method

Abstract

The invention provides an adaptive layer normalization hardware acceleration system and method. The system comprises a data input module, a parallel calculation module, an adaptive parameter generation module, a reciprocal square root operation module and a normalization calculation output module. The data input module is configured to: obtain an input vector and context information; the parallel computing module is configured to perform parallel computing according to an input vector to obtain a mean value and a variance; the adaptive parameter generation module is configured to input the input vector and the context information into the parameter model to obtain a scaling parameter and an offset parameter; the reciprocal square root operation module is configured to calculate a root extraction negation value of the variance by using a fast reciprocal square root algorithm; and the normalization calculation output module is configured to calculate a normalization result according to the input vector, the mean value, the scaling parameter, the offset parameter and the root extraction negation value so as to solve the problem of low layer normalization calculation efficiency in the layer normalization process at present.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of neural networks, and in particular to an adaptive layer normalization hardware acceleration system and method. BACKGROUND

[0002] With the wide application of deep neural networks in natural language processing, image recognition, speech synthesis and other fields, the structure of neural network models is becoming more and more complex, and higher requirements are put forward for the convergence speed and generalization ability of neural network models. Layer normalization technology is a normalization technology commonly used in neural network models, which can accelerate the model training speed and improve the model convergence stability. As an important means to improve the stability of model training, it is widely introduced into various network architectures. Among them, the adaptive layer normalization algorithm has better adaptability and generalization ability due to its ability to dynamically adjust according to input feature changes and task context information, and has been widely applied.

[0003] In order to meet the needs of deep neural network models for efficient training and stable convergence, various hardware implementation schemes for layer normalization have been proposed at present. Through the customization of hardware circuit structure, the specific calculation process of layer normalization is optimized to realize high-speed data processing and calculation. Then, due to the data dependency of calculating variance and mean, the current work still adopts sequential calculation mode, resulting in large overall hardware delay and low throughput. SUMMARY

[0004] The present application provides an adaptive layer normalization hardware acceleration system and method to solve the technical problem of low layer normalization calculation efficiency in the current layer normalization process.

[0005] The first aspect of the present application provides an adaptive layer normalization hardware acceleration system, comprising:

[0006] a data input module, a parallel computing module, an adaptive parameter generation module, a square root reciprocal operation module, and a normalization calculation output module connected in communication;

[0007] The data input module is configured to:

[0008] obtain an input vector and context information; the context information is determined by the input vector;

[0009] The parallel computing module is configured to:

[0010] perform parallel calculation according to the input vector, and simultaneously obtain mean and variance;

[0011] The adaptive parameter generation module is configured to:

[0012] inputting the input vector and the context information into a parameter model to obtain a scaling parameter and an offset parameter; the parameter model is generated by training historical input vectors and historical context information;

[0013] The square root reciprocal operation module is configured to:

[0014] Based on the variance, the inverse square root of the variance is calculated using a fast square root reciprocal algorithm;

[0015] The normalization calculation output module is configured to:

[0016] According to the input vector, the mean, the scaling parameter, the offset parameter, and the inverse square root value, a normalization result is calculated.

[0017] In this embodiment, the parallel calculation module is configured with an addition tree; the addition tree includes a plurality of adders;

[0018] The addition tree is configured to:

[0019] According to the input vector, an input vector sum value and an input vector sum square value are calculated; the input vector sum value is:

[0020]

[0021] The sum square value is:

[0022]

[0023] In this embodiment, the mean is:

[0024]

[0025] In the formula, x k is an input vector; n is any positive integer;

[0026] The variance is:

[0027]

[0028] In this embodiment, the adaptive parameter generation module is further configured to:

[0029] Obtain a neural network model; the neural network model is configured with a first parameterization function and a second parameterization function;

[0030] Train the neural network model using historical input vectors and historical context information to obtain a parameter model;

[0031] Input the input vector and the context information into the parameter model to obtain a scaling parameter and an offset parameter.

[0032] In this embodiment, the scaling parameter is:

[0033] γ(c) = f γ (c) ;

[0034] wherein c is context information; f γ is a first parameterized function;

[0035] The offset parameter is:

[0036] β(c) = f β (c) ;

[0037] wherein f β is a second parameterized function.

[0038] In this embodiment, the square root reciprocal operation module comprises:

[0039] a first conversion unit, a subtractor, a Newton approximation unit, a second conversion unit;

[0040] The first conversion unit is configured to:

[0041] convert the variance from an integer to a floating point number;

[0042] The subtractor is configured to:

[0043] subtract the floating point number from a magic constant to obtain a first numerical value; the magic constant is obtained using the exponent bit characteristics of the floating point number;

[0044] The Newton approximation unit is configured to:

[0045] perform a Newton approximation operation according to the first numerical value and a second numerical value to obtain a target numerical value; the second numerical value is float ÷ 2;

[0046] wherein float is a floating point number;

[0047] The second conversion unit is configured to:

[0048] convert the target numerical value from a floating point number to an integer.

[0049] In this embodiment, the square root reciprocal operation module is further configured to:

[0050] obtain the output result of the second conversion unit and re-input it to the first conversion unit to repeatedly calculate the inverse square root value of the output result until a set number of times is reached to obtain the inverse square root value of the variance.

[0051] In this embodiment, the normalization result is:

[0052]

[0053] The second aspect of the present application provides a self-adaptive layer normalization hardware acceleration method, applied to the self-adaptive layer normalization hardware acceleration system of any one of the first aspect, comprising:

[0054] obtaining an input vector and context information; the context information is determined by the input vector;

[0055] performing parallel calculation according to the input vector, and simultaneously obtaining a mean value and a variance;

[0056] inputting the input vector and the context information into a parameter model to obtain a scaling parameter and an offset parameter; the parameter model is generated by training historical input vectors and historical context information;

[0057] based on the variance, using a fast square root reciprocal algorithm to calculate an inverse root value of the variance;

[0058] calculating a normalization result according to the input vector, the mean value, the scaling parameter, the offset parameter and the inverse root value.

[0059] In this embodiment, the normalization result is:

[0060]

[0061] wherein, x k is an input vector; μ is a mean value, γ(c) is a scaling parameter, β(c) is an offset parameter, is an inverse root value of the variance.

[0062] The present application provides a self-adaptive layer normalization hardware acceleration system and method, the system comprising: a data input module, a parallel calculation module, an adaptive parameter generation module, a square root reciprocal operation module and a normalization calculation output module connected in communication; the data input module is configured to: obtain an input vector and context information; the context information is determined by the input vector; the parallel calculation module is configured to: perform parallel calculation according to the input vector, and simultaneously obtain a mean value and a variance; the adaptive parameter generation module is configured to: input the input vector and the context information into a parameter model to obtain a scaling parameter and an offset parameter; the parameter model is generated by training historical input vectors and historical context information; the square root reciprocal operation module is configured to: based on the variance, using a fast square root reciprocal algorithm to calculate an inverse root value of the variance; the normalization calculation output module is configured to: calculate a normalization result according to the input vector, the mean value, the scaling parameter, the offset parameter and the inverse root value, so as to improve the calculation efficiency of layer normalization in the layer normalization process. BRIEF DESCRIPTION OF DRAWINGS

[0063] In order to make the technical solutions of the present application clearer, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0064] Figure 1 The flow chart of the adaptive layer normalization hardware acceleration system in the present application runs;

[0065] Figure 2 The structural diagram of the reciprocal square root operation module in the present application;

[0066] Figure 3 The structural diagram of the addition tree in the present application.

[0067] Explanation of reference signs:

[0068] 1-data input module; 2-parallel computing module; 21-adder; 3-adaptive parameter generation module; 4-reciprocal square root operation module; 41-first conversion unit; 42-subtractor; 43-Newton approximation unit; 44-second conversion unit; 5-normalization calculation output module. DETAILED DESCRIPTION

[0069] In order to make the technical solutions of the present application clearer, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0070] For example, with the wide application of deep neural networks in natural language processing, image recognition, speech synthesis and other fields, the model structure is becoming more and more complex, and higher requirements are put forward for the convergence speed and generalization ability of the model. As an important means to improve the stability of neural network training, normalization technology is widely introduced into various network architectures. Among them, layer normalization (LayerNorm) is a normalization method commonly used in recurrent neural networks (RNN), transformers (Transformer) and other structures.

[0071] Layer Normalization (LN) is a normalization technique commonly used in deep neural networks, which can accelerate model training and improve convergence stability, especially in structures such as recurrent neural networks and transformers. However, traditional layer normalization hardware implementations generally lack flexibility and lack context awareness. Specifically, the scaling parameter γ and the translation parameter β in the normalization process are often set as static constants, which cannot be dynamically adjusted according to input features or task context, severely restricting their adaptability and generalization ability in complex application scenarios such as multi-task learning and style transfer.

[0072] To address the low efficiency of layer normalization computation, Google team proposed an adaptive layer normalization (AdaLN) mechanism in their T5 (Text-to-Text Transfer Transformer) model in 2019. This method dynamically adjusts the γ and β parameters by introducing conditional inputs (such as task identifiers or style vectors), enabling the normalization process to have context awareness, significantly improving the model's expression ability and convergence efficiency in multi-task processing and style transfer.

[0073] On this basis, the present application proposes a hardware-optimized adaptive layer normalization implementation scheme, which effectively integrates the AdaLN mechanism into the underlying hardware architecture.

[0074] For example, unlike batch normalization, layer normalization performs standardization operations on all feature dimensions within the sample, that is, for the input vector x = [x1, x2, x3, …, xN], the mean and variance are calculated and normalized: n

[0075]

[0076] In terms of hardware implementation, since the calculation of σ 2 requires the known μ value, current layer normalization mainly relies on serial or semi-parallel structures to complete the calculation of mean and variance, with limited overall computational efficiency, making it difficult to meet the high throughput and low power consumption requirements of large-scale models. In addition, in current mainstream layer normalization implementations, the normalization parameters γ and β are usually static weights, lacking the ability to perceive the task or input context, limiting their application performance in multi-task learning, transfer learning, and multi-modal processing.

[0077] For example, the basic operation of basic layer normalization is to normalize each layer of input features. Assuming the input vector is x, its mean is μ, and its standard deviation is σ, then the output y of LN can be represented as:​

[0078]

[0079] To increase the expressive power of the model, trainable scaling parameters γ and offset parameters β are usually added, so the output variable of layer normalization is:

[0080]

[0081] Where γ and β are trainable parameters for scaling and shifting the standardized data. Because it does not depend on the batch dimension (Batch), it has obvious advantages in small batch or sequence modeling scenarios, and is widely used in GPT, BERT and other models.

[0082] To solve the above problems, the application provides a self-adaptive layer normalization hardware acceleration system and method, which will be described below.

[0083] As shown in Figure 1 , it is a flow chart of the self-adaptive layer normalization hardware acceleration system in the application.

[0084] The first aspect of the application provides a self-adaptive layer normalization hardware acceleration system, which comprises:

[0085] The data input module 1, the parallel computing module 2, the adaptive parameter generation module 3, the square root reciprocal operation module 4 and the normalization calculation output module 5 are communicatively connected.

[0086] The data input module 1 is configured to:

[0087] Obtain an input vector and context information; the context information is determined by the input vector; the input vector itself is local and only contains information of the current data point (such as a single word, an image block or user behavior), but lacks understanding of the surrounding environment. The context information refers to external knowledge or environment related to the input vector, which is used to enhance the understanding of the input by the model.

[0088] The parallel computing module 2 is configured to:

[0089] Parallel computing according to the input vector, while obtaining the mean and variance; the mean is:

[0090]

[0091] In the formula, x k is the input vector; n is any positive integer;

[0092] The variance is:

[0093]

[0094] Specifically, in the solving of sigma 2 , mathematical conversion is performed to express it as:

[0095]

[0096] From the above formula, the calculation of sigma is changed to the calculation of sigma and The parallel operation of sigma 2 and mu 2 is realized, which greatly improves the operation speed.

[0097] Exemplarily, the context information or external condition c can be a loss function in the training process or a manually defined training parameter. For example, in an image compression model, the purpose of normalizing the image features is to reduce the sensitivity of the model to the distribution deviation of the input features, enhance the generalization ability, and speed up the model convergence. By setting the external condition c (defined as the compression ratio size here), the retention degree of high-level image information in the model inference can be dynamically adjusted, so that gamma (c) and beta (c) can perceive the change of the model compression ratio, thereby adjusting the perception ability of the model to high-level information, obtaining output features with different compression ratios, and thus realizing the function of dynamically adjusting the number of bits per pixel (bpp).

[0098] The parallel computing module 2 is configured with an addition tree, as shown in Figure 3 ; the addition tree includes a plurality of adders 21;

[0099] The addition tree is configured to:

[0100] According to the input vector, the sum value of the input vector and the sum square value of the input vector are calculated; the sum value of the input vector is:

[0101]

[0102] The sum square value is:

[0103]

[0104] Exemplarily, before calculating the mean and variance, the average value and the sum of squares of the data are calculated first, and then sigma 2 is quickly calculated through simple subtraction. The new algorithm is mathematically elegant and has low computational complexity. Assuming that the number of input vectors is n, and the input vectors are x1, x2, …, x n , the alternative algorithm process is as follows:

[0105] Input: x i withi∈1,2,…,n.

[0106] for(i=1;i≤n;i++)

[0107] sum+=x[i]

[0108] sqr+=x[i]*x[i];

[0109] μ=sum / n;

[0110] σ=sqr / n-μ 2 .

[0111] Output:μ,σ.

[0112] wherein the adder tree is as shown in Figure 3 for fast calculation of and so as to obtain the sum of input vector required for calculation of μ and the sum of square of input vector required for calculation of σ.

[0113] The adaptive parameter generation module 3 is configured to:

[0114] input the input vector and the context information into a parameter model to obtain a scaling parameter and an offset parameter; the parameter model is generated by training historical input vectors and historical context information.

[0115] The adaptive parameter generation module 3 is further configured to:

[0116] obtain a neural network model; the neural network model is configured with a first parameterized function f γ and a second parameterized function f β ; train the neural network model by using historical input vectors and historical context information to obtain a parameter model; input the input vector and the context information into the parameter model to obtain the scaling parameter and the offset parameter.

[0117] The scaling parameter is:

[0118] γ(c)=f γ (c);

[0119] wherein c is the context information; f γ is the first parameterized function.

[0120] The offset parameter is:

[0121] β(c)=f β (c);

[0122] wherein f β is the second parameterized function.

[0123] Specifically, adaptive normalization (adaLN) adjusts the parameters of γ and β by introducing an adaptive mechanism. Specifically, adaLN adaptively adjusts these parameters according to certain characteristics of the input. Assuming that there is a context information or external condition c, the scaling parameter and the offset parameter of adaLN can be represented as γ(c) and β(c):

[0124]

[0125] where γ(c) and β(c) are calculated according to the context information or external condition c. This can be achieved by a small neural network (usually a fully connected layer): γ(c) = f γ (c), β(c) = f β (c), f γ and f β are parameterized functions that can be learned.

[0126] The square root reciprocal operation module 4 is configured to:

[0127] Based on the variance, the reciprocal value of the square root of the variance is calculated using a fast square root reciprocal algorithm.

[0128] Exemplarily, in previous research and practical applications, the calculation of the square root reciprocal has a high frequency, and therefore an efficient algorithm is urgently needed to improve the operation performance. Since multiplication and addition are highly optimized in computer architecture, for example, linear transformations such as γ(c) · (x-μ) + β(c) can be calculated quickly. However, the reciprocal of the standard deviation involves square root and division operations, both of which are relatively slow in hardware execution, especially the division operation overhead. Therefore, the current calculation of the square root reciprocal often becomes a key factor affecting the overall performance. The following is the regular code:

[0129] float y = 1 / sqrt(x);

[0130] It is worth noting that in the actual implementation of the expression the square root function is not directly called in the code. Instead, the process is completely composed of basic operations such as multiplication, bit shift, etc., which are highly optimized in hardware, and therefore has extremely high execution efficiency. In fact, the calculation of the square root reciprocal is not an exact solution, but an approximate algorithm whose error is controlled within about 1%, but compared with the conventional exact calculation method, the operation speed is improved by more than three times. This approximate method significantly improves the overall performance of the normalization operation under the premise of acceptable accuracy.

[0131] Its square root reciprocal algorithm is as follows:

[0132] Input: x, y, number, const float threehalfs = 1.5F

[0133] x = number * 0.5F

[0134] y = number

[0135] i = *(long*)&y

[0136] i = 0x5f3759df - (i >> 1)

[0137] y = *(float*)&i

[0138] y = y * (threehalfs - (x * y * y)) / / 1st iteration

[0139] y = y * (threehalfs - (x * y * y)) / / 2nd iteration

[0140] return y

[0141] wherein the magic constant 0x5f3759df is a "magic number" which is an integer number obtained through experience and mathematical derivation, and which utilizes the exponent bit characteristics of a floating point number to quickly generate an initial approximation value of 1 / √number by subtracting the initial guess value. The value of the magic constant is optimized according to a large number of experiments. In the algorithm, y is used to store the approximate result and will undergo an initial guess and iterative optimization; x is an intermediate variable, and its initial value is number*0.5F (i.e., number / 2), 0.5F being 0.5 of the float type, which is used as a coefficient in the iterative calculation.

[0142] At the hardware implementation level, the architecture of the reciprocal square root operation module 4 is as shown in Figure 2 .

[0143] The reciprocal square root operation module 4 comprises:

[0144] a first conversion unit 41, a subtracter 42, a Newton approximation unit 43, and a second conversion unit 44.

[0145] The first conversion unit 41 is configured to:

[0146] convert the variance from an integer to a floating point number.

[0147] The subtracter 42 is configured to:

[0148] subtract the floating point number from a magic constant to obtain a first value; the magic constant is obtained by utilizing the exponent bit characteristics of a floating point number.

[0149] The Newton approximation unit 43 is configured to:

[0150] According to the first value and the second value, perform a Newton approximation operation to obtain a target value; the second value is float ÷ 2;

[0151] In the formula, float is a floating-point number;

[0152] The second conversion unit 44 is configured to:

[0153] Convert the target value from a floating-point number to an integer.

[0154] The input data (i.e., the variance σ 2 ) is first converted from an integer to a floating-point number by the first conversion unit 41, then the floating-point number is divided by two and the magic number is subtracted, then the Newton approximation operation is performed by the Newton approximation unit 43, and finally the integer is converted back by the second conversion unit 44, i.e., the approximate result after taking the reciprocal of the square root is obtained.

[0155] The reciprocal of square root operation module 4 is further configured to:

[0156] The output result of the second conversion unit 44 is obtained and re-input to the first conversion unit 41, and the reciprocal of the square root value of the output result is repeatedly calculated until a set number of times is reached, to obtain the reciprocal of the square root value of the variance. If a more accurate approximation value is required, the reciprocal of the square root operation module 4 can perform two or more Newton iteration operations to improve the accuracy of the reciprocal of the square root value of the variance.

[0157] The normalization calculation output module 5 is configured to:

[0158] According to the input vector, the mean value, the scaling parameter, the offset parameter, and the reciprocal of the square root value, calculate the normalization result for use by the subsequent layers of the deep neural network. The normalization result is:

[0159]

[0160] The application provides a self-adaptive layer normalization hardware acceleration system. In existing algorithm design, model accuracy and algorithm complexity are often given priority, while parallelism and delay bottleneck in hardware implementation are ignored. In order to better adapt to high parallelism and low delay hardware architecture, the original algorithm structure is systematically transformed to reduce serial dependence and improve module concurrent execution capability, thereby significantly optimizing overall inference efficiency and resource occupation while maintaining accuracy. The application introduces a structure-level optimization method at the hardware level, which simultaneously completes the calculation of mean and variance through parallelization, greatly improving the throughput and real-time responsiveness of the normalization module when processing large-scale input data, and significantly reducing system delay. Compared with the serial calculation structure, the self-adaptive layer normalization hardware acceleration system provided by the application significantly improves performance while maintaining accuracy. The application realizes a context-aware adaptive normalization mechanism at the hardware level, which not only improves the calculation efficiency and adaptability of the system, but also provides a new technical path and reference paradigm for subsequent deep learning accelerator design. Integrating the self-adaptive layer normalization hardware acceleration system into existing GPUs, FPGAs or other dedicated chips significantly improves the computing efficiency of the chips.

[0161] The application provides a self-adaptive layer normalization hardware acceleration system with the following beneficial ways:

[0162] 1. The mean and variance calculation in the current layer normalization has serial dependence, and the calculation efficiency is limited. The application transforms the layer normalization variance calculation formula, converts the variance calculation into the operation form of and , realizes the parallelization of mean and variance calculation, and greatly improves the operation speed. Through the parallel calculation optimization of mean and variance, the serial dependence in the calculation process is reduced, and the calculation time is greatly shortened; the efficient square root reciprocal fast algorithm further improves the overall calculation efficiency, meeting the demand of large-scale model high throughput and low power consumption.

[0163] 2. The method of calculating the reciprocal of the standard deviation is low in efficiency and consumes a long time; the application adopts the square root reciprocal fast algorithm of the specific numerical calculation strategy, which improves the operation speed to three times under the condition of ensuring about 1% error, effectively improving the overall calculation efficiency.

[0164] Exemplarily, compared with the current layer normalization method, the advantages of the application are shown in the following table:

[0165] Figure 1 : Dimensional analysis table between the current layer normalization method and the layer normalization method of the application

[0166]

[0167] The second aspect of the application provides a self-adaptive layer normalization hardware acceleration method, applied to the self-adaptive layer normalization hardware acceleration system described in any of the above embodiments, comprising:

[0168] obtaining an input vector and context information; the context information is determined by the input vector;

[0169] performing parallel calculation according to the input vector, and obtaining a mean value and a variance value at the same time;

[0170] inputting the input vector and the context information into a parameter model to obtain a scaling parameter and an offset parameter; the parameter model is generated by training historical input vectors and historical context information;

[0171] based on the variance value, using a fast square root reciprocal algorithm to calculate the reciprocal square root value of the variance value;

[0172] calculating a normalization result according to the input vector, the mean value, the scaling parameter, the offset parameter and the reciprocal square root value.

[0173] the normalization result is:

[0174]

[0175] wherein, x k is the input vector; μ is the mean value, γ(c) is the scaling parameter, β(c) is the offset parameter, is the reciprocal square root value of the variance value.

[0176] It should be noted that the effects of the above method embodiments can refer to the effects of the above system embodiments, which will not be repeated here.

[0177] The above detailed description of the specific embodiments of the application has further described the purposes, technical solutions and beneficial effects of the embodiments of the application. It should be understood that the above is only a specific embodiment of the application, and is not used to limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solutions of the embodiments of the application shall be included in the protection scope of the embodiments of the application.

Claims

1. An adaptive layer normalization hardware acceleration system, characterized in that, include: The communication connection includes a data input module (1), a parallel computing module (2), an adaptive parameter generation module (3), a square root reciprocal operation module (4), and a normalized calculation output module (5). The data input module (1) is configured as follows: Obtain the input vector and context information; the context information is determined by the input vector. The parallel computing module (2) is configured as follows: Parallel computation is performed based on the input vector to obtain both the mean and variance. The adaptive parameter generation module (3) is configured as follows: The input vector and the context information are input into the parametric model to obtain the scaling parameter and the offset parameter; the parametric model is generated by training with historical input vectors and historical context information. The square root reciprocal calculation module (4) is configured as follows: Based on the variance, the square root inverse value of the variance is calculated using the fast square root reciprocal algorithm; The normalization calculation output module (5) is configured as follows: The normalization result is calculated based on the input vector, mean, scaling parameter, offset parameter, and square root inverse value.

2. The adaptive layer normalization hardware acceleration system according to claim 1, characterized in that, The parallel computing module (2) is configured with an addition tree; the addition tree includes: a plurality of adders (21); The addition tree is configured as follows: Based on the input vectors, calculate the sum of the input vectors and the square of the sum of the input vectors; the sum of the input vectors is: The sum of squares is:

3. The adaptive layer normalization hardware acceleration system according to claim 1, characterized in that, The mean is: In the formula, x k Let n be the input vector; n is any positive integer. The variance is:

4. The adaptive layer normalization hardware acceleration system according to claim 1, characterized in that, The adaptive parameter generation module (3) is further configured to: Obtain a neural network model; the neural network model is configured with a first parameterization function and a second parameterization function; The neural network model is trained using historical input vectors and historical context information to obtain a parametric model; The input vector and the context information are input into the parameter model to obtain the scaling parameter and the offset parameter.

5. The adaptive layer normalization hardware acceleration system according to claim 4, characterized in that, The scaling parameter is: γ(c)=f γ (c); In the formula, c represents context information; f γ This is the first parameterized function; The offset parameter is: β(c)=f β (c); In the formula, f β This is the second parameterized function.

6. The adaptive layer normalization hardware acceleration system according to claim 1, characterized in that, The square root reciprocal calculation module (4) includes: First conversion unit (41), subtractor (42), Newton approximation unit (43), second conversion unit (44); The first conversion unit (41) is configured as follows: Convert the variance from an integer to a floating-point number; The subtractor (42) is configured as follows: Subtracting the floating-point number from the magic constant yields the first value; the magic constant is obtained using the exponent characteristics of the floating-point number. The Newton approximation unit (43) is configured as follows: Based on the first and second values, Newton's approximation is performed to obtain the target value; the second value is: float÷2; In the formula, float represents a floating-point number; The second conversion unit (44) is configured as follows: Convert the target value from a floating-point number to an integer.

7. The adaptive layer normalization hardware acceleration system according to claim 6, characterized in that, The square root reciprocal calculation module (4) is further configured as follows: The output of the second conversion unit (44) is obtained and re-inputted into the first conversion unit (41). The square root of the output is repeatedly calculated until a set number of times is reached to obtain the square root of the variance.

8. The adaptive layer normalization hardware acceleration system according to claim 1, characterized in that, The normalization result is as follows:

9. An adaptive layer normalization hardware acceleration method, applied to an adaptive layer normalization hardware acceleration system according to any one of claims 1 to 8, characterized in that, include: Obtain the input vector and context information; The context information is determined by the input vector; Parallel computation is performed based on the input vector to obtain both the mean and variance. The input vector and the context information are input into the parametric model to obtain the scaling parameter and the offset parameter; the parametric model is generated by training with historical input vectors and historical context information. Based on the variance, the square root inverse value of the variance is calculated using the fast square root reciprocal algorithm; The normalization result is calculated based on the input vector, mean, scaling parameter, offset parameter, and square root inverse value.

10. The adaptive layer normalization hardware acceleration method according to claim 9, characterized in that, The normalization result is as follows: In the formula, x k The input vector is μ, γ(c) is the scaling parameter, and β(c) is the offset parameter. It is the square root of the variance, inverted.

Images