Hardware acceleration circuit, data processing acceleration method, chip and accelerator

Abstract

The application relates to a hardware acceleration circuit, a data processing acceleration method, a chip and an accelerator. The circuit comprises a natural logarithm module, which is used for obtaining a natural logarithm value of an i-th data element in n data elements of a data set and a natural logarithm value of a mean square deviation of the n data elements, wherein n is greater than 1; an addition and subtraction module, which is used for obtaining a subtraction operation result between the natural logarithm value of the i-th data element and the natural logarithm value of the mean square deviation; an exponential function module, which is used for obtaining an exponential function value of the subtraction operation result; and a multiplication module, which is used for obtaining a multiplication operation result between the exponential function value and a mask tensor of the i-th data element, so as to obtain a specific function value corresponding to the i-th data element. The scheme provided in the application embodiment is beneficial to improving the accuracy of the obtained standardized function value.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a hardware acceleration circuit, a data processing acceleration method, a chip, and an accelerator. Background Technology

[0002] Since its inception, artificial intelligence has seen its theories and technologies mature, and its application areas have continuously expanded. Taking deep learning based on neural networks as an example, it has high requirements for both the quantity and quality of training data. To improve the training effect of neural networks, factors detrimental to training can be eliminated from the training data. For example, data normalization methods can be used to process the raw data.

[0003] Data standardization is widely used in deep learning and related technologies. In these technologies, the function value of the standardization function can be calculated using general-purpose computing units such as central processing units (CPUs) or graphics processing units (GPUs). However, when the processing of neural networks is performed by hardware circuits such as deep learning accelerators (DLAs) or neural network processing units (NPUs), if the standardization function layer (such as a normalization layer) is located in the middle layer of the neural network, it will lead to job migration overhead between the DLA / NPU and the CPU / GPU. This makes the scheme of using the CPU / GPU to determine the standardization function value inefficient, resulting in increased system bandwidth and higher power consumption. Summary of the Invention

[0004] To address or partially address the problems existing in related technologies, this application provides a hardware acceleration circuit, a data processing acceleration method, a chip, and an accelerator, which are beneficial for improving the speed of obtaining layer normalization function values ​​while meeting the accuracy requirements of the layer normalization function.

[0005] The first aspect of this application provides a hardware acceleration circuit, comprising: The Natural Logarithm module is used to obtain the natural logarithm of the i-th data element and the natural logarithm of the mean square error of the n data elements in a dataset, where n is greater than 1. The addition / subtraction module is used to obtain the result of the subtraction operation between the natural logarithm of the i-th data element and the natural logarithm of the mean square error; The exponential function module is used to obtain the exponential function value of the subtraction result; The multiplication module is used to obtain the result of the multiplication operation between the exponential function value and the mask tensor of the i-th data element, so as to obtain the specific function value corresponding to the i-th data element.

[0006] A second aspect of this application provides an artificial intelligence chip, which includes the hardware acceleration circuit described above.

[0007] A third aspect of this application provides a data processing acceleration method, the method comprising: To obtain the natural logarithm of the i-th data element and the natural logarithm of the mean square error of the n data elements in a dataset, where n is greater than 1; Obtain the result of the subtraction operation between the natural logarithm of the i-th data element and the natural logarithm of the mean square error; To obtain the exponential function value of the subtraction result; Obtain the result of the multiplication operation between the exponential function value and the mask tensor of the i-th data element to obtain the specific function value corresponding to the i-th data element.

[0008] A fourth aspect of this application provides an artificial intelligence accelerator, comprising: Processor; and The memory stores executable code, which, when executed by the processor, causes the processor to perform the methods described above.

[0009] The fifth aspect of this application provides a computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method described above.

[0010] In some embodiments of this application, the standardization function is deformed so that the variance values ​​corresponding to at least some elements in the input dataset are transformed from the denominator to the numerator. This reduces the accuracy loss that may occur when the reciprocal of the variance value approaches zero, especially when the additive values ​​involved in the variance calculation are large. This helps to improve the accuracy of the obtained layer standardization function values ​​while ensuring processing speed.

[0011] Furthermore, obtaining the natural logarithmic and exponential function values ​​via table lookup avoids complex exponential, natural logarithmic, and reciprocal operations, thus improving data processing speed during layer normalization function calculation and obtaining layer normalization function values ​​more quickly. On the other hand, it reduces the excessive hardware circuit area and cost incurred in implementing exponential, natural logarithmic, and reciprocal operations.

[0012] Furthermore, the precision of Layer Norm functions is increased for 8-bit DLA or other hardware architectures in related technologies.

[0013] Furthermore, based on the above transformation of the normalized function, when using a lookup table (LUT) to obtain at least one of the exponential function value, natural logarithmic value, etc., a lookup table with fewer entries can be used, which is more acceptable for hardware implementation.

[0014] Furthermore, LUT combinational logic circuits are too expensive for integers (INT) 16 (2^16 = 65536 entries), and the performance is unacceptable if time-sharing based circuits are used because it requires 65536 cycles to complete a single LUT result. The embodiments of this application, by using INT 8 or INT 10 after data conversion, can significantly reduce the number of lookup cycles to obtain the lookup result of a single LUT, thus improving processing speed.

[0015] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0016] The above and other objects, features and advantages of this application will become more apparent from the more detailed description of exemplary embodiments thereof in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments thereof.

[0017] Figure 1 This is a schematic diagram illustrating layer standardization in one embodiment of this application; Figure 2 This is a schematic diagram of the structure of a neural network shown in one embodiment of this application; Figure 3 This is a schematic diagram of the structure of a neural network for classification shown in one embodiment of this application; Figure 4 This is a structural block diagram of a hardware acceleration circuit according to an embodiment of this application; Figure 5 This is a structural block diagram of a hardware acceleration circuit according to another embodiment of this application; Figure 6 This is a structural block diagram of a basic lookup table circuit unit according to an embodiment of this application; Figures 7 to 9 This is a structural block diagram of the hardware acceleration circuit of some other embodiments of this application; Figure 10 This is a schematic flowchart of a data processing acceleration method according to an embodiment of this application; Figure 11 This is a schematic flowchart of a data processing acceleration method according to another embodiment of this application; Figure 12 This is a structural block diagram of an artificial intelligence accelerator according to an embodiment of this application. Detailed Implementation

[0018] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0019] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a” and “the” as used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0020] It should be understood that although the terms "first," "second," "third," etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0021] In the process of using or training artificial intelligence models, data standardization can be performed. Data standardization not only refers to standardization before data is input into the model, but also to standardization of the output data of each layer in the model after input. For example, tools can be used to observe whether the output results and weight distribution of each layer in the network model are within acceptable ranges. Generally, after standardization, the output of each layer is fixed within a certain range.

[0022] For standardization methods of bounded data, since the data boundaries are fixed, the scaling is obviously related to the number of data points and the minimum and maximum values. The original data can be scaled proportionally within a certain range by performing calculations on the data's size, minimum, and maximum values. For example, processing image data from [0, 255] to [-1, 1] satisfies the standard of a mean of 0 and a standard deviation of 1; this is a fixed scaling of bounded data.

[0023] When training a neural network using gradient descent, the feature distribution of the training data changes continuously as the network depth increases. To ensure the stability of the data feature distribution, standardization can be implemented, resulting in a larger learning rate and faster convergence speed. Standardization also helps prevent overfitting, making the training process smoother.

[0024] By standardizing the data, at least some of the effects shown below can be obtained.

[0025] On the one hand, data standardization can unify the units of measurement, smoothing gradients between different layers or batches of data and preventing gradient explosion or vanishing. After data standardization, because the original data values ​​are scaled to a smaller range, the "gradient gap" between the original data is also reduced overall, smoothing the gradient and preventing large oscillations in a particular layer of the model. Excessive data oscillations will lead to greater changes in the derivatives of the various parameters to be trained, and for differentiation, smoother is better. This way, when the model backpropagates and calculates derivatives, the model loss will tend to be stable, making it less prone to gradient explosion or vanishing problems.

[0026] On the other hand, eliminating the negative impact of outliers on model training accelerates convergence. Outliers are data with large variance in the original data structure, which is represented on the number line as being far away from other sample data. In gradient descent, a method primarily used in deep learning, the loss error between the model's output and label is calculated as an average error. Therefore, when backpropagating the gradient using gradient descent, the parameters are updated using the mean error.

[0027] On the other hand, eliminating the negative impact of noisy data on the model and preventing overfitting is crucial. Data standardization can reduce the influence of noisy data on the model output. Standardization removes dimensional interference, and wavelet denoising removes random noise. Noise often has a large gradient relative to its surrounding data; data standardization reduces this gradient, thus eliminating noise.

[0028] For example, data can be standardized in the following ways: mean-variance standardization, maximum standardization, absolute maximum standardization, maximum-minimum standardization, norm standardization, quartile standardization, etc. Among them, mean-variance standardization can scale data to a standard normal distribution N[0,1] with a mean of 0 and a standard deviation of 1, and is a widely used standardization method.

[0029] Depending on the object being standardized, standardization can be categorized into: Layer Normalization (LN), Instance Normalization (IN), Group Normalization (GN), and Batch Normalization (BN).

[0030] Figure 1 This is a schematic diagram illustrating layer standardization in one embodiment of this application.

[0031] See Figure 1 Let's take Layer Normalization (LN) as an example. Layer normalization involves standardizing each channel (C) and spatial dimension (H, W) individually, unaffected by batch size, and can be applied to recurrent neural networks (RNNs). For instance, LN function values ​​can be obtained through job migration between DLA / NPU and CPU / GPU, implemented in PyTorch by calling `torch.nn.LayerNorm(out_channels,H,W)`. However, CPU / GPU efficiency is low, and job migration leads to performance degradation, increased system bandwidth, and higher power consumption.

[0032] Hardware solutions for Layer Norms in Deep Learning Accelerators (DLAs) or Neural Processing Units (NPUs) are not common. INT 8 is the most widely used precision in DLAs / NPUs, but due to its wide dynamic range in the intermediate values, it is difficult to provide an 8-bit hardware solution for Layer Norms. Specifically, layer normalization normalizes all neurons in a given layer. The layer normalization functions are shown in equations (1) to (3).

[0033] Equation (1) Equation (2) Equation (3) in, This is data that needs to be standardized. The data has been standardized; var represents the variance. It is the mean squared error. It should be noted that the value in equation (1) can be... Replace with ε is used to improve numerical stability to prevent the denominator from accidentally becoming zero.

[0034] As can be seen from equations (1) to (3), due to the complex square root and reciprocal (1 / square root) in the Layer Norm function, a dedicated hardware pipeline for Layer Norm is not feasible for large-scale computational applications (due to area cost). Layer Norm is a normalization function applied to the C, H, and W dimensions to obtain zero mean and unit variance, which is beneficial for training time and network performance. During inference, the circuit should dynamically calculate the mean / variance of the feature map to complete Layer Norm. The data standardization process may involve calculations of complex square roots and reciprocals (1 / square root).

[0035] Considering that dedicated hardware pipelines for Layer Norm functions are impractical for large-scale computing power implementations—for example, increased computing power leads to exorbitant hardware costs—one approach to obtaining Layer Norm functions employs a lookup table (LUT) for searching 16-bit integers (INT16). However, a 16-bit LUT is a large table requiring significant storage space; for example, a table containing 2^16 (65536) entries would require substantial static random access memory (SRAM) / dynamic random access memory (DRAM) to store the data, resulting in excessively high costs for the LUT combinational logic circuitry. Furthermore, the processing time for a single lookup is excessively long when using a 16-bit LUT. Using time-sharing based circuitry would be unacceptable in terms of performance, as it would require 65536 cycles to complete a single LUT result.

[0036] This application provides a hardware acceleration circuit, chip, data processing acceleration method, and accelerator that transforms the layer normalization function so that at least some of the logical components in the transformed layer normalization function can be implemented using multiple lookup tables that occupy less storage space. This can balance the power consumption, bandwidth, performance, and accuracy of determining the function value of the layer normalization function, thus meeting the needs of neural networks.

[0037] For example, in an INT8 DLA / NPU, the multiply-accumulate (MAC) operation typically multiplies INT8 format data by INT8 format data, with the accumulation part being a typical 32-bit convolution or matrix multiplication calculation. Directly calculating the reciprocal of the integer value would result in significant computational loss. Therefore, this application proposes a method for hardware acceleration of Layer Norm functions. By transforming Layer Norm functions and representing them using natural logarithms, the complex calculations of square roots and reciprocals (1 / square root) are reduced, effectively lowering hardware costs and reducing the number of lookup table entries from 65535 to 256 or 1024.

[0038] Figure 2 This is a schematic diagram of the structure of a neural network shown in one embodiment of this application.

[0039] See Figure 2 The diagram illustrates the topology of a neural network 100, including an input layer, hidden layers, and an output layer. This neural network 100 is capable of performing calculations or operations based on data elements I1 and I2 received from the input layer, and generating output data O1 and O2 based on the results of the calculations.

[0040] For example, neural network 100 can be a deep neural network (DNN) that includes one or more hidden layers. Figure 2 The neural network 100 in the diagram includes an input layer L1, two hidden layers L2 and L3, and an output layer L4. DNNs include, but are not limited to, Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs).

[0041] It should be noted that, Figure 2 The four layers shown are for illustrative purposes only and should not be construed as limiting the scope of this application. For example, a neural network may include more or fewer hidden layers. Figure 2 Each of the layers shown can be preceded by a normalization layer to perform normalization processing on the input data for that layer.

[0042] Nodes in different layers of the neural network 100 can be connected to each other for data transmission. For example, a node can receive data from other nodes to perform calculations on the received data and output the calculation results to nodes in other layers.

[0043] Each node can determine its output data based on the output data and weights received from nodes in previous layers. For example, Figure 2 middle This represents the weight between the first node of the first layer and the first node of the second layer. This represents the output data of the first node in the first layer. Let represent the bias value of the first node in the second layer. Then, the output data of the first node in the second layer can be represented as: The calculation method for the output data of other nodes is similar and will not be described in detail here.

[0044] In some embodiments, the neural network is configured with a normalization layer, such as an LN layer, which can normalize the input data of the hidden layers, etc., corresponding to the LN layer.

[0045] In some embodiments, the neural network is configured with activation function layers, such as soft max function layers, which can convert the result values ​​for each class into probability values.

[0046] In some embodiments, the neural network is configured with a loss function layer after the flexible maximum function layer, which is capable of calculating the loss as an objective function for training or learning.

[0047] Understandably, a neural network can respond to the data to be processed, and after processing the data, obtain a recognition result; the data to be processed may include at least one of voice data, text data, and image data.

[0048] A typical type of neural network is a classification neural network. Classification neural networks determine the category to which a data element belongs by calculating the probability of that element corresponding to each class.

[0049] Figure 3 This is a schematic diagram of the structure of a neural network for classification, as shown in one embodiment of this application.

[0050] See Figure 3 The neural network 200 for classification in this embodiment may include a hidden layer 210, a fully connected layer (FC layer) 220, a flexible maximum function layer 230, and a loss function layer 240. At least some of the layers, such as the hidden layer 210 and the fully connected layer 220, may each have a normalization layer preceding them.

[0051] like Figure 3As shown, in response to the data to be classified, the neural network 200 normalizes the data using a normalization layer, and then performs calculations sequentially according to the order of hidden layer 210 and FC layer 220. The FC layer 220 outputs a calculation result s, which corresponds to the classification probability of the data element. The FC layer 220 may include multiple nodes corresponding to multiple classes, and each node outputs a result value corresponding to the probability that the data element is classified into the corresponding class. For example, see also... Figure 2 FC layer 220 corresponds to Figure 2 The output layer L4 has two nodes, corresponding to two categories (Class 1 and Class 2). The output value of one node can be a result value representing the probability that a data element is classified into Class 1, and the output value of the other node can be a result value representing the probability that a data element is classified into Class 2. The FC layer 220 outputs the calculation result s to the flexible maximal function layer 230, which converts the calculation result s into a probability value y and can also perform standardization on the probability value y.

[0052] The flexible maximum function layer 230 outputs the probability value y to the loss function layer 240, which can calculate the cross-entropy loss L of the result s based on the probability value y.

[0053] During the backpropagation learning process, the flexible maximum function layer 230 calculates the gradient of the cross-entropy loss L. Then, the FC layer 220 performs gradient learning based on the cross-entropy loss L. For example, the weights of the FC layer 220 can be updated according to the gradient descent algorithm. Further, subsequent learning processing can be performed in the hidden layer 210.

[0054] The neural network 200 can be implemented in software, in hardware circuits, or a combination of both. For example, in the case of hardware circuit implementation, the normalization layer, hidden layer 210, fully connected (FC) layer 220, flexible maximum function layer 230, and loss function layer 240 are all implemented in hardware circuits, which can be integrated into a single AI chip or distributed across multiple chips. This configuration avoids the data migration between other layers of the neural network and processors such as CPUs / GPUs when the normalization layer is implemented in a CPU / GPU, thereby improving the efficiency of neural network data processing, reducing data processing latency and power consumption, and avoiding increased bandwidth usage.

[0055] The technical solutions of the embodiments of this application are described in detail below with reference to the accompanying drawings.

[0056] Figure 4This is a structural block diagram of a hardware acceleration circuit according to an embodiment of this application. In this application, the hardware acceleration circuit can be used, for example, but not limited to, to implement the normalization layer in the neural network 200 described above. The hardware acceleration circuit can be, for example, but not limited to, a circuit component in a CPLD (Complex Programming logic device) chip, an FPGA (Field Programmable Gate Array) chip, a dedicated chip, etc.

[0057] To facilitate understanding of this application, the layer normalization function is explained below. Assuming there is an array X, the formula for calculating the Layer Norm function value of the i-th element xi can be shown in equations (1) to (3). Since the relationships shown in equations (4) to (5) exist: Equation (4) Equation (5) Equation (6) can be obtained based on equations (1), (4), and (5). Equation (6) In the formula, y(x)i represents the Layer Norm function value of the i-th element xi, e is the natural constant, ln represents the logarithm (natural logarithm) with the natural number e as the base, xi represents the i-th element of array X, var represents the variance of the i-th element, and maski represents the mask tensor used to characterize the sign of the i-th element xi.

[0058] See Figure 4 The hardware acceleration circuit in this embodiment includes a natural logarithm module 310, an addition and subtraction module 320, and an exponential function module 330.

[0059] The natural logarithm module 310 is used to obtain the natural logarithm of the i-th data element xi among n data elements in a dataset and the natural logarithm of the mean square error of the n data elements. The natural logarithm can be calculated by dedicated hardware circuitry or determined by looking up a table or other methods.

[0060] The addition / subtraction module 320 is used to obtain the result of subtraction between the natural logarithm of the i-th data element xi and the natural logarithm of the mean square error. It is worth noting that in this application, the addition / subtraction module 320 can have three cases: including an adder but not a subtractor, including a subtractor but not an adder, and including both an adder and a subtractor.

[0061] The exponential function module 330 is used to obtain the exponential function value of the subtraction result.

[0062] The multiplication module 340 is used to obtain the result of the multiplication operation between the exponential function value and the mask tensor of the i-th data element, so as to obtain the specific function value corresponding to the i-th data element xi.

[0063] The mask tensor is used to represent the sign of the i-th data element xi, such as a positive or negative sign. For example, if the i-th data element is -0.5, the mask tensor is used to represent "-"; if the i-th data element is 0.5, the mask tensor is used to represent "+". The mask tensor maski of the i-th data element xi can be shown in equation (7).

[0064] Equation (7) In this embodiment, the specific function can be a layer normalization function, which can be expressed using an exponential function and the natural logarithm. Specific functions include, but are not limited to, Layer Norm functions, instance normalization functions, and group normalization functions.

[0065] In some embodiments, data element xi may be processed by an acceleration circuit to process initial data. For example, it can be obtained after processing according to equation (2) above. For example, the initial data is the input data of the neural network or the output data of an intermediate layer of the neural network. In addition, the data element xi can also be data obtained from a hardware acceleration circuit such as a CPU.

[0066] Data element xi is the initial data processed by the acceleration circuit. The following example illustrates the process after data processing. Specifically, the addition / subtraction module is also used to output the i-th initial data point out of the n initial data points in the initial dataset B. The result of the subtraction operation with the average of the n initial data is used to obtain the data set A containing the n data elements xi.

[0067] In some embodiments, the variances of n data elements are first obtained, and then the natural logarithm of the mean squared error of the n data elements is obtained based on the variances. The variance of the i-th data element can be calculated by the acceleration circuit. Alternatively, the variance of the i-th data element can be obtained externally by the acceleration circuit, such as by the CPU.

[0068] For example, a dataset A contains n data elements, where 0 ≤ i ≤ n-1. The variance of the i-th data element can be calculated by the acceleration circuit as shown below.

[0069] Specifically, the multiplication module is also used to obtain the result of the multiplication operation between the i-th data element and the i-th data element. That is, the result of squaring the i-th data element.

[0070] The addition and subtraction module is also used to perform addition operations on the squares of n data elements. .

[0071] The aforementioned hardware acceleration circuit also includes a shift module, used to perform a right shift operation on the sum of the squares of n data elements to obtain the variance var of the n data elements (which can be expressed by the formula). (represented by the variance var), the mean squared error of the i-th data element is obtained. Here, n is determined based on the correction data and is set by the software. mul is the scaling factor. For example, the variance var is calculated by accumulating the squares of each data element xi, and then saturated to INT 8 using the scaling factor mul and the right shift bit rshift, thus obtaining the mean square error of an 8-bit integer type. .

[0072] In some embodiments, the mean of n initial data points can be obtained by the addition / subtraction module 320. Alternatively, the mean of n initial data points can be data obtained from a hardware acceleration circuit, such as a CPU.

[0073] The example is taken where the mean of n initial data points can be obtained by the addition / subtraction module 320.

[0074] The addition / subtraction module 320 is also used for n initial data. The algorithm performs an accumulation operation to obtain the result of adding n initial data points. The shift module also performs a right shift operation on the result of adding n initial data points to obtain the mean of those n initial data points. For example, this can be achieved by accumulating all... Then, by saturating to int8 using a scaling factor mul and a right shift bit rishift, the average value of an 8-bit integer type is obtained, which can be expressed by the formula: .

[0075] Understandably, the scaling factor mul and the number of right shifts rishifts for the two shift operations described above can be different depending on the actual needs. Understandably, the two shift operations can be implemented using the same shift circuit, or they can be implemented using two independently configured shift circuits.

[0076] In some embodiments, the hardware acceleration circuit can also accelerate the initial data. Format conversion can be performed to reduce the bandwidth required for data standardization. For example, the initial data can be reduced. The number of bits used can be used to reduce the bandwidth required.

[0077] Specifically, the hardware acceleration circuit also includes a first conversion circuit, which is used to obtain n initial data points from the initial dataset. , initial data of n The data is converted from the first bit width (N2 bits) to the second bit width (N0 bits) and then output to the addition / subtraction module. For example, the initial data can be converted from an N2-bit (e.g., 32-bit) floating-point number to an 8-bit integer, and this 8-bit integer can be used as the i-th initial data. .

[0078] The hardware acceleration circuit provided in this embodiment effectively reduces the number of square root functions and reciprocal functions by transforming the normalized function, thus helping to reduce the complexity of the hardware acceleration circuit. Furthermore, eliminating large denominators helps improve the computational accuracy for the normalized function.

[0079] In some embodiments, the natural logarithmic value and the exponential function value in equation (6) can be obtained using a lookup table to improve data processing speed. For example, the values ​​can be obtained separately using a lookup table. , The function value of z, where z is - .

[0080] For example, transforming equation (6) yields equation (7).

[0081] Equation (7) Where LUT0 represents the second lookup table, LUT1 represents the third lookup table, and LUT2 represents the first lookup table.

[0082] Specifically, the exponential function module includes a first lookup table module, which is used to output the exponential function value, such as the function value of ez, based on the subtraction result of the first lookup table.

[0083] Furthermore, the natural logarithm module may include some or all of the second lookup table module and the third lookup table module. The second lookup table module is used to output the natural logarithm value corresponding to the i-th data element based on the second lookup table, such as... The function value. The third lookup table module is used to output the natural logarithmic value of the mean squared error of n data elements based on the variance of n data elements and the third lookup table, such as The function value. It should be noted that the third lookup table establishes var and The mapping relationship between them, rather than and The mapping relationship between them eliminates the need for square root operations, which helps to further reduce the complexity of hardware acceleration circuits.

[0084] In this embodiment, referring to equation (1), the square function values ​​of each data element are... The process of obtaining the reciprocal of the addition result and the subsequent multiplication process are transformed to avoid the precision loss that may be caused by the reciprocal approaching 0 when the addition result is large, which is conducive to improving the accuracy of the obtained layer normalization function value.

[0085] Furthermore, obtaining the values ​​of each square root function, natural logarithm function, and exponential function by looking up tables avoids complex square root, natural logarithm, and reciprocal operations, thus improving the data processing speed in the layer normalization function calculation process and obtaining the layer normalization (Layer Norm) function values ​​more quickly. On the other hand, it avoids the excessive hardware circuit area and high cost incurred in implementing complex square root, natural logarithm, and reciprocal operations.

[0086] Figure 5 This is a structural block diagram of a hardware acceleration circuit according to another embodiment of this application. See also... Figure 5 The hardware acceleration circuit in this embodiment includes: a second lookup table module 410, a third lookup table module 420, a first lookup table module 450, a subtractor 430, a conversion circuit 440, and a multiplication module 340. In this embodiment, the second lookup table module 410, the third lookup table module 420, and the first lookup table module 450 are each implemented by an independent lookup table circuit, and are also referred to as the second lookup table circuit, the third lookup table circuit, and the first lookup table circuit. It is understood that in other embodiments of this application, some or all of the lookup table modules may also be implemented by software modules.

[0087] The second lookup table circuit is used to respond to the index value of the i-th data element in the data set, and outputs the natural logarithmic value corresponding to the i-th data element based on the second lookup table. The index value of the data element is data with a bit width of N0 bits.

[0088] In one embodiment, the index values ​​of multiple i-th data elements are sequentially input into the second lookup table module 410. The second lookup table circuit sequentially outputs the natural logarithmic values ​​corresponding to each data element in the second lookup table. Each natural logarithmic value in the second lookup table is data with a bit width of N1 bits.

[0089] Understandably, the index value of the data can be the data itself, or it can be obtained by transforming the data, such as a portion of the data extracted from it.

[0090] The second lookup table can be used to implement the mapping relationship between the index value of a data element and its natural logarithmic value. This second lookup table allows the natural logarithmic value of a data element to be determined through a pre-defined mapping relationship without the need for complex function calculations.

[0091] Similarly, the third lookup table module 420 is used to output the natural logarithm of the mean squared error of the n data elements in response to the index value of the variance of the n data elements. The index value of the variance of the data elements is data with a bit width of N3 bits.

[0092] The index values ​​of the variances of n data elements are input to the third lookup table module 420. The third lookup table circuit outputs the natural logarithm of the mean squared variances of the n data elements in the third lookup table. Each natural logarithm in the third lookup table is a data with a width of N4 bits.

[0093] Understandably, the index value of the data can be the data itself, or it can be obtained by transforming the data, such as a portion of the data extracted from it.

[0094] A third lookup table can be used to establish a mapping between the index value of variance (the square of the mean square deviation) and the natural logarithm of the mean square deviation. This third lookup table allows the natural logarithm of the mean square deviation to be determined through a pre-defined mapping without requiring complex function calculations.

[0095] Subtractor 430 is used to output the result of the subtraction operation between the output of the second lookup table and the output of the third lookup table.

[0096] In one embodiment, the subtractor 430 takes a lookup result with a bit width of N5 bits as input, such as 10 bits of data as input, and outputs a subtraction result with a bit width of N6 bits (e.g., 32 bits).

[0097] The hardware acceleration circuit may further include a conversion circuit 440, which, in response to a state control signal, converts the N6-bit subtraction result output by the addition / subtraction module into a corresponding index value. The index value output by the conversion circuit 440 is N7-bit data. For example, it converts a 32-bit subtraction result into an 8-bit index value.

[0098] In one embodiment, the conversion circuit 440 may include a leading zero count (LZC) circuit and a shifter. The leading zero count circuit outputs the number of leading zeros in the subtraction result to the shifter. The leading zero count is the number of zeros that appear between the most significant bit of the binary data and the first 1.

[0099] In one specific implementation, the shifter uses the number of leading zeros as the shift bit. The subtraction result is shifted by this number of bits, outputting a shifted data with a width of N7 bits. That is, N7 consecutive bits are truncated from the subtraction result, starting with a leading 1 and moving towards the least significant bit, serving as the index value of the subtraction result. It is understandable that the specific configuration of the conversion circuit 440 can be determined based on the specific data structure of the index value.

[0100] In some embodiments, the first lookup table to the third lookup table are stored in different storage areas of the storage module, and each of the first lookup table circuits to the third lookup table circuits is configured with a basic lookup table circuit unit, and each independently completes the lookup operation.

[0101] In this embodiment of the application, the storage module may be, for example, RAM (Random-Access Memory), ROM (Read-Only Memory), FLASH, etc.

[0102] In some embodiments, a lookup table for the ln() function can be generated as follows. Taking ln(xi) as an example, based on the characteristics that the natural logarithm rapidly approaches negative infinity (data overflow) when xi is less than 0.01, and that the difference between natural logarithmic values ​​can be ignored when xi is greater than 15, xi is limited to, for example, the range [0.01, 15]. First, the range of xi can be divided into 256 points. Then, ln(xi) is calculated for each point, and all these results are quantized to map them to the range [-512, 512]. Next, all these quantized values ​​are filled into a table.

[0103] The first lookup table circuit is used to output an exponential function value corresponding to the output data of the conversion circuit 440 in response to the index value. This index value is N7-bit data.

[0104] The first lookup table circuit sequentially outputs the exponential function values ​​corresponding to the results of each subtraction operation in the first lookup table. Each exponential function value in the first lookup table is N8-bit data.

[0105] Understandably, the index value of the data can be the data itself, or it can be obtained by transforming the data, such as a portion of the data extracted from it.

[0106] The first lookup table can be used to establish a mapping relationship between the result of subtraction and the value of the exponential function. This first lookup table allows the exponential function value of data to be determined through a pre-defined mapping relationship without the need for complex function calculations.

[0107] The first lookup table circuit is used to respond to the index value of the subtraction result and output the exponential function value corresponding to the subtraction result based on the first lookup table. The index value of the subtraction result is a data with a width of N7 bits.

[0108] In some embodiments, a lookup table for the exponential function can be generated as follows: The result of subtracting the natural logarithm of the i-th data element from the natural logarithm of its mean square error is negative or 0. When the negative value is small enough that the difference between exponential function values ​​can be ignored, the range of the subtraction result is limited to, for example, [-10, 0]. Since the subtraction result is negative or 0, the base-e exponential function value of the subtraction result is standardized to the range (0, 1).

[0109] In one specific implementation, the index value of the input data element in the second lookup table (LUT0) is a fixed-point integer with a bit width of 8 (N0) bits. Each output natural logarithmic value is data with a bit width of 10 (N1) bits. The index value of the variance of the n input data elements in the third lookup table (LUT1) is data with a bit width of 8 (N3) bits, and each output natural logarithmic value is data with a bit width of 10 (N4) bits. The result of the subtraction operation of multiple natural logarithmic values ​​is data with a bit width of 32 (N6) bits, and the index value of the subtraction result is data with a bit width of 8 (N7) bits. The first lookup table (LUT2) inputs data with a bit width of 8 (N7) bits and outputs data with a bit width of 8 (N8) bits. The input of the multiplication module 340 is data with a bit width of 8 (N9) bits, and the output is data with a bit width of 8 (N10) bits. That is, N0, N3, N7, and N8~N10 are all 8 bits, N2 (the initial data bit width) and N6 are 32 bits, and N1, N4, and N5 (the input data bit width of the subtractor) are 10 bits. In other words, the second and third lookup tables have 8-bit input and 10-bit output, while the first lookup table has 8-bit input and 8-bit output.

[0110] Therefore, in this embodiment, the value range of each data that needs to be processed in the process of obtaining a specific function value (such as the layer normalization function value) can be limited to a certain range, thereby facilitating the implementation of the solution with less bit width data and corresponding hardware circuitry. For example, when the first lookup table and the second lookup table are 8-bit input and 8-bit output, and the third lookup table is 10-bit input and 8-bit output, the storage space occupied by the three lookup tables is at most (2×2^8) = 512 plus (1×2^10) = 1024 entries, for a total of 1536 entries. This is significantly reduced compared to the 65536 entries required by the 16-bit solution, and the hardware lookup circuitry and bandwidth occupied are also significantly reduced accordingly. On the other hand, the lookup speed can be improved within the allowable accuracy range, thereby further accelerating the circuit response speed and reducing power consumption. The hardware circuit solution provided by this embodiment, which is basically based on 8 bits, can effectively balance important indicators such as circuit cost, power consumption, bandwidth, performance, and data accuracy.

[0111] It is understandable that N0 to N10 can be other values, for example, they can be in the range of [1, 32]. In some embodiments, N0, N1, N3, N4, N5, N7 to N10 can be values ​​of 9, 10, 11, 12, that is, N0, N1, N3, N4, N5, N7 to N10 can be in the range of [8, 12]. N0, N1, N3, N4, N8 to N10 can also be unequal.

[0112] See Figure 6 As shown, in a specific implementation, the basic lookup table circuit unit 20 includes a logic circuit 21, an input terminal group 22, a control terminal group 23, and an output terminal group 24. The input terminal group 22 inputs the data of the lookup table into the logic circuit 21. The logic circuit 21 selects the value corresponding to the index value (also called the address) input from the control terminal group 23 and outputs it from the output terminal group 24. The logic circuit 21 can be, for example, a logic gate circuit or a logic switch circuit. It is understood that in this application, a terminal group refers to a set of connection terminals, including one or more connection terminals. If the control terminal group 23 has A control terminals and the output terminal group 24 has B output terminals, the basic lookup table circuit unit 20 is referred to as A-input B-output.

[0113] The basic lookup table circuit unit 20 can perform lookup output based on the stored lookup table. Taking the second lookup table as an example, the lookup table also has A inputs and B outputs. The data element input to the lookup table control terminal is the index value with a bit width of A bits, and the output data is the exponential function value with a bit width of B bits. It can be understood that the second lookup table in the storage module only stores the true value of the natural logarithmic value, and the basic lookup table circuit unit 20 is used to realize the mapping relationship between the index value and the true value of the natural logarithmic value.

[0114] To better understand the lookup process in the embodiments of this application, Table 1 below shows a specific example of a first lookup table. This table has N7-bit input and N8-bit output, where N7 and N8 are both 8. The input data to the control terminal of the first lookup table can be an index value with a bit width of N7 bits, and the output data can be an exponential function value with a bit width of N8 bits. For ease of understanding, the data in Table 1 are all represented in decimal format. It can be understood that the first lookup table in the storage module only stores the true value of the exponential function value. The lookup table circuit is used to implement the mapping relationship between the index value and the exponential function value. The subtraction operation result and the integer value of the table are included in Table 1 for better understanding of this application.

[0115] Table 1

[0116] As shown in Table 1, the result of a subtraction operation is a negative number or 0, and the range of the subtraction result is defined as [-10, 0]. For table lookup, the range [-10, 0] is discretized into 256 as shown in the "Subtraction Result" column (i.e., ...). There are 10 points, and the corresponding exponential function value for each point is shown in the column "Exponential Function Value". The result of each subtraction operation corresponds to an integer value in the range [0, 255] shown in the column "Index Value". Each exponential function value corresponds to an integer value in the range [0, 255] shown in the column "Table Integer Value". The data in the column "Exponential Function Value" is stored as the true value in the first lookup table of the storage module. The table can be looked up by the index value.

[0117] Figure 7 This is a structural block diagram of a hardware acceleration circuit according to another embodiment of this application. In this embodiment, the first lookup table module to the third lookup table module of the lookup table circuit 30 share a basic lookup table circuit unit 20, and the subtractor and the adder share an addition operation circuit 402.

[0118] See Figure 6 and Figure 7 The hardware acceleration circuit in this embodiment includes a lookup table circuit 30, an addition / subtraction module 400, a shift circuit 600, a conversion circuit 700, a multiplication module, and a storage module 10. In this embodiment, the multiplication module includes a first multiplier 390 and a second multiplier 500 that are independently configured. It is understood that in other embodiments, the functions of the first multiplier 390 and the second multiplier 500 can be achieved by time-division multiplexing the same multiplier.

[0119] The lookup table circuit 30 includes a basic lookup table circuit unit 20, which includes a logic circuit 21, an input group 22, a control group 23, and an output group 24. The input group 22 is connected to the storage module 10. The logic circuit 21 is used to: in a first time period, responding to the index value of the i-th data element input from the control group 23, output the natural logarithmic value corresponding to the i-th data element from the output group 24 based on a second lookup table; in a second time period following the first time period, responding to the index value of the variance values ​​of the n data elements input from the control group 23, output the natural logarithmic value of the mean square error of the n data elements from the output group 24 based on a third lookup table; in a third time period following the second time period, responding to the index value of the subtraction result between the above two natural logarithmic values ​​input from the control group 23, output the exponential function value corresponding to the subtraction result from the output group 24 based on a first lookup table.

[0120] In one specific implementation, storage module 10 includes a first storage area, where the first lookup table to the third lookup table are stored in a time-sharing manner. Since only one storage area needs to be configured for time-sharing storage of any one of the three lookup tables, the storage space occupied by the lookup tables is effectively reduced, thus reducing hardware costs.

[0121] In another specific implementation, the storage module 10 includes a first storage area to a third storage area, and the first lookup table to the third lookup table are respectively stored in one of the three storage areas.

[0122] In one specific implementation, the basic lookup table circuit unit 20 further includes a state control terminal group, used to configure the basic lookup table circuit unit 20 to an M1-bit input, M2-bit output state in response to a first state control signal during a portion of the time period from the first to the third time period, and to configure the basic lookup table circuit unit 20 to an M3-bit input, M4-bit output state in response to a second state control signal during another portion of the time period. At least one pair of M1 and M3, and M2 and M4, are not equal. That is, M1 and M3 are not equal, and / or M2 and M4 are not equal. This scheme is more suitable for situations where the input / output data bit widths of the first to the third lookup tables are not completely the same.

[0123] In another specific implementation, the basic lookup table circuit unit 20 can be fixed as M1-bit input and M2-bit output. This scheme is more suitable for situations where the input / output data bit widths of the first lookup table to the third lookup table are the same.

[0124] Understandably, in this embodiment, the lookup table circuit 30 further includes a first selector 40 and a second selector 50. The first selector 40 is used to selectively output the index value of the i-th data element input via different data input channels, the index value of the variance of the i-th data element, and the index value of the subtraction result between the natural logarithms to the control terminal group 23 of the basic lookup table circuit unit 20. The second selector 50 is used to output different lookup data output from the output terminal group 24 of the basic lookup table circuit unit 20 to their respective corresponding data output channels.

[0125] The addition / subtraction module 400 can be used to obtain the i-th data element, and to obtain the subtraction result between the two natural logarithmic values ​​before the third time period.

[0126] In some embodiments, the addition / subtraction module includes an adder and a subtractor. The adder is used to obtain a first addition operation result, and the subtractor is used to obtain a subtraction operation result between the natural logarithm of the i-th data element and the natural logarithm of the mean square error of the n data elements. The adder and subtractor are configured independently of each other, or the adder and subtractor share a single addition operation unit.

[0127] The second multiplier 500 is used to obtain the square value of the data element xi.

[0128] In some embodiments, to facilitate the calculation of the mean squared error and / or the mean, the hardware acceleration circuit may further include a third selector 60, used to output each initial data point before the first time period to the addition / subtraction module 400 for accumulation processing to obtain the mean, or to obtain the natural logarithm of the i-th data element obtained from a lookup table. The output is given to the addition / subtraction module to obtain the subtraction result between the natural logarithm of the i-th data element and the natural logarithm of the mean square error of the n data elements.

[0129] In one embodiment, the hardware acceleration circuit may further include a fifth selector 80, used to input the mean of the initial data output by the shift circuit 600 into the inverter circuit 406 to obtain each data element, or to input the natural logarithm of the mean square error of n data elements obtained from a lookup table into the inverter circuit 406 to obtain the subtraction result between the natural logarithm of the i-th data element and the natural logarithm of the mean square error. The addition / subtraction module 400 includes an addition circuit 402, a fourth selector 404, and an inverter circuit 406.

[0130] The inverter circuit 406 is used to output the negative of the mean of the initial data from the shift circuit 600, or the negative of the natural logarithm of the mean square error of n data elements.

[0131] The fourth selector 404 is used to input the square values ​​of each data element xi output by the second multiplier 500 to the addition circuit 402 in sequence to obtain the sum of the square values ​​of each data element, or to input the negative of the average value mean or the negative of the natural logarithm of the square root of the mean square of each initial data output by the inverter circuit 406 to the addition circuit 402.

[0132] In one specific embodiment, see Figure 7 As shown, the addition circuit 402 performs an operation on the initial data. By superimposing, we obtain Then, after processing by shift circuit 600, the average value (mean) is obtained. Inverter circuit 406 outputs the negative value of the average value (mean) to adder circuit 402, which outputs the i-th initial data. The result of the addition operation with the negative value of the mean is the i-th data element xi. The i-th data element xi is output to the second multiplier 500 and the lookup table circuit 30. The lookup table circuit 30 outputs the natural logarithm of the i-th data element xi through the second lookup table, and this natural logarithm is input to the third selector 60. On the other hand, the second multiplier 500 performs a square operation on the i-th data element xi and outputs the square value. The addition operation circuit 402 performs a summation operation on the square values ​​of each data element xi to obtain the cumulative square result of each data element. Then, the square accumulation result is processed by the shift circuit 600 to obtain... And use it as the variance var.

[0133] In some embodiments, the addition / subtraction module 400 and the second multiplier 500 together form a multiply-accumulator module. The output of the multiply-accumulator module is N6 (e.g., 32) bits of data. The conversion circuit 700 converts the variance var from N6 bits of data to N7 bits of data (e.g., 8 bits or 10 bits) and outputs it to the lookup table circuit 30, which outputs the natural logarithm of the mean square deviation of n data elements through a third lookup table. The natural logarithm of the i-th data element xi is selected and output to the addition circuit 402 via the third selector 60. The natural logarithm of the mean square error of the n data elements is output to the inverter circuit 406 via the fifth selector 80. The inverter circuit 406 outputs the negative value of the natural logarithm of the mean square error, which is then output to the addition circuit 402 via the fourth selector 404. The addition circuit 402 outputs the addition result of the natural logarithm of the i-th data element xi and the negative value of the natural logarithm of the mean square error, which is also the subtraction result between the natural logarithm of the i-th data element xi and the natural logarithm of the mean square error. The addition module 400 outputs the subtraction result to the lookup table circuit 30, which outputs the exponential function value corresponding to the subtraction result through the first lookup table.

[0134] The first multiplier 390 performs a multiplication operation on the exponential function value and the mask tensor of the i-th data element to obtain the normalized function value.

[0135] In this embodiment, by reusing the basic lookup table circuit unit, only one basic lookup table circuit unit is needed to meet the lookup requirements of three lookup table modules, effectively reducing the area and cost of the hardware acceleration circuit. The subtractor and adder share the addition operation circuit, further reducing the circuit area. The switching of the conversion circuit between different states can adapt to different states of the basic lookup table circuit unit, facilitating lookups with different data bit widths, thereby improving the flexibility and applicability of the hardware acceleration circuit.

[0136] Figure 8 This is a structural block diagram of a hardware acceleration circuit according to another embodiment of this application. The hardware acceleration circuit of this embodiment includes: a first lookup table module, a second lookup table module, a third lookup table module, an adder, a conversion circuit 440, and a subtractor 430.

[0137] See Figure 8 This embodiment and Figure 5 The difference in the hardware acceleration circuit is that, in this embodiment, the second lookup table module and the third lookup table module share the first basic lookup table circuit unit 20A, and the first lookup table module is configured with a second basic lookup table circuit unit 20B. The first basic lookup table circuit unit 20A has an M1 bit input and an M2 bit output, and the second basic lookup table circuit unit 20B has an M3 bit input and an M4 bit output. At least one pair of M1 and M3, and M2 and M4, are not equal.

[0138] In this embodiment, the second and third lookup tables both have M1-bit input and M2-bit output, while the first lookup table has M3-bit input and M4-bit output. In a specific example, the second and third lookup tables have 8-bit input and 10-bit output, while the first lookup table has 8-bit input and 8-bit output.

[0139] In one specific implementation, the second lookup table and the third lookup table are stored in the first storage area of ​​the storage module in a time-sharing manner, while the first lookup table is stored in the second storage area of ​​the storage module.

[0140] The first basic lookup table circuit unit 20A includes a first input group, a first control group, a first output group, and a first logic circuit. The first input group is connected to the first storage area of ​​the storage module. The first logic circuit is used to: in a first time period, responding to the index value of the i-th data element input from the first control group, output the exponential function value corresponding to the i-th data element from the first output group based on the first lookup table. In a second time period after the first time period, responding to the index value of the variance of the n data elements input from the first control group, output the natural logarithm of the mean square deviation of the n data elements from the first output group based on the second lookup table.

[0141] The second basic lookup table circuit unit 20B includes a second input terminal group, a second control terminal group, a second output terminal group, and a second logic circuit. The second input terminal group is connected to the second storage area of ​​the storage module. The second logic circuit is used to: in a third time period after the second time period, in response to the index value of the subtraction result between two natural logarithmic values ​​input from the second control terminal group, output the exponential function value corresponding to the subtraction result from the second output terminal group based on the third lookup table.

[0142] It is understood that in this embodiment, the hardware acceleration circuit also includes a sixth selector 70. The sixth selector 70 is used to selectively output the index values ​​of multiple data elements input via different data input channels and the index value of the variance of the i-th data element to the first control terminal group of the first basic lookup table circuit unit 20A. For example, if the second lookup table and the third lookup table are both 8-bit inputs and 10-bit outputs, they can be implemented based on the first basic lookup table circuit unit 20A; if the first lookup table is 8-bit inputs and 8-bit outputs, it can be implemented based on a second basic lookup table circuit unit 20B, which is different from the first basic lookup table circuit unit 20A.

[0143] In this embodiment, by reusing a basic lookup table circuit unit for lookup table modules with the same data bit width, and configuring separate basic lookup table circuit units for lookup table modules with different data bit widths, the complexity of circuit control can be reduced.

[0144] Figure 9 This is a structural block diagram of a hardware acceleration circuit according to another embodiment of this application. The hardware acceleration circuit of this embodiment includes a first lookup table module 450, a second lookup table module 410, a third lookup table module 420, a subtractor 800, a subtractor 430, a conversion circuit 440, and a multiplier. In this embodiment, the first lookup table module 450, the second lookup table module 410, and the third lookup table module 420 are implemented by independent lookup table circuits.

[0145] See Figure 9 This embodiment and Figure 5Similarly, the main differences are as follows. This embodiment includes a subtractor 800 in its hardware acceleration circuit. The subtractor 800 outputs the result of subtracting multiple initial data points from the initial dataset and their average, to obtain a data set containing multiple data elements. Through this subtraction operation, the value range of the data elements can be reduced, thereby facilitating the implementation of the solution using less bit-width data and corresponding hardware circuitry. It should be noted that the average value can be obtained by processing multiple initial data points using the hardware acceleration circuit, or it can be obtained from a source such as a CPU; this is not limited here.

[0146] This application also provides embodiments of a data processing acceleration method for obtaining layer normalization function values.

[0147] The data processing acceleration method provided in some embodiments of this application uses a simple LUT circuit (such as three 8-bit LUTs) to reduce hardware costs (small combinational logic circuits).

[0148] The data processing acceleration method provided in some embodiments of this application uses a simple 8-bit lookup table, which can adapt to the 8-bit data processing process of existing 8-bit DLA hardware architecture.

[0149] The data processing acceleration method provided in some embodiments of this application uses a simple 8-bit lookup table, which can reduce system bandwidth requirements.

[0150] The data processing acceleration method provided in some embodiments of this application can reduce DRAM / SRAM memory usage by reducing the size of the lookup table.

[0151] The data processing acceleration method provided in some embodiments of this application, when using time-division circuits, can reduce the lookup table size to obtain a smaller LUT latency (e.g., each 8-bit LUT requires 256 cycles).

[0152] The data processing acceleration methods provided in some embodiments of this application can be used for multi-step or one-time hardware implementation.

[0153] The data processing acceleration method provided in some embodiments of this application can increase the accuracy of Layer Norm functions compared with INT 8 DLA or other hardware architecture solutions in related technologies.

[0154] The data processing acceleration methods provided in some embodiments of this application are implemented using DLA / NPU hardware accelerators, supporting end-to-end training or inference without the need for CPU or GPU.

[0155] The data processing acceleration methods provided in some embodiments of this application can support the Layer Norm of the intermediate or final layers of Convolutional Neural Networks (CNNs) or Transformer Networks.

[0156] Figure 10 This is a schematic flowchart illustrating a data processing acceleration method according to an embodiment of this application. See also... Figure 10 A data processing acceleration method includes steps S1010 to S1030.

[0157] In step S1010, the natural logarithm of the i-th data element and the natural logarithm of the mean square error of the n data elements in the dataset are obtained, where n is greater than 1.

[0158] In step S1020, the result of the subtraction operation between the natural logarithm of the i-th data element and the natural logarithm of the mean square error is obtained.

[0159] In step S1030, the exponential function value of the subtraction result is obtained.

[0160] In step S1040, the result of the multiplication operation between the exponential function value and the mask tensor of the i-th data element is obtained to obtain the specific function value corresponding to the i-th data element.

[0161] In some embodiments, the above method may further include the following operations: First, obtain the square results of each of the n data elements. Then, obtain the addition result of the square results of the n data elements. Next, perform a right shift operation on the addition result to obtain the variance of the n data elements, so as to obtain the natural logarithm of the mean square error of the n data elements based on the variance.

[0162] In some embodiments, data elements can be obtained by preprocessing the initial data. Specifically, the above method may further include the following operation: outputting the result of subtracting the i-th initial data point from the n initial data points of the initial dataset from the average of the n initial data points, to obtain a dataset containing n data elements.

[0163] In some embodiments, the received initial data can be converted first to reduce bandwidth usage. Specifically, the above method may further include the following operations: obtaining n initial data points from the initial dataset, and converting each of the n initial data points from a first-bit wide data set to a second-bit wide data set.

[0164] In some embodiments, the average value of multiple initial data can be obtained in the following manner. Specifically, the above method may further include the following operations: obtaining the addition result of n initial data, performing a right shift operation on the addition result of n initial data, thereby obtaining the average value of n initial data.

[0165] To improve the speedup effect of hardware acceleration methods and reduce hardware costs, lookup tables can be used to calculate complex functions.

[0166] For example, obtaining the exponential function value of the subtraction result can include: outputting the exponential function value of the subtraction result based on the first lookup table.

[0167] For example, obtaining the natural logarithm of the i-th data element and the natural logarithm of the mean squared error of the n data elements can include: outputting the natural logarithm of the i-th data element based on a second lookup table, and outputting the natural logarithm of the mean squared error of the n data elements based on the variance of the n data elements and a third lookup table.

[0168] Figure 11 This is a schematic flowchart illustrating a data processing acceleration method according to another embodiment of this application.

[0169] See Figure 11 A data processing acceleration method includes steps S1110 to S1160.

[0170] In step S1110, the result of the subtraction operation between the i-th initial data and the average of the n initial data in the initial dataset is output to obtain a dataset containing n data elements.

[0171] In step S1120, the natural logarithm value corresponding to the i-th data element is output based on the second lookup table.

[0172] In step S1130, based on the variance of the n data elements and the third lookup table, the natural logarithm of the mean squared error of the n data elements is output.

[0173] In step S1140, the result of the subtraction operation between the natural logarithm of the i-th data element and the natural logarithm of the mean square error is obtained.

[0174] In step S1150, the exponential function value of the subtraction result is output based on the first lookup table.

[0175] In step S1160, the result of the multiplication operation between the exponential function value and the mask tensor is obtained to obtain the specific function value corresponding to the i-th data element.

[0176] In some embodiments, multiple lookup tables can be accessed by time-division multiplexing the lookup table circuitry.

[0177] For example, the above method may further include: in response to a state control signal, converting the subtraction result from data with a width of N6 bits to data with a width of N7 bits, so as to input the data with a width of N7 bits into a lookup table.

[0178] The second lookup table has N0 bits as input and N1 bits as output, the third lookup table has N3 bits as input and N4 bits as output, and the first lookup table has N7 bits as input and N8 bits as output. The values ​​of N0, N1, N3, N4, N7, and N8 are in the range [8, 12].

[0179] For example, the above method can be used to implement the standardized function layer of a neural network, which is used to classify the data to be processed. The data to be processed includes at least one of speech data, text data, and image data.

[0180] The relevant features of the data processing acceleration method in this application embodiment can be found in the relevant content of the foregoing hardware acceleration circuit embodiment, and will not be repeated here.

[0181] In one specific embodiment, the 32-bit floating-point number xi' is first converted into an 8-bit integer.

[0182] Then, the mean of multiple xi's is obtained. Specifically, the n xi's are first summed to obtain the first accumulated value. Here, the number of xi's is the number of data points in the last dimension of the input tensor. Next, the first accumulated value is converted into an 8-bit integer average by right shifting by m bits. Here, the m bits in the right shift are determined based on the correction data and are set by the software.

[0183] Next, obtain the difference x_hat between xi' and mean.

[0184] The variance var can be obtained before, during, or after obtaining the difference x_hat. Specifically, the squares of the differences x_hat of the n xi can be accumulated to obtain a second accumulated value. Then, the second accumulated value is converted into an 8-bit integer type variance by right shifting it n bits.

[0185] Then, obtain the mask tensor (maski) corresponding to the difference x_hat.

[0186] Next, the natural logarithmic value A (such as a 10-bit integer) corresponding to the difference x_hat is obtained through the second lookup table circuit.

[0187] Then, the natural logarithm B (e.g., a 10-digit integer) corresponding to the square root of the variance var is obtained through the third lookup table circuit.

[0188] Next, obtain the difference C between A and B.

[0189] Then, the exponential function value D of the difference C is obtained through the first lookup table circuit.

[0190] Next, the exponential function value D is multiplied by maski to obtain the standardized data E corresponding to xi'.

[0191] The data processing acceleration method according to the embodiments of this application can be applied to artificial intelligence accelerators. Figure 12 This is a schematic diagram of the structure of an artificial intelligence accelerator according to an embodiment of this application. See also... Figure 12 The artificial intelligence accelerator 1200 includes a memory 1210 and a processor 1220.

[0192] The artificial intelligence accelerator 1220 can be a general-purpose processor, such as a CPU (Central Processing Unit), or an artificial intelligence processor (IPU) for performing artificial intelligence operations. Artificial intelligence operations can include machine learning operations, neuromorphic operations, etc. Machine learning operations include neural network operations, k-means operations, support vector machine operations, etc. The artificial intelligence processor can, for example, include one or a combination of GPU (Graphics Processing Unit), DLA (Deep Learning Accelerator), NPU (Neural-Network Processing Unit), DSP (Digital Signal Processing Unit), Field-Programmable Gate Array (FPGA), and Application Specific Integrated Circuit (ASIC). This application does not limit the specific type of processor.

[0193] Memory 1210 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by processor 1220 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 1210 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 1210 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital multifunction optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-high density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0194] The memory 1210 stores executable code, which, when processed by the processor 1220, can cause the processor 1220 to execute part or all of the methods described above.

[0195] In one possible implementation, the artificial intelligence accelerator may include multiple processors, each of which can independently run various assigned tasks. This application does not limit the processors or the tasks they run.

[0196] It is understood that, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or in software program modules.

[0197] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the artificial intelligence processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage module can be any suitable magnetic or magneto-optical storage medium, such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), etc.

[0198] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0199] In one possible implementation, an artificial intelligence chip is also disclosed, which includes the aforementioned hardware acceleration circuitry.

[0200] In one possible implementation, a board is also disclosed, which includes a storage device, an interface device, a controller, and the aforementioned artificial intelligence chip; wherein the artificial intelligence chip is connected to the storage device, the controller, and the interface device respectively; the storage device is used to store data; the interface device is used to realize data transmission between the artificial intelligence chip and external devices; and the controller is used to monitor the status of the artificial intelligence chip.

[0201] In one possible implementation, an electronic device is disclosed that includes the aforementioned artificial intelligence chip. The electronic device includes data processing devices, robots, computers, printers, scanners, tablets, smart terminals, mobile phones, dashcams, navigators, sensors, cameras, servers, cloud servers, cameras, camcorders, projectors, watches, headphones, mobile storage, wearable devices, vehicles, home appliances, and / or medical devices. Vehicles include airplanes, ships, and / or vehicles; home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs.

[0202] Furthermore, the method according to this application can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing some or all of the steps in the method described above.

[0203] Alternatively, this application may be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) storing executable code (or computer program or computer instruction code) thereon, which, when executed by a processor of an electronic device (or server, etc.), causes the processor to perform part or all of the steps of the methods described above according to this application.

[0204] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A hardware acceleration circuit, characterized in that, include: The natural logarithm module is used to obtain the natural logarithm of the i-th data element among n data elements in a dataset and the natural logarithm of the mean square error of the n data elements, where n is greater than 1. The addition / subtraction module is used to obtain the subtraction result between the natural logarithm of the i-th data element and the natural logarithm of the mean square error; The exponential function module is used to obtain the exponential function value of the subtraction result; The multiplication module is used to obtain the result of the multiplication operation between the exponential function value and the mask tensor of the i-th data element, so as to obtain the layer normalization function value corresponding to the i-th data element; The exponential function module includes a first lookup table module, which is used to output the exponential function value of the subtraction result based on the first lookup table. And / or the natural logarithm module includes some or all of the second lookup table module and the third lookup table module; The second lookup table module is used to output the natural logarithm of the i-th data element based on the second lookup table; The third lookup table module is used to output the natural logarithm of the mean squared error of the n data elements based on the variance of the n data elements and the third lookup table. The hardware acceleration circuit includes a first lookup table module, a second lookup table module, and a third lookup table module. At least a portion of the first lookup table module, the second lookup table module, and the third lookup table module share a first basic lookup table circuit unit. At least one other module among the first lookup table module, the second lookup table module, and the third lookup table module is configured with a second basic lookup table circuit unit. The first basic lookup table circuit unit has M1 bit input and M2 bit output, and the second basic lookup table circuit unit has M3 bit input and M4 bit output. At least one pair of M1 and M3, and M2 and M4, are not equal. The hardware acceleration circuit further includes a second conversion circuit, which is used to convert the subtraction result output by the addition and subtraction module from data with a bit width of N6 bits to data with a bit width of N7 bits in response to a state control signal, and input it to the second basic lookup table circuit unit.

2. The hardware acceleration circuit as described in claim 1, characterized in that, The multiplication module is also used to obtain the square operation result of each of the n data elements; The addition / subtraction module is also used to obtain the addition result of the square operation of the n data elements; The hardware acceleration circuit also includes a shift module for performing a right shift operation on the addition result to obtain the variance of the n data elements, so as to obtain the natural logarithm of the mean square error of the n data elements based on the variance.

3. The hardware acceleration circuit as described in claim 2, characterized in that: The addition / subtraction module includes an adder and a subtractor. The adder is used to obtain the result of the addition operation, and the subtractor is used to obtain the result of the subtraction operation between the natural logarithm of the i-th data element and the natural logarithm of the mean square error. The adders and subtractors are configured independently of each other; or... The adder and subtractor share a single addition operation unit.

4. The hardware acceleration circuit as described in claim 3, characterized in that, The addition / subtraction module is also used to output the result of the subtraction operation between the i-th initial data in the initial dataset and the average of the n initial data, so as to obtain the data set containing the n data elements.

5. The hardware acceleration circuit as described in claim 4, characterized in that: It also includes a first conversion circuit, which is used to obtain n initial data in the initial dataset, convert the n initial data from the first bit wide data to the second bit wide data respectively, and output them to the addition and subtraction module.

6. The hardware acceleration circuit as described in claim 4, characterized in that, The addition / subtraction module is also used to obtain the addition result of the n initial data; The shift module is also used to perform a right shift operation on the addition result of the n initial data, thereby obtaining the average value of the n initial data.

7. The hardware acceleration circuit as described in claim 1, characterized in that, The hardware acceleration circuit includes at least two lookup table modules selected from the first to the third lookup table modules, wherein: Each of the at least two lookup table modules is configured with a basic lookup table circuit unit; or... The at least two lookup table modules share a basic lookup table circuit unit.

8. The hardware acceleration circuit as described in claim 7, characterized in that: The second lookup table has N0 bits as input and N1 bits as output, the third lookup table has N3 bits as input and N4 bits as output, and the first lookup table has N7 bits as input and N8 bits as output. The values ​​of N0, N1, N3, N4, N7, and N8 are in the range [8, 12].

9. An artificial intelligence chip, characterized in that: The chip includes the hardware acceleration circuitry as described in any one of claims 1 to 8.

10. A method for accelerating data processing, characterized in that, include: Obtain the natural logarithm of the i-th data element among n data elements in a dataset and the natural logarithm of the mean square error of the n data elements, where n is greater than 1; Obtain the result of the subtraction operation between the natural logarithm of the i-th data element and the natural logarithm of the mean square error; Obtain the exponential function value of the subtraction result; The result of the multiplication operation between the exponential function value and the mask tensor of the i-th data element is obtained to obtain the layer normalization function value corresponding to the i-th data element; Obtaining the exponential function value of the subtraction result includes: outputting the exponential function value of the subtraction result based on a first lookup table; and / or, obtaining the natural logarithm of the i-th data element and the natural logarithm of the mean square error includes: outputting the natural logarithm of the i-th data element based on a second lookup table, and outputting the natural logarithm of the mean square error of the n data elements based on the variance of the n data elements and a third lookup table; The method further includes, in response to a state control signal, converting the subtraction result from data with a bit width of N6 bits to data with a bit width of N7 bits, and inputting it into a second basic lookup table circuit unit.

11. The method as described in claim 10, characterized in that, Also includes: Obtain the square result of each of the n data elements; Obtain the sum of the squares of the n data elements; A right shift operation is performed on the result of the addition operation to obtain the variance of the n data elements, and the natural logarithm of the mean square error of the n data elements is obtained based on the variance.

12. The method as described in claim 11, characterized in that, Also includes: Output the result of subtracting the i-th initial data point from the n initial data points in the initial dataset from the average of the n initial data points, to obtain the dataset containing the n data elements.

13. The method as described in claim 12, characterized in that, Also includes: Obtain n initial data points from the initial dataset, and convert each of the n initial data points from a first-bit wide data point to a second-bit wide data point.

14. The method as described in claim 12, characterized in that, Also includes: Obtain the result of the addition operation of the n initial data; The result of the addition operation of the n initial data is right-shifted to obtain the average value of the n initial data.

15. The method as described in claim 10, characterized in that: The second lookup table has N0 bits as input and N1 bits as output, the third lookup table has N3 bits as input and N4 bits as output, and the first lookup table has N7 bits as input and N8 bits as output. The values ​​of N0, N1, N3, N4, N7, and N8 are in the range [8, 12].

16. The method according to any one of claims 10 to 15, characterized in that: The method is used to implement a standardized function layer in a neural network, which is used to classify the data to be processed; wherein, The data to be processed includes at least one of voice data, text data, and image data.

17. An artificial intelligence accelerator, characterized in that, include: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 10 to 16.

Images