Neural Network Computing Method, Device, Board, and Computer-Readable Storage Medium

Abstract

The present disclosure relates to a method, apparatus, board, and computer-readable storage medium for neural network computing. The neural network computing apparatus of the present disclosure is included in an integrated circuit device, which includes a general-purpose interconnect interface and other processing devices. The neural network computing apparatus interacts with other processing devices to jointly complete a computing operation specified by a user. The integrated circuit device may further include a storage device, which is respectively connected to the neural network computing apparatus and other processing devices and is used for data storage of the neural network computing apparatus and other processing devices.

Description

Technical Field

[0001] The present disclosure generally relates to the field of neural networks. More specifically, the present disclosure relates to methods, devices, boards, and computer-readable storage media for neural network computing. Background Art

[0002] In order to process massive amounts of data, modern neural networks continuously increase the depth, width, and resolution of the network, resulting in an increasing model storage capacity and computational complexity. As a result, neural networks are extremely resource-consuming during training and inference. Therefore, those skilled in the art have been trying their best to reduce energy consumption.

[0003] Quantized neural networks can compress the model, reduce computational power consumption, and accelerate model inference and training without loss of accuracy. The quantization approach is to use low-bit fixed-point numbers to replace floating-point numbers in the network, and the data scale can be reduced from the original 32 bits (single-precision floating-point numbers) to 8 bits or 4 bits. Quantized networks have such advantages and are already common technical means in this technical field.

[0004] During training quantization, in addition to quantizing weights and activation values, it is also necessary to quantize the backpropagation gradient. The computational complexity of the backpropagation gradient is about twice that of the forward propagation. A large number of experiments have shown that during the training process of fixed-point quantization, the quantization accuracy requirement for the backpropagation gradient is higher than that for the forward propagation weights and activation values. That is to say, the quantization bit width of the backpropagation gradient must be higher than that of the forward propagation weights and activation values. Therefore, the quantization bit width of the backpropagation gradient cannot be made low enough at present and generally remains at 16 bits, and the benefit of compressing the model is limited. This has become the main factor restricting the quantization of the fixed-point network training process at the edge, and without being able to effectively complete the training, neural networks cannot be applied.

[0005] Therefore, there is a technical problem at present, that is, the existing quantization methods cannot effectively reduce energy consumption during both the inference and training processes. Summary of the Invention

[0006] To at least partially solve the technical problems mentioned in the background art, the solution of the present disclosure provides a method, device, board, and computer-readable storage medium for neural network computing.

[0007] In one aspect, the present disclosure discloses a neural network computing device, including a control unit, a quantization unit, and a computing unit. The control unit is used to provide segmentation parameters; the quantization unit is used to quantize floating-point data according to the segmentation parameters to generate fixed-point data; the computing unit is used to calculate the neural network using the fixed-point data.

[0008] In another aspect, the present disclosure discloses a multiplication unit, including a multiplier, an addition module, and a fixed-point to floating-point converter. The multiplier is used to multiply a first fixed-point data and a second fixed-point data to generate a fixed-point product; the addition module is used to add a plurality of quantization offset coefficients corresponding to the first fixed-point data and the second fixed-point data to generate a quantization offset coefficient sum; the fixed-point to floating-point converter is used to convert the fixed-point product into floating-point data according to the quantization offset coefficient sum.

[0009] In another aspect, the present disclosure discloses an integrated circuit device including the foregoing neural network computing device or multiplication unit, and discloses a board including the foregoing integrated circuit device.

[0010] In another aspect, the present disclosure discloses a method for computing a neural network, including: receiving floating-point data for computing the neural network, the floating-point data falling within a numerical distribution; based on a segmentation parameter, segmenting the numerical distribution into a first interval and a second interval; determining whether the floating-point data falls within the first interval; if so, quantizing the floating-point data according to the segmentation parameter to generate fixed-point data; and computing the neural network using the fixed-point data.

[0011] In another aspect, the present disclosure discloses a method for computing a neural network based on floating-point data, including: presetting a plurality of quantization intervals, each quantization interval corresponding to a quantization formula, each quantization formula exhibiting a different gradient of quantization; determining that the floating-point data falls within a specific interval of the plurality of quantization intervals; and quantizing the floating-point data according to the quantization formula corresponding to the specific interval to generate fixed-point data; and computing the neural network using the fixed-point data.

[0012] In another aspect, the present disclosure discloses a method for computing a neural network based on floating-point data, including: presetting a plurality of quantization intervals; determining that the floating-point data falls within a specific interval of the plurality of quantization intervals; and quantizing the floating-point data according to the specific interval to generate fixed-point data; setting an N-bit flag in the data structure of the fixed-point data, the flag bit recording the specific interval, where N is a positive integer; and computing the neural network using the fixed-point data.

[0013] In another aspect, the present disclosure discloses a method for computing a neural network, including: providing a first segmentation parameter and a second segmentation parameter; quantizing a first floating-point data according to the first segmentation parameter to generate a first fixed-point data; quantizing a second floating-point data according to the second segmentation parameter to generate a second fixed-point data; performing a multiplication operation on the first fixed-point data and the second fixed-point data to generate intermediate data; and computing the neural network according to the intermediate data.

[0014] In another aspect, the present disclosure discloses a method for forward propagation in a neural network, including: receiving activation values and weights required for calculating this layer; providing a first segmentation parameter, a first offset value, and a first segmentation value corresponding to the activation values; providing a second segmentation parameter, a second offset value, and a second segmentation value corresponding to the weights; quantizing the activation values according to the first segmentation parameter, the first offset value, and the first segmentation value to generate first fixed-point data; quantizing the weights according to the second segmentation parameter, the second offset value, and the second segmentation value to generate second fixed-point data; performing a multiplication operation on the first fixed-point data and the second fixed-point data to generate intermediate data; performing a floating-point calculation on the intermediate data to generate activation values of the next layer; and repeating the above steps to perform calculations for each layer to complete the neural network.

[0015] In another aspect, the present disclosure discloses a method for backward propagation in a neural network, where the neural network includes weights and weight fixed-point data after fixed-point processing of the weights, including: receiving error values of the next layer; providing a segmentation parameter, an offset value, and a segmentation value corresponding to the error values of the next layer; quantizing the error values of the next layer according to the segmentation parameter, the offset value, and the segmentation value to generate error value fixed-point data; performing a multiplication operation on the error value fixed-point data and the weight fixed-point data to generate the gradient of the weights; performing a fixed-point calculation on the gradient to generate error values of this layer; and adjusting the weights according to the error values of this layer.

[0016] In another aspect, the present disclosure discloses a method for training a neural network, including: performing forward propagation; calculating error values of the next layer according to the activation values of the next layer; performing backward propagation; and adjusting the weights according to the error values of this layer. The step of performing forward propagation includes: receiving activation values and weights required for calculating this layer; quantizing the activation values according to a first segmentation parameter, a first offset value, and a first segmentation value corresponding to the activation values to generate first fixed-point data; quantizing the weights according to a second segmentation parameter, a second offset value, and a second segmentation value corresponding to the weights to generate second fixed-point data; performing a multiplication operation on the first fixed-point data and the second fixed-point data to generate intermediate data; and performing a floating-point calculation on the intermediate data to generate activation values of the next layer. The step of performing backward propagation includes: quantizing the error values of the next layer according to a third segmentation parameter, a third offset value, and a third segmentation value corresponding to the error values of the next layer to generate third fixed-point data; performing a multiplication operation on the second fixed-point data and the third fixed-point data to generate the gradient of the weights; and performing a fixed-point calculation on the gradient to generate error values of this layer.

[0017] In another aspect, the present disclosure discloses an electronic device, including one or more processors and a memory. Computer-executable instructions are stored in the memory, and when the computer-executable instructions are run by the one or more processors, the electronic device is caused to execute any one of the methods described above.

[0018] In another aspect, the present disclosure discloses a computer-readable storage medium, including computer-executable instructions, and when the computer-executable instructions are run by one or more processors, any one of the methods described above is executed.

[0019] The present disclosure proposes a technical solution for adaptive non-uniform low-bit quantization training. By introducing technical means such as multiple quantization intervals, non-uniform fixed-point data formats, and corresponding hardware, the technical problem that existing quantization methods cannot effectively reduce energy consumption during both the inference and training processes is solved, achieving technical effects of reducing the memory occupancy of the network model, reducing the consumption of model training resources, and improving the training speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] By reading the following detailed description with reference to the accompanying drawings, the above and other objects, features, and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the drawings, several embodiments of the present disclosure are shown in an exemplary but non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0021] Figure 1 is an input-output diagram showing the sigmoid function;

[0022] Figure 2 is a schematic diagram showing a four-layer structure of a neural network;

[0023] Figure 3 is a schematic diagram showing a neural network computing device according to an embodiment of the present disclosure;

[0024] Figure 4 is a schematic diagram showing a control unit according to an embodiment of the present disclosure;

[0025] Figure 5 is a curve diagram showing the possible numerical distribution of input data according to an embodiment of the present disclosure;

[0026] Figure 6 is a schematic diagram showing an 8-bit non-uniform fixed-point quantization data structure according to an embodiment of the present disclosure;

[0027] Figure 7 is a flowchart showing a method for computing a neural network according to another embodiment of the present disclosure;

[0028] Figure 8 is a schematic diagram showing a multiplication unit according to another embodiment of the present disclosure;

[0029] Figure 9 is a flowchart showing a method for implementing a neural network for multiplication calculations according to another embodiment of the present disclosure;

[0030] Figure 10 is a schematic diagram showing a non-uniform fixed-point quantization data structure according to another embodiment of the present disclosure;

[0031] Figure 11 is a curve graph showing the possible numerical distribution of input data according to another embodiment of the present disclosure;

[0032] Figure 12 is a schematic diagram showing a neural network computing device according to another embodiment of the present disclosure;

[0033] Figure 13 is a flowchart showing a method for forward propagation according to another embodiment of the present disclosure;

[0034] Figure 14 is a flowchart showing a method for backpropagation according to another embodiment of the present disclosure;

[0035] Figure 15 is a structural diagram showing an integrated circuit device according to another embodiment of the present disclosure; and

[0036] Figure 16 is a structural diagram showing a board card according to another embodiment of the present disclosure. Detailed implementation manners

[0037] Next, the technical solutions in the embodiments of the present disclosure will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are some, but not all, of the embodiments of the present disclosure. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present disclosure without creative efforts shall fall within the protection scope of the present disclosure.

[0038] It should be understood that the terms "first", "second", "third", and "fourth", etc. in the claims, the description, and the drawings of the present disclosure are used to distinguish different objects, rather than to describe a specific order. The terms "including" and "comprising" used in the description and claims of the present disclosure indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0039] It should also be understood that the terms used in this disclosure specification are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in this disclosure specification and the claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include the plural forms. It should be further understood that the term "and / or" used in this disclosure specification and the claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0040] As used in this specification and the claims, the term "if" can be interpreted as "when", "once", "in response to determining" or "in response to detecting" depending on the context.

[0041] The specific embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0042] A neural network is constructed based on the concept of neurons. A neuron is like a kind of perceptron. Generally, the activation function of a perceptron is a step function, while the common activation function of a neuron is the sigmoid function, and its definition is as follows:

[0043]

[0044] The input-output graph of the sigmoid function is as Figure 1 shown. It can map a real number into the interval of 0 and 1 and is suitable for binary classification.

[0045] A neural network is a system of multiple neurons connected according to certain rules. Taking a convolutional neural network as an example, it is generally composed of the following four layer structures: an input layer, a convolution layer, a pooling layer, and a fully connected layer. Figure 2 It is a schematic diagram showing the four-layer structure of the neural network 200.

[0046] The input layer 201 intercepts part of the information from the input data and presents it in the form of a feature matrix, which carries the features corresponding to this part of the information. The input data here can include but is not limited to image data, voice data, or text data.

[0047] The convolutional layer 202 is configured to receive the feature matrix from the input layer 201 and extract features from the input data through convolutional operations. In actual use, multiple convolutional layers can be constructed. Taking image data as an example, the convolutional layers in the first half are used to capture local and detailed information of the image. For example, each pixel of the output image is only the result of calculating the values in a very small range of the input image. The receptive field of the subsequent convolutional layers gradually increases, which is used to capture more complex and abstract information of the image. After the operations of multiple convolutional layers, the abstract representations of the image at various different scales are finally obtained. Although the feature extraction of the input image is completed through convolutional operations, the information volume of the feature image is too large and the dimension is too high, which not only consumes a lot of computing time but also easily leads to overfitting, and further dimensionality reduction is required.

[0048] The pooling layer 203 is configured to replace a certain area of the data with a single value, which is usually the maximum value or the average value of all the values in that area. If the maximum value is used, it is called max pooling; if the average value is used, it is called average pooling. Through pooling, the model size can be reduced and the computing speed can be improved without losing too much information.

[0049] The fully connected layer 204 acts as a classifier in the entire convolutional neural network 200, which is equivalent to a feature space transformation. It extracts and integrates all the useful information before, and with the non-linear mapping of the aforementioned activation function, theoretically, multiple fully connected layers can simulate any non-linear transformation to perform information comparison based on different classifications, so as to determine whether the input data is similar to the comparison target.

[0050] A neural network requires weights as the parameters of the model, and the optimal solution for the weights is obtained during the training process of the neural network. In addition, some parameters such as the connection method of the neural network, the number of network layers, and the number of nodes in each layer are not obtained through learning but are set in advance. These artificially set parameters are called hyperparameters.

[0051] A neural network is divided into forward propagation and backward propagation. The so-called forward propagation is to calculate forward in the direction as shown Figure 2 and calculate the state value and activation value of each neuron in turn from the input layer to the output layer. Backward propagation is to calculate from the output end to the input end in reverse, which is to obtain the gradient value.

[0052] When the input data is being computed in a neural network, it is generally desired to find the solution where the loss function reaches its minimum value, indicating the result closest to the real situation. However, the loss function of a neural network is very complex, making it difficult to obtain the optimal analytical expression. The common approach is to calculate the negative direction of the gradient, because the maximum value in the negative gradient direction is the direction in which the loss function decreases the most. The backpropagation algorithm is used to find this gradient value, and then the model parameters are updated based on gradient descent. Therefore, the backpropagation algorithm starts from the output layer of the neural network model and uses the chain rule of function differentiation to calculate the model gradient layer by layer backward, hoping to obtain the solution where the loss function is minimized.

[0053] By calculating the gradient in this way, each neuron is only computed once and there is no repeated calculation. The fundamental reason why this calculation direction is efficient is that when calculating the gradient, the previous-level unit depends on the calculation of the subsequent-level unit. By first obtaining the gradient value of the subsequent-level unit and then calculating the previous-level unit, the already computed results can be fully utilized, avoiding repeated calculations.

[0054] In neural network training, regularization penalties and ReLU activation functions are often used. Regularization penalties are used because in the case of overfitting, in order to match all the data in the test set, high-order functions with poor generalization will produce large fluctuations. These fluctuations cause the derivative to become very large, and large parameters are required to fit all the data. Therefore, adding a penalty term can punish the situation where the parameters are very large, avoiding these parameters with large fluctuations.

[0055] And when the model has N layers, theoretically the activation rate of neurons will decrease by 2 N times. ReLU can better enable the sparsified model to extract relevant features and fit the training data. In addition, compared with other activation functions, ReLU has the following advantages: for linear functions, ReLU has stronger expressive power, especially in deep networks; for non-linear functions, the gradient of ReLU is constant in the non-negative interval, so there is no vanishing gradient problem, maintaining the convergence speed of the model in a stable state. The vanishing gradient problem refers to the fact that when the gradient is less than 1, the error between the predicted value and the true value decays once for each layer of propagation. If the sigmoid function mentioned above is used as the activation function in a deep model, this phenomenon is particularly obvious, resulting in the stagnation of model convergence.

[0056] Due to the effects of the regularization penalty and the ReLU activation function, the weights, activation values, and gradients of each layer are not uniformly distributed in terms of numerical values. Moreover, the numerical distributions of the weights, activation values, and gradients exhibit the characteristic of approaching 0 during the entire training process, similar to a Gaussian-like distribution. Considering that the numerical values will be concentrated near 0, and uniform quantization will quantize all these values close to 0 to 0, which seriously affects the direction of network training. For example, if 7-bit uniform quantization is adopted, the numerical values with an absolute value less than will be quantized to 0. For training, such quantization is intolerable. In the existing uniform quantization scheme, the factors limiting the network training accuracy mainly come from the quantization error of the gradients, especially the quantization error of the gradients near 0 in the numerical distribution.

[0057] The present disclosure provides an adaptive non-uniform low-bit quantization scheme, which is applicable to the training tasks and inference tasks of various neural networks (such as convolutional neural networks, recurrent neural networks, graph neural networks, etc.). As mentioned above, during the process of improving neural networks, two types of parameters will be involved. One is the general parameters, that is, the parameter data obtained through training, such as the weight w and the offset b in y = wx + b; the other is called hyperparameters, which are the parameter data set before starting learning and cannot be obtained through training.

[0058] In the embodiments of the present disclosure, the apparatuses and various devices, units, modules, etc. described below can be implemented in the form of hardware circuits, such as digital circuits or analog circuits. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. Unless otherwise specified, the artificial intelligence processors mentioned in the embodiments can be any suitable hardware processors, such as CPUs, GPUs, FPGAs, DSPs, and ASICs, etc. Unless otherwise specified, the memories, storage devices, and storage units can be any suitable magnetic storage medium or magneto-optical storage medium, such as resistive random access memory RRAM (Resistive Random Access Memory), dynamic random access memory DRAM (Dynamic Random Access Memory), static random access memory SRAM (Static Random-Access Memory), enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), high-bandwidth memory HBM (High-Bandwidth Memory), hybrid memory cube HMC (Hybrid Memory Cube), etc.

[0059] Optionally, when the following devices, various apparatuses, units, and modules are implemented by ASIC, their advantages over other hardware implementations lie in aspects such as power consumption, reliability, and area, especially when applied to high-performance and low-power mobile terminals.

[0060] An embodiment of the present disclosure is a neural network computing device. The neural network computing device divides the numerical distribution of floating-point input data into a first interval and a second interval, and different quantization methods are used for floating-point data falling into different intervals. A schematic diagram of this embodiment is as Figure 3 shown.

[0061] The neural network computing device of this embodiment includes a control unit 31, a quantization unit 32, and a computing unit 33. The control unit 31 is used to provide various parameters and hyperparameters required during the quantization process; the quantization unit 32 is used to quantize the floating-point data according to the parameters and hyperparameters to generate fixed-point data; the computing unit 33 is used to calculate the neural network using the fixed-point data. Figure 3 Each unit of

[0062] The control unit 31 is used to provide the segmentation parameter α, the offset value shift, and the segmentation value x p etc. As Figure 4 shown, the control unit 31 includes an input / output module 311, a segmentation parameter generator 312, an offset value generator 313, and a segmentation value generator 314.

[0063] The input / output module 311 serves as a channel for signal transmission between the control unit 31 and external units. It can output signals when a control requirement occurs, and at the same time, it can receive control signals from external units and send the control signals to the segmentation parameter generator 312, the offset value generator 313, or the segmentation value generator 314.

[0064] The segmentation parameter generator 312 is used to generate the hyperparameter: the segmentation parameter α. The segmentation parameter α is used to define the aforementioned first interval and second interval. If the input data x (floating-point number) is distributed in the numerical distribution D, the relationship between the segmentation parameter α and the first interval A and the second interval B is:

[0065] A = {x|x < 2 α max{abs(D)}, x ∈ D} (1)

[0066] B = {x|x ≥ 2 α max{abs(D)}, x ∈ D} (2)

[0067] Refer to Figure 5, the Gaussian-distributed curve in the figure is the possible value distribution of the input data x. The input data x will basically fall entirely within the range of the value distribution D. The first interval A is a symmetric interval around 0, i.e., the range between the two dashed lines in the figure, and the range outside the dashed lines is the second interval B. When the input data x is less than 2 α max{abs(D)}, it means that the input data x falls within the first interval A; when the input data x is greater than or equal to 2 α max{abs(D)}, it means that the input data x falls within the second interval B.

[0068] The segmentation parameter α determines Figure 5 the position of the dashed line in

[0069]

[0070] , that is, it determines the range of the first interval A. The smaller the absolute value of the segmentation parameter α, the larger the range of the first interval A. The segmentation parameter α is related to the quantization bit width b. The quantization bit width b represents the number of bits of the fixed-point number, where the quantization bit width b is a positive integer and the segmentation parameter α is a negative integer. In this embodiment, the relationship between the two is as follows: Taking the quantization bit width b as 8 (i.e., an 8-bit fixed-point number) as an example, the segmentation parameter α can be optionally one of -4, -5, -6, -7.

[0071] The offset value generator 313 is used to generate the offset value shift. The offset value shift represents the quantization offset amount, and the value generator 313 generates the offset value shift according to the following expression:

[0072]

[0073] where the ceil function returns the smallest integer greater than or equal to the expression.

[0074] The segmentation value generator 314 is used to generate the segmentation value x p , and the segmentation value x p represents the dividing line between the first interval A and the second interval B on Figure 5 the abscissa, that is, Figure 4 the abscissa value corresponding to the dashed line in p :.

[0075] x p = 2 α max{abs(D)} (4)

[0076] The aforementioned segmentation parameter α, offset value shift, and segmentation value x p are the parameters required in the quantization process. After being generated by the control unit 31, they will be sent to the quantization unit 32 through the input / output module 311.

[0077] Optionally, in one embodiment, the control unit 31 may also be implemented in other hardware circuit ways, which will not be elaborated here.

[0078] Return to Figure 3 , the quantization unit 32 includes: an absolute value calculator 321, a comparator 322, a two-way selector 323, an adder 324, a quantizer 325, and a bus converter 326.

[0079] The absolute value calculator 321 receives the input data x, takes the absolute value of the input data x in floating-point format, and outputs the absolute value abs(x). The comparator 322 receives the absolute value abs(x) and the segmentation value x p , and compares the absolute value abs(x) with the segmentation value x p . If the segmentation value x p is greater than the absolute value abs(x), referring to Equation (1), it means that the input data x falls within the first interval A. The comparator 322 sets the value of the flag bit flag to 1. If the segmentation value x p is less than or equal to the absolute value abs(x), referring to Equation (2), it means that the input data x falls within the second interval B, and the value of the flag bit flag is set to 0. The value of the flag bit flag reflects whether the input data x falls within the first interval A or the second interval B.

[0080] The two-way selector 323 receives the segmentation parameter α from the control unit 31 and determines whether to output the segmentation parameter α or 0 based on the value of the flag bit flag. When the value of the flag bit flag is 1, it means that the input data x falls within the first interval A, and the values within the first interval A including 0 are easily quantized to 0. In this embodiment, the quantization accuracy is adjusted for the values within the first interval A. Therefore, the two-way selector 323 sets the output value to the segmentation parameter α for the subsequent stage to adjust the accuracy. When the value of the flag bit flag is 0, it means that the input data x falls within the second interval B, and the values within this interval do not need to adjust the accuracy. Therefore, the two-way selector 323 sets the output value to 0.

[0081] The adder 324 receives the output of the two-way selector 323 and the offset value shift from the control unit 31, and adds the offset value shift to the output value of the two-way selector 323 to generate a quantization offset value s. That is, when the value of the flag bit flag is 1, the quantization offset value s is the segmentation parameter α plus the offset value shift; when the value of the flag bit flag is 0, the quantization offset value s is the offset value shift.

[0082] The quantizer 325 quantizes the input data x into n-bit fixed-point data, and its quantization formula is:

[0083]

[0084] Among them, 2 s is called the quantization interval, the round function is the result of the rounding operation, and x[n - 1:0] is the fixed-point data after quantization of the input data x. However, the output x[n - 1:0] of the quantizer 325 is only intermediate data and not the final quantization result.

[0085] The bus converter 326 is used to combine the value of the flag bit flag with the intermediate data x[n - 1:0] to generate the fixed-point data x[n:0]. More specifically, it adds the value of the flag bit flag to the intermediate data x[n - 1:0], so that the final fixed-point data x[n:0] is n + 1 bits.

[0086] In another scenario, if the value of the flag bit flag is not important for the calculation unit 33, this embodiment may not include the bus converter 326, and the output x[n - 1:0] of the quantizer 325 is the final quantization result and is directly transmitted to the calculation unit 33.

[0087] Optionally, in one embodiment, the quantization unit 32 may also be implemented in other hardware circuit ways, which will not be elaborated here.

[0088] This embodiment is paired with a new non-uniform fixed-point quantization data structure. The data format of the non-uniform fixed-point data x[n:0] will be described below. Figure 6 An 8-bit non-uniform fixed-point quantization data structure is shown, which includes a sign bit 61, a value bit 62, and a flag bit 63. The sign bit 61 is 1 bit and is the most significant bit (MSB), used to record the positive or negative sign of the fixed-point data. The flag bit 63 is 1 bit and is the least significant bit (LSB), used to record the value of the flag bit flag. The middle 6 bits are the value bit 62, used to record the value of the fixed-point data. The output x[n - 1:0] of the quantizer 325 corresponds to the values of the sign bit 61 and the value bit 62 of the non-uniform fixed-point quantization data structure, and the bus converter 326 then records the value of the flag bit flag in the flag bit 63 to generate the complete fixed-point data x[n:0]. Based on Figure 6 the non-uniform fixed-point quantization data structure, the relationship between the corresponding floating-point input data x and the sign bit 61, the value bit 62, and the flag bit 63 is:

[0089] x = (-1) sign × value × 2 α·flag × interval (6)

[0090] Among them, sign is the value of the sign bit 61, and value is the value of the value bit 62.

[0091] If the number of bits of the fixed-point number is not 8 but n, in this fixed-point data structure, its most significant bit and least significant bit are also the sign bit 61 and the flag bit 63, and the middle n - 2 bits are the numerical bits 62.

[0092] Return to Figure 3 , in the quantization unit 32, the floating-point input data x is converted into fixed-point data x[n:0], and the fixed-point data x[n:0] is transmitted to the calculation unit 33 for calculation. The calculation unit 33 can perform specific calculations according to actual needs, such as Figure 2 the convolution calculation of the fully connected layer 204 in . After the calculation is completed, an intermediate result y[n:0] is generated. The intermediate result y[n:0] is also fixed-point data. The calculation unit 33 then restores the intermediate result y[n:0] to floating-point data y, and the entire calculation process is completed.

[0093] The foregoing description takes the input data x as the activation value as an example, but the present disclosure is not limited thereto. That is, in a neural network, any data that needs to be quantized (such as weights and gradients, etc.) can be converted into fixed-point data by using the quantization unit 32.

[0094] In summary, this embodiment implements a method for calculating a neural network. Figure 7 The flowchart showing this method is described based on the foregoing hardware design. It can be understood that it is not limited to the hardware design of the present disclosure, such as digital circuits or analog circuits. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. The artificial intelligence processor mentioned in the embodiment can be any suitable hardware processor, such as CPU, GPU, FPGA, DSP, and ASIC, etc. Without special instructions, the memory, storage device, and storage unit can be any suitable magnetic storage medium or magneto-optical storage medium, such as resistive random access memory RRAM, dynamic random access memory DRAM, static random access memory SRAM, enhanced dynamic random access memory EDRAM, high bandwidth memory HBM, hybrid memory cube HMC, etc.

[0095] In step 71, the absolute value operator 321 receives the floating-point data of the neural network calculation, and the floating-point data falls within the numerical distribution. Refer to Figure 3 , the absolute value operator 321 receives the input data x, takes the absolute value of the input data x in floating-point format, and outputs the absolute value abs(x). It should be particularly emphasized that in this embodiment, a series of floating-point data will be received, and each floating-point data is processed according to the Figure 7 process.

[0096] In step 72, the control unit 31 divides the numerical distribution into a first interval and a second interval based on the segmentation parameter. Refer to Figure 4, the segmentation parameter generator 312 of the control unit 31 generates a segmentation parameter α. The segmentation parameter α defines a first interval A and a second interval B according to equations (1) and (2).

[0097] In step 73, the comparator 322 determines whether the floating-point data falls within the first interval. Refer to Figure 3 , the comparator 322 receives the absolute value abs(x) and the segmentation value x p , and compares the absolute value abs(x) with the segmentation value x p . If the segmentation value x p is greater than the absolute value abs(x), it indicates that the input data x falls within the first interval A. The comparator 322 sets the value of the flag bit flag to 1. If the segmentation value x p is less than or equal to the absolute value abs(x), it indicates that the input data x falls within the second interval B, and the value of the flag bit flag is set to 0.

[0098] If the floating-point data falls within the first interval, step 74 is executed. The multiplexer 323, adder 324, quantizer 325, and bus converter 326 quantize the floating-point data according to the segmentation parameter to generate fixed-point data. Refer to Figure 3 , the multiplexer 323 receives the segmentation parameter α from the control unit 31 and determines whether to output the segmentation parameter α or 0 based on the value of the flag bit flag. Since the floating-point data falls within the first interval, the value of the flag bit flag is 1, and the multiplexer 323 sets the output value to the segmentation parameter α. The quantization offset value s output by the adder 324 is the segmentation parameter α plus the offset value shift. The quantizer 325 quantizes the input data x into intermediate data x according to equation (5) q , and the bus converter 326 then adds the value of the flag bit flag to the intermediate data x q to generate fixed-point data x[n:0].

[0099] If the floating-point data does not fall within the first interval, step 75 is executed. The multiplexer 323, adder 324, quantizer 325, and bus converter 326 do not quantize the floating-point data according to the segmentation parameter, but quantize the floating-point data according to the following expression to generate fixed-point data x q :

[0100]

[0101] interval = 2 shift (8)

[0102]

[0103] These expressions have no substantial difference from Equation (3) and Equation (5). The difference lies in that since the floating-point data falls within the second interval B, the value of the flag bit flag is 0, and the two-way selector 323 sets the output value to 0. The quantization offset value s output by the adder 324 is only the offset value shift. The quantizer 325 quantizes the input data x into an intermediate data x q , and then the bus converter 326 adds the value of the flag bit flag to the intermediate data x q to generate the fixed-point data x[n:0].

[0104] In step 76, the computing unit 33 calculates the neural network using the fixed-point data x[n:0]. Referring to Figure 3 , the computing unit 33 can perform specific calculations according to actual needs. After the calculation is completed, an intermediate result y[n:0] is generated. The intermediate result y[n:0] is also fixed-point data. The computing unit 33 then restores the intermediate result y[n:0] to floating-point data y, thus completing the entire calculation process.

[0105] Since this embodiment can generate the flag bit flag through the comparator 322 to record whether the input data x falls within the first interval A or the second interval B, enabling the quantization unit 32 to automatically select an appropriate precision for quantization, this embodiment is an "adaptive" system. Furthermore, the precisions of the first interval A and the second interval B are different. The first interval A including 0 uses 2 α to divide the precision more finely to avoid a large number of data close to 0 being quantized to 0. Therefore, this embodiment is still a "non-uniform" system.

[0106] For a neural network, an important calculation that the computing unit 33 in the foregoing embodiment can perform is matrix multiplication. Matrix multiplication involves a large number of multiplications of fixed-point numbers. Based on the Figure 6 fixed-point data structure, a multiplication unit with a special structure is proposed.

[0107] When performing multiplication calculation on two fixed-point data with the Figure 6 fixed-point data structure, assume that the first fixed-point data x1 falls within the first numerical distribution, the second fixed-point data x2 falls within the second numerical distribution, the first segmentation parameter α1 is used to divide the first numerical distribution into a first interval and a second interval, the second segmentation parameter α2 is used to divide the second numerical distribution into a third interval and a fourth interval, the first flag value flag1 is used to reflect whether the first fixed-point data x1 falls within the first interval or the second interval, and the second flag value flag2 is used to reflect whether the second fixed-point data x2 falls within the third interval or the fourth interval. Based on Equation (6), the floating-point product x1 × x2 is:[[]]

[0108]

[0109] Another embodiment of the present disclosure is a multiplication unit, the schematic diagram of which is as shown in Figure 8 to implement the calculation of Equation (10). The multiplication unit 80 of this embodiment includes a multiplier 81, an addition module 82, and a fixed-point to floating-point converter 83.

[0110] The multiplier 81 is used to multiply the first fixed-point data x1[n-1:0] and the second fixed-point data x2[n-1:0] to generate a fixed-point product y[2n-1:0], that is, to implement the The first fixed-point data x1[n-1:0] and the second fixed-point data x2[n-1:0] can come from Figure 3 the output of the quantizer 325, or the output of the bus converter 326, and then remove the lowest bit flag bit flag in the data structure, leaving the sign bit and the numerical bit. Since both the first fixed-point data x1[n-1:0] and the second fixed-point data x2[n-1:0] are n bits, the fixed-point product y[2n-1:0] is 2n bits.

[0111] The addition module 82 is used to add a plurality of quantization offset coefficients corresponding to the first fixed-point data x1[n-1:0] and the second fixed-point data x2[n-1:0] to generate a quantization offset coefficient sum, that is, to implement the As shown in Equation (8), the of Equation (10) is equal to Therefore, the quantization offset coefficient described above involves the segmentation parameter α, the flag value flag, and the offset value shift.

[0112] The addition module 82 includes a first selector 821, a second selector 822, and an adder 823. The first selector 821 sets the first output value to the first segmentation parameter α1 or 0 according to the first flag value flag1. When the first flag value flag1 is 1, that is, when the value of the flag bit x1[n] is 1, it means that the first fixed-point data x1[n-1:0] falls within the first interval, and the first segmentation parameter α1 needs to participate in the calculation, so the first segmentation parameter α1 is output; when the first flag value flag1 is 0, that is, when the value of the flag bit x1[n] is 0, it means that the first fixed-point data x1[n-1:0] falls within the second interval, and the first segmentation parameter α1 does not participate in the calculation, so 0 is output. Similarly, the second selector 822 sets the second output value to the second segmentation parameter α2 or 0 according to the second flag value flag2, and its specific operation is the same as that of the first selector 821 and will not be repeated here.

[0113] The adder 823 adds the first offset value shift1, the second offset value shift2, the first output value, and the second output value to generate the quantization offset coefficient sum s, that is, α1·flag1 + α2·flag2 + shift1 + shift2. The first offset value shift1 and the second offset value shift2 are calculated according to Equation (3).

[0114] The fixed-point to floating-point converter 83 converts the fixed-point product y[2n-1:0] into floating-point data y according to the quantization offset coefficient sum s. The fixed-point to floating-point converter 83 includes a power-of-two calculator 831 and a multiplier 832. The power-of-two calculator 831 uses the quantization offset coefficient sum s as the power to generate the quantization offset value 2 s to achieve The multiplier 832 multiplies the fixed-point product y[2n-1:0] by the quantization offset value 2 s to obtain the floating-point product x1×x2.

[0115] Optionally, in one embodiment, the multiplication unit 80 may also have other hardware circuit implementation methods, which will not be elaborated here.

[0116] [[ID=]15]In summary, this embodiment implements a method for executing a multiplication calculation neural network. The flowchart of this method is as Figure 9 shown. The process of this embodiment is described based on the foregoing hardware design. It can be understood that it is not limited to the hardware design of the present disclosure, such as digital circuits or analog circuits. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. The artificial intelligence processor mentioned in the embodiment can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the memory, storage device, and storage unit can be any suitable magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM, a dynamic random access memory DRAM, a static random access memory SRAM, an enhanced dynamic random access memory EDRAM, a high-bandwidth memory HBM, a hybrid memory cube HMC, etc.

[0117] In step 91, the control unit 31 provides the first segmentation parameter α1 and the second segmentation parameter α2. More specifically, the segmentation parameter generator 312 of the control unit 31 generates the first segmentation parameter α1 and the second segmentation parameter α2.

[0118] In step 92, the quantization unit 32 quantizes the first floating-point data x1 according to the first segmentation parameter α1 to generate the first fixed-point data x1[n:0]. Among them, the comparator 322 generates a segmentation value x based on the first segmentation parameter α1 and according to Equation (4) p, the first numerical distribution is divided into a first interval A and a second interval B; then it is determined whether the first floating-point data x1 falls within the first interval A, and the first interval A is defined by Equation (1). If the first floating-point data x1 falls within the first interval A, the two-way selector 323, adder 324, and quantizer 325 generate intermediate fixed-point data x1[n-1:0] according to the following expression:

[0119]

[0120]

[0121]

[0122] If the first floating-point data x1 does not fall within the first interval A, the two-way selector 323, adder 324, and quantizer 325 generate intermediate fixed-point data x1[n-1:0] according to the following expression:

[0123]

[0124]

[0125]

[0126] Finally, the bus converter 326 adds the flag bit flag to generate the complete first fixed-point data x1[n:0].

[0127] In step 93, the quantization unit 32 quantizes the second floating-point data x2 according to the second segmentation parameter α2 to generate the second fixed-point data x2[n:0]. Similarly, the comparator 322 generates the segmentation value x based on the second segmentation parameter α2 and according to Equation (4) p , the second numerical distribution is divided into a third interval and a fourth interval; then it is determined whether the second floating-point data x2 falls within the third interval, and the third interval is defined by Equation (1). If the second floating-point data x2 falls within the third interval, the two-way selector 323, adder 324, and quantizer 325 generate the second fixed-point data x2[n-1:0] according to the following expression:

[0128]

[0129]

[0130]

[0131] If the second floating-point data x2 does not fall within the third interval, the two-way selector 323, adder 324, and quantizer 325 generate intermediate fixed-point data x2[n-1:0] according to the following expression:

[0132]

[0133]

[0134]

[0135] Finally, the last bus converter 326 fills in the flag bit flag to generate the complete second fixed-point data x2[n:0].

[0136] In step 94, the multiplication unit 80 performs a multiplication operation on the first fixed-point data x1[n:0] and the second fixed-point data x2[n:0] to generate an intermediate data y. As mentioned above, the flag bit of the data structure of the fixed-point data in the present disclosure is used to record the intervals of the first fixed-point data x1[n:0] and the second fixed-point data x2[n:0], the sign bit is used to record the positive and negative signs of the first fixed-point data x1[n:0] and the second fixed-point data x2[n:0], and the numerical bit is used to record the numerical values V1 and V2 of the first fixed-point data x1[n:0] and the second fixed-point data x2[n:0]. The multiplier 81 adds the numerical values of the sign bits of the first fixed-point data x1[n:0] and the second fixed-point data x2[n:0] to generate a sign sum value sign t ; The task of the first selector 821 is equivalent to multiplying the first segmentation parameter α1 by the numerical value of the flag bit x1[n] of the first fixed-point data to generate a first parameter multiplication value pm1; the task of the second selector 822 is equivalent to multiplying the second segmentation parameter α2 by the numerical value of the flag bit x2[n] of the second fixed-point data to generate a second parameter multiplication value pm2; finally, the fixed-point to floating-point converter 83 executes the following expression to generate the intermediate data:

[0137]

[0138] In step 95, a neural network is calculated based on the intermediate data. In the neural network architecture as Figure 2 shown, the input data can be diverse, such as image, voice, text data, etc. These data will undergo a large number of multiplication operations in the input layer 201, convolutional layer 202, pooling layer 203, and fully connected layer 204. All these multiplication operations can be implemented using the aforementioned steps until the inference process is completed, and finally, these image, voice, and text data are recognized. In addition to the input data, the parameters in the neural network can also be converted into fixed-point data using the aforementioned steps for multiplication operations with the input data.

[0139] During the quantization process, data close to 0 will be quantized to 0, which affects the calculation. Therefore, in the foregoing multiple embodiments, the numerical distribution of the input data is divided into two: one is a positive and negative symmetric interval including 0, and the other is an interval other than the foregoing. However, the present disclosure does not limit the number of intervals. As long as the numerical distributions in different ranges need to be quantized using different precisions, they can be appropriately divided into multiple intervals.

[0140] When the number of intervals exceeds two, Figure 6 only the size of the flag bit 63 in the data structure needs to be adjusted. Taking three or four intervals as an example, in order to completely record that the data falls into one of these three or four intervals, the flag bit 63 requires 2 bits, as shown by the flag bit 10 in Figure 10 That is to say, if the numerical distribution is divided into N intervals, the flag bit requires ceil[log2N] bits.

[0141] Another embodiment of the present disclosure is a neural network computing device. The difference from the foregoing embodiments is that the neural network computing device in this embodiment divides the numerical distribution of floating-point data into a first interval, a second interval, and a third interval, and different quantization methods will be adopted when the floating-point data falls into different intervals. As shown in Figure 11 According to different quantization precisions, the numerical distribution D is divided into a first interval A, a second interval B, and a third interval C.

[0142] This embodiment uses two segmentation parameters to define these three intervals. The segmentation parameter α1 is used to define the first interval A and the second interval B, and the segmentation parameter α2 is used to define the second interval B and the third interval C. The relationship between the segmentation parameters α1, α2 and the first interval A, the second interval B, and the third interval C is:

[0143]

[0144]

[0145]

[0146] When calculating the fixed-point data x that falls into the first interval A q the following expression can be used:

[0147]

[0148]

[0149]

[0150] When calculating the fixed-point data x that falls into the second interval B q the following expression can be used:

[0151]

[0152]

[0153]

[0154] When calculating the fixed-point data x falling within the third interval C q the following expression can be used:

[0155]

[0156] interval3 = 2 shift

[0157]

[0158] The schematic diagram of this embodiment is as shown in Figure 12 and there is no significant difference from the framework of Figure 3 except for the control unit 121, comparator 122, and multiplexer 123.

[0159] Compared with the control unit 31 of Figure 3 the control unit 121 outputs the segmentation parameters α1, α2 to the multiplexer 123 and outputs the first segmentation value x corresponding to the segmentation parameter α1 p1 and the second segmentation value x corresponding to the segmentation parameter α2 p2 to the comparator 122, and the expressions are as follows:

[0160]

[0161]

[0162] Compared with the comparator 322 of Figure 3 the comparator 122 is implemented by a two-stage comparison circuit. The first stage compares the absolute value abs(x) of the input data x with the first segmentation value x p1 If the absolute value abs(x) is less than the first segmentation value x p1 it means that the input data x falls within the first interval A, so there is no need to compare it with the second segmentation value x p2 and the output flag value flag is 00. If the absolute value abs(x) is not less than the first segmentation value x p1 it means that the input data x falls within the second interval B or the third interval C, and then it enters the second-stage circuit to compare the absolute value abs(x) of the input data x with the second segmentation value x p2 If the absolute value abs(x) is less than the second segmentation value x p2, it indicates that the input data x falls within the second interval B, so the output flag value flag is 01. If the absolute value abs(x) is not less than the second division value x p2 , it indicates that the input data x falls within the third interval C, so the output flag value flag is 10.

[0163] Compared with Figure 3 The three-way selector 123 receives the segmentation parameters α1 and α2 and determines the output to be the segmentation parameters α1, α2 or 0 based on the value of the flag bit flag. When the value of the flag bit flag is 00, it indicates that the input data x falls within the first interval A, so the three-way selector 123 sets the output value to the segmentation parameter α1. When the value of the flag bit flag is 01, it indicates that the input data x falls within the second interval B, so the three-way selector 123 sets the output value to the segmentation parameter α2. When the value of the flag bit flag is 10, it indicates that the input data x falls within the third interval C, so the three-way selector 123 sets the output value to 0.

[0164] The operations of the remaining components are the same as those of the Figure 3 corresponding components, so they will not be elaborated. Optionally, in one embodiment, Figure 12 The embodiment of

[0165] Figure 12 The embodiment is illustrated with 3 intervals. The present disclosure is not limited to the number of intervals, and those skilled in the art can easily extend it to multiple intervals without creative input.

[0166] Another embodiment of the present disclosure is a method for forward propagation in a neural network, that is, the inference process of the neural network. The method can utilize Figure 3 or Figure 12 The device of Figure 3 to implement. For the convenience of description, the following will be described in conjunction with the Figure 13 embodiment of

[0167] In step 1301, the absolute value calculator 321 receives the activation value x1 and the weight value x2 required for the calculation of this layer. The activation value x1 and the weight value x2 in floating-point format are input to the absolute value calculator 321 as input data, and the absolute value abs(x) is output.

[0168] In step 1302, the control unit 31 provides the first segmentation parameter α1, the first offset value shift1, and the first segmentation value x corresponding to the activation value x1 p1 . The first segmentation parameter α1, the first offset value shift1, and the first segmentation value x p1 have all been described in the foregoing embodiments and will not be elaborated here. Among them, the first segmentation value x p1 can be obtained through the calculation of Equation (4).

[0169] In step 1303, the control unit 31 provides the second segmentation parameter α2, the second offset value shift2, and the second segmentation value x corresponding to the weight value x2 p2 . Among them, the second segmentation value x p2 can be obtained through the calculation of Equation (4).

[0170] In step 1304, the quantization unit 32 quantizes the activation value x1 according to the first segmentation parameter α1, the first offset value shift1, and the first segmentation value x p1 to generate the first fixed-point data. The comparator 322 determines whether the activation value x1 falls within the first interval A based on the first segmentation value x p1 . The first interval A is defined by Equation (1). If the activation value x1 falls within the first interval A, the multiplexer 323, the adder 324, and the quantizer 325 generate the intermediate fixed-point data x1[n-1:0] according to the following expression:

[0171]

[0172]

[0173]

[0174] If the activation value x1 does not fall within the first interval A, the multiplexer 323, the adder 324, and the quantizer 325 generate the intermediate fixed-point data x1[n-1:0] according to the following expression:

[0175]

[0176]

[0177]

[0178] Finally, the bus converter 326 fills in the flag bit flag to generate the complete first fixed-point data x1[n:0].

[0179] In step 1305, the quantization unit 32 quantizes the weight value x2 according to the second segmentation parameter α2, the second offset value shift2, and the second segmentation value x p2 , to generate the second fixed-point data. More specifically, the weight value x2 falls within the second numerical distribution, and the second numerical distribution is divided into a third interval and a fourth interval. The comparator 322 determines whether the weight value x2 falls within the third interval based on the second segmentation value x p2 . If the weight value x2 falls within the third interval, the multiplexer 323, the adder 324, and the quantizer 325 generate the intermediate fixed-point data x2[n - 1:0] according to the following expression:

[0180]

[0181]

[0182]

[0183] If the weight value x2 does not fall within the third interval, the multiplexer 323, the adder 324, and the quantizer 325 generate the intermediate fixed-point data x2[n - 1:0] according to the following expression:

[0184]

[0185]

[0186]

[0187] Finally, the bus converter 326 fills in the flag bit flag to generate the complete second fixed-point data x2[n:0].

[0188] In step 1306, the calculation unit 33 performs a multiplication operation on the first fixed-point data x1[n:0] and the second fixed-point data x2[n:0] to generate intermediate data. In this embodiment, the calculation unit 33 has Figure 8The structure of the multiplication unit 80 multiplies the first fixed-point data x1[n:0] and the second fixed-point data x2[n:0] to generate intermediate data y. As described above, the flag bit of the data structure of the fixed-point data in the present disclosure is used to record the intervals of the first fixed-point data x1[n:0] and the second fixed-point data x2[n:0], the sign bit is used to record the positive and negative signs of the first fixed-point data x1[n:0] and the second fixed-point data x2[n:0], and the numerical bit is used to record the numerical values V1 and V2 of the first fixed-point data x1[n:0] and the second fixed-point data x2[n:0]. The multiplier 81 adds the numerical values of the sign bits of the first fixed-point data x1[n:0] and the second fixed-point data x2[n:0] to generate a sign sum value sign t ; the task of the first selector 821 is equivalent to multiplying the first segmentation parameter α1 and the numerical value of the flag bit x1[n] of the first fixed-point data to generate a first parameter multiplication value pm1; the task of the second selector 822 is equivalent to multiplying the second segmentation parameter α2 and the numerical value of the flag bit x2[n] of the second fixed-point data to generate a second parameter multiplication value pm2; finally, the fixed-point to floating-point converter 83 executes the following expression to generate intermediate data:

[0189]

[0190] This intermediate data is the output result of this layer and also the input data of the next layer, that is, the activation value. The method returns to step 1301, inputs the activation value of the next layer obtained in this step to the next layer, and repeats the execution until all layers are fully calculated.

[0191] In step 1307, the neural network inference is completed. In the neural network architecture as Figure 2 shown, the input data can be diverse, such as image, voice, text data, etc. These data repeatedly perform the multiplication operations of the foregoing steps multiple times in the input layer 201, convolutional layer 202, pooling layer 203, and fully connected layer 204 until the inference process is completed, and finally these image, voice, and text data are recognized.

[0192] Another embodiment of the present disclosure is a method for backpropagation in a neural network, which quantifies the backpropagation gradient by propagating error values. The method can also utilize Figure 3 or Figure 12 hardware to implement. For the convenience of description, the following will be described in conjunction with the Figure 3 embodiment Figure 14A flowchart showing the method of this embodiment is presented. The process of this embodiment is described based on the aforementioned hardware design. It should be understood that it is not limited to the hardware design of the present disclosure, such as digital circuits or analog circuits. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. The artificial intelligence processor mentioned in the embodiment can be any suitable hardware processor, such as CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the memory, storage device, and storage unit can be any suitable magnetic storage medium or magneto-optical storage medium, such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high bandwidth memory (HBM), hybrid memory cube (HMC), etc.

[0193] In step 1401, the quantization unit 32 receives the error value x of the next layer R . Referring to Figure 2 , in backpropagation, the error value of the output value is propagated backward in the direction of the fully connected layer 204, pooling layer 203, convolutional layer 202, and input layer 201. For the final output node, the difference between the activation value generated by the network and the actual value is taken as the error value x of the next layer R , and the error value x of the next layer R is Figure 3 the input data x of the device of

[0194] In step 1402, the control unit 31 provides the segmentation parameter α R corresponding to the error value x of the next layer R , offset value shift R and segmentation value x pR . It should be noted that the forward segmentation parameter α, offset value shift, and segmentation value x p are different from the reverse segmentation parameter α R , offset value shift R and segmentation value x pR . Among them, the segmentation value x pR can also be obtained through the calculation of Equation (4).

[0195] In step 1403, the quantization unit 32 quantizes the error value x of the next layer R according to the segmentation parameter α R , offset value shift pR and segmentation value x R to generate error value fixed-point data. More specifically, the comparator 322 determines whether the error value x of the next layer pR falls within the first interval A based on the segmentation value x R . The first interval A is defined by Equation (1). If the error value x of the next layerR If it falls within the first interval A, the two-way selector 323, adder 324, and quantizer 325 generate intermediate fixed-point data x according to the following expression R [n-1:0]:

[0196]

[0197]

[0198]

[0199] For example, the next-layer error value x R If it does not fall within the first interval A, the two-way selector 323, adder 324, and quantizer 325 generate intermediate fixed-point data x according to the following expression R [n-1:0]:

[0200]

[0201]

[0202] <{

[0203] Finally, the bus converter as 326 adds the flag bit flag to generate the complete fixed-point data x R [n:0].

[0204] In step 1404, the computing unit 33 performs a multiplication operation on the error value fixed-point data x R [n:0] and the weight fixed-point number x2[n:0] to generate the gradient of the weight x2. The weight fixed-point number x2[n:0] can be quantized in the forward propagation process (step 1305). In this embodiment, the computing unit 33 has Figure 8 the structure of the multiplication unit 80, and performs a multiplication operation on the error value fixed-point data x R [n:0] and the weight fixed-point data x2[n:0] to generate the gradient of the weight x2. As described above, the flag bit of the data structure of the fixed-point data in the present disclosure is used to record the interval of the error value fixed-point data x R [n:0] and the weight fixed-point data x2[n:0], the sign bit is used to record the positive and negative signs of the error value fixed-point data x R [n:0] and the weight fixed-point data x2[n:0], and the numerical bit is used to record the numerical value V R [n:0] and the weight fixed-point data x2[n:0], V2. The multiplier 81 adds the numerical values of the sign bits of the error value fixed-point data x R [n:0] and the weight fixed-point data x2[n:0] to generate the sign sum value sign R [n:0] and the weight fixed-point data x2[n:0] to generate the sign sum value signt ; the task of the first selector 821 is equivalent to multiplying the segmentation parameter α R and the sign bit x of the error value fixed-point data R [n] to generate a first parameter multiplication value pm1; the task of the second selector 822 is equivalent to multiplying the second segmentation parameter α2 and the sign bit x2[n] of the weight fixed-point data to generate a second parameter multiplication value pm2; finally, the fixed-point to floating-point converter 83 executes the following expression to generate the gradient of the weight x2:

[0205]

[0206] In step 1405, the quantization unit 32 performs fixed-point calculation on the gradient to generate the error value of this layer. The detailed process of performing fixed-point calculation has been described above and will not be repeated here.

[0207] In step 1406, the control unit 31 adjusts the weight x2 according to the error value of this layer. The backpropagation algorithm starts from the output layer of the neural network model, uses the chain rule of function derivation, and regresses layer by layer to find the model gradient to adjust the weight x2.

[0208] Another embodiment of the present disclosure is a method for training a neural network. Generally, training a neural network is to use the output value obtained by forward propagation, calculate its error value, and then backpropagate the error value back to the input end, and adjust the weights of this layer according to the error values of each layer, so that the inference structure of the neural network model is closer to the actual situation. In other words, the method for training a neural network in this embodiment includes Figure 13 the forward propagation process to obtain the activation value of the next layer, and then calculate the error value of the next layer according to the activation value of the next layer, and then execute Figure 13 the backpropagation process to generate the error value of this layer, and then adjust the appropriate weights according to the error value of this layer. Push back layer by layer to obtain the appropriate weights for each layer.

[0209] Figure 15 is a structural diagram showing an integrated circuit device 1500 according to an embodiment of the present disclosure. As Figure 15 shown, the integrated circuit device 1500 includes a computing device 1502, and the computing device 1502 is the neural network computing device in the foregoing multiple embodiments. In addition, the integrated circuit device 1500 further includes a general interconnect interface 1504 and other processing devices 1506.

[0210] The other processing device 1506 can be one or more types of processors such as a central processing unit, a graphics processing unit, an artificial intelligence processor, etc. The number thereof is not limited and is determined according to actual needs. The other processing device 1506 serves as an interface between the computing device 1502 and external data and control, and performs functions including but not limited to data transfer and basic control such as starting and stopping the computing device 1502. The other processing device 1506 can also cooperate with the computing device 1502 to jointly complete computing tasks.

[0211] The general-purpose interconnect interface 1504 can be used to transfer data and control instructions between the computing device 1502 and the other processing device 1506. For example, the computing device 1502 can obtain the required input data from the other processing device 1506 via the general-purpose interconnect interface 1504 and write it into the storage unit on the chip of the computing device 1502. Further, the computing device 1502 can obtain control instructions from the other processing device 1506 via the general-purpose interconnect interface 1504 and write them into the control cache on the chip of the computing device 1502. Alternatively or optionally, the general-purpose interconnect interface 1504 can also read the data in the storage module of the computing device 1502 and transfer it to the other processing device 1506.

[0212] The integrated circuit device 1500 further includes a storage device 1508, which can be connected to the computing device 1502 and the other processing device 1506 respectively. The storage device 1508 is used to store the data of the computing device 1502 and the other processing device 1506, and is particularly suitable for data that cannot be fully stored in the internal storage of the computing device 1502 or the other processing device 1506 for the required operations.

[0213] According to different application scenarios, the integrated circuit device 1500 can be used as a system-on-chip (SOC) for devices such as mobile phones, robots, drones, video capture, etc., thereby effectively reducing the core area of the control part, improving the processing speed and reducing the overall power consumption. In this case, the general-purpose interconnect interface 1504 of the integrated circuit device 1500 is connected to some components of the device. Some of these components can be, for example, a camera, a display, a mouse, a keyboard, a network card, or a wifi interface.

[0214] The present disclosure also discloses a chip or an integrated circuit chip, which includes the integrated circuit device 1500. The present disclosure also discloses a chip packaging structure, which includes the above chip.

[0215] Another embodiment of the present disclosure is a board card, which includes the above chip packaging structure. Refer to Figure 16 , in addition to including a plurality of the above chips 1602, the board card 1600 can also include other supporting components, and the supporting components include a storage device 1604, an interface device 1606, and a control device 1608.

[0216] The memory device 1604 is connected to the chip 1602 in the chip package structure via a bus 1616 for storing data. The memory device 1604 may include multiple groups of memory cells 1610 .

[0217] Interface device 1606 is electrically connected to chip 1602 within the chip package. Interface device 1606 facilitates data transmission between chip 1602 and an external device 1612 (e.g., a server or computer). In this embodiment, interface device 1606 is a standard PCIe interface. Data to be processed is transferred from the server to chip 1602 via the standard PCIe interface, enabling data transfer. Calculation results from chip 1602 are also transmitted back to external device 1612 via interface device 1606.

[0218] The control device 1608 is electrically connected to the chip 1602 to monitor the status of the chip 1602. Specifically, the chip 1602 and the control device 1608 can be electrically connected via an SPI interface. The control device 1608 can include a microcontroller (MCU).

[0219] Another embodiment of the present disclosure is an electronic device or apparatus, which includes the above-mentioned board 1600. Depending on different application scenarios, the electronic device or apparatus may include a data processing device, a robot, a computer, a printer, a scanner, a tablet computer, a smart terminal, a mobile phone, a driving recorder, a navigator, a sensor, a camera, a server, a cloud server, a camera, a video camera, a projector, a watch, a headset, a mobile storage, a wearable device, a vehicle, a household appliance, and / or a medical device. The vehicle includes an airplane, a ship, and / or a vehicle; the household appliance includes a television, an air conditioner, a microwave oven, a refrigerator, an electric rice cooker, a humidifier, a washing machine, an electric light, a gas stove, and a range hood; and the medical device includes an MRI machine, an ultrasound machine, and / or an electrocardiograph.

[0220] Another embodiment of the present disclosure is an electronic device, comprising one or more processors and a memory, wherein the memory stores computer executable instructions. When the computer executable instructions are executed by the one or more processors, the electronic device executes the method as described above, in particular, executes the method as described above. Figure 7 、 Figure 9 、 Figure 13 and Figure 14 The method described.

[0221] Another embodiment of the present disclosure is a computer-readable storage medium storing computer-executable instructions for computing data in a computing device. When the computer-executable instructions are run by one or more processors, the method described above is executed, in particular, the method as described in Figure 7 , Figure 9 , Figure 13 and Figure 14 is executed.

[0222] The present disclosure proposes a technical solution for adaptive non-uniform low-bit fixed-point training. By introducing technical means such as multiple quantization intervals, non-uniform fixed-point data formats, and corresponding hardware, the technical problem that the quantization bit width is not low enough to effectively reduce energy consumption is solved, and the following technical effects are achieved:

[0223] 1. Reduction in the memory occupancy of the network model, reduction in the consumption of model training resources, and improvement in training speed.

[0224] 2. When using the neural network model of the present disclosure for inference, the inference speed can be accelerated and the accuracy degradation can be reduced.

[0225] 3. When training the neural network model of the present disclosure, problems such as training crashes and non-convergence can be avoided.

[0226] The foregoing can be better understood according to the following clauses:

[0227] Clause A1. A method for calculating a neural network based on floating-point data, comprising: presetting a plurality of quantization intervals, each quantization interval corresponding to a quantization formula, each quantization formula presenting different gradients of quantization; determining that the floating-point data falls within a specific interval of the plurality of quantization intervals; quantizing the floating-point data according to the quantization formula corresponding to the specific interval to generate fixed-point data; and

[0228] calculating the neural network using the fixed-point data.

[0229] Clause A2. The method according to Clause A1, wherein the plurality of quantization intervals include a first interval and a second interval, and the first interval is:

[0230] x|x < 2 α max{abs(D)}

[0231] where x is the floating-point data, α is a first segmentation parameter, and D is the data distribution of the floating-point data.

[0232] Clause A3. The method according to Clause A2, wherein if the specific interval is the first interval, the quantization formula is:

[0233]

[0234] interval = 2 shift ×2 α

[0235]

[0236] where x q is the fixed-point data, the ceil function returns the smallest integer greater than or equal to the expression, and b is the quantization bit width.

[0237] Clause A4. The method according to Clause A2, wherein the second interval is:

[0238] x|x≥2 α max{abs(D)}

[0239] where x is the floating-point data, α is the first segmentation parameter, and D is the data distribution.

[0240] Clause A5. The method according to Clause A4, wherein if the specific interval is the second interval, the quantization formula is:

[0241]

[0242] interval = 2 shift

[0243]

[0244] where x q is the fixed-point data, the ceil function returns the smallest integer greater than or equal to the expression, and b is the quantization bit width.

[0245] Clause A6. The method according to Clause A2, wherein the plurality of quantization intervals further includes a third interval, and the third interval is:

[0246] x|x>2 β max{abs(D)}

[0247] where β is the second segmentation parameter and is greater than α.

[0248] Clause A7. The method according to Clause A6, wherein if the specific interval is the second interval, the quantization formula is:

[0249]

[0250] interval = 2 shift ×2 β

[0251]

[0252] where x q is the fixed-point data, the ceil function returns the smallest integer greater than or equal to the expression, and b is the quantization bit width.

[0253] Clause A8. The method according to Clause A6, wherein if the specific interval is the third interval, the quantization formula is:

[0254]

[0255] interval = 2 shift

[0256]

[0257] where x q is the fixed-point data, the ceil function returns the smallest integer greater than or equal to the expression, and b is the quantization bit width.

[0258] Clause A9. The method according to Clause A2, wherein the numerical distribution is a Gaussian distribution, and the first interval is a positive and negative symmetric interval including 0.

[0259] Clause A10. The method according to Clause A1, wherein the precision of the multiple quantization intervals is non-uniform.

[0260] Clause A11. The method according to Clause A1, wherein the floating-point data is one of a weight value, an activation value, and a gradient.

[0261] Clause A12. A method for calculating a neural network based on floating-point data, comprising: presetting a plurality of quantization intervals; determining that the floating-point data falls within a specific interval of the plurality of quantization intervals; quantizing the floating-point data according to the specific interval to generate fixed-point data; setting an N-bit flag bit in a data structure of the fixed-point data, the flag bit recording the specific interval, where N is a positive integer; and calculating the neural network using the fixed-point data.

[0262] Clause A13. The method according to Clause A12, wherein the data structure further includes: a sign bit for recording the positive or negative sign of the fixed-point data; and a numerical bit for recording the numerical value of the fixed-point data.

[0263] Clause A14. The method according to Clause A13, wherein the highest bit of the data structure is the sign bit, and the lowest bit is the flag bit.

[0264] Clause A15. The method according to Clause A12, wherein the plurality of quantization intervals are 2 intervals, and N is equal to 1.

[0265] Clause A16. The method according to Clause A12, wherein the plurality of quantization intervals are 3 or 4 intervals, and N is equal to 2.

[0266] Clause A17. An electronic device, comprising: one or more processors; and a memory storing computer-executable instructions that, when run by the one or more processors, cause the electronic device to perform the method according to any one of Clauses A1-16.

[0267] Clause A18. A computer-readable storage medium comprising computer-executable instructions that, when run by one or more processors, perform the method according to any one of Clauses A1-16.

[0268] The embodiments of the present disclosure have been introduced in detail above. Specific examples are used herein to elaborate on the principles and implementation manners of the present disclosure. The description of the above embodiments is only used to help understand the method and its core idea of the present disclosure; at the same time, for those of ordinary skill in the art, according to the idea of the present disclosure, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present disclosure.

Claims

1. A method for calculating a neural network based on floating-point data, comprising: Presetting a plurality of quantization intervals, each quantization interval corresponding to a quantization formula, each quantization formula showing different gradients of quantization, wherein the plurality of quantization intervals include a first interval and a second interval, and the quantization accuracy of the first interval is finer than that of the second interval; determining that the floating-point data falls within a specific interval of the plurality of quantization intervals, the floating-point data being one of a weight, an activation value, and a gradient; Quantizing the floating-point data according to the quantization formula corresponding to the specific interval to generate fixed-point data; And Calculating the neural network using the fixed-point data, wherein the input data of the input layer of the neural network includes at least one of image data, voice data, and text data, the numerical distribution of the floating-point data is a Gaussian distribution, and the first interval is a positive and negative symmetric interval including 0; wherein, the first interval is: x|x < 2 α max{abs(D)} Where x is the floating-point data, α is the first segmentation parameter, and D is the data distribution of the floating-point data; Where if the specific interval is the first interval, the quantization formula is: interval = 2 shift × 2 α where x q is the fixed-point data, the ceil function returns the smallest integer greater than or equal to the expression, and b is the quantization bit width.

2. The method according to claim 1, wherein the second interval is: x | x ≥ 2 α max{abs(D)} Among them, x is the floating-point data, α is the first segmentation parameter, and D is the data distribution.

3. The method according to claim 2, wherein if the specific interval is the second interval, the quantization formula is: interval = 2 shift Among them, x q For the fixed-point data, the ceil function returns the smallest integer greater than or equal to the expression, and b is the quantization bit width.

4. The method according to claim 1, wherein the plurality of quantization intervals further include a third interval, and the third interval is: x | x > 2 β max{abs(D)} Among them, β is a second segmentation parameter and is greater than α.

5. The method according to claim 4, wherein if the specific interval is the second interval, the quantization formula is: interval = 2 shift × 2β Among them, x q For the fixed-point data, the ceil function returns the smallest integer greater than or equal to the expression, and b is the quantization bit width.

6. The method according to claim 4, wherein if the specific interval is the third interval, the quantization formula is: interval = 2 shift Among them, x q For the fixed-point data, the ceil function returns the smallest integer greater than or equal to the expression, and b is the quantization bit width.

7. A method for calculating a neural network based on floating-point data, comprising: Presetting a plurality of quantization intervals, each quantization interval corresponding to a quantization formula, each quantization formula showing different gradients of quantization, wherein the plurality of quantization intervals include a first interval and a second interval, and the quantization accuracy of the first interval is finer than that of the second interval; Determining that the floating-point data falls within a specific interval of the plurality of quantization intervals; Quantizing the floating-point data according to the quantization formula corresponding to the specific interval to generate fixed-point data, the floating-point data being one of a weight, an activation value, and a gradient; Setting an N-bit flag in the data structure of the fixed-point data, the flag bit recording the specific interval, where N is a positive integer; And Calculating the neural network using the fixed-point data, wherein the input data of the input layer of the neural network includes at least one of image data, voice data, and text data, the numerical distribution of the floating-point data is a Gaussian distribution, and the first interval is a positive and negative symmetric interval including 0; Wherein, the first interval is: x|x < 2 α max{abs(D)} Where x is the floating-point data, α is the first segmentation parameter, and D is the data distribution of the floating-point data; Where if the specific interval is the first interval, the quantization formula is: interval = 2 shift × 2 α where x q is the fixed-point data, the ceil function returns the smallest integer greater than or equal to the expression, and b is the quantization bit width.

8. The method according to claim 7, wherein the data structure further comprises: a sign bit for recording the positive or negative sign of the fixed-point data; and a numerical bit for recording the numerical value of the fixed-point data.

9. The method according to claim 8, wherein the highest bit of the data structure is the sign bit and the lowest bit is the flag bit.

10. The method according to claim 7, wherein the plurality of quantization intervals are 2 intervals and N is equal to 1.

11. The method according to claim 7, wherein the plurality of quantization intervals are 3 or 4 intervals and N is equal to 2.

12. An electronic device, comprising: one or more processors; and a memory storing computer-executable instructions, which when run by the one or more processors cause the electronic device to perform the method according to any one of claims 1-11.

13. A computer-readable storage medium comprising computer-executable instructions, which when run by one or more processors perform the method according to any one of claims 1-11.

Images