Methods, apparatuses, and computer readable storage media for training a neural network

Abstract

The present disclosure discloses devices, methods and integrated circuit boards for training neural networks, wherein the devices are embodied in computing devices included in a combined processing apparatus that can also include general-purpose interconnect interfaces and other processing devices. The computing devices interact with the other processing devices to collectively perform user-specified computing operations. The combined processing apparatus can also include storage devices connected to the computing devices and the other processing devices, respectively, for data of the computing devices and the other processing devices. The present disclosure can accelerate training of neural networks.

Description

Technical Field

[0001] This disclosure generally relates to the field of artificial intelligence. More specifically, this disclosure relates to methods, apparatus, integrated circuits, boards, and computer-readable storage media for training neural networks via hardware platforms. Background Technology

[0002] With the continuous development of artificial intelligence technology, how to efficiently train neural networks to obtain good neural network models has become a focus of attention. Existing neural networks typically use floating-point data to perform calculations during training to achieve good training results. Although floating-point data has relatively high precision, it places higher hardware requirements on the hardware platform running the neural network during training, such as larger storage space and higher power consumption. Furthermore, in some training scenarios, using relatively lower precision fixed-point data can achieve the same or similar training results as floating-point data, making the application of floating-point data unnecessary in some situations.

[0003] Furthermore, data with relatively low bit widths can represent a limited range of data. For example, a 16-bit floating-point number can only represent a value with a minimum precision of 5.9604 × 10⁻⁶. -8 However, when performing backpropagation for neural network training, it is often possible to obtain values ​​less than 5.9604 × 10⁻⁶. -8 The value obtained exceeds the range that a 16-bit floating-point number can represent. If a 16-bit floating-point number is still used to represent this value, it will be 0. Clearly, such a zeroing operation will significantly impact the accuracy and precision of the calculation. Summary of the Invention

[0004] To address some or all of the problems mentioned above and to provide a way to efficiently train neural networks, this disclosure provides the following technical solutions in several aspects.

[0005] In one aspect, this disclosure provides an apparatus for training a neural network, wherein training the neural network includes iteratively performed forward propagation and backpropagation, the apparatus comprising: scaling circuitry configured to scale a loss value obtained from the forward propagation according to a scaling factor to obtain a scaled loss value; update circuitry configured to perform an update operation in the backpropagation based on the scaled loss value; and adjustment circuitry configured to adjust the scaling factor at least according to gradient data in the backpropagation for scaling the loss value in the next generation of backpropagation.

[0006] In yet another aspect, this disclosure provides an integrated circuit that includes the device described above and which will be discussed in several embodiments below.

[0007] In yet another aspect, this disclosure provides a board that includes the device described above and which will be discussed in several embodiments below.

[0008] In another aspect, this disclosure provides a method for training a neural network, wherein training the neural network includes iteratively performed forward propagation and back propagation, the method comprising: scaling a loss value obtained from the forward propagation according to a scaling factor to obtain a scaled loss value; performing an update operation in the back propagation based on the scaled loss value; and adjusting the scaling factor at least according to gradient data in the back propagation for scaling the loss value in the next generation of back propagation.

[0009] In another aspect, this disclosure provides an apparatus for training a neural network. The apparatus includes at least one processor. The apparatus also includes at least one memory storing computer program code, said at least one memory and said computer program code being configured to utilize the processor to cause the apparatus to perform the foregoing method and several embodiments of the method described below.

[0010] In one aspect, this disclosure provides a computer-readable storage medium storing a computer program for training a neural network that, when run by one or more processors, implements the foregoing method and several embodiments of the method described below.

[0011] Through the aforementioned devices, methods, integrated circuits, boards, and computer-readable storage media for training neural networks, the disclosed solution can adaptively or timely adjust the magnitude of the loss value during backpropagation of neural network training. This prevents errors caused by the limited range of low-precision data types during backpropagation updates, thereby improving the accuracy and precision of the computation. Furthermore, the adjustment of the loss value makes quantizing high-precision data into low-precision data for computation more applicable, thus expanding the computational scenarios of neural networks. Moreover, since the disclosed solution supports quantizing high-precision data into low-precision data for related computations within the neural network, the computational scenarios are not limited by the number of bits supported by the processor chip, thus expanding the processor's application scenarios. Additionally, the use of low-precision data types (e.g., using fixed-point numbers for neural network operations such as multiplication and addition) accelerates the neural network training process and reduces computational overhead and power consumption. Furthermore, neural networks trained using the disclosed solution can be widely applied in various fields such as image processing, speech recognition, and data acquisition, significantly improving efficiency and cost-effectiveness in these areas. Attached Figure Description

[0012] The above-described features of the invention can be better understood by referring to the accompanying drawings, and its numerous objects, features and advantages will be apparent to those skilled in the art, wherein like reference numerals denote like elements, and wherein:

[0013] Figure 1 This is an exemplary block diagram illustrating a neural network to which the technical solutions disclosed herein can be applied;

[0014] Figure 2 This is a functional block diagram illustrating an apparatus for training a neural network according to an embodiment of this disclosure;

[0015] Figure 3 This is an exemplary flowchart illustrating update operations in forward and backward propagation in a neural network according to embodiments of this disclosure;

[0016] Figure 4 It is a graph illustrating the principle of quantization error according to an embodiment of this disclosure;

[0017] Figure 5 This is a flowchart illustrating a method for training a neural network according to an embodiment of this disclosure;

[0018] Figure 6 This is a structural diagram illustrating a combined processing apparatus according to an embodiment of the present disclosure; and

[0019] Figure 7 This is a schematic diagram illustrating the structure of a circuit board according to an embodiment of the present disclosure. Detailed Implementation

[0020] Embodiments of the invention will now be described with reference to the accompanying drawings. It should be understood that, for the sake of simplicity and clarity, reference numerals may be repeated in the drawings to indicate corresponding or similar elements where deemed appropriate. Furthermore, numerous specific details are set forth herein to provide a thorough understanding of the embodiments described herein. However, those skilled in the art can practice the various embodiments described herein without these specific details, guided by the teachings of this disclosure. In other instances, well-known methods, processes, and components have not been described in detail to avoid unnecessarily obscuring the embodiments described herein. Moreover, this description should not be construed as limiting the scope of the embodiments described herein.

[0021] As mentioned above, the solution disclosed herein is mainly applied to the field of artificial intelligence, especially to the efficient training of neural networks. Therefore, in order to facilitate the understanding of the solution disclosed herein, the neural network architecture and its working principle involved in this disclosure will be introduced first below.

[0022] A neural network (NN) is a mathematical model that mimics the structure and function of biological neural networks. Neural networks perform computations through a large number of interconnected neurons. Therefore, a neural network is a computational model composed of numerous interconnected nodes (or neurons). Each node represents a specific output function called an activation function. Each connection between two neurons represents a weighted value of the signal passing through that connection, called a weight, which is analogous to the network's memory. The output of a neural network varies depending on the connections between neurons, as well as the weights and activation functions. In a neural network, the neuron is the basic unit. It receives a certain number of inputs and a bias, and when a signal (value) arrives, it is multiplied by a weight. A connection links one neuron to another neuron in another layer or within the same layer, and each connection is accompanied by an associated weight. Additionally, the bias is an extra input to the neuron; it is always 1 and has its own connection weight. This ensures that the neuron will activate even if all inputs are empty (all zeros).

[0023] In applications, if a non-linear function isn't applied to the neurons in a neural network, the network is merely a linear function and no more powerful than a single neuron. If we want the output of a neural network to be between 0 and 1—for example, in cat / dog identification—outputs close to 0 can be interpreted as cats, and outputs close to 1 as dogs. To achieve this, activation functions, such as the sigmoid activation function, are introduced into the neural network. The return value of this activation function is typically a number between 0 and 1. Therefore, activation functions introduce non-linearity into the neural network, narrowing the computational results to a smaller range. In practice, how the activation function is expressed is not important; what matters is parameterizing a non-linear function with weights, which can be changed to alter the non-linear function.

[0024] Figure 1 This is an exemplary block diagram illustrating a neural network 100 to which the technical solutions disclosed herein can be applied. For example... Figure 1 As shown in the figure, the neural network 100 includes an input layer and an output layer, as well as a plurality of hidden layers located between the input layer and the output layer, which are exemplarily shown in the figure as convolutional layers, activation layers, pooling layers and fully connected layers.

[0025] The neurons in the input layer are called input neurons. In this example, three input neurons are shown, each receiving three input signals x1, x2, and x3. As the first layer in a neural network, the input layer receives the required input signals (values) and passes them to the next layer. Typically, the input layer does not perform any operations on the input signals (values) and has no associated weights or biases. For specific neural networks, such as convolutional neural networks (CNNs), their input layers can process multidimensional data. Commonly, the input layer of a one-dimensional CNN receives a one-dimensional or two-dimensional array, where the one-dimensional array is typically a time or spectral sample; the two-dimensional array can contain multiple channels; the input layer of a two-dimensional CNN receives a two-dimensional or three-dimensional array; the input layer of a three-dimensional CNN receives a four-dimensional array, and so on. In some specific applications, preprocessing operations can also be performed on the data at the input layer, such as mean removal, normalization, and dimensionality reduction.

[0026] Hidden layers contain neurons (nodes) used to apply different transformations to the input data. Figure 1 The neural network shown includes four hidden layers: a convolutional layer with four neurons (nodes), an activation layer with four neurons, a pooling layer with two neurons, and a fully connected layer with six neurons. Finally, the values ​​processed by the fully connected layer are passed to the output layer. The neurons in the output layer are called output neurons. The output layer receives the output from the last hidden layer. Figure 1 In the neural network shown, the output layer has two neurons and two output signals y1 and y2. As can be seen from the hidden layers, each neuron in each hidden layer may or may not be connected to any neuron in the next layer, depending on the specific hidden layer. For example, neurons in the activation layer and pooling layer are partially connected, while those in the pooling layer and the fully connected layer are fully connected.

[0027] The exemplary hidden layers in this example are briefly described below. It should be understood that the descriptions of the various hidden layers described above are merely exemplary and not restrictive, and the technical solutions disclosed herein are not intended to limit the scope of the examples. Figure 1 The limitations of the hidden layer structure of the neural network shown are explained, and those skilled in the art can, based on the teachings of this disclosure, [address the limitations of the hidden layer structure]. Figure 1 The neural network structure shown can be modified, for example, by adding one or more layers or removing layers as needed for the application. Figure 1 One or more layers in the structure shown, and these operations are still covered within the technical solutions covered by this disclosure.

[0028] As the first hidden layer in this example—the convolutional layer—its function is typically to extract features from the input data. It can contain multiple convolutional kernels, each element of which corresponds to a weight coefficient and a bias, similar to a neuron in a feedforward neural network. When processing image data, in the convolutional layer, each feature in the image is first perceived locally, and then a higher-level synthesis operation is performed on the local features to obtain global information. Convolutional layer parameters include kernel size, stride, and padding, which together determine the size of the output feature map and are hyperparameters of the convolutional neural network. In application, each neuron in the convolutional layer is connected to multiple neurons in a region located close to it in the previous layer; the size of this region depends on the kernel size. During operation, the convolutional kernel systematically scans the input features, performing element-wise multiplication (multiply-add) on the input features and adding the bias.

[0029] The activation layer that receives the output of the convolutional layer essentially performs a non-linear mapping on the output. Commonly used activation functions include: Sigmoid, Tanh, ReLU, Leaky, ELU, and Maxout. After applying these activation functions, the output of the previous layer becomes more complex, thereby enhancing the expressive power of the neural network model.

[0030] Pooling layers are primarily used for feature dimensionality reduction, compressing the amount of data and parameters, reducing overfitting, and improving the model's fault tolerance. Common pooling methods include max pooling and average pooling. After feature extraction in convolutional layers and processing by activation layers, the output feature map is passed to pooling layers for feature selection and information filtering. Pooling layers contain predefined pooling functions that replace the result of a single point in the feature map with the statistics of its neighboring regions. The selection of pooling regions is the same as the feature map scanning step of the convolutional kernel; this step can be controlled by the pooling size, stride, and padding.

[0031] After the preceding convolution, activation, and pooling, the signal processing flow of the neural network reaches the fully connected layer, which is located at the end of the hidden layers in this example neural network. In the fully connected layer, the feature map loses its spatial topological structure, is unfolded into a vector, and output through the activation function. The fully connected layer can non-linearly combine the extracted features to obtain the output; that is, the fully connected layer itself is not expected to have feature extraction capabilities, but rather attempts to utilize existing high-order features to achieve the learning objective. Additionally, operations such as Local Normalization (LRN) and data augmentation can be performed in the fully connected layer to increase the robustness of the neural network.

[0032] Despite Figure 1 Not shown in the diagram, each layer in the neural network has one or more operators associated with that layer (which will be combined). Figure 2(To describe in detail) to perform corresponding computational operations. In a neural network, an operator is a mapping from one function space to another. Broadly speaking, any operation performed on any function can be considered an operator. In short, an operator can be a mapping, relation, or transformation. For example, for convolutional layers (or other layers that need to perform convolution operations), there are convolution operators, which can be concretized as an expression of one or more convolution calculation formulas. By using this convolution operator to compute with the input data and the convolution kernel, the result value after the convolution operation can be obtained.

[0033] Above Figure 1 The neural network structure and the function of its nodes are illustrated and described exemplarily. In practical applications, to obtain a good neural network model, a large amount of sample data (including input and output) is provided in advance to train the initial neural network. After training, the trained neural network is obtained. This trained neural network can provide a correct output for inputs from future real-world environments.

[0034] Before discussing the training of a neural network, we need to define a loss function. A loss function is a function that measures the performance of a neural network in performing a specific task. In some embodiments, the loss function can be obtained as follows: during the training of a neural network, for each sample data point, the output value is obtained by passing it along the network. Then, the difference between this output value and the expected value is squared. The resulting loss function is the distance between the predicted value and the true value, and the purpose of training the neural network is to reduce this distance, or the value of the loss function. In some embodiments, the loss function can be expressed as:

[0035]

[0036] In the above formula, y represents the expected value. This refers to the actual result obtained by processing each sample data in the sample dataset through a neural network, where i is the index of each sample data in the sample dataset. This represents the expected value y and the actual result. The error value between them. m is the number of sample data in the sample dataset.

[0037] Let's take a practical application scenario of cat and dog identification as an example. Assume a dataset consists of images of cats and dogs. If the image is of a dog, the corresponding label is 1; if the image is of a cat, the corresponding label is 0. This label corresponds to the expected value y in the formula above. When feeding each sample image to the neural network, the goal is to obtain the recognition result through the neural network, i.e., whether the animal in the image is a cat or a dog. To calculate the loss function, it is necessary to traverse every sample image in the dataset and obtain the actual result corresponding to each sample image. Then, calculate the loss function according to the definition above. If the value of the loss function (referred to as the "loss value") is relatively large, for example, exceeding a predetermined threshold, it means that the neural network has not been trained properly. In this case, it is necessary to use the aforementioned backpropagation process to further adjust the weights.

[0038] When starting to train a neural network, the weights need to be randomly initialized. In most cases, the initialized neural network will not provide good training results. During training, assuming you start with a very poor neural network, you can train it to obtain a network with high accuracy.

[0039] The training process of a neural network consists of two stages. The first stage is the forward signal processing operation (referred to as the "forward propagation process" in this disclosure), where training proceeds from the input layer through the hidden layers and finally to the output layer. The second stage is the backward propagation gradient operation (referred to as the "backward propagation process" in this disclosure), where training proceeds from the output layer to the hidden layers and finally to the input layer, adjusting the weights and biases of each layer in the neural network according to the gradient.

[0040] During forward propagation, the input value is fed into the input layer of the neural network. After passing through the corresponding operations performed by relevant operators in multiple hidden layers, the output layer of the neural network yields the so-called predicted value. When the input value is provided to the input layer of the neural network, it may not perform any operations or may undergo some necessary preprocessing depending on the application scenario. In the hidden layers, the second hidden layer obtains the intermediate predicted value from the first hidden layer, performs calculations and activation operations, and then passes the obtained intermediate predicted value to the next hidden layer. The same operations are performed in subsequent layers, and finally, the output value is obtained at the output layer of the neural network. After the forward processing of the forward propagation process, an output value called the predicted value is usually obtained. To calculate the error, the predicted value can be compared with the actual output value to obtain the corresponding error value.

[0041] During backpropagation, the chain rule of differential calculus can be used to update the weights of each layer, aiming to obtain a lower error value in the next forward propagation compared to the previous one. In the chain rule, the derivatives of the error values ​​of the weights of the last layer of the neural network are first calculated (these derivatives are called gradients). Then, these gradients are used to calculate the gradients of the penultimate layer in the neural network. This process is repeated until the gradient corresponding to each weight in the neural network is obtained. Finally, the corresponding gradient is subtracted from each weight in the neural network, thus updating the weights to reduce the error value. Similar to the various operators used in the forward propagation process (referred to as forward operators in this disclosure), there are also corresponding backward operators in the backpropagation process. For example, the convolution operators in the aforementioned convolutional layers include the forward convolution operators in the forward propagation process and the deconvolution operators in the backpropagation process.

[0042] During the training of a neural network, each forward propagation of the signal and each corresponding backward propagation of the error results in an update of the network's weights using the gradient; this is called an iteration. To obtain a neural network with the expected accuracy, a large sample dataset is needed during training, but inputting the entire dataset into a computing device (such as a computer) at once is nearly impossible. Therefore, to solve this problem, the sample dataset needs to be divided into multiple blocks and passed to the computer block by block. Each block undergoes forward propagation and then a corresponding backward propagation update of the neural network's weights. When a complete sample dataset passes through the neural network once and returns a weight update, this process is called an epoch. In practice, passing the complete dataset once to the neural network is insufficient; the complete dataset needs to be passed multiple times within the same neural network, i.e., multiple epochs are required to ultimately obtain a neural network with the expected accuracy.

[0043] During neural network training, users typically desire as fast a training speed and as high an accuracy rate as possible. However, these expectations are often influenced by the data type of the neural network data. In many applications, neural network data is represented using high-precision data formats (e.g., floating-point numbers). Taking convolution operations during forward propagation and deconvolution operations during backpropagation as examples, when these operations are performed on a computing device's central processing unit ("CPU") and graphics processing unit ("GPU"), almost all inputs, weights, and gradients are floating-point data to ensure data precision.

[0044] Taking floating-point format as a high-precision data format as an example, according to computer architecture, for the same length of floating-point and fixed-point operations, floating-point operations are more complex and require more logic devices to construct the floating-point unit. Thus, the size of a floating-point unit is larger than that of a fixed-point unit. Furthermore, floating-point units consume more resources, making the power consumption difference between fixed-point and floating-point operations typically orders of magnitude, resulting in significant differences in computational cost. However, experiments have shown that fixed-point operations are faster than floating-point operations with minimal precision loss. Therefore, using fixed-point operations to handle large amounts of neural network operations (such as convolution and fully connected operations) in artificial intelligence chips is a feasible solution. For example, floating-point data involving the inputs, weights, and gradients of forward convolution, forward fully connected, backward convolution, and backward fully connected operators can be quantized before fixed-point operations, and the low-precision data can be converted back to high-precision data after the operator operations are completed.

[0045] Taking the weights of a neural network as an example, where the quantized weights are all 8-bit fixed-point numbers (a lower precision type compared to floating-point numbers), a neural network often has millions of connections, with almost all space occupied by the weights of the neurons, and these weights may all be different floating-point numbers. The weights of each layer tend to follow a normal distribution within a certain interval, such as (-3.0, 3.0). The maximum and minimum values ​​corresponding to the weights of each layer in the neural network are stored, and each floating-point value is represented by an 8-bit fixed-point number. Within the range of the maximum and minimum values, the interval is linearly divided into 256 quantization intervals, each represented by an 8-bit fixed-point number. For example, within the interval (-3.0, 3.0), byte 0 represents -3.0, byte 255 represents 3.0, and so on, with byte 128 representing 0.

[0046] During the quantization process, two quantization parameters are typically involved: shift and n. shift is the position of the fixed-point data (i.e., the "point position parameter" disclosed herein), and n is the bit width of the fixed-point data (i.e., the "bit width parameter" disclosed herein). n can be initially set manually, while shift is calculated using the following formula based on the distribution range of the data to be quantized and n:

[0047]

[0048] Where Z is the maximum absolute value of the data to be quantized, max(|F|). Let F represent the floating-point data before quantization, and I represent the n-bit fixed-point number after quantization. The fixed-point transformation from F to I can be calculated using the following formula:

[0049] F≈I×2shift (3)

[0050] Where step = 2 s Given the quantization step size (minimum quantization interval), the resulting fixed-point number I can be expressed by the following formula:

[0051]

[0052] When it is necessary to convert the quantized fixed-point number to a floating-point number, a dequantization operation can be performed. The dequantized value will then be... It can be expressed by the following formula:

[0053]

[0054] It can be seen that the fixed-point number obtained through the above quantization is beneficial for accelerating neural network training, reducing chip size, and significantly reducing computational overhead. Specifically, the above quantization operation can be performed on neuron data and weight data during the forward propagation of the training neural network, and on gradient data used for update operations during the back propagation of the training neural network. When some related operators are introduced into the above quantization operation (to be discussed later)... Figure 3 When describing the process in detail, quantization operations can be optimized to fully utilize them while minimizing the introduction of excessive quantization overhead, thereby accelerating neural network training, improving training accuracy, and reducing computational costs.

[0055] As mentioned earlier, although the quantization operations described above bring significant technical advantages to neural network training, in some application scenarios, when the loss value is expressed using a relatively low-precision data type, such as the 16-bit floating-point number in the previous example (which has relatively low precision compared to 32-bit, 64-bit, or higher-bit floating-point numbers), the limitation of the numerical representation range will inevitably have an adverse impact on the update operation in backpropagation. Therefore, this disclosure proposes an effective loss value adjustment scheme to overcome the above-mentioned shortcomings, improve the training accuracy of neural networks, and thus accelerate training.

[0056] Figure 2 This is a functional block diagram illustrating a device 200 for training a neural network according to an embodiment of this disclosure. Figure 2As shown, the device 200 includes a scaling circuit 202, an update circuit 204, and an adjustment circuit 206. In one or more embodiments, the device 200 may further include a quantization circuit 208. According to the scheme disclosed herein, the scaling circuit can be configured to scale the loss value obtained during the forward propagation of the training neural network according to a scaling factor to obtain a scaled loss value. By amplifying the loss value, errors in subsequent update operations caused by the limited numerical representation range of low-precision data can be effectively avoided, thereby preventing delays and inefficiencies in the entire training process. For ease of understanding, assuming the loss value disclosed herein is represented as loss, it can be scaled using the following formula:

[0057] Loss_scale = loss × scale (6)

[0058] In equation (6) above, scale represents the scaling factor disclosed herein, while Loss_scale represents the scaled scaling factor.

[0059] After scaling the loss value, the update circuit can be configured to perform an update operation in backpropagation based on the scaled loss value. In one embodiment, this update operation may involve updating the weights and updating the gradient data propagated from the previous layer to the next layer in the backpropagation direction. As previously mentioned, the update operation in backpropagation involves quantization operations on various types of data, such as the quantization operation from high-precision data (e.g., floating-point numbers) to low-precision data (e.g., fixed-point numbers) and the dequantization operation from low-precision data to high-precision data, as described above. To this end, the quantization circuit 208 of device 200 can be configured to perform a quantization operation on the operational data according to quantization parameters, and to determine whether to update the aforementioned quantization parameters based on the operational data. Further, the adjustment circuit can be configured to adjust the aforementioned scaling factor for scaling the loss value in the next generation of backpropagation when the quantization circuit determines to update the quantization parameters. In some embodiments, the scaling factor can be adjusted based on the gradient data in backpropagation.

[0060] Depending on the application scenario, the operational data described above can include various types of data during the neural network training process. For example, in one embodiment, the operational data can include gradient data. Based on this, the quantization parameter can include a first point position parameter or a first first width parameter applied to the quantization gradient data, and the adjustment circuit can be configured to adjust the scaling factor for scaling the loss value in the next-generation backpropagation when the first point position parameter or the first first width parameter is updated.

[0061] In one embodiment, the operational data may include gradient data and neuron data, and the quantization parameter may include a second point location parameter or a second bit width parameter for the gradient data and neuron data. Based on this, the adjustment circuit disclosed herein can be configured to adjust a scaling factor for scaling the loss value in the next-generation backpropagation when the second point location parameter or the second bit width parameter is updated.

[0062] In one embodiment, the operational data may include gradient data, neuron data, and weight data, the quantization parameter includes a third point position parameter or a third bit width parameter for the three, and the adjustment circuit may be configured to adjust the scaling factor for scaling the loss value in the next-generation backpropagation when the third point position parameter or the third bit width parameter is updated.

[0063] Depending on the computational scenario, quantization operations can be executed in different ways. Therefore, the quantization circuit disclosed herein may include a positive quantization circuit and a negative quantization circuit. The positive quantization circuit can be configured to quantize high-precision data type (e.g., floating-point) operation data into low-precision data type (e.g., fixed-point) operation data according to quantization parameters, while the negative quantization circuit can be configured to negatively quantize low-precision data type operation data into high-precision data type operation data according to quantization parameters. In some embodiments, in addition to performing quantization operations, the quantization circuit disclosed herein can also determine whether to update the quantization parameters. To this end, in one embodiment, the quantization circuit can be configured to determine whether to update the quantization parameters based on the quantization error of the operation data. In one implementation scenario, the quantization circuit disclosed herein can be configured to perform calculations based on the average of the operation data before and after quantization to determine the aforementioned quantization error.

[0064] In the backpropagation update operation, a key task is to update the weight data effectively and accurately. To this end, the update circuit disclosed herein can be configured to obtain weight gradient data using a scaled loss value. Correspondingly, the dequantization circuit in the aforementioned quantization circuit can be configured to dequantize the weight gradient data from a low-precision data type (e.g., fixed-point number) to a high-precision data type (e.g., floating-point number). Furthermore, the update circuit can be configured to update the weights using a scaling factor and the high-precision data type weight gradient data. For example, when the obtained loss value is a 16-bit floating-point number, it can be amplified by a scaling factor before the backpropagation update operation (which includes quantizing to a fixed-point number for calculation), and when calculating the weight gradient, it can be converted to high-precision data (e.g., a 32-bit floating-point number). Then, the 32-bit floating-point number can be scaled down inversely proportional to the scaling factor to calculate the weight gradient data, thereby updating the weights. This quantization and update process will be further described below.

[0065] Based on the aforementioned equation (6), when using Loss_scale for backpropagation (including quantization and calculation of the operation data of the corresponding layer), the weight gradient w_grad used for updating the weights of each layer can be obtained. Next, to ensure the accuracy of the calculation, w_grad can be converted into high-precision data (for example, converting it from a relatively low-precision 16-bit floating-point number to a relatively high-precision 32-bit floating-point number), and the actual value of the weight gradient is calculated using the following equation (7), and this actual value is used to update the weights:

[0066] W_grad_real=w_grad / scale (7)

[0067] Regarding the scaling factor "scale" disclosed herein, it can be adjusted in several ways. In one embodiment, the scaling factor can be adjusted based on one or more hyperparameters. In another embodiment, the scaling factor can be determined based on the data distribution of the gradient data. In one embodiment, the scaling factor can also be determined based on a preset threshold and the maximum value of the gradient data.

[0068] The above combination Figure 2 The device disclosed herein is described. It should be understood that the device disclosed herein can be applied to one or more layers in a neural network. In particular, when the neural network is a multi-layered structure, i.e., when the neural network includes multiple intermediate hidden layers, the quantization circuit of this disclosure can be configured to determine whether to update the quantization parameters for each layer. When the quantization circuit determines at any layer that the quantization parameters need to be updated, for example, based on the aforementioned quantization error, the adjustment circuit will dynamically adjust the scaling factor accordingly for scaling the loss value in the next-generation backpropagation.

[0069] Figure 3 This is an exemplary flowchart illustrating update operations in forward and backward propagation in a neural network 300 according to an embodiment of this disclosure.

[0070] like Figure 3 As shown by the dashed box, the neural network 300 can be implemented as including a computation block 301 in the forward propagation, a gradient update block 302 and a weight update block 303 in the backpropagation process. For ease of understanding and description of the scheme disclosed herein, Figure 3 The neural network shown can be viewed as a network that includes only a single hidden layer (e.g., a convolutional layer) or a network that includes only one type of operation (convolutional operation only). Those skilled in the art will understand from the above and below description that the scheme disclosed herein is also applicable to situations where the hidden layer includes multiple layers or various other types of operations.

[0071] Further shown in Figure 3 The operators mentioned above include the quantize operator, the forward convolution operator convFwd, the weight gradient operator convBpFilter, and the input data gradient operator convBpData. The following description follows the forward and backward propagation processes of training the neural network (including weight and gradient updates). Figure 3 The process in the process. It should be noted here that, in Figure 3 In the diagram, "x" represents the input neuron data, "w" represents the weights, "dx" represents the input gradient, "dy" represents the output gradient, the content within "[]" represents the specific data type, "paramx" represents the quantization parameters of the neuron data, "paramw" represents the quantization parameters of the weights, and "paramdy" represents the quantization parameters of the gradient.

[0072] First, the input neuron data x[fp32] and initial weights w[fp32] can be received. As shown in parentheses, both are 32-bit floating-point numbers. It is understood that 32 bits here is merely exemplary, and they could also be 16 bits or other bit-width floating-point numbers. As previously mentioned, both can be quantized to fixed-point numbers by quantization operations performed, for example, by the quantization circuit of this disclosure. For this purpose, in one embodiment, a quantization operator “quantify” can be implemented on the quantization circuit of this disclosure. The quantization operator may include a quantization strategy operator and a quantization parameter operator. In one embodiment, the quantization strategy operator may be used at least to determine whether to perform a quantization parameter update operation, while the quantization parameter operator may be used at least to determine the quantization parameters and perform the quantization operation on the neural network data of the high-precision data type (floating-point numbers in the example of this disclosure) using the quantization parameters.

[0073] In some application scenarios, the aforementioned quantization strategy operator can be responsible for calculating the quantization error diff. bit And quantitative cycle trend value diff update Since determining the quantization error is crucial for adjusting the quantization period and data bit width, it will be described in detail below.

[0074] Suppose that in a scenario, the data to be quantized is F = [f1, f2, ..., f m The data obtained after quantization using n-bit fixed-point quantization is: The quantization error can then be calculated using the following formula:

[0075]

[0076] Right now:

[0077]

[0078] When diff bit If the quantization bit width is greater than the threshold th, then we can consider increasing the quantization bit width by t bits, so that the new quantization bit width is n = n + t, where th and t are both variable hyperparameters.

[0079] As can be seen, the determination of the quantization error above involves the calculation of the mean function mean(), and the calculation method of this quantization error has the following significance:

[0080] like Figure 4 The curves 1 and 2 shown represent two floating-point data distributions. Assuming one quantization interval is [a, b], floating-point values ​​between [a, c] are quantized to a, and floating-point values ​​between [c, b] are quantized to b. Assuming the data follows a Gaussian distribution P(x) ~ G(0, σ), then the mean before quantization is...

[0081]

[0082] The quantified mean is:

[0083]

[0084] And from Figure 4 As can be seen, the slope of the tangent line at point c is k. Through derivation and approximate calculations, the larger |K| is (i.e., the more concentrated the distribution) and the larger the quantization interval, the greater the mean. f and The greater the difference, the greater the error in the final training result after quantization. Experiments show that the more concentrated the data distribution, the greater the error it introduces to the final training result. Therefore, the difference between the mean before and after quantization can be used to simulate the actual error brought about by quantization. To prevent this error from increasing, the quantization interval should be reduced, i.e., the quantization bit width should be increased, when quantizing data with a more concentrated distribution (|K|). This disclosure is based on this theoretical foundation and, after considering the impact of quantization error on training accuracy and effect, proposes a scheme with variable quantization period and data bit width.

[0085] To reduce the computational cost of calculating quantization parameters during training, instead of calculating the quantization parameters for each generation based on the current data to be quantized, the quantization parameters can be updated at regular intervals. In generations without updates, the stored quantization parameters from the previous update can be used when quantizing the data. As long as the update interval is chosen appropriately, this will not lead to a loss of training accuracy because the changes in the data to be quantized (e.g., weights and gradients) are relatively stable during training, exhibiting a certain degree of continuity and similarity. A simple approach is to use a fixed update period, but a fixed update period has poor adaptability; therefore, this disclosure also proposes an adaptive update period adjustment.

[0086] Let the interval "Interval" (i.e., the quantization period) be used to update the quantization parameters. The calculation method is as follows:

[0087] First, we introduce shift, which is the moving average m over the training iteration cycle.

[0088] m (i) ←α×shift+(1-a)×m (i-1) (12)

[0089] Introducing diff to measure the trend of shift changes update1 :

[0090] diff update1 =|m (i) -m (i-1) | (13)

[0091] diff update1 The larger the value, the more drastic the change in the numerical range, requiring a higher update frequency; that is, the smaller the Interval.

[0092] The diff measures the trend of the fixed-point width n. update2 :

[0093]

[0094] diff update2 The larger the value, the greater the quantization bit width required, necessitating bit width updates and higher interval frequencies.

[0095] By considering both of the above measures simultaneously, we obtain the aforementioned quantitative cycle trend value, diff. update as follows:

[0096] diff update =max(diff update1 diff update2 (15)

[0097] Finally, Interva1 was calculated:

[0098]

[0099] In the equation above, α, β, γ, δ, t, and th can be hyperparameters, and β and γ can be either empirical values ​​or hyperparameters. Furthermore, conventional hyperparameter optimization methods are generally applicable to β and γ.

[0100] The above describes how to calculate the quantization error diff for the quantization strategy operator. bit And quantitative cycle trend value diff updateThe details are described in detail. In one implementation scenario, the inputs to the quantization strategy operator disclosed herein may include data before and after quantization, quantization parameters (mainly using the moving average m of shift), quantization period I (which can be either an input or an output), and output quantization bit width, wherein the quantization period and output quantization bit width can be passed as inputs to the quantization parameter operator.

[0101] Furthermore, the input to the quantization parameter operator can include the data to be quantized, quantization parameters (including the number of points shift, the sliding average of the number of points m, scaling factor, etc.), data bit width (indicating the bit width used in the output quantized data), and quantization period. In some application scenarios, the quantization period can be a variable controlling whether the quantization operator should calculate the quantization parameters. For example, when the quantization period is equal to 0, quantization parameter statistics can be performed. Of course, this setting is merely exemplary, and those skilled in the art can assign other meanings to the quantization period based on the teachings of this document, or use different forms for control. In other application scenarios, if quantization parameters are statistically analyzed, the new quantization parameters need to be updated to the address of the old quantization parameters. Otherwise, the quantization operation will still use the old quantization parameters.

[0102] Depending on the implementation method or application scenario, the quantization operator can quantize the data at the same layer within the entire board. The resulting quantization parameters can then be synchronized across multiple machines and multiple cards, or they can be left unsynchronized. When synchronization is not performed, each board can maintain its own set of quantization parameters. Additionally, each processor core synchronizes after calculating its quantization parameters, and the results are combined to obtain the final global quantization parameters.

[0103] Return to Figure 3The processing flow, after the quantization operation of the aforementioned quantization operator, yields new quantization parameters and quantized fixed-point numbers, namely the quantization parameters "paramx" and "paramw" and the quantized data "x[int8]" (corresponding to an 8-bit fixed-point neuron) and "w[int8]" (corresponding to an 8-bit fixed-point weight). These four items are then fed into the forward convolution operator convFwd for computation, resulting in a floating-point result y[fp16] (corresponding to 16-bit floating-point data). In some scenarios, the output of the forward convolution operator convFwd can also be 32-bit floating-point data. During this convolution operation, the forward convolution operator convFwd can perform operations such as multiplication and addition on the fixed-point neuron data and weight data. For example, when the input neuron data is image data and the corresponding weights are convolution kernels (filters), the convolution operation can be the multiplication and summation of the corresponding elements of the image matrix and the filter, followed by the addition of the bias b, to obtain a feature map as the output. To maintain the output data as floating-point data, the convolution operator disclosed herein can also incorporate a dequantization operator, which can be implemented, for example, through the dequantization circuit in the quantization circuit disclosed herein. Thus, the output result can be dequantized into 16-bit floating-point data y[fp16]. The dequantization operation here may involve using the aforementioned quantization parameters paramx and paramw to determine the step size during dequantization, i.e., step in the aforementioned equation (5), thereby dequantizing the fixed-point number into a high-precision floating-point number.

[0104] As mentioned earlier, based on the training results obtained from the forward propagation process, the loss function LossDiff can be determined, which can be achieved, for example, by combining... Figure 1 The method described in Equation (1) is used to obtain the loss value, which will not be repeated here. After obtaining the loss value of LossDiff, according to the scheme disclosed herein, it can be scaled using a scaling circuit (i.e., performing the “scaling” operation shown in the figure) for use in the backpropagation operation. For example, Equation (6) can be used to determine the scaled loss value.

[0105] Next, the training process proceeds to backpropagation, which involves backpropagation gradient update block 302 and weight update block 303. To this end, this disclosure implements two operators via update circuitry: the weight gradient operator "convBpFilter" and the input data gradient operator "convBpData" shown in the figure. In one or more embodiments, convBpData can calculate the gradient of the input neuron data x. Based on the chain rule, the formula for calculating the gradient of x is:

[0106]

[0107] Furthermore, the function of convBpFilter can be to calculate the gradient of the weight w. Based on the chain rule, the formula for calculating the gradient of w can be derived as follows:

[0108]

[0109] In the two formulas above, w, x, and δ represent the weights, the input, and the gradient data from the previous layer, respectively. The 'rot180' function represents a convolution operation, while the 'rot180' function represents rotating the data by 180 degrees.

[0110] The specific operations involved in the backpropagation process are described below as examples to facilitate understanding of the operations involved in the backpropagation gradient update block and the weight update block.

[0111] During backpropagation, for a neural network with two or more layers and multiple neurons, for each layer, the input gradient data, such as a vector (equivalent to the adjusted dy[fp16] in the figure), is first weighted and summed with the weight matrix of the current layer to calculate the output gradient vector of the current layer (equivalent to dx[fp16] in the figure). This involves a dequantization operation from fixed-point to floating-point numbers performed by the dequantization circuit. Furthermore, this input gradient vector (equivalent to dy[fp16] in the figure) is operated on with the input neuron data during forward propagation (e.g., bitwise multiplication) to obtain the gradient of the weights of the current layer (equivalent to dw[fp32] in the figure). Then, it can be scaled using the previous equation (7) to obtain the actual weight gradient. Finally, the weights of the current layer (equivalent to w[fp32] in the figure) can be updated based on the obtained actual gradient of the weights of the current layer.

[0112] Based on the above processing, the disclosed scheme can obtain the current input gradient data by scaling the loss value of the loss function LossDiff. Then, it is quantized using the quantify operator to obtain 8-bit fixed-point input gradient data dy[int8] and quantization parameters paramdy for this gradient data. Next, the corresponding weight quantization parameters paramw, the 8-bit fixed-point weights w[int8], along with the previously obtained input gradient dy[int8] and gradient quantization parameters paramdy, are fed into the input data gradient operator convBpData for computation to obtain the output gradient dx[fp16] of this layer, which serves as the input gradient data (i.e., dy[fp16]) for the next layer (if it exists) in the backpropagation direction. Furthermore, the input gradient dy[int8], the quantization parameter paramdy of the gradient data, along with the corresponding quantization parameter paramx from the forward propagation process of this layer and the aforementioned quantized neuron data x[int8], can be fed into the weight gradient operator convBpFilter to obtain the weight gradient dw[fp32] of this layer. Then, through the solver, the updated weight w[fp32] of this layer can be calculated based on dw[fp32] (after the corresponding scaling as described above) for use in the next forward propagation process.

[0113] Based on the above description, those skilled in the art can understand that during backpropagation, the weights w and the input neuron data x can reuse the data used in the forward propagation. Therefore, during backpropagation, only the gradient dy needs to be quantized. The result of quantizing the gradient dy is the quantized parameter paramdy and the quantized data dy[int8]. There is no need to quantize the input neuron data and weight data again, thereby reducing the number of data quantizations and shortening the training time.

[0114] The above combination Figure 3 The training scheme and the quantization operations involved in this disclosure are described in detail. As previously mentioned, when the quantization circuit of this disclosure determines that the quantization parameters need to be updated during backpropagation, the adjustment circuit of this disclosure will adjust the scaling factor accordingly. In one embodiment, the adjusted scaling factor can be calculated based on the following formula:

[0115] t = floor(log2(th / m1)) (19)

[0116] scale_new=scale×pow(2,t) (20)

[0117] In the above formula, th is a hyperparameter, which can be set according to the size of the neuron data, for example, 512. m1 can be the maximum value of the gradient data during backpropagation. floor() represents the floor function, while pow(x,y) represents the function of calculating x to the power of y (or raised to the power of y).

[0118] Figure 5 This is a flowchart illustrating a method 500 for training a neural network according to an embodiment of this disclosure, wherein the process of training the neural network includes iteratively performed forward propagation and backward propagation. Figure 5 As shown, at step 502, method 500 scales the loss value obtained from the forward propagation according to a scaling factor to obtain a scaled loss value. Then, at step 504, method 500 performs an update operation in the backpropagation based on the scaled loss value. In one embodiment, this update operation may include gradient updates and weight updates. At step 506, method 500 adjusts the scaling factor at least according to the gradient data in the backpropagation for scaling the loss value in the next generation of backpropagation. (Based on the foregoing...) Figure 3 As described above, those skilled in the art will understand that quantization operations are interspersed within the aforementioned update operations, thereby advantageously accelerating the execution of the update operations. Furthermore, the output of step 506 is the adjusted scaling factor, and this scaling factor is fed back to step 502 for further scaling of the loss value. It can be seen that method 500 can be executed repeatedly until the loss value of the neural network reaches the expected value, thereby completing the training of the neural network.

[0119] Based on the above description, those skilled in the art will understand that method 500 can be combined with this disclosure. Figure 2 The described device 200 is used to perform this, therefore combining Figure 2 The description of the specific operation of device 200 also applies to the steps performed by method 500, and will not be repeated here.

[0120] Figure 6 This is a structural diagram illustrating a combined processing apparatus 600 according to an embodiment of the present disclosure. Figure 6 As shown, the combined processing device 600 includes a computing processing device 602, an interface device 604, other processing devices 606, and a storage device 608. Depending on the application scenario, the computing processing device may include one or more computing devices 610, which can be configured to perform the functions described herein. Figure 1-5 The operations described herein. In particular, in some application scenarios, computing device 610 may include components disclosed herein. Figure 2 The described device 200, and performs the combination Figure 5 The steps described.

[0121] In different embodiments, the computing processing apparatus disclosed herein can be configured to perform user-specified operations. In exemplary applications, the computing processing apparatus can be implemented as a single-core artificial intelligence processor or a multi-core artificial intelligence processor. Similarly, one or more computing devices included within the computing processing apparatus can be implemented as an artificial intelligence processor core or a portion of the hardware structure or circuitry of an artificial intelligence processor core to implement, for example, various circuits disclosed herein, such as scaling circuits, updating circuits, quantization circuits, or adjustment circuits. When multiple computing devices are implemented as artificial intelligence processor cores or portions of the hardware structure of artificial intelligence processor cores, the computing processing apparatus disclosed herein can be considered to have a single-core structure or a homogeneous multi-core structure.

[0122] In exemplary operation, the computing processing device disclosed herein can interact with other processing devices through an interface device to jointly complete user-specified operations. Depending on the implementation, the other processing devices disclosed herein may include one or more types of processors such as a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and an artificial intelligence processor, both general-purpose and / or special-purpose processors. These processors may include, but are not limited to, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, the computing processing device disclosed herein can be considered to have a single-core structure or a homogeneous multi-core structure. However, when the computing processing device and other processing devices are considered together, they can be considered to form a heterogeneous multi-core structure.

[0123] In one or more embodiments, the other processing device may serve as an interface between the computing processing device disclosed herein (which may be specifically embodied in artificial intelligence, such as neural network operations) and external data and control, performing basic controls including but not limited to data transfer, starting and / or stopping the computing device. In another embodiment, the other processing device may also cooperate with the computing processing device to jointly complete computational tasks.

[0124] In one or more embodiments, the interface device can be used to transfer data and control commands between a computing processing device and other processing devices. For example, the computing processing device can obtain input data from other processing devices via the interface device and write it to on-chip storage (or memory) of the computing processing device. Further, the computing processing device can obtain control commands from other processing devices via the interface device and write them to on-chip control cache of the computing processing device. Alternatively or optionally, the interface device can also read data from the storage device of the computing processing device and transmit it to other processing devices.

[0125] Additionally or optionally, the combined processing apparatus disclosed herein may further include a storage device. As shown in the figures, the storage device is connected to both the computational processing device and the other processing device. In one or more embodiments, the storage device may be used to store data from the computational processing device and / or the other processing device, such data as the operational data disclosed herein, including but not limited to pre-quantized or post-quantized neuron data, weight data, and / or gradient data. In some embodiments, this data may be data that cannot be fully stored in the internal or on-chip storage of the computational processing device or other processing device.

[0126] In some embodiments, this disclosure also discloses a chip (e.g. Figure 7 The chip shown is 702. In one implementation, the chip is a system-on-chip (SoC) and integrates one or more such... Figure 6 The combined processing unit shown is illustrated. This chip can be connected to external interface devices (such as...). Figure 7 The external interface device 706 shown is connected to other related components. These related components may be, for example, a camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. In some applications, the chip may integrate other processing units (e.g., video codecs) and / or interface modules (e.g., DRAM interfaces). In some embodiments, this disclosure also discloses a chip package structure that includes the aforementioned chip. In some embodiments, this disclosure also discloses a board that includes the aforementioned chip package structure. The following will be combined with… Figure 7 This board is described in detail.

[0127] Figure 7 This is a schematic diagram illustrating the structure of a board 700 according to an embodiment of this disclosure. For example... Figure 7As shown, the board includes a storage device 704 for storing data, which includes one or more storage cells 710. This storage device can be connected and transmit data with the controller 708 and the aforementioned chip 702 via, for example, a bus. Furthermore, the board also includes an external interface device 706 configured for data relay or switching between the chip (or a chip in a chip package) and an external device 712 (e.g., a server or computer). For example, data to be processed can be transferred from the external device to the chip via the external interface device. Alternatively, the calculation results of the chip can be transmitted back to the external device via the external interface device. Depending on the application scenario, the external interface device can have different interface forms; for example, it can adopt a standard PCIe interface.

[0128] In one or more embodiments, the controller in the disclosed board can be configured to regulate the state of the chip. Therefore, in one application scenario, the controller may include a microcontroller (MCU) for regulating the operating state of the chip.

[0129] Based on the above combination Figure 6 and Figure 7 Based on the description, those skilled in the art will understand that this disclosure also discloses an electronic device or apparatus that may include one or more of the aforementioned boards, one or more of the aforementioned chips, and / or one or more of the aforementioned combined processing apparatus.

[0130] Depending on the application scenario, the electronic devices or apparatus disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablets, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The electronic devices or apparatus disclosed herein can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus disclosed herein can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing, such as cloud computing, edge computing, and terminal applications. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present disclosure can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling and collaborative work of end-to-cloud or cloud-edge-end integration.

[0131] It should be noted that, for the sake of brevity, this disclosure describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solutions disclosed herein are not limited by the order of the described actions. Therefore, based on the disclosure or teachings of this document, those skilled in the art will understand that some steps can be performed in a different order or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this disclosure can be considered optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions disclosed herein. In addition, depending on the solution, the description of some embodiments in this disclosure may have different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this disclosure can also be referred to the relevant descriptions of other embodiments.

[0132] In terms of specific implementation, based on the disclosure and teachings of this document, those skilled in the art will understand that several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the electronic device or apparatus embodiments described above, this document divides them based on logical functions, but in actual implementation, there may be other division methods. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationships between different units or components, the connections discussed above in conjunction with the accompanying drawings can be direct or indirect couplings between units or components. In some scenarios, the aforementioned direct or indirect couplings involve communication connections utilizing interfaces, where the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0133] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this disclosure. Additionally, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically independently.

[0134] In some implementation scenarios, the integrated unit described above can be implemented as a software program module. If implemented as a software program module and sold or used as an independent product, the integrated unit can be stored in a computer-readable storage device (CMSDD). Therefore, when the disclosed solution is embodied in a software product (e.g., a computer-readable storage medium), the software product can be stored in a memory, which may include several instructions to cause a computer device (e.g., a personal computer, server, or network device) to execute some or all of the steps of the method described in the embodiments of this disclosure. The aforementioned memory may include, but is not limited to, various media capable of storing program code, such as USB flash drives, flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0135] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the circuit's hardware structure may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as CPUs, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage units or storage devices can be any suitable storage medium (including magnetic storage media or magneto-optical storage media), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), ROM, and RAM.

[0136] While numerous embodiments of this disclosure have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and intent of this disclosure. It should be understood that various alternatives to the embodiments of this disclosure described herein may be employed in the practice of this disclosure. The appended claims are intended to define the scope of this disclosure and therefore cover equivalents or alternatives within the scope of these claims.

[0137] The foregoing can be better understood in accordance with the following terms:

[0138] Clause A1. An apparatus for training a neural network, wherein training the neural network includes iteratively performed forward propagation and backward propagation, the apparatus comprising:

[0139] A scaling circuit configured to scale the loss value obtained from the forward propagation according to a scaling factor to obtain a scaled loss value;

[0140] An update circuit configured to perform the update operation during backpropagation based on the scaled loss value; and

[0141] An adjustment circuit is configured to adjust the scaling factor based at least on the gradient data in the backpropagation, so as to scale the loss value in the next generation of backpropagation.

[0142] Clause A2, the device according to Clause A1, wherein said device further includes quantization circuitry configured to:

[0143] Perform a quantization operation on the operation data involved in the update operation according to the quantization parameters; and

[0144] The decision to update the quantization parameters is based on the operational data.

[0145] Clause A3, the device according to Clause A1 or A2, wherein the operating data includes the gradient data, the quantization parameters include a first point position parameter or a first bit width parameter, and the adjustment circuitry is configured to adjust the scaling factor for scaling the loss value in next-generation backpropagation when the first point position parameter or the first bit width parameter is updated.

[0146] Clause A4, the device according to any one of Clauses A1-A3, wherein the operational data includes the gradient data and neuron data, the quantization parameters include a second point position parameter or a second bit width parameter, and the adjustment circuitry is configured to adjust the scaling factor for scaling the loss value in next-generation backpropagation when the second point position parameter or the second bit width parameter is updated.

[0147] Clause A5, the device according to any one of Clauses A1-A4, wherein the operational data includes the gradient data, neuron data, and weight data, the quantization parameter includes a third point position parameter or a third bit width parameter, and the adjustment circuitry is configured to adjust the scaling factor for scaling the loss value in next-generation backpropagation when the third point position parameter or the third bit width parameter is updated.

[0148] Clause A6. The device according to any one of Clauses A1-A5, wherein the quantization circuit includes a positive quantization circuit and an inverse quantization circuit, wherein the positive quantization circuit is configured to quantize high-precision data type operation data into low-precision data type operation data according to the quantization parameters, and the inverse quantization circuit is configured to inverse quantize low-precision data type operation data into high-precision data type operation data according to the quantization parameters.

[0149] Clause A7. The device according to any one of Clauses A1-A6, wherein the high-precision data type is a floating-point data type and the low-precision data type is a fixed-point data type.

[0150] Clause 8. The apparatus according to any one of Clauses A1-A7, wherein the update circuit is configured to obtain weight gradient data in the gradient data using the scaled loss value, and the dequantization circuit is configured to dequantize the weight gradient data into weight gradient data of a high-precision data type, and the update circuit is configured to update the weights using the scaling factor and the weight gradient data of the high-precision data type.

[0151] Clause 9. The device according to any one of clauses A1-A8, wherein the adjustment circuitry is configured to determine the scaling factor based on the data distribution of the gradient data.

[0152] Clause 10. The device according to any one of clauses A1-A9, wherein the adjustment circuit is configured to determine the scaling factor based on the maximum value of the gradient data and a preset threshold.

[0153] Clause A11, the device according to any one of clauses A1-A10, wherein the quantization circuit is configured to determine whether to update the quantization parameters based on the quantization error of the operating data.

[0154] Clause A12, the device according to any one of clauses A1-A11, wherein the quantization circuit is configured to perform a calculation based on the mean of the operating data before and after quantization to determine the quantization error.

[0155] Clause A13, the device according to any one of Clauses A1-A12, wherein the neural network comprises a multi-layer structure formed by the connection of a plurality of neurons, and wherein the quantization circuitry is configured to determine, for each layer, whether to update the quantization parameters, and when the quantization circuitry determines to update the quantization parameters at any layer, the adjustment circuitry is configured to dynamically adjust the scaling factor for scaling the loss value in next-generation backpropagation.

[0156] Clause A14, an integrated circuit comprising the device described in any one of clauses A1-A13.

[0157] Clause A15, a board that includes the device described in any one of clauses A1-A14.

[0158] Clause A16. A method for training a neural network, wherein training the neural network includes iteratively performed forward propagation and backward propagation, the method comprising:

[0159] The loss value obtained from the forward propagation is scaled according to a scaling factor to obtain a scaled loss value;

[0160] The update operation in the backpropagation is performed based on the scaled loss value; and

[0161] The scaling factor is adjusted at least based on the gradient data in the backpropagation so as to scale the loss value in the next generation of backpropagation.

[0162] Clause A17. The method according to Clause A16, wherein the method further comprises performing a quantization operation, the quantization operation including:

[0163] Quantization is performed on the operational data involved in the update operation according to the quantization parameters; and

[0164] The decision to update the quantization parameters is based on the operational data.

[0165] Clause A18, the method according to Clause A16 or A17, wherein the operational data includes the gradient data, the quantization parameter includes a first point position parameter or a first width parameter, and the adjustment includes:

[0166] When the first point position parameter or the first bit width parameter is updated, the scaling factor is adjusted to scale the loss value in the next-generation backpropagation.

[0167] Clause A19, the method according to any one of Clauses A16-A18, wherein the operational data includes the gradient data and the neuron data, the quantization parameter includes a second point position parameter or a second bit width parameter, and the adjustment includes:

[0168] When the second point position parameter or the second bit width parameter is updated, the scaling factor is adjusted to scale the loss value in the next-generation backpropagation.

[0169] Clause A20, the method according to any one of Clauses A16-A19, wherein the operational data includes the gradient data, neuron data, and weight data, the quantization parameter includes a third point position parameter or a third bit width parameter, and the adjustment includes:

[0170] When the third point position parameter or the third bit width parameter is updated, the scaling factor is adjusted to scale the loss value in the next-generation backpropagation.

[0171] Clause A21. The method according to any one of Clauses A16-A20, wherein the quantization operation includes a positive quantization operation and a negative quantization operation, wherein the positive quantization operation includes quantizing operation data of a high-precision data type into operation data of a low-precision data type according to the quantization parameters, and the negative quantization operation includes dequantizing operation data of a low-precision data type into operation data of a high-precision data type according to the quantization parameters.

[0172] Clause A22, the method according to any one of Clauses A16-A21, wherein the high-precision data type is a floating-point data type and the low-precision data type is a fixed-point data type.

[0173] Clause A23, the method according to any one of Clauses A16-A22, wherein the update operation in the backpropagation includes obtaining weight gradient data in the gradient data using the scaled loss value, and the dequantization operation includes dequantizing the weight gradient data into weight gradient data of a high-precision data type, and the update operation in the backpropagation includes updating the weights using the scaling factor and the weight gradient data of the high-precision data type.

[0174] Clause A24, the method of any one of Clauses A16-A23, wherein the adjustment includes determining the scaling factor based on the data distribution of the gradient data.

[0175] Clause A25, the method of any one of Clauses A16-A24, wherein the adjustment includes determining the scaling factor based on the maximum value of the gradient data and a preset threshold.

[0176] Clause A26, the method of any one of Clauses A16-A25, wherein the quantization includes determining whether to update the quantization parameter based on the quantization error of the operating data.

[0177] Clause A27, the method according to any one of Clauses A16-A26, wherein the quantization includes performing a calculation based on the mean of the operational data before and after quantization to determine the quantization error.

[0178] Clause A28, the method according to any one of clauses A16-A27, wherein the neural network comprises a multi-layered structure formed by the connection of a plurality of neurons, and wherein the method comprises:

[0179] For each layer, determine whether to update the quantization parameters; and

[0180] When the quantization parameter is determined to be updated at any layer, the scaling factor is dynamically adjusted to scale the loss value in the next-generation backpropagation.

[0181] Clause A29. A computer-readable storage medium storing a computer program for training a neural network, which, when run by one or more processors, implements the method according to any one of claims A16-28.

[0182] Clause A30. An apparatus for training a neural network, comprising: at least one processor; at least one memory storing computer program code, said at least one memory and said computer program code being configured to utilize said processor to cause the apparatus to perform the method according to any one of Clauses A16-A28.

[0183] It should be understood that the terms "first," "second," "third," and "fourth," etc., in the claims, specification, and drawings of this disclosure are used to distinguish different objects, not to describe a specific order. The terms "comprising" and "including" as used in the specification and claims of this disclosure indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0184] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0185] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0186] The embodiments of this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this disclosure. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of this disclosure, and on the specific implementation methods and application scope of this disclosure, are all within the scope of protection of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

Claims

1. An apparatus for training a neural network, wherein training the neural network includes iteratively performed forward propagation and backward propagation, the apparatus comprising: A scaling circuit configured to scale the loss value obtained from the forward propagation according to a scaling factor to obtain a scaled loss value; An update circuit is configured to perform the update operation in the backpropagation based on the scaled loss value; as well as An adjustment circuit is configured to adjust the scaling factor based at least on the gradient data in the backpropagation, so as to scale the loss value in the next generation of backpropagation; The application areas of the neural network include: image processing and speech recognition.

2. The device according to claim 1, wherein the device further comprises a quantization circuit configured to: Perform a quantization operation on the operation data involved in the update operation according to the quantization parameters; and The decision to update the quantization parameters is based on the operational data.

3. The device of claim 2, wherein the operational data includes the gradient data, the quantization parameter includes a first point position parameter or a first bit width parameter, and the adjustment circuitry is configured to adjust the scaling factor for scaling the loss value in next-generation backpropagation when the first point position parameter or the first bit width parameter is updated.

4. The device of claim 2, wherein the operational data includes the gradient data and neuron data, the quantization parameter includes a second point position parameter or a second bit width parameter, and the adjustment circuitry is configured to adjust the scaling factor for scaling the loss value in next-generation backpropagation when the second point position parameter or the second bit width parameter is updated.

5. The device of claim 2, wherein the operational data includes the gradient data, neuron data, and weight data, the quantization parameter includes a third point position parameter or a third bit width parameter, and the adjustment circuitry is configured to adjust the scaling factor for scaling the loss value in next-generation backpropagation when the third point position parameter or the third bit width parameter is updated.

6. The device according to claim 2, wherein the quantization circuit includes a positive quantization circuit and a negative quantization circuit, wherein the positive quantization circuit is configured to quantize high-precision data type operation data into low-precision data type operation data according to the quantization parameters, and the negative quantization circuit is configured to negative quantize low-precision data type operation data into high-precision data type operation data according to the quantization parameters.

7. The device according to claim 6, wherein the high-precision data type is a floating-point data type and the low-precision data type is a fixed-point data type.

8. The apparatus of claim 6, wherein the update circuit is configured to obtain weight gradient data in the gradient data using the scaled loss value, and the dequantization circuit is configured to dequantize the weight gradient data into weight gradient data of a high-precision data type, and the update circuit is configured to update the weights using the scaling factor and the weight gradient data of the high-precision data type.

9. The device of claim 1, wherein the adjustment circuit is configured to determine the scaling factor based on the data distribution of the gradient data.

10. The device of claim 1, wherein the adjustment circuit is configured to determine the scaling factor based on the maximum value of the gradient data and a preset threshold.

11. The device of claim 2, wherein the quantization circuit is configured to determine whether to update the quantization parameters based on the quantization error of the operating data.

12. The apparatus of claim 11, wherein the quantization circuit is configured to perform a calculation based on the mean of the operating data before and after quantization to determine the quantization error.

13. The device according to any one of claims 2-8, 11-12, wherein the neural network comprises a multilayer structure formed by connecting a plurality of neurons, and wherein the quantization circuit is configured to determine, for each layer, whether to update the quantization parameter, and when the quantization circuit determines to update the quantization parameter at any layer, the adjustment circuit is configured to dynamically adjust the scaling factor for scaling the loss value in next-generation backpropagation.

14. An integrated circuit comprising the device according to any one of claims 1-13.

15. A board comprising the device according to any one of claims 1-13.

16. A method for training a neural network, wherein training the neural network includes iteratively performed forward propagation and backward propagation, the method comprising: The loss value obtained from the forward propagation is scaled according to a scaling factor to obtain a scaled loss value; The update operation in the backpropagation is performed based on the scaled loss value; as well as The scaling factor is adjusted at least based on the gradient data in the backpropagation so that it can be used for scaling the loss value in the next generation of backpropagation; The application areas of the neural network include: image processing and speech recognition.

17. The method of claim 16, wherein the method further comprises performing a quantization operation, the quantization operation comprising: Quantization is performed on the operation data involved in the update operation according to the quantization parameters; as well as The decision to update the quantization parameters is based on the operational data.

18. The method of claim 17, wherein the operational data includes the gradient data, the quantization parameter includes a first point position parameter or a first width parameter, and the adjustment includes: When the first point position parameter or the first bit width parameter is updated, the scaling factor is adjusted to scale the loss value in the next-generation backpropagation.

19. The method of claim 17, wherein the operational data includes the gradient data and the neuron data, the quantization parameter includes a second point position parameter or a second bit width parameter, and the adjustment includes: When the second point position parameter or the second bit width parameter is updated, the scaling factor is adjusted to scale the loss value in the next-generation backpropagation.

20. The method of claim 17, wherein the operational data includes the gradient data, neuron data, and weight data, the quantization parameter includes a third point position parameter or a third bit width parameter, and the adjustment includes: When the third point position parameter or the third bit width parameter is updated, the scaling factor is adjusted to scale the loss value in the next-generation backpropagation.

21. The method according to claim 17, wherein the quantization operation includes a positive quantization operation and a negative quantization operation, wherein the positive quantization operation includes quantizing operation data of a high-precision data type into operation data of a low-precision data type according to the quantization parameters, and the negative quantization operation includes negative quantizing operation data of a low-precision data type into operation data of a high-precision data type according to the quantization parameters.

22. The method of claim 21, wherein the high-precision data type is a floating-point data type and the low-precision data type is a fixed-point data type.

23. The method of claim 21, wherein the update operation in backpropagation includes obtaining weight gradient data in the gradient data using the scaled loss value, and the dequantization operation includes dequantizing the weight gradient data into weight gradient data of a high-precision data type, and the update operation in backpropagation includes updating the weights using the scaling factor and the weight gradient data of the high-precision data type.

24. The method of claim 16, wherein the adjustment includes determining the scaling factor based on the data distribution of the gradient data.

25. The method of claim 16, wherein the adjustment includes determining the scaling factor based on the maximum value of the gradient data and a preset threshold.

26. The method of claim 17, wherein the quantization includes determining whether to update the quantization parameters based on the quantization error of the operating data.

27. The method of claim 26, wherein the quantization comprises performing a calculation based on the mean of the operational data before and after quantization to determine the quantization error.

28. The method according to any one of claims 17-23, 26-27, wherein the neural network comprises a multi-layer structure formed by connecting a plurality of neurons, and wherein the method comprises: For each layer, determine whether to update the quantization parameters; as well as When the quantization parameter is determined to be updated at any layer, the scaling factor is dynamically adjusted to scale the loss value in the next-generation backpropagation.

29. A computer-readable storage medium storing a computer program for training a neural network, which, when run by one or more processors, implements the method according to any one of claims 16-28.

30. An apparatus for training a neural network, comprising: At least one processor; At least one memory storing computer program code, the at least one memory and the computer program code being configured to utilize the processor to cause the device to perform the method according to any one of claims 16-28.

Images