Methods, apparatuses, and computer-readable storage media for training a neural network

Abstract

The present disclosure discloses devices, methods and integrated circuit boards for training neural networks, wherein the devices are embodied in computing devices included in a combined processing apparatus that can also include general-purpose interconnect interfaces and other processing devices. The computing devices interact with the other processing devices to collectively perform user-specified computing operations. The combined processing apparatus can also include storage devices connected to the computing devices and the other processing devices, respectively, for data of the computing devices and the other processing devices. The present disclosure can accelerate training of neural networks.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to the field of artificial intelligence. More specifically, the present disclosure relates to methods, devices, integrated circuits, boards, and computer-readable storage media for training a neural network by a hardware platform. BACKGROUND

[0002] With the continuous development of the field of artificial intelligence, how to efficiently train a neural network to obtain a good neural network model has become a focus of attention. Existing neural networks usually use floating-point data to perform operations in training to obtain good training results. Although floating-point data has relatively high data precision, it puts higher hardware requirements on the hardware platform running the neural network during training, such as larger storage space and higher power consumption. In addition, in some training scenarios, the use of fixed-point data with relatively low precision can also achieve the same or similar training effect as floating-point data, so that the application of floating-point data is not necessary in some cases.

[0003] In addition, the data range that can be represented by relatively low bit-width data is limited. For example, the minimum precision value that can be represented by 16-bit floating-point data is 5.9604x10 -8 However, when performing backpropagation in neural network training, it is often possible to obtain a value less than 5.9604x10 -8 In other words, the value obtained is beyond the range that can be represented by a 16-bit floating-point number. At this time, if the 16-bit floating-point number is still used to represent the value, the value is 0. Obviously, such zeroing operation will have a significant impact on the accuracy and precision of the calculation. SUMMARY

[0004] To solve some or all of the problems mentioned above, a way of efficiently training a neural network is provided, and the present disclosure provides the following technical solutions in various aspects.

[0005] In one aspect, the present disclosure provides a device for training a neural network, wherein training the neural network includes forward propagation and backpropagation performed iteratively, the device comprising: a scaling circuit configured to scale a loss value obtained from the forward propagation according to a scaling factor to obtain a scaled loss value; an update circuit configured to perform an update operation in the backpropagation based on the scaled loss value; a quantization circuit configured to perform a quantization operation on operation data according to a quantization parameter, and determine whether to update the quantization parameter based on the operation data; an adjustment circuit configured to adjust the scaling factor for scaling the loss value in the next generation of backpropagation when the quantization circuit determines to update the quantization parameter.

[0006] In yet another aspect, the disclosure provides an integrated circuit comprising the apparatus as described above and to be discussed in the embodiments below.

[0007] In yet another aspect, the disclosure provides a board card comprising the apparatus as described above and to be discussed in the embodiments below.

[0008] In another aspect, the disclosure provides a method for training a neural network, wherein training the neural network comprises forward propagation and backward propagation performed iteratively, the method comprising: scaling a loss value obtained from the forward propagation according to a scaling factor to obtain a scaled loss value; performing an update operation in the backward propagation based on the scaled loss value; performing a quantization operation on operation data according to a quantization parameter, and determining whether to update the quantization parameter based on the operation data; and adjusting the scaling factor for scaling the loss value in the next generation of backward propagation when it is determined to update the quantization parameter.

[0009] In yet another aspect, the disclosure provides an apparatus for training a neural network. The apparatus comprises at least one processor. The apparatus further comprises at least one memory storing computer program code, the at least one memory and the computer program code configured to, with the processor, cause the apparatus to perform the method as described above and the embodiments of the method described below.

[0010] In one aspect, the disclosure provides a computer-readable storage medium storing a computer program for training a neural network, which, when executed by one or more processors, implements the method as described above and the embodiments of the method described below.

[0011] By means of the device, method, integrated circuit, board and computer readable storage medium for training a neural network as described above, in the back propagation of training the neural network, the size of the loss value can be adaptively or timely adjusted, so as to avoid the influence of the result error caused by the too small data expression range of the low-precision data type when the low-precision type of operation data is used for back updating operation, thereby improving the accuracy and precision of the operation. In addition, due to the adjustment of the loss value by the present scheme, it also makes it more applicable to quantize high-precision data into low-precision data to participate in the operation, thereby expanding the operation scene of the neural network. Further, since the present scheme supports quantizing high-precision data into low-precision data to participate in the relevant operation within the neural network, it also makes the calculation scene not limited by the number of operation bits that can be supported by the processor chip, thereby expanding the use scene of the processor. In addition, due to the use of low-precision data type (for example, using fixed-point numbers to perform relevant operations of the neural network such as multiplication and addition operations), the training process of the neural network is also accelerated and the calculation overhead and power consumption are reduced by the present scheme. In addition, the neural network trained by the present scheme can be widely used in various fields such as image processing, speech recognition, data acquisition, etc., which greatly improves the efficiency and cost of the related fields. BRIEF DESCRIPTION OF DRAWINGS

[0012] The above-mentioned features of the present application can be better understood by the following description in conjunction with the accompanying drawings, in which the same reference numerals indicate the same elements and in which:

[0013] Figure 1 is an exemplary block diagram illustrating a neural network to which the technical scheme of the present disclosure can be applied;

[0014] Figure 2 is a functional block diagram of a device for training a neural network according to an embodiment of the present disclosure;

[0015] Figure 3 is an exemplary flowchart illustrating forward propagation and update operation in back propagation in a neural network according to an embodiment of the present disclosure;

[0016] Figure 4 is a curve diagram illustrating the principle of quantization error according to an embodiment of the present disclosure;

[0017] Figure 5 is a flowchart of a method for training a neural network according to an embodiment of the present disclosure;

[0018] Figure 6 is a structural diagram of a combination processing device according to an embodiment of the present disclosure; and

[0019] Figure 7FIG. 1 is a structural schematic diagram illustrating a board card according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] Embodiments of the present application will now be described with reference to the accompanying drawings. It should be understood that the drawings are not necessarily to scale and that, for simplicity and clarity, the drawings have not necessarily been drawn to scale so as to illustrate certain aspects of the embodiments. In addition, the application will be described in connection with the exemplary embodiments illustrated in the drawings and the following description. However, it should be understood that there are many alternative ways of implementing the application. The application is described, however, in the context of the foregoing embodiments for the purpose of illustrating the general principles of the application. The application should be appreciated that the application is adapted to do have broader applications not specifically described herein. Those of ordinary skill in the art will recognize immediately upon reading this description that the application can be practiced with various claims not necessarily encompassed by this description. Accordingly, the application is not to be limited in scope by the embodiments described but is instead defined by the following claims.

[0021] As mentioned before, the scheme of the present disclosure is mainly applied in the field of artificial intelligence, especially in the efficient training of neural networks. Therefore, in order to facilitate the understanding of the scheme of the present disclosure, the neural network architecture and its working principle involved in the present disclosure will be introduced first.

[0022] A neural network ("Neural Network", abbreviated as "NN") is a mathematical model that simulates the structure and function of a biological neural network. The neural network is calculated by a large number of neuron connections. Therefore, the neural network is a computational model composed of a large number of nodes (or "neurons") connected to each other. Each node represents a specific output function, called activation function. The connection between each two neurons represents a weighted value of the signal passing through the connection, called weight, which is equivalent to the memory of the neural network. The output of the neural network depends on the connection between the neurons and the difference in weight and activation function. In a neural network, a neuron is the basic unit of a neural network. It obtains a certain number of inputs and a bias, which is multiplied by a weight when the signal (value) arrives. The connection is to connect a neuron to another neuron in the same layer or another layer, and the connection is accompanied by a weight associated with it. In addition, the bias is an additional input to the neuron, which is always 1 and has its own connection weight. This ensures that even if all inputs are empty (all 0), the neuron will be activated.

[0023] In applications, if no non-linear function is applied to the neurons in the neural network, the neural network is just a linear function and it is not more powerful than a single neuron. If the output of a neural network is to be between 0 and 1, for example, in the cat-dog discrimination example, an output close to 0 can be considered as a cat and an output close to 1 can be considered as a dog. To achieve this goal, an activation function, such as a sigmoid activation function, is introduced in the neural network. The return value of the activation function is usually a number between 0 and 1. Therefore, the activation function is used to introduce non-linearity into the neural network, which shrinks the result of the neural network operation into a smaller range. In fact, how the activation function is expressed is not important, what is important is that a non-linear function is parameterized by some weights, which can be changed to change the non-linear function.

[0024] Figure 1 is an exemplary block diagram showing a neural network 100 to which the technical solution of the present disclosure can be applied. As shown in Figure 1 , the neural network 100 includes an input layer and an output layer and a plurality of hidden layers between the input layer and the output layer, which are exemplarily shown as a convolutional layer, an activation layer, a pooling layer and a fully connected layer in the figure.

[0025] The neurons of the input layer are called input neurons, and in this example, 3 input neurons are drawn, which receive 3 input signals x1, x2, x3. The input layer, as the first layer in the neural network, accepts the input signals (values) that need to be input and passes them to the next layer. Generally, the input layer does not operate on the input signals (values) and has no associated weights and biases. For a specific neural network, such as a convolutional neural network, the input layer can process multi-dimensional data. Commonly, the input layer of a one-dimensional convolutional neural network receives a one-dimensional or two-dimensional array, where the one-dimensional array is usually a time or frequency sample; the two-dimensional array can contain multiple channels; the input layer of a two-dimensional convolutional neural network receives a two-dimensional or three-dimensional array; the input layer of a three-dimensional convolutional neural network receives a four-dimensional array, and so on. In some specific application scenarios, preprocessing operations can also be performed on the data at the input layer, such as mean removal, normalization and dimensionality reduction operations, etc.

[0026] The hidden layers contain neurons (nodes) for applying different transformations to the input data. In Figure 1 , the neural network includes four hidden layers, i.e., a convolutional layer including 4 neurons (nodes), an activation layer including 4 neurons, a pooling layer including 2 neurons, and a fully connected layer including 6 neurons. Finally, the operation values of the fully connected layer are passed to the output layer. The neurons of the output layer are called output neurons. The output layer receives the output from the last hidden layer. In Figure 1In the neural network shown, the output layer has two neurons and two output signals y1 and y2. As can be seen from the hidden layers, each neuron in each hidden layer may or may not be connected to any neuron in the next layer, depending on the specific hidden layer. For example, neurons in the activation layer and pooling layer are partially connected, while those in the pooling layer and the fully connected layer are fully connected.

[0027] The exemplary hidden layers in this example are briefly described below. It should be understood that the descriptions of the various hidden layers described above are merely exemplary and not restrictive, and the technical solutions disclosed herein are not intended to limit the scope of the examples. Figure 1 The limitations of the hidden layer structure of the neural network shown are explained, and those skilled in the art can, based on the teachings of this disclosure, [address the limitations of the hidden layer structure]. Figure 1 The neural network structure shown can be modified, for example, by adding one or more layers or removing layers as needed for the application. Figure 1 One or more layers in the structure shown, and these operations are still covered within the technical solutions covered by this disclosure.

[0028] As the first hidden layer in this example—the convolutional layer—its function is typically to extract features from the input data. It can contain multiple convolutional kernels, each element of which corresponds to a weight coefficient and a bias, similar to a neuron in a feedforward neural network. When processing image data, in the convolutional layer, each feature in the image is first perceived locally, and then a higher-level synthesis operation is performed on the local features to obtain global information. Convolutional layer parameters include kernel size, stride, and padding, which together determine the size of the output feature map and are hyperparameters of the convolutional neural network. In application, each neuron in the convolutional layer is connected to multiple neurons in a region located close to it in the previous layer; the size of this region depends on the kernel size. During operation, the convolutional kernel systematically scans the input features, performing element-wise multiplication (multiply-add) on the input features and adding the bias.

[0029] The activation layer that receives the output of the convolutional layer essentially performs a non-linear mapping on the output. Commonly used activation functions include: Sigmoid, Tanh, ReLU, Leaky, ELU, and Maxout. After applying these activation functions, the output of the previous layer becomes more complex, thereby enhancing the expressive power of the neural network model.

[0030] The pooling layer is mainly used for feature dimension reduction, compressing the number of data and parameters, reducing overfitting, and improving the fault tolerance of the model. Generally, the pooling method mainly includes maximum pooling and average pooling. After the feature extraction by the convolution layer and the processing by the activation layer, the output feature map is transmitted to the pooling layer for feature selection and information filtering. The pooling layer contains a preset pooling function, which replaces the result of a single point in the feature map with the statistical quantity of the adjacent region of the feature map. The pooling layer selects the pooling region and scans the feature map with the convolution kernel in the same step, which can be controlled by the pooling size, step, and padding.

[0031] After the foregoing convolution + activation + pooling, the signal processing procedure of the neural network reaches the fully connected layer, which is located at the last part of the hidden layer in the example neural network. The feature map loses the spatial topology structure in the fully connected layer, is unfolded as a vector, and is output through an excitation function. The fully connected layer can perform nonlinear combination on the extracted features to obtain the output. The fully connected layer itself is not expected to have feature extraction capability, but attempts to use the existing high-order features to complete the learning goal. In addition, local normalization (LRN), data enhancement, and other operations can be performed in the fully connected layer to increase the robustness of the neural network.

[0032] Although not shown in Figure 1 , at each layer in the neural network, there is one or more operators associated with the layer (to be described in detail in connection with Figure 2 ) to perform corresponding computing operations. An operator is a mapping from a function space to a function space in the neural network. Broadly speaking, any operation on a function can be considered an operator. In short, an operator can be a mapping, a relationship, or a transformation. For example, there is a convolution operator for the convolution layer (or other layers that need to perform convolution operations), which can be embodied as an expression of one or more convolution calculation formulas. By using the convolution operator to calculate the input data and the convolution kernel, the result value after the convolution operation can be obtained.

[0033] The foregoing exemplary description is made for the neural network structure and the functions of the nodes thereof shown in Figure 1 . In actual applications, in order to obtain a good neural network model, a large amount of sample data (including input and output) is provided in advance to train the initial neural network. After the training is completed, the trained neural network is obtained. The trained neural network can give a correct output for the input of the real environment in the future.

[0034] Before discussing the training of a neural network, a loss function needs to be defined. The loss function is a function that measures the performance of a neural network in performing a certain task. In some embodiments, the loss function can be obtained as follows: during the training of a neural network, for each sample data, the output value is obtained by passing the sample data through the neural network, and then the output value is subtracted from the expected value and squared. The loss function thus calculated is the distance between the predicted value and the true value, and the purpose of training the neural network is to reduce the value of the distance or loss function. In some embodiments, the loss function can be expressed as:

[0035]

[0036] In the above formula, y represents the expected value, the actual result obtained by each sample data in the sample data set through the neural network, and i is the index of each sample data in the sample data set. represents the error value between the expected value y and the actual result . m is the number of sample data in the sample data set.

[0037] Taking the actual application scenario of cat and dog identification as an example. Assume that a data set consists of pictures of cats and dogs. If the picture is a dog, the corresponding label is 1, and if the picture is a cat, the corresponding label is 0. This label is the expected value y in the above formula. When each sample picture is passed to the neural network, the actual purpose is to obtain the identification result of the animal in the picture, i.e., whether it is a cat or a dog. In order to calculate the loss function, each sample picture in the sample data set must be traversed to obtain the actual result of each sample picture, and then the loss function is calculated according to the above definition. If the value of the loss function (referred to as the “loss value” for short) is relatively large, for example, exceeds a predetermined threshold, it indicates that the neural network has not been trained well, and at this time, the weight values need to be further adjusted by means of the aforementioned back propagation process.

[0038] When starting to train the neural network, the weight values need to be randomly initialized. In most cases, the initialized neural network cannot provide a good training result. During the training process, it is assumed that a very poor neural network is started, and through training, a network with high accuracy can be obtained.

[0039] The training process of the neural network is divided into two stages. The first stage is the forward processing operation of the signal (referred to as the “forward propagation process” in the present disclosure), and the training passes from the input layer through the hidden layer and finally reaches the output layer. The second stage is the gradient operation of the back propagation (referred to as the “backward propagation process” in the present disclosure), and the training passes from the output layer to the hidden layer and finally to the input layer, and the weight values and biases of each layer of the neural network are adjusted according to the gradient.

[0040] In the forward propagation process, an input value is input to the input layer of the neural network, and a corresponding operation performed by a plurality of related operators of the hidden layers can obtain an output of a so-called prediction value from the output layer of the neural network. When the input value is provided to the input layer of the neural network, it can not be operated or some necessary preprocessing can be done according to the application scenario. In the hidden layer, the second hidden layer obtains the prediction intermediate result value from the first hidden layer and performs a calculation operation and an activation operation, and then passes the obtained prediction intermediate result value to the next hidden layer. The same operation is performed in the subsequent layers, and finally the output value is obtained at the output layer of the neural network. After the forward processing in the forward propagation process, an output value called a prediction value can usually be obtained. In order to calculate the error, the prediction value can be compared with the actual output value to obtain the corresponding error value.

[0041] In the backward propagation process, the chain rule of differential calculus can be used to update the weights of each layer in order to obtain a lower error value in the next forward propagation process. In the chain rule, the derivatives of the error value corresponding to the last layer of the neural network are first calculated (these derivatives are called gradients). Then, these gradients are used to calculate the gradients of the second-to-last layer in the neural network. This process is repeated until the gradients corresponding to each weight in the neural network are obtained. Finally, each weight in the neural network is subtracted by the corresponding gradient, thereby updating the weight once to reduce the error value. Similar to the use of various operators (referred to as forward operators in this disclosure) in the forward propagation process, there are corresponding backward operators in the corresponding backward propagation process. For example, for the convolution operator in the convolution layer described above, it includes a forward convolution operator in the forward propagation process and a deconvolution operator in the backward propagation process.

[0042] In the process of training the neural network, the weights in the neural network are updated by gradient once per forward propagation process of the signal forward processing and the corresponding backward propagation process of the error. This is called an iteration. In order to obtain a neural network with expected accuracy, a very large sample data set is needed in the training process, and it is almost impossible to input the sample data set into a computing device (such as a computer) at one time. Therefore, in order to solve this problem, the sample data set needs to be divided into multiple blocks, and each block is delivered to the computer. After the forward propagation process of the forward processing, the weights of the neural network are updated in the backward propagation process. When a complete sample data set is processed once by the neural network and the corresponding weight update is returned, this process is called an epoch. In practice, it is not enough to pass a complete data set in the neural network, and the complete data set needs to be passed in the same neural network multiple times, that is, multiple epochs, to finally obtain a neural network with expected accuracy.

[0043] In the process of training the neural network, the user usually wants to train faster and more accurately, but such expectations are usually affected by the data type of the neural network data. In many application scenarios, the data of the neural network is represented by high-precision data formats (such as floating-point numbers). Taking the convolution operation in the forward propagation process and the reverse convolution operation in the backward propagation process as an example, when these two operations are performed on the central processing unit (“CPU”) and the graphics processing unit (“GPU”) of the computing device, in order to ensure data precision, almost all inputs, weights and gradients are floating-point type data.

[0044] Taking the floating-point type format as an example of a high-precision data format, according to the computer architecture, based on the operation representation rules of floating-point numbers and fixed-point numbers, for floating-point operations and fixed-point operations of the same length, the floating-point operation calculation mode is more complex and requires more logic devices to form a floating-point operation unit. In terms of volume, the volume of the floating-point operation unit is larger than that of the fixed-point operation unit. Further, the floating-point operation unit needs to consume more resources to process, resulting in a significant difference in computing cost between fixed-point operations and floating-point operations. However, according to experiments, fixed-point operations are faster than floating-point operations, and the accuracy loss is not large, so it is feasible to use fixed-point operations to process a large number of neural network operations (such as convolution and full connection operations) in artificial intelligence chips. For example, the floating-point type data of the inputs, weights and gradients related to the forward convolution, forward full connection, reverse convolution and reverse full connection operators can be quantized and then operated by fixed-point numbers, and the low-precision data can be converted into high-precision data after the operator operation is completed.

[0045] Taking an example in which the quantized objects are the weights of a neural network, and the quantized weights are all 8-bit fixed-point numbers (a low-precision type relative to floating-point numbers), since there are often millions of connections in a neural network, almost all the space is occupied by the weights of the neuron connections, and these weights can all be different floating-point numbers. The weights of each layer tend to be normally distributed in a certain interval, for example, (-3.0, 3.0). The maximum and minimum values corresponding to the weights of each layer in the neural network are saved, and each floating-point value is represented by an 8-bit fixed-point number. Among them, 256 quantization intervals are linearly divided in the maximum and minimum value range, and each quantization interval is represented by an 8-bit fixed-point number. For example: in the interval (-3.0, 3.0), byte 0 represents -3.0, and byte 255 represents 3.0. In this way, byte 128 represents 0.

[0046] In the process of performing the quantization operation, two quantization parameters are usually involved: shift and n, where shift is the position of the point of the fixed-point number (i.e., the “point position parameter” of the present disclosure), and n is the bit width of the fixed-point number (i.e., the “bit width parameter” of the present disclosure). n can be initially set artificially, and shift is calculated by using the following formula based on the distribution range of the data to be quantized and n:

[0047]

[0048] where Z is the maximum value max(|F|) of the absolute value of the data F to be quantized. Using F to represent the floating-point data before quantization and I to represent the n-bit fixed-point number after quantization, the fixed-point conversion method from F to I can be calculated by the following formula:

[0049] F≈I×2 shift (3)

[0050] where step = 2 s is the quantization step (the smallest quantization interval), and the obtained fixed-point number I can be expressed by the following formula:

[0051]

[0052] When it is necessary to convert the fixed-point number obtained after quantization into a floating-point number, the dequantization operation can be performed, and the value after dequantization can be expressed by the following formula:

[0053]

[0054] It can be seen that the quantized fixed-point numbers described above are beneficial to the accelerated training of neural networks, the reduction of chip size and the significant reduction of computational overhead. In particular, the quantization operations described above can be performed on neuron data and weight data in the forward propagation of training a neural network, and the quantization operations can be performed on gradient data used for update operations in the backward propagation of training a neural network. When some related operators (to be described in detail later) are introduced in the quantization operations described above, the quantization operations can be optimized so as to make full use of the quantization operations while introducing as little quantization overhead as possible, thereby accelerating the training of neural networks, improving the training accuracy and reducing the computational overhead. Figure 3

[0055] As described above, although the quantization operations described above bring obvious technical advantages to the training of neural networks, in some application scenarios, when the loss value is expressed by a relatively low-precision data type, for example, the 16-bit floating-point number in the previous example (which is relatively low-precision compared with the 32-bit, 64-bit or higher floating-point number), due to the limitation of the numerical expression range, it will inevitably have an adverse effect on the update operation in the backward propagation. Therefore, the present disclosure proposes an effective loss value adjustment scheme to overcome the above defects and improve the training accuracy of neural networks, thereby accelerating the training.

[0056] Figure 2 is a functional block diagram illustrating a device 200 for training a neural network according to an embodiment of the present disclosure. As shown in Figure 2 , the device 200 includes a scaling circuit 202, an update circuit 204, a quantization circuit 206 and an adjustment circuit 208. According to the scheme of the present disclosure, the scaling circuit can be configured to scale the loss value obtained in the forward propagation in the process of training a neural network according to a scaling factor to obtain a scaled loss value. By amplifying the loss value, it can be effectively avoided that the subsequent update operation is wrong due to the limitation of the numerical expression range of the low-precision data, thereby causing the delay and inefficiency of the entire training process. For ease of understanding, it is assumed that the loss value of the present disclosure is represented as loss, which can be scaled by using the following formula:

[0057] Loss_scale = loss x scale (6)

[0058] In the above formula (6), scale represents the scaling factor of the present disclosure, and Loss_scale represents the scaled scaling factor.

[0059] ​After performing the scaling operation on the loss value, the update circuit can be configured to perform an update operation in the backpropagation based on the scaled loss value. In one embodiment, the update operation here can involve an update to the weight values and gradient data updates that are passed from a previous layer to a next layer in the direction of the backpropagation. As previously mentioned, the update operation in the backpropagation involves quantization operations on various types of data, such as the quantization operation from high-precision data (e.g., floating-point numbers) to low-precision data (e.g., fixed-point numbers) and the dequantization operation from low-precision data to high-precision data as previously described. To this end, the quantization circuit of the device 200 can be configured to perform a quantization operation on operation data according to a quantization parameter and determine whether to update the quantization parameter based on the operation data. Further, the adjustment circuit can be configured to adjust the scaling factor for scaling the loss value in the next generation of backpropagation when the quantization circuit determines to update the quantization parameter.

[0060] According to different application scenarios, the operation data described above can include various types of data within the neural network training process. For example, in one embodiment, the operation data can include gradient data. Based on this, the quantization parameter can include a first bit position parameter or a first bit width parameter applied to quantize the gradient data, and the adjustment circuit can be configured to adjust the scaling factor for scaling the loss value in the next generation of backpropagation when the first bit position parameter or the first bit width parameter is updated.

[0061] In one embodiment, the operation data can include gradient data and neuron data, and the quantization parameter can include a second bit position parameter or a second bit width parameter for the gradient data and the neuron data. Based on this, the adjustment circuit of the present disclosure can be configured to adjust the scaling factor for scaling the loss value in the next generation of backpropagation when the second bit position parameter or the second bit width parameter is updated.

[0062] In one embodiment, the operation data can include gradient data, neuron data, and weight data, and the quantization parameter can include a third bit position parameter or a third bit width parameter for the three types of data, and the adjustment circuit can be configured to adjust the scaling factor for scaling the loss value in the next generation of backpropagation when the third bit position parameter or the third bit width parameter is updated.

[0063] Depending on the computing scenario, the quantization operation can have different execution manners. In view of this, the quantization circuit of the present disclosure can include a forward quantization circuit and a reverse quantization circuit, wherein the forward quantization circuit can be configured to quantize operation data of a high-precision data type (e.g., a floating-point type) into operation data of a low-precision data type (e.g., a fixed-point type) according to a quantization parameter, and the reverse quantization circuit can be configured to reverse quantize operation data of a low-precision data type into operation data of a high-precision data type according to the quantization parameter. In some embodiments, in addition to performing the quantization operation, the quantization circuit of the present disclosure can also make a judgment on whether to update the quantization parameter. To this end, in one embodiment, the quantization circuit can be configured to determine whether to update the quantization parameter according to a quantization error of the operation data. In one implementation scenario, the quantization circuit of the present disclosure can be configured to perform an operation based on the mean values of the operation data before and after quantization to determine the aforementioned quantization error.

[0064] In the update operation of back propagation, a key task is how to effectively and accurately update the weight data. To this end, the update circuit of the present disclosure can be configured to obtain weight gradient data by using the scaled loss value. Accordingly, the reverse quantization circuit in the aforementioned quantization circuit can be configured to reverse quantize the weight gradient data from a low-precision data type (e.g., a fixed-point number) into weight gradient data of a high-precision data type (e.g., a floating-point number). Further, the update circuit can be configured to update the weight by using the scaling factor and the weight gradient data of the high-precision data type. For example, when the obtained loss value is a 16-bit length floating-point number, it can be enlarged by the scaling factor before the reverse update operation (which includes the operation of quantizing into a fixed-point number for calculation), and when calculating the weight gradient, it can be converted into a high-precision data (e.g., a 32-bit length floating-point number). Then, the 32-bit length floating-point number can be reduced in inverse proportion to the scaling factor to calculate the weight gradient data, thereby updating the weight. Regarding the quantization and update process, further description will be made below.

[0065] Based on the aforementioned formula (6), when back propagation is performed by using Loss_scale (including the quantization operation and calculation of the operation data of the corresponding layer), the weight gradient w_grad used for updating the weight of each layer can be obtained. Then, in order to ensure the accuracy of the calculation, w_grad can be converted into a high-precision type data (e.g., from a relatively low-precision 16-bit floating-point number to a relatively high-precision 32-bit floating-point number), and the actual value of the weight gradient is calculated by the following formula (7), and the actual value is used to update the weight:

[0066] W_grad_real=w_grad / scale (7)

[0067] As for the scaling factor "scale" of the present disclosure, it can be adjusted in various ways. In one embodiment, the scaling factor can be adjusted according to one or more hyperparameters. In another embodiment, the scaling factor can be determined according to the data distribution of the gradient data. In one embodiment, the scaling factor can also be determined based on a preset threshold and the maximum value of the gradient data.

[0068] The above is described in connection with Figure 2 The device of the present disclosure is described. It is to be understood that the device of the present disclosure can be applied to one or more layers in a neural network. In particular, when the neural network is a multi-layered layer structure, i.e., the neural network includes multiple intermediate hidden layers, the quantization circuit of the present disclosure can be configured to determine whether to update the quantization parameter for each layer. When the quantization circuit determines that the quantization parameter needs to be updated at any layer, for example, based on the aforementioned quantization error determination that the quantization parameter needs to be updated, the adjustment circuit will correspondingly dynamically adjust the scaling factor for scaling the loss value in the next generation of backpropagation.

[0069] Figure 3 is an exemplary flowchart illustrating the update operation in the forward propagation and backpropagation in the neural network 300 according to an embodiment of the present disclosure.

[0070] As Figure 3 indicated by the dashed box in Figure 3 The neural network shown can be considered as a network including only a single hidden layer (e.g., a convolutional layer) or a network including only one type of operation (only convolution operation), while those skilled in the art will understand from the above and below description that the scheme of the present disclosure is equally applicable to the case where the hidden layer includes multiple layers or multiple other types of operations.

[0071] Further shown in Figure 3 are the aforementioned multiple operators, which can specifically include the quantization operator "quantify", the forward convolution operator "convFwd", the weight gradient operator "convBpFilter", and the input data gradient operator "convBpData". The flow in Figure 3 will be described below in the order of the forward propagation process and the backpropagation process (including the operations of weight update and gradient update) of the training neural network. It is to be noted that in Figure 3In the above equation, "x" represents input neuron data, "w" represents weight, "dx" represents input gradient, "dy" represents output gradient, the content in "[]" represents specific data type, "paramx" represents quantization parameter of neuron data, "paramw" represents quantization parameter of weight, and "paramdy" represents quantization parameter of gradient.

[0072] First, input neuron data x[fp32] and initial weight w[fp32] can be received. As shown in the brackets, both are floating point numbers with 32-bit length. It can be understood that 32 bits here is only exemplary, which can also be 16 bits or other bit-width floating point numbers. As mentioned before, quantization operation performed by, for example, quantization circuit of the present disclosure, can quantize both to fixed point numbers. For this purpose, in an embodiment, quantization operator "quantify" can be implemented on the quantization circuit of the present disclosure. The quantization operator can include quantization strategy operator and quantization parameter operator. In an embodiment, the quantization strategy operator can be used at least to determine whether to perform update operation of quantization parameter, and the quantization parameter operator can be used at least to determine quantization parameter and perform the quantization operation on neuron data of high-precision data type (floating point number in the example of the present disclosure) using the quantization parameter.

[0073] In some application scenarios, the above-mentioned quantization strategy operator can be responsible for calculating quantization error diff bit and quantization period trend value diff update . Since the determination of quantization error is of great significance to the adjustment of quantization period, data bit width, etc., it will be described in detail below.

[0074] Suppose in a scenario, the data to be quantized is F = [f1, f2,..., f m ], and the data obtained after quantization using n-bit fixed point is The quantization error can be calculated by the following equation:

[0075]

[0076] That is:

[0077]

[0078] When diff bit is greater than threshold value th, then it can be considered to increase the quantization bit width by t bits, so that the new quantization bit width is n = n + t, where th and t are both variable hyperparameters.

[0079] It can be seen that the determination of the above quantization error involves the calculation of the mean function mean(), and the calculation method of the quantization error has the following significance:

[0080] As Figure 4 shown in the curves 1 and 2 of two floating-point number data distributions, assuming that one of the quantization intervals is [a, b], [a, c], the floating-point number between a and b is quantized to a, and the floating-point number between c and b is quantized to b, assuming that the data satisfies the Gaussian distribution P(x) ~ G(0, σ), the mean before quantization is

[0081]

[0082] The mean after quantization is:

[0083]

[0084] And it can be seen from Figure 4 that the slope of the tangent line at c is k. Through derivation and approximation calculation, the greater |K| (that is, the more concentrated the distribution) and the greater the quantization interval, the greater the difference between mean f and . Through experiments, it can be known that the more concentrated the data distribution, the greater the error caused by quantization to the final result of training, so the difference between the means before and after quantization can be used to simulate the actual error caused by quantization to training. In order to keep the error from increasing, when quantizing data with a more concentrated distribution (|K|), the quantization interval should be reduced, that is, the quantization bit width should be increased. The present disclosure is based on such a theoretical basis, and after considering the influence of quantization error on training accuracy and effect, a scheme of variable quantization period and data bit width is proposed.

[0085] In order to reduce the calculation consumption caused by calculating the quantization parameters in the training process, the quantization parameters can not be calculated according to the current data to be quantized every generation, but can be updated at intervals. In the generation without updating, the quantization data can use the quantization parameters obtained by the last update. As long as the update interval is selected appropriately, this will not cause a loss of training accuracy, because the to-be-quantized data (such as weight and gradient data) in the training process is relatively stable, has certain continuity and similarity. A simple way is to use a fixed update period, but the fixed update period has poor adaptability, so the present disclosure also proposes an adaptive update period adjustment.

[0086] The quantization parameters are updated at intervals "Interval" generations (that is, the quantization period), and the calculation method is as follows:

[0087] First, introduce the shift moving average m

[0088] m (i) ← α × shift + (1 - α) × m (i-1) (12)

[0089] Introducing diff to measure the trend of shift changes update1 :

[0090] diff update1 =|m (i) -m (i-1 (13)

[0091] diff update1 The larger the value, the more drastic the change in the numerical range, requiring a higher update frequency; that is, the smaller the Interval.

[0092] The diff measures the trend of the fixed-point width n. update2 :

[0093]

[0094] diff update2 The larger the value, the greater the quantization bit width required, necessitating bit width updates and higher interval frequencies.

[0095] By considering both of the above measures simultaneously, we obtain the aforementioned quantitative cycle trend value, diff. update as follows:

[0096] diff update =max(diff update1 diff update2 (15)

[0097] Finally, the Interval is calculated:

[0098]

[0099] In the equation above, α, β, γ, δ, t, and th can be hyperparameters, and β and γ can be either empirical values ​​or hyperparameters. Furthermore, conventional hyperparameter optimization methods are generally applicable to β and γ.

[0100] The above describes how to calculate the quantization error diff for the quantization strategy operator. bit And quantitative cycle trend value diff update The details are described in detail. In one implementation scenario, the inputs to the quantization strategy operator disclosed herein may include data before and after quantization, quantization parameters (mainly using the moving average m of shift), quantization period I (which can be either an input or an output), and output quantization bit width, wherein the quantization period and output quantization bit width can be passed as inputs to the quantization parameter operator.

[0101] Further, the input of the quantization parameter operator can include the data to be quantized, the quantization parameter (including the number of points shift, the sliding average of the number of points m, the scaling factor, etc.), the data bit width (indicating which bit width the quantized data is output), and the quantization period. In some application scenarios, the quantization period can be a variable that controls whether the quantization operator needs to calculate the quantization parameter. For example, when the quantization period is equal to 0, the statistics of the quantization parameter can be performed. Of course, the settings here are only exemplary, and those skilled in the art can also give the quantization period other meanings or use different forms to control based on the teachings herein. In some other application scenarios, if the statistics of the quantization parameter are performed, the new quantization parameter needs to be updated to the address of the old quantization parameter. Otherwise, the old quantization parameter will still be used for this quantization operation.

[0102] According to different implementation manners or application scenarios, the quantization operator can quantize the same layer data in the entire board card at present. The quantization parameter obtained thereby can be synchronized among multiple machines and multiple cards, or not. When not synchronized, a copy of the quantization parameter can be maintained in each board card. Additionally, after each processor core calculates the quantization parameter, the final global quantization parameter is obtained by synchronization.

[0103] Returning to Figure 3The processing flow of the quantization operator is shown in FIG. 3. As shown in FIG. 3, the quantization operator can receive the input data x[fp16] and the weight data w[fp16] as inputs, and can output the quantized data x[int8] and w[int8] as outputs. The quantization operator can also receive the quantization parameters paramx and paramw as inputs. The quantization parameters paramx and paramw can be determined based on the training result of the forward propagation process. For example, the quantization parameters paramx and paramw can be determined by the following equations:

[0104] As mentioned above, based on the training result obtained from the forward propagation process, the loss function LossDiff can be determined, which can be obtained by combining the equation (1) described above, and the details are not repeated here. After obtaining the loss value of the LossDiff, according to the scheme of the present disclosure, the scaling circuit can be used to scale it (i.e., perform the “scaling” operation shown in the figure) for operation in the back propagation. For example, the scaled loss value can be determined using equation (6). Figure 1

[0105] Next, the training process will enter the back propagation process, which involves the reverse gradient update block 302 and the weight update block 303 to perform the back propagation process. For this purpose, the present disclosure implements two operators, i.e., the weight gradient operator “convBpFilter” and the input data gradient operator “convBpData” shown in the figure, through the update circuit. In one or more embodiments, the function of convBpData can be to calculate the gradient of the input neuron data x. According to the chain rule of derivation, the gradient calculation formula of x can be obtained as:

[0106] ​

[0107] Further, the function of the convBpFilter can be to calculate the gradient of the weight w, and according to the chain rule of derivation, the gradient calculation formula of w can be derived as:

[0108]

[0109] In the above two formulas, w, x, δ respectively represent the weight, input and input gradient data from the previous layer, represents the convolution operation, and the rot180 function represents rotating the data by 180 degrees.

[0110] The specific operations involved in the back propagation process are described below to facilitate understanding of the operations involved in the reverse gradient update block and the weight update block.

[0111] In the back propagation process, for a neural network including a plurality of neurons of two or more layers, for each layer, firstly, the input gradient data such as a vector (equivalent to the adjusted dy[fp16] in the figure) and the weight matrix of the current layer are weighted and summed to calculate the output gradient vector of the current layer (equivalent to dx[fp16] in the figure), which involves the dequantization operation from fixed-point number to floating-point number performed by the dequantization circuit. In addition, the input gradient vector (equivalent to dy[fp16] in the figure) is operated (for example, multiplied by the phase) with the input neuron data in the forward propagation process to obtain the gradient of the weight of the current layer (equivalent to dw[fp32] in the figure). Then, the actual weight gradient can be obtained by scaling it using the above formula (7). Then, the weight of the current layer (equivalent to w[fp32] in the figure) can be updated according to the obtained actual gradient of the weight of the current layer.

[0112] Based on the above processing, the disclosed scheme can obtain the current input gradient data by scaling the loss value of the loss function LossDiff. Then, it is quantized using the quantify operator to obtain 8-bit fixed-point input gradient data dy[int8] and quantization parameters paramdy for this gradient data. Next, the corresponding weight quantization parameters paramw, the 8-bit fixed-point weights w[int8], along with the previously obtained input gradient dy[int8] and gradient quantization parameters paramdy, are fed into the input data gradient operator convBpData for computation to obtain the output gradient dx[fp16] of this layer, which serves as the input gradient data (i.e., dy[fp16]) for the next layer (if it exists) in the backpropagation direction. Furthermore, the input gradient dy[int8], the quantization parameter paramdy of the gradient data, along with the corresponding quantization parameter paramx from the forward propagation process of this layer and the aforementioned quantized neuron data x[int8], can be fed into the weight gradient operator convBpFilter to obtain the weight gradient dw[fp32] of this layer. Then, through the solver, the updated weight w[fp32] of this layer can be calculated based on dw[fp32] (after the corresponding scaling as described above) for use in the next forward propagation process.

[0113] Based on the above description, those skilled in the art can understand that during backpropagation, the weights w and the input neuron data x can reuse the data used in the forward propagation. Therefore, during backpropagation, only the gradient dy needs to be quantized. The result of quantizing the gradient dy is the quantized parameter paramdy and the quantized data dy[int8]. There is no need to quantize the input neuron data and weight data again, thereby reducing the number of data quantizations and shortening the training time.

[0114] The above combination Figure 3 The training scheme and the quantization operations involved in this disclosure are described in detail. As previously mentioned, when the quantization circuit of this disclosure determines that the quantization parameters need to be updated during backpropagation, the adjustment circuit of this disclosure will adjust the scaling factor accordingly. In one embodiment, the adjusted scaling factor can be calculated based on the following formula:

[0115] t = floor(log2(th / m1)) (19)

[0116] scale_new=scale×pow(2,t) (20)

[0117] where th in the above equation is a hyper-parameter which can be set, for example, according to the size of the neuron data, such as 512. ml can be the maximum value of the gradient data in backpropagation. The floor() represents a floor function, and the pow(x, y) function represents the y-th power (or exponent) of x.

[0118] Figure 5 is a flow chart illustrating a method 500 for training a neural network according to embodiments of the present disclosure, where the process of training the neural network includes forward propagation and backpropagation which are iteratively performed. As shown in Figure 5 , at step 502, the method 500 scales the loss value obtained from the forward propagation according to a scaling factor to obtain a scaled loss value. Then, at step 504, the method 500 performs an update operation in the backpropagation based on the scaled loss value. In one embodiment, the update operation here can include gradient update and weight update. At step 506, the method 500 performs a quantization operation on operation data according to a quantization parameter, and determines whether to update the quantization parameter based on the operation data. According to the foregoing description in connection with Figure 3 , one skilled in the art can understand that the quantization operation is interspersed in the aforementioned update operation, thereby advantageously accelerating the execution of the update operation. Then, at step 508, upon determining to update the quantization parameter, the scaling factor is adjusted for scaling the loss value in the next generation of backpropagation. In other words, the output of step 508 is the adjusted scaling factor, and the scaling factor is fed back to step 502 for new scaling of the loss value. As can be seen, the method 500 can be repeatedly performed until the loss value of the neural network reaches the expected value, thereby completing the training of the neural network.

[0119] Based on the above description, one skilled in the art can understand that the method 500 can be performed by the device 200 described in the present disclosure in connection with Figure 2 , and thus the description of the specific operation of the device 200 in connection with Figure 2 also applies to the steps performed by the method 500, which will not be described here.

[0120] Figure 6 is a structural diagram illustrating a combined processing apparatus 600 according to embodiments of the present disclosure. As shown in Figure 6 , the combined processing apparatus 600 includes a computing processing apparatus 602, an interface apparatus 604, other processing apparatus 606, and a storage apparatus 608. According to different application scenarios, the computing processing apparatus can include one or more computing devices 610, which can be configured to perform the operations described in the present disclosure in connection with the accompanying Figures 1-5 . In particular, in some application scenarios, the computing devices 610 can include the device described in the present disclosure in connection with Figure 2The described apparatus 200, and performs the operations in connection with Figure 5 The described steps.

[0121] In different embodiments, the computing processing apparatus of the present disclosure can be configured to perform user-specified operations. In exemplary applications, the computing processing apparatus can be implemented as a single-core artificial intelligence processor or a multi-core artificial intelligence processor. Similarly, one or more computing devices included in the computing processing apparatus can be implemented as an artificial intelligence processor core or a partial hardware structure or circuit of an artificial intelligence processor core to implement, for example, various types of circuits disclosed by the present disclosure, such as scaling circuits, updating circuits, quantization circuits, or adjustment circuits. When multiple computing devices are implemented as an artificial intelligence processor core or a partial hardware structure of an artificial intelligence processor core, the computing processing apparatus of the present disclosure can be considered to have a single-core structure or a homogeneous multi-core structure.

[0122] In exemplary operations, the computing processing apparatus of the present disclosure can interact with other processing apparatuses through an interface apparatus to jointly complete user-specified operations. Depending on the implementation, the other processing apparatuses of the present disclosure can include one or more types of processors, such as a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an artificial intelligence processor, and the like general-purpose and / or special-purpose processors. These processors can include, but are not limited to, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and the like, and the number thereof can be determined according to actual needs. As previously mentioned, the computing processing apparatus of the present disclosure can be considered to have a single-core structure or a homogeneous multi-core structure. However, when the computing processing apparatus and the other processing apparatuses are considered together, they can be considered to form a heterogeneous multi-core structure.

[0123] In one or more embodiments, the other processing apparatuses can serve as an interface for external data and control for the computing processing apparatus of the present disclosure (which can be embodied as a relevant computing device for artificial intelligence, such as neural network operations), and perform basic controls including, but not limited to, data transfer, starting and / or stopping of the computing device, and the like. In further embodiments, the other processing apparatuses can also cooperate with the computing processing apparatus to jointly complete computing tasks.

[0124] In one or more embodiments, the interface device can be used to transfer data and control instructions between the computing processing device and other processing devices. For example, the computing processing device can obtain input data from other processing devices via the interface device and write the input data into a storage device (or memory) on the computing processing device chip. Further, the computing processing device can obtain control instructions from other processing devices via the interface device and write the control instructions into a control buffer on the computing processing device chip. Alternatively or optionally, the interface device can also read data from the storage device of the computing processing device and transfer the data to other processing devices.

[0125] Additionally or alternatively, the combined processing device of the present disclosure can further include a storage device. As shown in the figures, the storage device is connected to the computing processing device and the other processing device, respectively. In one or more embodiments, the storage device can be used to save data of the computing processing device and / or the other processing device, which can be, for example, operation data of the present disclosure, including but not limited to pre-quantized or post-quantized neuron data, weight data, and / or gradient data. In some embodiments, the data can be data that cannot be saved in the internal or on-chip storage device of the computing processing device or the other processing device.

[0126] In some embodiments, the present disclosure further discloses a chip (e.g., chip 702 shown in Figure 7 In one implementation, the chip is a System on Chip (SoC) and integrates one or more combined processing devices as shown in Figure 6 The chip can be connected to other related components through an external interface device (e.g., external interface device 706 shown in Figure 7 The related components can be, for example, a camera, a display, a mouse, a keyboard, a network card, or a wifi interface. In some application scenarios, other processing units (e.g., a video codec) and / or interface modules (e.g., a DRAM interface) can be integrated on the chip. In some embodiments, the present disclosure further discloses a chip package structure including the chip described above. In some embodiments, the present disclosure further discloses a board including the chip package structure described above. The board will be described in detail below. Figure 7

[0127] Figure 7 is a structural schematic diagram showing a board 700 according to an embodiment of the present disclosure. As shown in Figure 7 ​As shown in FIG. 7, the board card includes a storage device 704 for storing data, which includes one or more storage units 710. The storage device can be connected and transmit data with the control device 708 and the chip 702 described above by means of, for example, a bus. Further, the board card also includes an external interface device 706, which is configured for the data relay or switching function between the chip (or the chip in the chip package structure) and an external equipment 712 (for example, a server or a computer, etc.). For example, the data to be processed can be delivered by the external equipment to the chip through the external interface device. For another example, the calculation result of the chip can be transmitted back to the external equipment via the external interface device. According to different application scenarios, the external interface device can have different interface forms, for example, it can adopt a standard PCIE interface, etc.

[0128] In one or more embodiments, the control device in the board card of the present disclosure can be configured for regulating the state of the chip. For this purpose, in one application scenario, the control device can include a micro controller unit (MCU) for regulating the working state of the chip.

[0129] According to the above description in combination with Figure 6 and Figure 7 , those skilled in the art can understand that the present disclosure also discloses an electronic equipment or device, which can include one or more of the above-mentioned board cards, one or more of the above-mentioned chips and / or one or more of the above-mentioned combined processing devices.

[0130] According to different application scenarios, the electronic device or apparatus of the present disclosure can include a server, a cloud server, a server cluster, a data processing apparatus, a robot, a computer, a printer, a scanner, a tablet computer, a smart terminal, a PC device, an Internet of Things terminal, a mobile terminal, a mobile phone, a vehicle record device, a navigator, a sensor, a camera, a camera, a video camera, a projector, a watch, a headset, a mobile storage, a wearable device, a visual terminal, an autonomous driving terminal, a vehicle, a household appliance, and / or a medical device. The vehicle includes an airplane, a ship, and / or a vehicle; the household appliance includes a television, an air conditioner, a microwave oven, a refrigerator, an electric rice cooker, a humidifier, a washing machine, an electric lamp, a gas stove, an oil smoke exhaust fan; the medical device includes a nuclear magnetic resonance instrument, a B-ultrasound instrument, and / or an electrocardiograph. The electronic device or apparatus of the present disclosure can also be applied to the fields of Internet, Internet of Things, data center, energy, transportation, public management, manufacturing, education, power grid, telecommunications, finance, retail, construction site, medical treatment, etc. Further, the electronic device or apparatus of the present disclosure can also be used in cloud, edge, terminal, etc. application scenarios related to artificial intelligence, big data, and / or cloud computing. In one or more embodiments, the electronic device or apparatus with high computing power according to the present disclosure can be applied to a cloud device (such as a cloud server), and the electronic device or apparatus with small power consumption can be applied to a terminal device and / or an edge device (such as a smart phone or a camera). In one or more embodiments, the hardware information of the cloud device and the hardware information of the terminal device and / or the edge device are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud device according to the hardware information of the terminal device and / or the edge device to simulate the hardware resources of the terminal device and / or the edge device, so as to complete the unified management, scheduling and collaborative work of end-cloud integration or cloud-edge integration.

[0131] It should be noted that for the purpose of simplicity, the present disclosure describes some methods and embodiments thereof as a series of actions and combinations thereof, but those skilled in the art can understand that the schemes of the present disclosure are not limited by the order of the described actions. Therefore, those skilled in the art can understand that some steps can be executed in other orders or simultaneously according to the disclosure or teaching of the present disclosure. Further, those skilled in the art can understand that the described embodiments of the present disclosure can be regarded as optional embodiments, i.e. the actions or modules involved therein are not necessarily essential for the implementation of one or more schemes of the present disclosure. In addition, the description of some embodiments of the present disclosure also has different focuses according to the schemes. Therefore, those skilled in the art can understand that the parts not described in detail in one embodiment of the present disclosure can also refer to the relevant description of other embodiments.

[0132] In terms of specific implementation, based on the disclosure and teachings of the present disclosure, those skilled in the art can understand that the several embodiments disclosed by the present disclosure can also be implemented in other manners not disclosed herein. For example, in terms of the units described in the aforementioned electronic device or apparatus embodiments, the units can be combined or integrated in another system, or some features or functions of the units can be selectively disabled. In terms of the connection relationship between the units or components, the connection between the units or components discussed in the foregoing can be direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling refers to communication connection using an interface, where the communication interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.

[0133] In the present disclosure, the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units. The aforementioned components or units can be located in the same place or distributed on a plurality of network units. In addition, according to actual needs, some or all of the units can be selected to achieve the purposes of the solutions described in the embodiments of the present disclosure. In addition, in some scenarios, multiple units in the embodiments of the present disclosure can be integrated into one unit or each unit physically exists separately.

[0134] In some implementation scenarios, the integrated units described above can be implemented in the form of software program modules. If implemented in the form of software program modules and sold or used as independent products, the integrated units can be stored in a computer readable memory. Based on this, when the solutions of the present disclosure are embodied in the form of a software product (for example, a computer readable storage medium), the software product can be stored in a memory, which can include a plurality of instructions to make a computer device (for example, a personal computer, a server or a network device, etc.) execute some or all of the steps of the method described in the embodiments of the present disclosure. The aforementioned memory can include, but is not limited to, a U disk, a flash disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0135] In some other implementation scenarios, the above-mentioned integrated units can also be implemented in the form of hardware, i.e., specific hardware circuits, which can include digital circuits and / or analog circuits, etc. The physical implementation of the hardware structure of the circuit can include but is not limited to physical devices, and the physical devices can include but are not limited to transistors or memristors, etc. In view of this, various apparatuses (e.g., computing apparatuses or other processing apparatuses) described herein can be implemented by appropriate hardware processors, such as CPUs, GPUs, FPGAs, DSPs, ASICs, etc. Further, the aforementioned storage units or storage devices can be any appropriate storage medium (including magnetic storage media or magneto-optical storage media, etc.), which can be, for example, resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high bandwidth memory (HBM), hybrid memory cube (HMC), ROM, and RAM, etc.

[0136] While the present disclosure has been illustrated and described with reference to various embodiments thereof, it will be clear to those skilled in the art that the embodiments are intended to be illustrative only and not limiting of the scope of the disclosure. Many changes, modifications, and variations will be apparent to those skilled in the art. It is therefore contemplated to cover any and all such changes, modifications, and variations in the scope of the appended claims. The following clauses are intended to define the scope of the disclosure and thus cover equivalents or alternatives within the scope of the claims.

[0137] The foregoing can be better understood in light of the following clauses:

[0138] Clause A1. A device for training a neural network, wherein training the neural network comprises forward propagation and backward propagation performed iteratively, the device comprising:

[0139] scaling circuitry configured to scale a loss value obtained from the forward propagation according to a scaling factor to obtain a scaled loss value;

[0140] updating circuitry configured to perform an update operation in the backward propagation based on the scaled loss value;

[0141] quantization circuit configured to perform a quantization operation on operational data according to a quantization parameter and determine whether to update the quantization parameter based on the operational data; and

[0142] adjustment circuit configured to adjust the scaling factor for scaling of the loss value in a next generation of backpropagation when the quantization circuit determines to update the quantization parameter.

[0143] Clause A2, the device of Clause A1, wherein the operational data comprises gradient data, the quantization parameter comprises a first point location parameter or a first bit width parameter, and the adjustment circuit is configured to adjust the scaling factor for scaling of the loss value in a next generation of backpropagation when the first point location parameter or the first bit width parameter is updated.

[0144] Clause A3, the device of Clause A1 or A2, wherein the operational data comprises gradient data and neuron data, the quantization parameter comprises a second point location parameter or a second bit width parameter, and the adjustment circuit is configured to adjust the scaling factor for scaling of the loss value in a next generation of backpropagation when the second point location parameter or the second bit width parameter is updated.

[0145] Clause A4, the device of any one of Clauses A1-A4, wherein the operational data comprises gradient data, neuron data, and weight data, the quantization parameter comprises a third point location parameter or a third bit width parameter, and the adjustment circuit is configured to adjust the scaling factor for scaling of the loss value in a next generation of backpropagation when the third point location parameter or the third bit width parameter is updated.

[0146] Clause A5, the device of any one of Clauses A1-A4, wherein the quantization circuit comprises a forward quantization circuit and a reverse quantization circuit, wherein the forward quantization circuit is configured to quantize operational data of a high precision data type to operational data of a low precision data type according to a quantization parameter, and the reverse quantization circuit is configured to dequantize operational data of a low precision data type to operational data of a high precision data type according to a quantization parameter.

[0147] Clause A6, the device of any one of Clauses A1-A5, wherein the high precision data type is a floating point data type and the low precision data type is a fixed point data type.

[0148] Clause A7, the device of any one of clauses A1-A6, wherein the update circuit is configured to obtain weight gradient data using the scaled loss value, and the dequantization circuit is configured to dequantize the weight gradient data into high-precision data type weight gradient data, and the update circuit is configured to update weights using the scaling factor and the high-precision data type weight gradient data.

[0149] Clause A8, the device of any one of clauses A1-A7, wherein the adjustment circuit is configured to adjust the scaling factor according to one or more hyperparameters.

[0150] Clause A9, the device of any one of clauses A1-A8, wherein the adjustment circuit is configured to determine the scaling factor according to a data distribution of the gradient data.

[0151] Clause A10, the device of any one of clauses A1-A9, wherein the adjustment circuit is configured to determine the scaling factor according to a maximum value of the gradient data and a preset threshold value.

[0152] Clause A11, the device of any one of clauses A1-A10, wherein the quantization circuit is configured to determine whether to update the quantization parameter according to a quantization error of the operation data.

[0153] Clause A12, the device of any one of clauses A1-A11, wherein the quantization circuit is configured to perform an operation based on a mean value of the operation data before quantization and after quantization to determine the quantization error.

[0154] Clause A13, the device of any one of clauses A1-A12, wherein the neural network comprises a multi-layer structure formed by connections of a plurality of neurons, and wherein the quantization circuit is configured to determine whether to update the quantization parameter for each layer, and when the quantization circuit determines to update the quantization parameter at any one layer, the adjustment circuit is configured to dynamically adjust the scaling factor for scaling of the loss value in the next generation of backpropagation.

[0155] Clause A14, an integrated circuit comprising the device of any one of clauses A1-A13.

[0156] Clause A15, a board card comprising the device of any one of clauses A1-A13.

[0157] Clause A16, a method for training a neural network, wherein training the neural network comprises forward propagation and backpropagation performed iteratively, the method comprising:

[0158] scaling the loss value obtained from the forward propagation according to a scaling factor to obtain a scaled loss value;

[0159] performing an update operation in the backward propagation based on the scaled loss value;

[0160] performing a quantization operation on operation data according to a quantization parameter, and determining whether to update the quantization parameter based on the operation data; and

[0161] adjusting the scaling factor for scaling of the loss value in a next generation of backward propagation upon determining to update the quantization parameter.

[0162] Clause A17, the method of Clause A16, wherein the operation data comprises gradient data, the quantization parameter comprises a first point location parameter or a first bit width parameter, and the method comprises:

[0163] adjusting the scaling factor for scaling of the loss value in a next generation of backward propagation upon updating the first point location parameter or the first bit width parameter.

[0164] Clause A18, the method of Clause A16 or Clause A17, wherein the operation data comprises gradient data and neuron data, the quantization parameter comprises a second point location parameter or a second bit width parameter, and the method comprises:

[0165] adjusting the scaling factor for scaling of the loss value in a next generation of backward propagation upon updating the second point location parameter or the second bit width parameter.

[0166] Clause A19, the method of any one of Clauses A16-A18, wherein the operation data comprises gradient data, neuron data, and weight data, the quantization parameter comprises a third point location parameter or a third bit width parameter, and the method comprises:

[0167] adjusting the scaling factor for scaling of the loss value in a next generation of backward propagation upon updating the third point location parameter or the third bit width parameter.

[0168] Clause A20, the method of any one of Clauses A16-A19, wherein the quantization operation comprises a forward quantization operation and a reverse quantization operation, wherein the forward quantization operation comprises quantizing operation data of a high precision data type into operation data of a low precision data type according to a quantization parameter, and the reverse quantization operation comprises dequantizing operation data of a low precision data type into operation data of a high precision data type according to a quantization parameter.

[0169] Clause A21. The method of any one of clauses A16-A20, wherein the high-precision data type is a floating-point data type and the low-precision data type is a fixed-point data type.

[0170] Clause A22. The method of any one of clauses A16-A21, wherein the update operation in the backward propagation comprises obtaining weight gradient data with the scaled loss value, and the dequantization operation comprises dequantizing the weight gradient data into high-precision data type weight gradient data, and the update operation in the backward propagation further comprises updating weights with the scaling factor and the high-precision data type weight gradient data.

[0171] Clause A23. The method of any one of clauses A16-A22, wherein the adjusting comprises adjusting the scaling factor according to one or more hyperparameters.

[0172] Clause A24. The method of any one of clauses A16-A23, wherein the adjusting comprises determining the scaling factor according to a data distribution of the gradient data.

[0173] Clause A25. The method of any one of clauses A16-A24, wherein the adjusting comprises determining the scaling factor according to a maximum value of the gradient data and a preset threshold value.

[0174] Clause A26. The method of any one of clauses A16-A25, wherein the quantizing comprises determining whether to update the quantization parameter according to a quantization error of the operation data.

[0175] Clause A27. The method of any one of clauses A16-A26, wherein the quantizing comprises performing an operation based on a mean value of the operation data before quantization and after quantization to determine the quantization error.

[0176] Clause A28. The method of any one of clauses A16-A27, wherein the neural network comprises a multi-layer structure formed by connections of a plurality of neurons, and wherein the quantizing comprises determining whether to update the quantization parameter for each layer, and dynamically adjusting the scaling factor for scaling of the loss value in the next generation of backward propagation when it is determined to update the quantization parameter at any layer.

[0177] Clause A29. A device for training a neural network, comprising: at least one processor; at least one memory storing computer program code, the at least one memory and the computer program code configured to, with the processor, cause the device to perform the method of any of clauses A16-A28.

[0178] Clause A30. A computer-readable storage medium storing a computer program for training a neural network, which when executed by one or more processors implements the method of any of clauses A16-A28.

[0179] It should be understood that the terms “first,” “second,” “third,” and “fourth” and the like in the claims, specification, and drawings of this disclosure are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. The terms “comprises,” “comprising,” “includes,” “including,” “contains,” “containing,” “has,” “having,” “united,” “unites,” “coupled,” “coupling,” and the like, when used in the specification and in the claims, indicate the presence of the stated feature, integer, step, operation, element, component, and / or group but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0180] It should also be understood that the terms used in the specification and the claims are not to be interpreted as specific language but are intended to be given their broadest interpretation. The use of the term “including” and “comprising” as used in the specification and the claims are intended to mean that there are no restrictions on the components, steps, compositions, and / or elements of the described features, compositions, and / or methods. The use of the term “consisting” as used in the specification and the claims is intended to mean that there are no additional components, steps, compositions, and / or elements present in the described features, compositions, and / or methods. The use of the term “consisting essentially of’ as used in the specification and the claims is intended to mean that there are additional components, steps, compositions, and / or elements present in the described features, compositions, and / or methods, but that these additional components, steps, compositions, and / or elements do not materially alter the basic and novel characteristics of the described features, compositions, and / or methods.

[0181] As used in this specification and claims, the terms “if’ and “when” can be interpreted to mean “upon” or “in response to a determination” or “in response to a detection” depending on the context. Similarly, the phrase “if it is determined” or “if [a described condition or event] is detected” can be interpreted to mean “upon determining” or “in response to a determination” or “upon detecting [the described condition or event]” or “in response to a detection [of the described condition or event]” depending on the context.

[0182] The above has been made a detailed introduction to the embodiments of the present disclosure, and the principles and implementation manners of the present disclosure are described by applying specific examples. The above embodiment description is only used to help understand the method of the present disclosure and its core idea. Meanwhile, the changes or deformations made by the person skilled in the art according to the idea of the present disclosure, based on the specific implementation manners and application range of the present disclosure, all belong to the protection range of the present disclosure. In summary, the content of the specification should not be understood as a limitation of the present disclosure.

Claims

1. A device for training a neural network for image processing or speech recognition, the device being a computing device in a combined processing apparatus, wherein training the neural network comprises forward propagation and backward propagation performed iteratively, the device comprising: a scaling circuit configured to scale a loss value obtained from the forward propagation according to a scaling factor to obtain a scaled loss value; an update circuit configured to perform an update operation in the backward propagation based on the scaled loss value; a quantization circuit configured to perform a quantization operation on operation data according to a quantization parameter, and determine whether to update the quantization parameter based on the operation data; and an adjustment circuit configured to adjust the scaling factor for scaling the loss value in a next generation of backward propagation when the quantization circuit determines to update the quantization parameter. 2.The device of claim 1, wherein the operation data comprises gradient data, the quantization parameter comprises a first point location parameter or a first bit width parameter, and the adjustment circuit is configured to adjust the scaling factor for scaling the loss value in a next generation of backward propagation when the first point location parameter or the first bit width parameter is updated. 3.The device of claim 1, wherein the operation data comprises gradient data and neuron data, the quantization parameter comprises a second point location parameter or a second bit width parameter, and the adjustment circuit is configured to adjust the scaling factor for scaling the loss value in a next generation of backward propagation when the second point location parameter or the second bit width parameter is updated. 4.The device of claim 1, wherein the operation data comprises gradient data, neuron data and weight data, the quantization parameter comprises a third point location parameter or a third bit width parameter, and the adjustment circuit is configured to adjust the scaling factor for scaling the loss value in a next generation of backward propagation when the third point location parameter or the third bit width parameter is updated. 5.The device of claim 1, wherein the quantization circuit comprises a forward quantization circuit and a reverse quantization circuit, wherein the forward quantization circuit is configured to quantize operation data of a high precision data type into operation data of a low precision data type according to a quantization parameter, and the reverse quantization circuit is configured to reverse quantize operation data of a low precision data type into operation data of a high precision data type according to a quantization parameter. 6.The device of claim 5, wherein the high precision data type is a floating point data type and the low precision data type is a fixed point data type. 7.The device of claim 5, wherein the update circuit is configured to obtain weight gradient data with the scaled loss value, and the reverse quantization circuit is configured to reverse quantize the weight gradient data into weight gradient data of a high precision data type, and the update circuit is configured to update weights with the scaling factor and the weight gradient data of the high precision data type. 8.The device of claim 1, wherein the adjustment circuit is configured to adjust the scaling factor according to one or more hyperparameters. ​ 9.The apparatus of claim 2, wherein the adjustment circuit is configured to determine the scaling factor according to a data distribution of the gradient data. 10.The apparatus of claim 2, wherein the adjustment circuit is configured to determine the scaling factor according to a maximum value of the gradient data and a preset threshold value. 11.The apparatus of claim 1, wherein the quantization circuit is configured to determine whether to update the quantization parameter according to a quantization error of the operation data. 12.The apparatus of claim 11, wherein the quantization circuit is configured to perform an operation based on a mean value of the operation data before quantization and after quantization to determine the quantization error. 13.The apparatus of any one of claims 1-12, wherein the neural network comprises a multi-layer structure formed by connections of a plurality of neurons, and wherein the quantization circuit is configured to determine whether to update the quantization parameter for each layer, and when the quantization circuit determines to update the quantization parameter at any one of the layers, the adjustment circuit is configured to dynamically adjust the scaling factor for scaling of the loss value in the next generation of backpropagation. 14.An integrated circuit comprising the apparatus of any one of claims 1-13. 15.A board card comprising the apparatus of any one of claims 1-13. 16.A method for training a neural network for image processing or speech recognition, performed by a computing device in a combined processing apparatus, wherein training the neural network comprises forward propagation and backpropagation performed iteratively, the method comprising: scaling a loss value obtained from the forward propagation according to a scaling factor to obtain a scaled loss value; performing an update operation in the backpropagation based on the scaled loss value; performing a quantization operation on operation data according to a quantization parameter, and determining whether to update the quantization parameter based on the operation data; and when it is determined to update the quantization parameter, adjusting the scaling factor for scaling of the loss value in the next generation of backpropagation. 17.The method of claim 16, wherein the operation data comprises gradient data, the quantization parameter comprises a first point position parameter or a first bit width parameter, and the method comprises: when the first point position parameter or the first bit width parameter is updated, adjusting the scaling factor for scaling of the loss value in the next generation of backpropagation. 18.The method of claim 16, wherein the operation data comprises gradient data and neuron data, the quantization parameter comprises a second point position parameter or a second bit width parameter, and the method comprises: when the second point position parameter or the second bit width parameter is updated, adjusting the scaling factor for scaling of the loss value in the next generation of backpropagation. 19.The method of claim 16, wherein the operation data comprises gradient data, neuron data and weight data, the quantization parameter comprises a third point position parameter or a third bit width parameter, and the method comprises: ​ The scaling factor is adjusted for scaling of the loss value in the next generation backpropagation when the third point position parameter or the third bit width parameter is updated.

20. The method of claim 16, wherein the quantization operation comprises a forward quantization operation and a reverse quantization operation, wherein the forward quantization operation comprises quantizing operation data of a high precision data type to operation data of a low precision data type according to a quantization parameter, and the reverse quantization operation comprises reverse quantizing operation data of a low precision data type to operation data of a high precision data type according to a quantization parameter.

21. The method of claim 20, wherein the high precision data type is a floating point data type and the low precision data type is a fixed point data type.

22. The method of claim 20, wherein the update operation in the backpropagation comprises obtaining weight gradient data using the scaled loss value, and the reverse quantization operation comprises reverse quantizing the weight gradient data to weight gradient data of a high precision data type, and the update operation in the backpropagation further comprises updating weights using the scaling factor and the weight gradient data of a high precision data type.

23. The method of claim 16, wherein the adjusting comprises adjusting the scaling factor according to one or more hyperparameters.

24. The method of claim 17, wherein the adjusting comprises determining the scaling factor according to a data distribution of the gradient data.

25. The method of claim 17, wherein the adjusting comprises determining the scaling factor according to a maximum value of the gradient data and a preset threshold value.

26. The method of claim 16, wherein the quantizing comprises determining whether to update the quantization parameter according to a quantization error of the operation data.

27. The method of claim 26, wherein the quantizing comprises performing an operation based on a mean value of the operation data before quantization and after quantization to determine the quantization error.

28. The method of any one of claims 16-27, wherein the neural network comprises a multi-layer structure formed by connections of a plurality of neurons, and wherein the quantizing comprises determining whether to update the quantization parameter for each layer, and dynamically adjusting the scaling factor for scaling of the loss value in the next generation backpropagation when it is determined to update the quantization parameter at any one layer.

Images