A neural network quantization training method, device and related products

Abstract

The present application discloses a neural network quantization training method, including: reading the input data and weight of the target quantization unit from a first storage area; then quantizing the input data and weight respectively to obtain the input data quantization result and the weight quantization result, and storing the input data quantization result and the weight quantization result in a second storage area; then based on the input data quantization result and the weight quantization result, performing forward propagation calculation on the target quantization unit to obtain output data; reading the input data quantization result and the weight quantization result from the second storage area; based on the input data quantization result, the weight quantization result and the output data read from the second storage area, performing back propagation calculation on the target quantization unit to obtain the weight gradient; based on the weight gradient and the optimizer algorithm, updating the weight. The embodiment of the present application can avoid repeated quantization, speed up the training speed of the neural network, and reduce memory overhead.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial neural networks (ANN), and in particular to a neural network quantization training method, device and related products. Background Art

[0002] Training is an important part of artificial neural network research, which is usually divided into three steps: forward propagation, back propagation and weight update. Quantization training is a training method for artificial neural networks, which refers to the training method of converting the floating-point data calculation of some layers with large computational complexity in the neural network into fixed-point data calculation.

[0003] However, in quantization training, only the original floating-point data is usually stored, which means that the original floating-point input data and floating-point weights that have been quantized during forward propagation need to be quantized again during back propagation, which is called secondary quantization. Due to the secondary quantization in the quantization training process, memory and time are wasted, which reduces the training speed of the neural network. Summary of the invention

[0004] The embodiments of the present application provide a neural network quantization training method, device and related products, which can avoid repeated quantization, speed up the calculation speed of the neural network, and reduce memory overhead.

[0005] In a first aspect, the present application provides a neural network quantization training method, the method comprising: reading the input data and weights of a target quantization unit from a first storage area, wherein the input data and weights are both floating-point data; quantizing the input data and weights respectively to obtain input data quantization results and weight quantization results, wherein the input data quantization results and weight quantization results are both fixed-point data; storing the input data quantization results and weight quantization results in a second storage area; performing forward propagation calculations on the target quantization unit based on the input data quantization results and weight quantization results to obtain output data; reading the input data quantization results and weight quantization results from the second storage area; performing back propagation calculations on the target quantization unit based on the input data quantization results, weight quantization results and output data read from the second storage area to obtain weight gradients; updating weights based on weight gradients and an optimizer algorithm.

[0006] It can be seen that the embodiment of the present application avoids repeated quantization operations by storing the input data quantization results and weight quantization results obtained by quantization during the forward propagation process in the second storage area, and then taking out the input data quantization results and weight quantization results from the second storage area for back propagation calculation. In addition, fixed-point data calculation is used instead of floating-point data calculation, which can reduce memory overhead and speed up the training speed of the neural network.

[0007] Based on the first aspect, in a possible embodiment, after the input data and the weight are quantized respectively to obtain the input data quantization result and the weight quantization result, the input data and the weight in the first storage area are deleted.

[0008] It can be seen that after the quantization operation is performed to obtain the input data quantization result and the weight quantization result, the original floating-point input data and floating-point weight in the first storage area are deleted, and the first storage area is released. Since fixed-point data occupies less memory space than floating-point data, the second storage area for storing the input data quantization result and the weight quantization result is obviously smaller than the first storage area, reducing memory overhead.

[0009] Based on the first aspect, in a possible embodiment, based on the input data quantization result, weight quantization result and output data read from the second storage area, back propagation calculation is performed on the target quantization unit to obtain the weight gradient, including: calculating the output data gradient based on the output data and the loss function; quantizing the output data gradient to obtain the output data gradient quantization result; based on the input data quantization result, weight quantization result and output data gradient quantization result, back propagation calculation is performed on the target quantization unit to obtain the weight gradient.

[0010] It can be seen that before performing the back-propagation calculation, only the output data gradient needs to be quantized, and there is no need to quantize the original floating-point input data and floating-point weights again. By reusing the input data quantization results and weight quantization results stored in the second storage area, the number of quantization times can be reduced and the training speed of the neural network can be accelerated.

[0011] Based on the first aspect, in a possible embodiment, the target quantization unit is any one of a neural network layer, a neuron or a channel.

[0012] It can be seen that the quantization training method provided in the embodiment of the present application is applicable to neural network layers, neurons or channels. This method has a wide range of applications and can improve the training speed of the entire neural network.

[0013] Based on the first aspect, in a possible embodiment, quantization is to convert floating-point data of a first bit width into fixed-point data of a second bit width.

[0014] Converting floating-point data calculations to fixed-point data calculations can increase calculation speed and reduce memory overhead.

[0015] Based on the first aspect, in a possible embodiment, the first bit width includes 64 bits and 32 bits, and the second bit width includes 8 bits and 16 bits.

[0016] It can be seen that the first bit width is higher than the second bit width. By converting the floating-point data calculation with a higher bit width into the fixed-point data calculation with a lower bit width, the calculation speed can be improved while reducing the memory overhead.

[0017] Based on the first aspect, in a possible embodiment, the optimizer algorithm includes a gradient descent algorithm, a momentum optimization algorithm, and an adaptive learning rate optimization algorithm.

[0018] It can be seen that there are many optimizer algorithms. The embodiment of the present application does not limit the selection of the optimizer algorithm. According to the type of neural network and actual training requirements, a suitable optimizer algorithm can be selected to update the weights of the neural network.

[0019] In a second aspect, an embodiment of the present application provides a neural network quantization training device, which includes: a reading unit, used to read the input data and weights of the target quantization unit from a first storage area, wherein the input data and weights are both floating-point data; a quantization unit, used to quantize the input data and weights respectively, to obtain input data quantization results and weight quantization results, wherein the input data quantization results and weight quantization results are both fixed-point data; a storage unit, used to store the input data quantization results and weight quantization results in a second storage area; a calculation unit, used to perform forward propagation calculations on the target quantization unit based on the input data quantization results and weight quantization results to obtain output data; the reading unit is also used to read the input data quantization results and weight quantization results from the second storage area; the calculation unit is also used to perform back propagation calculations on the target quantization unit based on the input data quantization results, weight quantization results and output data read from the second storage area, thereby obtaining weight gradients; the calculation unit is also used to update weights based on weight gradients and an optimizer algorithm.

[0020] Each unit of the above-mentioned device is specifically used to implement the method described in the first aspect.

[0021] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, wherein the processor and the memory may be interconnected via a bus or may be integrated together. The processor executes a code stored in the memory to implement the method described in the first aspect.

[0022] In a fourth aspect, the present application provides a computer-readable storage medium, including a program or instruction, which, when executed on a computer device, enables the computer device to execute the method described in the first aspect.

[0023] It can be understood that any one of the above aspects may be implemented together with any other aspect or multiple aspects, or may be implemented independently.

[0024] The embodiment of the present application reads the original input data and weights from the first storage area, quantizes the input data and weights to obtain input data quantization results and weight quantization results, and then deletes the original floating-point input data and floating-point weights stored in the first storage area, thereby reducing memory overhead; by storing the above-mentioned input data quantization results and weight quantization results in the second storage area, and then reading the input data quantization results and weight quantization results from the second storage area for back propagation calculation, repeated quantization operations on input data and weights are avoided, and there is almost no time overhead when reusing the quantization results, which can speed up the training speed of the neural network; using fixed-point data calculations with lower bit widths instead of floating-point data calculations with higher bit widths can reduce memory overhead and speed up the calculation speed at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments.

[0026] Figure 1 is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0027] Figure 2 is a structural schematic diagram of a combined processing device provided in an embodiment of the present application;

[0028] Figure 3 It is a structural schematic diagram of a board provided in an embodiment of the present application;

[0029] Figure 4 It is a schematic diagram of the structure of an artificial neural network provided in an embodiment of the present application;

[0030] Figure 5 is a schematic diagram of a floating point data format provided in an embodiment of the present application;

[0031] Figure 6 is a schematic diagram of a fixed-point data format provided in an embodiment of the present application;

[0032] Figure 7 It is a schematic diagram of the forward propagation process of a neural network layer provided in an embodiment of the present application;

[0033] Figure 8 It is a schematic diagram of the back propagation process of a neural network layer provided in an embodiment of the present application;

[0034] Fig. 9 This is a schematic diagram of the back propagation process of another neural network layer provided in an embodiment of the present application;

[0035] Fig.10 It is a flowchart of a neural network quantization training method provided in an embodiment of the present application;

[0036] Fig.11 It is a schematic diagram of a neural network quantization training device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] Before introducing the specific method steps of the embodiments of the present application, the hardware environment involved in the present application is first introduced.

[0038] Figure 1 1 is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application. Figure 1 As shown, the electronic device includes a memory 101, a processor 102, an input device 103 and an output device 104, wherein the electronic device may further include a bus 105, and the processor 102, the input device 103, the output device 104 and the memory 101 may be interconnected through the bus. Optionally, the electronic device may further include an instruction storage unit, which is disposed adjacent to the processor 102. Further optionally, the instruction storage unit is integrated with the processor 102, that is, the instruction storage unit is an on-chip storage unit of the processor 102. In this way, when the processor 102 needs to execute a program in the memory 101, the electronic device first loads the program in the memory 101 into the instruction storage unit, and then the processor 102 can access the above-mentioned instruction storage unit to execute the program in the instruction storage unit.

[0039] The processor 102 may be a central processing unit (CPU), an intelligence processing unit (NPU), a graphics processing unit (GPU) or an image processing unit, and this application does not limit this. According to the difference of the processor 102, the neural network quantization training method proposed in the embodiment of this application can be applied to artificial intelligence application fields such as image recognition processing, deep learning processing, computer vision processing, intelligent robot processing, natural language processing, etc., and execute complex functional programs in the field of artificial intelligence.

[0040] Figure 2 2 is a schematic diagram of a structure of a combined processing device 200 provided in an embodiment of the present application. Figure 2 As shown, the combined processing device 200 includes a computing processing device 202, an interface device 204, other processing devices 206, and a storage device 208. According to different application scenarios, the computing processing device may include one or more computing devices 210, which may be configured to perform the combined processing described herein. Fig.10 The method described below will be Fig.10 Provide a detailed description.

[0041] In different embodiments, the computing and processing device of the present application can be configured to perform user-specified operations. In an exemplary application, the computing and processing device can be implemented as a single-core artificial intelligence processor or a multi-core artificial intelligence processor. Similarly, one or more computing devices included in the computing and processing device can be implemented as an artificial intelligence processor core or a partial hardware structure of an artificial intelligence processor core. When multiple computing devices are implemented as an artificial intelligence processor core or a partial hardware structure of an artificial intelligence processor core, with respect to the computing and processing device of the present application, it can be regarded as having a single-core structure or a homogeneous multi-core structure.

[0042] In an exemplary operation, the computing and processing device of the present application can interact with other processing devices through an interface device to jointly complete the operation specified by the user. Depending on the implementation, other processing devices of the present application may include one or more types of processors in general and / or special processors such as a central processing unit (CPU), a graphics processing unit (GPU), an artificial intelligence processor, etc. These processors may include, but are not limited to, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As before, only with respect to the computing and processing device of the present application, it can be regarded as having a single-core structure or a homogeneous multi-core structure. However, when the computing and processing device and other processing devices are considered together, the two can be regarded as forming a heterogeneous multi-core structure.

[0043] In one or more embodiments, the other processing device can serve as an interface between the computing device of the present application (which can be embodied as an artificial intelligence such as a computing device related to neural network computing) and external data and control, and perform basic controls including but not limited to data handling, starting and / or stopping the computing device, etc. In other embodiments, the other processing device can also cooperate with the computing device to jointly complete computing tasks.

[0044] In one or more embodiments, the interface device can be used to transmit data and control instructions between the computing and processing device and other processing devices. For example, the computing and processing device can obtain input data from other processing devices via the interface device and write it into a storage device (or memory) on the computing and processing device chip. Further, the computing and processing device can obtain control instructions from other processing devices via the interface device and write them into a control cache on the computing and processing device chip. Alternatively or optionally, the interface device can also read data from the storage device of the computing and processing device and transmit it to other processing devices.

[0045] Additionally or optionally, the combined processing device of the present application may further include a storage device. Figure 2 As shown, the storage device is connected to the computing and processing device and other processing devices respectively. In one or more embodiments, the storage device can be used to store data of the computing and processing device and / or other processing devices. For example, the data can be data that cannot be fully stored in the internal or on-chip storage device of the computing and processing device or other processing devices.

[0046] In some embodiments, the present application also discloses a chip (eg Figure 3 In one implementation, the chip is a system on chip (SoC) and integrates one or more Figure 2 The chip can be connected to an external interface device (such as Figure 3 The external interface device 306 shown in the figure is connected to other related components. The related components may be, for example, a camera, a display, a mouse, a keyboard, a network card or a wifi interface. In some application scenarios, other processing units (such as a video codec) and / or interface modules (such as a DRAM interface) may be integrated on the chip. In some embodiments, the present application also discloses a chip packaging structure, which includes the above-mentioned chip. In some embodiments, the present application also discloses a board card, which includes the above-mentioned chip packaging structure. The following will be combined with Figure 3 The board is described in detail.

[0047] Figure 3 2 is a schematic diagram showing the structure of a board 300 according to an embodiment of the present application. Figure 3As shown in , the board includes a storage device 304 for storing data, which includes one or more storage units 32. The storage device can be connected and data can be transmitted with the control device 308 and the chip 302 mentioned above by means of, for example, a bus. Further, the board also includes an external interface device 306, which is configured for data relay or switching between the chip (or the chip in the chip packaging structure) and an external device 312 (such as a server or computer, etc.). For example, the data to be processed can be transmitted to the chip by the external device through the external interface device. For another example, the calculation result of the chip can be transmitted back to the external device via the external interface device. According to different application scenarios, the external interface device can have different interface forms, for example, it can adopt a standard PCIE interface, etc.

[0048] In one or more embodiments, the control device in the board of the present application may be configured to regulate the state of the chip. To this end, in an application scenario, the control device may include a microcontroller unit (MCU) to regulate the working state of the chip.

[0049] Combined with the above Figure 2 and Figure 3 Based on the description, those skilled in the art can understand that the present application also discloses an electronic device or apparatus, which may include one or more of the above-mentioned boards, one or more of the above-mentioned chips and / or one or more of the above-mentioned combined processing devices.

[0050] In order to facilitate understanding of the embodiments of the present application, some terms involved in the present application are explained below.

[0051] Artificial neural network: It can be referred to as neural network for short. It is a computing model that abstracts the way human brain neural network processes and memorizes information from the perspective of information processing. The neural network can be applied to a single or multiple processor cores in the above electronic device 100, board 200 or combination device 300. Figure 4 This is a schematic diagram of the structure of an artificial neural network. A neural network is composed of a large number of nodes (or neurons) connected to each other. Different connection methods form different networks. Each node represents a specific output function, called an activation function, and the connection between each two nodes represents the weighted value of a signal, such as Figure 4 W ij and W jk , they are called weights, which are equivalent to the memory of artificial neural networks. Figure 4As shown in the figure, the artificial neural network consists of three layers: input layer, hidden layer and output layer, each layer has multiple neurons. Among them, the input layer receives signals and data from the outside world; the output layer realizes the output of the system processing results; the hidden layer is also called the middle layer, which is between the input layer and the output layer and cannot be observed from the outside of the system. The middle layer may have one or more layers. The more hidden layers, the more complex the neural network structure. Artificial neural networks are usually approximations of certain algorithms or functions, or expressions of certain logical strategies. The output of the neural network changes with the network connection mode, weights and activation functions. At present, artificial neural networks have outstanding performance in pattern recognition, intelligent robots, automatic control, biology, medicine, economy and other fields, showing good intelligent characteristics.

[0052] Floating point data (float point number): Floating point format is a data representation format commonly used in computers. The data represented by the floating point format is called floating point data. The decimal point position of floating point data is floating, such as Figure 5 As shown in the figure, a mantissa, a base, an exponent, and a sign Ms are used to express real numbers. The mantissa is sometimes also called a significant figure. Floating-point numbers use the exponent to achieve the effect of floating decimal points, so that a wider range of real numbers can be flexibly expressed. Common floating-point formats include single-precision 32-bit floating-point numbers (float32) and double-precision 64-bit floating-point numbers (double64).

[0053] Fixed-point data (fixed point number): Fixed-point format is also a data representation format commonly used in computers. The data represented by the fixed-point format is called fixed-point data. Figure 6 It is a fixed-point data representation format in which the position of the decimal point of the fixed-point data is fixed, that is, the number of integer digits and decimal digits are fixed. It should be understood that Figure 6 The decimal point indicated in is not represented in the machine, but is agreed in advance at a fixed position. It is only a logical setting. Different logical decimal point positions represent different values. Commonly used fixed-point data types are 8-bit integers (int8) and 16-bit integers (int16). Generally speaking, the range of fixed-point data represented by the fixed-point format is limited, but the required processing hardware is relatively simple, while the range of floating-point values ​​represented by the floating-point format is larger, but the required processing hardware is relatively complex. However, the memory space occupied by fixed-point data is often smaller than that of floating-point data. Taking int16 and float32 as examples, the memory occupied by the former in the electronic device 100 is 75% less than that of the latter.

[0054] The following introduces the application scenarios involved in the embodiments of the present application.

[0055] Learning (or training) is an important part of artificial neural network research. The adaptability of neural networks is achieved through learning. According to changes in the environment, the processor needs to adjust the weight parameters of the neural network to improve the behavior of the system. The training process of artificial neural networks can be simply divided into three steps: forward propagation, back propagation, and weight update. Among them, forward propagation is the process of calculating the predicted value by the input training data; back propagation is to use the loss function (loss function) or objective function (objective function) to measure the difference between the predicted value obtained by forward propagation and the target value, so as to reversely find the partial derivative of the model parameters, including weights and biases; weight update is to update the weights of each layer of the neural network in the neural network model according to the gradient of each weight and the optimizer algorithm. For example, if the predicted value of the neural network model is higher than the target value, the weight is adjusted in the above way to make the predicted value lower. In general, neural network training is the processor constantly adjusting the weights until the neural network model can predict the desired target value.

[0056] Quantization training is a training method for artificial neural networks. It refers to a training method in which the processor converts the layers with large computational workloads in the neural network, such as convolution and linear calculations, from floating-point data calculations to fixed-point data calculations. Quantization here means data reduction, that is, the low-bit representation of data. Specifically, it converts higher-bit-width floating-point data into lower-bit-width fixed-point data. Fixed-point data calculations are obviously more efficient than floating-point data calculations, thereby achieving the purpose of computing acceleration. However, in quantization training, the memory usually only stores the original floating-point data, resulting in the original floating-point input data and floating-point weights that have been quantized during forward propagation. They need to be quantized again by the processor during back propagation, which is a secondary quantization situation. For example, if Figure 7 As shown, in the forward propagation process, the input data of a neural network layer is X, X can be an output data of the previous layer, and W is the weight of this layer. X and W are both floating point data, and X and W are both stored in the first storage area, which can be Figure 1 A portion of the memory 101 of the electronic device 100, or Figure 2 A portion of the storage device 208 of the combined processing device 200 may also be Figure 3A part of the storage unit 32 of the board. The operation process within the layer is invisible outside the layer. The operation within the layer mainly includes quantization and the forward propagation calculation corresponding to the layer. First, the processor reads X and W stored in the first storage area, and then the processor quantizes the input data X and weight W in the layer to obtain quantify_X and quantify_W, wherein quantify_X and quantify_W are both fixed-point data, and then the processor uses quantify_X and quantify_W to perform the forward propagation calculation of the layer, such as convolution calculation, linear summation calculation, etc., to obtain the output data Y of the layer, wherein the output data Y is floating-point data, which can be used as the input data of the next layer of neural network. Since quantization training usually only stores the original floating-point data in the electronic device, that is, only the input data X and weight W in the first storage area are saved, therefore, the above-mentioned input data quantization result quantify_X and weight quantization result quantify_W are released after being used in the forward propagation calculation and will not be stored in the memory. After completing the forward propagation process of the entire neural network, the reverse propagation process will begin. Figure 8 Here is an example of the back propagation process of a neural network layer. It should be noted that: Figure 7 and Figure 8 Refers to the same layer of the neural network. Since quantify_X and quantify_W obtained during the forward propagation have been released, Figure 8 The back propagation calculation still needs to use the input data quantization result and the weight quantization result. Therefore, the processor needs to read the input data X and the weight W stored in the first storage area again, and then the processor quantizes X and W from floating-point data to fixed-point data again within the layer. Figure 8 The Y_grad in is the gradient of the output data Y. The processor can calculate Y_grad through the loss function and the output data Y. Figure 8 As shown, Y_grad is also quantized from floating-point data to fixed-point data quantify_Y_grad within the layer, and then the processor uses quantify_X, quantify_W, and quantify_Y_grad to perform the corresponding back propagation calculation of the layer to obtain the gradient X_grad of the input data X and the gradient W_grad of the weight W. From the above content, it can be clearly seen that the quantization operation is repeatedly performed on some data during the forward propagation and back propagation process, that is, there is secondary quantization, and quantization itself has time and memory overhead, which will have the defects of slow calculation speed and large memory overhead.

[0057] In order to solve the above problems, the embodiment of the present application discloses a neural network quantization training method, which can be applied to a single or multiple processor cores in the above electronic device 100, the combined processing device 200 or the board 300. By storing the input data quantization results and weight quantization results obtained by quantization in the forward propagation process in the second storage area, the processor reuses the above quantization results for back propagation calculation in the back propagation process, avoiding repeated quantization, which not only speeds up the calculation speed of the neural network, but also reduces the memory overhead. For the convenience of description, the following only takes the electronic device 100 as an example for explanation.

[0058] See also Fig.10 , Fig.10 : is a flowchart of a neural network quantization training method provided in an embodiment of the present application, and the method may include the following steps:

[0059] S1001: The processor 102 reads input data and weights of a target quantization unit from a first storage area.

[0060] Among them, the input data and weights are all floating-point data, and the target quantization unit is the part of the neural network that needs to perform quantization training. Quantization training refers to the training method in which the processor converts floating-point data calculations into fixed-point data calculations. The neural network has multiple layers. The neural network can be a deep neural network (Deep Neural Networks, DNN), a convolutional neural network (Convolutional Neural Networks, CNN), a recurrent neural network (Recurrent Neural Networks, RNN), or other types of neural networks. This application does not specifically limit the type of neural network. The above-mentioned first storage area can be Figure 1 A portion of the memory 101 of the electronic device 100, or Figure 2 A portion of the storage device 208 of the combined processing device 200 may also be Figure 3 This application does not specifically limit the part of the storage unit 32 of the board.

[0061] In a possible embodiment, the target quantization unit is any one of a neural network layer, a neuron or a channel. For example, assuming that the neural network to be quantized is a deep neural network, such as Figure 4As shown, the deep neural network includes an input layer, multiple hidden layers and an output layer. The target quantization unit in the neural network that needs to perform quantization training is a neural network layer, which can be any layer of the multiple hidden layers of the deep neural network. The input data and weights of this layer are floating point data. It can be understood that the output data of the previous layer can be used as the input data of this layer (i.e., the current layer), and the output data of this layer can be used as the output data of the next layer. If the neural network is a convolutional neural network, the target quantization unit can be any convolutional layer in the convolutional neural network, and of course it can also be a channel of a certain layer, or a neuron of a certain layer. It should be noted that Figure 7 The operation diagram inside a certain layer of the neural network is only given as an example. The layer can have multiple input data, weights and output data, and this application does not limit it.

[0062] S1002: The processor 102 quantizes the input data and the weights respectively to obtain an input data quantization result and a weight quantization result.

[0063] Among them, the input data quantization results and weight quantization results are both fixed-point data.

[0064] In a possible embodiment, the quantization is that the processor 102 converts the floating point data of the first bit width into the fixed point data of the second bit width.

[0065] In a possible embodiment, the first bit width includes 64 and 32 bits, and the second bit width includes 8 and 16 bits. That is to say, floating-point data includes single-precision 32-bit floating-point numbers and double-precision 64-bit floating-point numbers, and fixed-point data includes 8-bit fixed-point integers and 16-bit fixed-point integers. It should be understood that the quantization of the present application is data simplification, that is, the low-bit representation of data. Specifically, the processor 102 converts floating-point data with a higher number of bits into fixed-point data with a lower number of bits. The essence is the conversion process of mapping data from a high dimension to a low dimension. It should be understood that there are many ways to implement the quantization operation of converting floating-point data into fixed-point data. For example, data quantization can be divided into symmetric and asymmetric types, and the symmetric type is divided into a type with a scaling factor and a type without a scaling factor. I will not introduce it too much here, and the present application does not limit the way to implement quantization.

[0066] In a possible embodiment, after the input data and weights are quantized to obtain the input data quantization results and weight quantization results, the input data and weights in the first storage area are deleted. Since the input data quantization results and weight quantization results obtained in S1002 are reused during back propagation, the input data and weights in floating point format in the first storage area are no longer needed. Therefore, after the quantization is completed, the input data and weights in the first storage area are released to reduce memory overhead.

[0067] S1003: The processor 102 stores the input data quantization result and the weight quantization result in the second storage area.

[0068] The second storage area may be Figure 1 A portion of the memory 101 of the electronic device 100, or Figure 2 A portion of the storage device 208 of the combined processing device 200 may also be Figure 3 A part of the storage unit 32 of the board is not specifically limited in this application. The first storage area and the second storage area can be different storage areas. Since the memory required to store floating-point data is often smaller than the memory required for fixed-point data, the space of the first storage area can be larger than the second storage area. Of course, the first storage area and the second storage area can also be the same storage area, that is, after deleting the original floating-point input data and floating-point weights in the storage area, the processor still stores the quantized input data quantization results and weight quantization results obtained after quantization in the first storage area.

[0069] S1004: The processor 102 performs forward propagation calculation based on the input data quantization result and the weight quantization result to obtain output data.

[0070] Specifically, the processor 102 performs forward propagation calculations on the target quantization unit based on the above-mentioned input data quantization results and weight quantization results, thereby obtaining the output data of the target quantization unit. The forward propagation process of the neural network refers to the process in which the processor executes each layer of the neural network in sequence from the input layer of the neural network and outputs the calculation results from the output layer of the neural network. It can be understood that arithmetic operations on data with lower bit widths are faster than operations on data with higher bit widths. For example, operations using 8-bit or 16-bit integers are often faster than operations using 32-bit floating-point numbers. Therefore, the processor quantizes the data and then performs corresponding calculations to improve the performance of the neural network.

[0071] For example, if Figure 7 In the forward propagation process shown, the target quantization unit is a neural network layer, the input data of the layer is X, the weight is W, X and W are both single-precision 32-bit floating point numbers, and X and W are pre-stored in the first storage area. First, the processor 102 reads X and W from the first storage area, and then the processor performs quantization operations on X and W respectively inside the neural network layer to obtain 8-bit fixed-point integers quantify_X and quantify_W, and then the processor performs the forward propagation calculation corresponding to the neural network layer based on quantify_X and quantify_W to obtain output data Y.

[0072] S1005: The processor 102 reads the input data quantization result and the weight quantization result from the second storage area.

[0073] Specifically, the processor 102 reads the input data quantization results and weight quantization results stored in the forward propagation process from the second storage area. It can be understood that the processor reads the input data quantization results and weight quantization results pre-stored in the second storage area for use in the back propagation calculation, avoiding repeated quantization of input data and weights, speeding up the calculation speed, and reducing memory overhead.

[0074] S1006: The processor 102 performs back propagation calculation based on the input data quantization result, the weight quantization result and the output data to obtain the weight gradient.

[0075] In a possible embodiment, the processor 102 first calculates the output data gradient based on the output data and the loss function, then quantizes the output data gradient to obtain the output data gradient quantization result, and finally performs the back propagation calculation corresponding to the target quantization unit based on the input data quantization result, the weight quantization result and the above-mentioned output data gradient quantization result read from the second storage area, thereby obtaining the weight gradient. The loss function is a functional formula used to measure the gap between the predicted value obtained by forward propagation and the target (actual) value, which helps to optimize the parameters of the neural network. Specifically, the predicted value obtained by forward propagation is matched with the target value, and then the loss can be calculated through the loss function, and then the partial derivative of the neural network parameters is reversely obtained by the loss function. The neural network parameters include weights and biases, and then the weights of the neural network are optimized by the optimizer to minimize the loss. It should be noted that this application does not limit the specific calculation formulas and implementation methods of the quantization operation, loss function, forward propagation calculation and back propagation calculation, and can be designed according to specific needs.

[0076] For example, following the example above, after the processor has completed the forward propagation process of the neural network, it starts to execute the backward propagation process of the neural network. Fig. 9As shown, in a certain layer of the neural network, the processor first calculates the gradient Y_grad of the output data Y by the loss function, and then quantizes Y_grad from floating-point data to fixed-point data to obtain the output data gradient quantization result quantify_Y_grad, and then reads the input data quantization result quantify_X and the weight quantization result quantify_W from the second storage area, and performs the corresponding back-propagation calculation of the layer in the back-propagation process based on quantify_X, quantify_W and quantify_Y_grad to obtain the input data gradient X_grad and the weight gradient W_grad. It can be understood that the processor reuses the input data quantization result and the weight quantization result to participate in the back-propagation calculation, avoiding repeated quantization of the input data and weights, and there is almost no time overhead in obtaining data from the memory, which can speed up the calculation speed and effectively shorten the back-propagation calculation time.

[0077] S1007: The processor 102 updates the weight based on the weight gradient and the optimizer algorithm.

[0078] In a possible embodiment, the optimizer includes a gradient descent method, a momentum optimization method, and an adaptive learning rate optimization algorithm. Among them, the gradient descent method (gradient descent, GD) includes a standard gradient descent method, a stochastic gradient descent (stochastic gradient descent, SGD) and a batch gradient descent method (batch gradient descent, BGD), etc.; the momentum optimization method is a change based on the gradient descent method, which has the effect of accelerating gradient descent, generally including a standard momentum optimization method (momentum), a Newton accelerated gradient (nesterov accelerated gradient, NAG) algorithm, etc.; the adaptive learning rate optimization algorithm mainly includes an AdaGrad algorithm, an RMSprop algorithm, an Adam algorithm, and an AdaDelta algorithm, etc. In actual application scenarios, a suitable optimizer algorithm can be selected to update weights in combination with specific problems, and this application is not limited.

[0079] It can be seen that the processor reduces memory overhead by reading the original input data and weights from the first storage area, then quantizing the input data and weights to obtain input data quantization results and weight quantization results, and then deleting the original input data and weights stored in the first storage area; the processor stores the above-mentioned input data quantization results and weight quantization results obtained by quantizing the forward propagation process in the second storage area, and then reads the above-mentioned input data quantization results and weight quantization results from the second storage area during the reverse propagation process for reverse propagation calculation, thereby avoiding repeated quantization operations on input data and weights, and there is almost no time overhead when reusing the quantization results, which can speed up the training speed of the neural network; the processor uses fixed-point data calculations with lower bit widths instead of floating-point data calculations with higher bit widths, which can reduce memory overhead and speed up the calculation speed at the same time.

[0080] It should be understood that the execution order of the above-mentioned steps S1003 and S1004 can be replaced, that is, before the processor executes the forward propagation calculation, the input data quantization results and the weight quantization results can be stored in the second storage area first, or the input data quantization results and the weight quantization results can be stored in the second storage area after the processor completes the forward propagation calculation.

[0081] Fig.11 It is a structural schematic diagram of a neural network quantization training device 1100 provided in an embodiment of the present application. The neural network quantization training device includes a reading unit 1101, a quantization unit 1102, a storage unit 1103 and a calculation unit 1104.

[0082] The reading unit 1101 is used to read the input data and weight of the target quantization unit from the first storage area, wherein the input data and weight are both floating-point data. The quantization unit 1102 is used to quantize the input data and weight respectively to obtain the input data quantization result and the weight quantization result, wherein the input data quantization result and the weight quantization result are both fixed-point data. The storage unit 1103 is used to store the input data quantization result and the weight quantization result in the second storage area. The calculation unit 1104 is used to perform forward propagation calculation on the target quantization unit based on the input data quantization result and the weight quantization result to obtain output data. The reading unit 1101 is also used to read the input data quantization result and the weight quantization result from the second storage area; the calculation unit 1104 is also used to perform back propagation calculation on the target quantization unit based on the input data quantization result, the weight quantization result and the output data read from the second storage area, so as to obtain the weight gradient; the calculation unit 1104 is also used to update the weight based on the weight gradient and the optimizer algorithm.

[0083] Each unit of the neural network quantization training device 1100 is specifically used to implement Fig.10For the sake of brevity, steps S1001 to S1007 in the embodiment of the neural network quantization training method are not repeated here.

[0084] Another embodiment of the present invention is a computer-readable storage medium, on which a computer program code for executing a neural network quantization training method is stored. When the computer program code is run by a server, the server includes a processor and a memory, the memory stores the aforementioned computer program code, and the processor runs the computer program code in the memory. In some implementation scenarios, the above-mentioned integrated unit can be implemented in the form of a software program module. If implemented in the form of a software program module and sold or used as an independent product, the integrated unit can be stored in a computer-readable memory. Based on this, when the solution of the present invention is embodied in the form of a software product (such as a computer-readable storage medium), the software product can be stored in a memory, which may include several instructions to enable a computer device (such as a personal computer, a server or a network device, etc.) to perform some or all of the steps of the method described in the embodiment of the present invention. The aforementioned memory may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0085] According to different application scenarios, the electronic equipment or device of the present invention may include a server, a cloud server, a server cluster, a data processing device, a robot, a computer, a printer, a scanner, a tablet computer, a smart terminal, a PC device, an Internet of Things terminal, a mobile terminal, a mobile phone, a driving recorder, a navigator, a sensor, a camera, a camera, a video camera, a projector, a watch, a headset, a mobile storage, a wearable device, a visual terminal, an automatic driving terminal, a vehicle, a household appliance, and / or a medical device. The vehicle includes an airplane, a ship, and / or a vehicle; the household appliance includes a television, an air conditioner, a microwave oven, a refrigerator, an electric rice cooker, a humidifier, a washing machine, an electric light, a gas stove, and a range hood; the medical device includes a nuclear magnetic resonance apparatus, a B-ultrasound apparatus, and / or an electrocardiograph. The electronic equipment or device of the present invention can also be applied to the Internet, the Internet of Things, a data center, energy, transportation, public administration, manufacturing, education, power grid, telecommunications, finance, retail, construction sites, medical treatment and other fields. Further, the electronic equipment or device of the present invention can also be used in the cloud, edge end, terminal and other application scenarios related to artificial intelligence, big data and / or cloud computing. In one or more embodiments, electronic devices or apparatuses with high computing power according to the solution of the present invention can be applied to cloud devices (such as cloud servers), while electronic devices or apparatuses with low power consumption can be applied to terminal devices and / or edge devices (such as smart phones or cameras). In one or more embodiments, the hardware information of the cloud device and the hardware information of the terminal device and / or edge device are compatible with each other, so that according to the hardware information of the terminal device and / or edge device, appropriate hardware resources can be matched from the hardware resources of the cloud device to simulate the hardware resources of the terminal device and / or edge device, so as to complete the unified management, scheduling and collaborative work of end-to-end or cloud-edge-to-end.

[0086] It should be noted that, for the purpose of simplicity, the present invention describes some methods and embodiments thereof as a series of actions and combinations thereof, but those skilled in the art will appreciate that the scheme of the present invention is not limited by the order of the described actions. Therefore, based on the disclosure or teaching of the present invention, those skilled in the art will appreciate that some of the steps therein may be performed in other orders or simultaneously. Further, those skilled in the art will appreciate that the embodiments described in the present invention may be regarded as optional embodiments, i.e., the actions or modules involved therein are not necessarily necessary for the implementation of one or some schemes of the present invention. In addition, depending on the different schemes, the present invention also has different emphases on the description of some embodiments. In view of this, those skilled in the art will appreciate that the parts not described in detail in a certain embodiment of the present invention may also refer to the relevant descriptions of other embodiments.

[0087] In terms of specific implementation, based on the disclosure and teachings of the present invention, those skilled in the art can understand that several embodiments disclosed in the present invention can also be implemented in other ways not disclosed herein. For example, with respect to the various units in the electronic device or device embodiments described above, this article splits them on the basis of considering the logical functions, and there may be other ways of splitting them in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features or functions in the units or components can be selectively disabled. In terms of the connection relationship between different units or components, the connection discussed in the above text in conjunction with the accompanying drawings can be a direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection using an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.

[0088] In the present invention, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units. The aforementioned components or units may be located at the same location or distributed on multiple network units. In addition, according to actual needs, some or all of the units may be selected to achieve the purpose of the scheme described in the embodiment of the present invention. In addition, in some scenarios, multiple units in the embodiment of the present invention may be integrated into one unit or each unit may exist physically separately.

[0089] In some other implementation scenarios, the above-mentioned integrated unit can also be implemented in the form of hardware, that is, a specific hardware circuit, which may include digital circuits and / or analog circuits, etc. The physical implementation of the hardware structure of the circuit may include but is not limited to physical devices, and the physical devices may include but are not limited to devices such as transistors or memristors. In view of this, the various devices described herein (such as computing devices or other processing devices) can be implemented by appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs, etc. Further, the aforementioned storage unit or storage device may be any appropriate storage medium (including magnetic storage media or magneto-optical storage media, etc.), which may be, for example, a variable resistance memory (Resistive Random Access Memory, RRAM), a dynamic random access memory (Dynamic Random Access Memory, DRAM), a static random access memory (Static Random Access Memory, SRAM), an enhanced dynamic random access memory (Enhanced Dynamic Random Access Memory, EDRAM), a high bandwidth memory (High Bandwidth Memory, HBM), a hybrid memory cube (Hybrid Memory Cube, HMC), ROM and RAM, etc.

[0090] The foregoing content can be better understood in accordance with the following terms:

[0091] Clause A1. A method for quantized training of a neural network, comprising:

[0092] Reading input data and weights of a target quantization unit from a first storage area, wherein the input data and weights are both floating-point data;

[0093] Quantizing the input data and the weight respectively to obtain an input data quantization result and a weight quantization result, wherein the input data quantization result and the weight quantization result are both fixed-point data;

[0094] storing the input data quantization result and the weight quantization result in a second storage area;

[0095] Based on the input data quantization result and the weight quantization result, performing forward propagation calculation on the target quantization unit to obtain output data;

[0096] Read the input data quantization result and the weight quantization result from the second storage area;

[0097] Based on the input data quantization result, the weight quantization result and the output data read from the second storage area, back-propagation calculation is performed on the target quantization unit to obtain a weight gradient;

[0098] Based on the weight gradients and an optimizer algorithm, the weights are updated.

[0099] Clause A2. According to the method described in Clause A1, after the input data and the weight are quantized respectively to obtain the input data quantization result and the weight quantization result, the input data and the weight in the first storage area are deleted.

[0100] Clause A3. The method according to clause A1 or clause A2, wherein the back propagation calculation is performed on the target quantization unit based on the input data quantization result, the weight quantization result and the output data read from the second storage area to obtain the weight gradient, comprising:

[0101] Based on the output data and the loss function, calculating the output data gradient;

[0102] quantizing the output data gradient to obtain an output data gradient quantization result;

[0103] Based on the input data quantization result, the weight quantization result and the output data gradient quantization result, back propagation calculation is performed on the target quantization unit to obtain the weight gradient.

[0104] Clause A4. According to the method described in Clause A3, the target quantization unit is any one of a neural network layer, a neuron or a channel.

[0105] Clause A5. The method according to clause A4, wherein the quantization is to convert floating-point data of a first bit width into fixed-point data of a second bit width.

[0106] Clause A6. The method of clause A5, wherein the first bit width includes 64 bits and 32 bits, and the second bit width includes 8 bits and 16 bits.

[0107] Clause A7. The method according to clause A6, wherein the optimizer algorithm includes a gradient descent algorithm, a momentum optimization algorithm, and an adaptive learning rate optimization algorithm.

[0108] Clause A8. A neural network quantization training apparatus, the apparatus comprising:

[0109] A reading unit, used for reading input data and weights of a target quantization unit from a first storage area, wherein the input data and weights are both floating point data;

[0110] A quantization unit, used to quantize the input data and the weight respectively to obtain an input data quantization result and a weight quantization result, wherein the input data quantization result and the weight quantization result are both fixed-point data;

[0111] A storage unit, used for storing the input data quantization result and the weight quantization result in a second storage area;

[0112] A calculation unit, configured to perform forward propagation calculation on the target quantization unit based on the input data quantization result and the weight quantization result to obtain output data;

[0113] The reading unit is further used to read the input data quantization result and the weight quantization result from the second storage area;

[0114] The calculation unit is further used to perform back propagation calculation on the target quantization unit based on the input data quantization result, the weight quantization result and the output data read from the second storage area, so as to obtain a weight gradient;

[0115] The computing unit is further configured to update the weight based on the weight gradient and an optimizer algorithm.

[0116] Item A9. According to the device described in Item A8, the storage unit is also used to delete the input data and the weight in the first storage area after quantizing the input data and the weight respectively to obtain the input data quantization result and the weight quantization result.

[0117] Clause A10. The apparatus according to clause A8 or clause A9, wherein the computing unit is specifically configured to:

[0118] Based on the output data and the loss function, calculating the output data gradient;

[0119] quantizing the output data gradient to obtain an output data gradient quantization result;

[0120] Based on the input data quantization result, the weight quantization result and the output data gradient quantization result, back propagation calculation is performed on the target quantization unit to obtain the weight gradient.

[0121] Item A11. According to the device described in Item A10, the target quantization unit is any one of a neural network layer, a neuron or a channel.

[0122] Clause A12. The apparatus of clause A11, wherein the quantization is to convert floating-point data of a first bit width into fixed-point data of a second bit width.

[0123] Clause A13. The apparatus of clause A12, wherein the first bit width includes 64 bits and 32 bits, and the second bit width includes 8 bits and 16 bits.

[0124] Clause A14. In the apparatus of clause A13, the optimizer algorithm comprises a gradient descent algorithm, a momentum optimization algorithm, and an adaptive learning rate optimization algorithm.

[0125] Clause A15. An electronic device comprising a processor and a memory;

[0126] The memory is used to store computer programs;

[0127] The processor is used to execute the computer program stored in the memory, so that the electronic device performs the method as described in any one of clauses A1 to A7.

[0128] Clause A16. A computer-readable storage medium comprising a program or instructions, when the program or instructions are executed on a computer device, the method as described in any one of Clauses A1 to A7 is performed.

[0129] The embodiments of the present invention are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A neural network quantization training method, characterized in that: The method comprises: Reading input data and weights of a target quantization unit from a first storage area, wherein the input data and weights are both floating-point data; Quantizing the input data and the weight respectively to obtain an input data quantization result and a weight quantization result, wherein the input data quantization result and the weight quantization result are both fixed-point data; storing the input data quantization result and the weight quantization result in a second storage area; Based on the input data quantization result and the weight quantization result, performing forward propagation calculation on the target quantization unit to obtain output data; Read the input data quantization result and the weight quantization result from the second storage area; Based on the input data quantization result, the weight quantization result and the output data read from the second storage area, back-propagation calculation is performed on the target quantization unit to obtain a weight gradient; Based on the weight gradient and the optimizer algorithm, updating the weight; The performing back propagation calculation on the target quantization unit based on the input data quantization result, the weight quantization result and the output data read from the second storage area to obtain the weight gradient comprises: Based on the output data and the loss function, calculating the output data gradient; quantizing the output data gradient to obtain an output data gradient quantization result; Based on the input data quantization result, the weight quantization result and the output data gradient quantization result, back propagation calculation is performed on the target quantization unit to obtain the weight gradient.

2. The method according to claim 1, characterized in that After the input data and the weight are quantized respectively to obtain the input data quantization result and the weight quantization result, the input data and the weight in the first storage area are deleted.

3. The method according to claim 1, characterized in that The target quantization unit is any one of a neural network layer, a neuron or a channel.

4. The method according to claim 3, characterized in that The quantization is to convert the floating point data of the first bit width into the fixed point data of the second bit width.

5. The method according to claim 4, characterized in that The first bit width includes 64 bits and 32 bits, and the second bit width includes 8 bits and 16 bits.

6. The method according to claim 5, characterized in that The optimizer algorithms include gradient descent algorithm, momentum optimization algorithm and adaptive learning rate optimization algorithm.

7. A neural network quantization training device, characterized in that: The device comprises: A reading unit, used for reading input data and weights of a target quantization unit from a first storage area, wherein the input data and weights are both floating point data; A quantization unit, used to quantize the input data and the weight respectively to obtain an input data quantization result and a weight quantization result, wherein the input data quantization result and the weight quantization result are both fixed-point data; A storage unit, used for storing the input data quantization result and the weight quantization result in a second storage area; A calculation unit, configured to perform forward propagation calculation on the target quantization unit based on the input data quantization result and the weight quantization result to obtain output data; The reading unit is further used to read the input data quantization result and the weight quantization result from the second storage area; The calculation unit is further used to perform back propagation calculation on the target quantization unit based on the input data quantization result, the weight quantization result and the output data read from the second storage area, so as to obtain a weight gradient; The computing unit is further configured to update the weight based on the weight gradient and the optimizer algorithm; In the aspect of performing back propagation calculation on the target quantization unit based on the input data quantization result, the weight quantization result and the output data read from the second storage area, so as to obtain the weight gradient, the calculation unit is used to: Based on the output data and the loss function, calculating the output data gradient; quantizing the output data gradient to obtain an output data gradient quantization result; Based on the input data quantization result, the weight quantization result and the output data gradient quantization result, back propagation calculation is performed on the target quantization unit to obtain the weight gradient.

8. An electronic device, characterized in that: including a processor and a memory; The memory is used to store computer programs; The processor is configured to execute the computer program stored in the memory, so that the electronic device executes the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The method comprises a program or an instruction, and when the program or the instruction is executed on a computer device, the method according to any one of claims 1 to 6 is executed.

Images