A training method, device and computing device for a neural network model

Abstract

The present invention discloses a training method, apparatus, and computing device for a neural network model. The method includes a forward propagation step, a backpropagation step, and a parameter update step. In the parameter update step, the following steps are performed: calculating parameter update values for the current network layer based on parameter gradients, generating a fourth block of floating-point numbers corresponding to the parameter update values, wherein the bit width of the fourth block of floating-point numbers is a third predetermined value; and updating the parameters of the current network layer based on the first block of floating-point numbers and the fourth block of floating-point numbers, generating a first block of floating-point numbers corresponding to the updated parameters.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a training method, apparatus, and computing device for a neural network model. Background Art

[0002] With the rapid development of artificial intelligence in recent years, AI, particularly deep learning algorithms, has been widely applied in fields such as visual imagery, speech recognition, and natural language processing. With the widespread adoption of edge intelligence, an increasing number of AI computations need to be performed on edge devices to deliver new interactive experiences. However, compared to servers, edge devices have lower computing power and less memory. Therefore, optimizing deep learning algorithms, which consume high computing power and storage, for deployment on edge devices is a hot topic in the AI field.

[0003] One method for optimizing deep learning algorithms, especially neural network models, is to quantize model parameters, that is, to quantize parameters such as single-precision floating-point numbers into integer format. Since the processor in the end device, whether it is a GPU or other processor, can process integer operations more quickly than floating-point operations, parameter quantization can be used to accelerate the deep learning model deployed on the end device, thereby speeding up the inference process.

[0004] However, low-precision neural network model training is still primarily based on high-precision floating-point computing platforms. As the network scale and training datasets of neural network models continue to grow, training neural network models requires enormous computing power, storage space, and power consumption. Research on quantization compression technology has shown that it has the performance advantages of low storage, low power consumption, and high computing power. Neural network model training includes three parts: forward propagation, backpropagation, and parameter update. Current research on low-precision quantization compression technology mainly focuses on the forward reasoning part of the neural network model, with less research on low-precision backpropagation and parameter update. Since the amount of computation required for backpropagation is twice that of forward propagation, quantized reverse gradient propagation is crucial.

[0005] During the training of low-precision neural network models, the parameter update values are too small compared to the high-precision parameter values (usually single-precision floating-point numbers), making it impossible to directly accumulate the parameter update values to the low-precision parameters. To address this problem, low-precision parameter values are usually used in forward propagation and directional propagation, while the corresponding high-precision floating-point values are additionally stored for parameter updates. This not only increases additional storage resources, but also requires frequent conversion operations between low-precision and high-precision data types, resulting in low training efficiency. Summary of the Invention

[0006] In view of the above problems, the present invention is proposed to provide a method, apparatus and computing device for training a neural network model that overcomes the above problems or at least partially solves the above problems.

[0007] According to one aspect of the present invention, a training method for a neural network model is provided, which is executed in a server, wherein the neural network model includes multiple network layers, and the method includes: a forward propagation step: obtaining a first block of floating-point numbers corresponding to the parameters of the current network layer; quantizing the activation value output by the previous network layer into a second block of floating-point numbers, wherein the bit width of the first block of floating-point numbers and the second block of floating-point numbers is a first predetermined value; calculating the activation value of the current network layer based on the first block of floating-point numbers and the second block of floating-point numbers, and outputting it to the next network layer; a backward propagation step: quantizing the activation value gradient output by the next network layer into a third block of floating-point numbers. Floating-point numbers, wherein the bit width of the third block of floating-point numbers is the second predetermined value; based on the third block of floating-point numbers and the first block of floating-point numbers, the activation value gradient of the previous network layer is calculated and output to the previous network layer; based on the third block of floating-point numbers and the second block of floating-point numbers, the parameter gradient of the current network layer is calculated; parameter updating step: based on the parameter gradient, the parameter update value of the current network layer is calculated, and the fourth block of floating-point numbers corresponding to the parameter update value is generated, wherein the bit width of the fourth block of floating-point numbers is the third predetermined value; based on the first block of floating-point numbers and the fourth block of floating-point numbers, the parameters of the current network layer are updated, and the first block of floating-point numbers corresponding to the updated parameters are generated.

[0008] Optionally, in the training method of the neural network model of the present invention, the current network layer is a fully connected layer, and the fully connected layer includes a linear processing unit and an activation function unit; the calculation of the activation value of the current network layer based on the first block of floating-point numbers and the second block of floating-point numbers includes: inputting the first block of floating-point numbers and the second block of floating-point numbers into the linear processing unit for processing, and outputting a linear value with a bit width of a third predetermined value; quantizing the linear value into a quantized linear value with a bit width of the first predetermined value; inputting the quantized linear value into the activation function unit for processing, and outputting an activation value with a bit width of the third predetermined value.

[0009] Optionally, in the training method of the neural network model of the present invention, the calculation of the activation value gradient of the previous network layer based on the third block of floating-point numbers and the first block of floating-point numbers includes: reversely deriving the third block of floating-point numbers based on the activation function adopted by the activation function unit to obtain a linear value gradient with a bit width of a third predetermined value; quantizing the linear value gradient into a quantized linear value gradient with a bit width of a second predetermined value; reversely deriving the quantized linear value gradient based on the first block of floating-point numbers to obtain the activation value gradient of the previous network layer; the calculation of the parameter gradient of the current network layer based on the third block of floating-point numbers and the second block of floating-point numbers includes: reversely deriving the linear value gradient based on the second block of floating-point numbers to obtain the parameter gradient of the current network layer.

[0010] Optionally, in the training method of the neural network model of the present invention, the current network layer is a convolution layer, and the convolution layer includes a convolution unit, a pooling unit and an activation function unit; the activation value of the current network layer is calculated based on the first block of floating-point numbers and the second block of floating-point numbers, including: inputting the first block of floating-point numbers and the second block of floating-point numbers into the convolution unit for processing, and outputting a linear value with a bit width of a third predetermined value; quantizing the linear value into a quantized linear value with a bit width of the first predetermined value; inputting the quantized linear value into the pooling unit for processing, and outputting a pooled value; inputting the pooled value into the activation function unit for processing, and outputting an activation value with a bit width of the third predetermined value.

[0011] Optionally, in the training method of the neural network model of the present invention, the calculation of the activation value gradient of the previous network layer based on the third block of floating-point numbers and the first block of floating-point numbers includes: reversely deriving the third block of floating-point numbers based on the activation function adopted by the activation function unit to obtain a pooling value gradient with a bit width of a third predetermined value; reversely deriving the pooling value gradient based on the pooling template adopted by the pooling unit to obtain a linear value gradient; quantizing the linear value gradient into a quantized linear value gradient with a bit width of a second predetermined value; reversely deriving the quantized linear value gradient based on the first block of floating-point numbers to obtain the activation value gradient of the previous network layer; the calculation of the parameter gradient of the current network layer based on the third block of floating-point numbers and the second block of floating-point numbers includes: reversely deriving the linear value gradient based on the second block of floating-point numbers to obtain the parameter gradient of the current network layer.

[0012] Optionally, in the training method of the neural network model of the present invention, the first predetermined value is smaller than the second predetermined value, and the second predetermined value is smaller than the third predetermined value.

[0013] Optionally, in the training method of the neural network model of the present invention, the first predetermined value is 8, the second predetermined value is 16, and the third predetermined value is 32.

[0014] Optionally, in the training method of the neural network model of the present invention, the parameters of the current network layer are updated based on the first block of floating-point numbers and the fourth block of floating-point numbers to generate the first block of floating-point numbers corresponding to the updated parameters, including: obtaining the fifth block of floating-point numbers corresponding to the delayed update value after the previous iteration, wherein the delayed update value is the part of the parameter update value that has not been updated to the parameter, and the bit width of the fifth block of floating-point numbers is the second predetermined value; shifting the mantissa of the fourth block of floating-point numbers right by a first predetermined value and adding them to the fifth block of floating-point numbers; calculating the difference between the first block of floating-point numbers and the fifth block of floating-point numbers after the mantissa is shifted right by a second predetermined value to obtain a first difference; calculating the sum of the first block of floating-point numbers and the fifth block of floating-point numbers, and then subtracting the first difference as the fifth block of floating-point numbers corresponding to the delayed update value after this iteration; and updating the first block of floating-point numbers to the first difference.

[0015] Optionally, in the training method of the neural network model of the present invention, the first predetermined value is the difference between the exponent of the fifth block of floating-point numbers and the exponent of the fourth block of floating-point numbers, and the second predetermined value is the bit width of the fifth block of floating-point numbers - 1.

[0016] Optionally, in the training method of the neural network model of the present invention, when performing the shift operation of the mantissa, the truncated part is added to the untruncated part in a rounded manner.

[0017] Optionally, in the training method of the neural network model of the present invention, the server includes an acceleration unit, and the training method is suitable for being executed by the acceleration unit.

[0018] Optionally, in the training method of the neural network model of the present invention, the acceleration unit is a neural network processing unit NPU or a graphics processing unit GPU.

[0019] Optionally, in the training method of the neural network model of the present invention, the server is deployed in a data center.

[0020] According to another aspect of the present invention, a training method for a neural network model is provided, which is executed in a terminal device, wherein the neural network model includes multiple network layers, and the method includes: a forward propagation step: obtaining a first block of floating-point numbers corresponding to the parameters of the current network layer; quantizing the activation value output by the previous network layer into a second block of floating-point numbers, wherein the bit widths of the first block of floating-point numbers and the second block of floating-point numbers are a first predetermined value; calculating the activation value of the current network layer based on the first block of floating-point numbers and the second block of floating-point numbers, and outputting it to the next network layer; a backward propagation step: quantizing the activation value gradient output by the next network layer into a third block of floating-point numbers. point numbers, wherein the bit width of the third block of floating-point numbers is the second predetermined value; the activation value gradient of the previous network layer is calculated based on the third block of floating-point numbers and the first block of floating-point numbers, and is output to the previous network layer; the parameter gradient of the current network layer is calculated based on the third block of floating-point numbers and the second block of floating-point numbers; a parameter updating step: calculating the parameter update value of the current network layer based on the parameter gradient, generating a fourth block of floating-point numbers corresponding to the parameter update value, wherein the bit width of the fourth block of floating-point numbers is the third predetermined value; updating the parameters of the current network layer based on the first block of floating-point numbers and the fourth block of floating-point numbers, and generating a first block of floating-point numbers corresponding to the updated parameters.

[0021] Optionally, in the training method of the neural network model of the present invention, the terminal device is a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a speaker computing device, a vehicle computing device, a wearable device including a computing device, or a home device including a computing device.

[0022] Optionally, in the training method of the neural network model of the present invention, the terminal device includes an acceleration unit, and the training method is suitable for being executed by the acceleration unit.

[0023] Optionally, in the training method of the neural network model of the present invention, the acceleration unit is a neural network processing unit NPU or a graphics processing unit GPU.

[0024] According to another aspect of the present invention, a training device for a neural network model is provided, comprising: a forward propagation module adapted to: obtain a first block of floating-point numbers corresponding to the parameters of a current network layer; quantize the activation values output by the previous network layer into a second block of floating-point numbers, wherein the bit widths of the first block of floating-point numbers and the second block of floating-point numbers are a first predetermined value; calculate the activation value of the current network layer based on the first block of floating-point numbers and the second block of floating-point numbers, and output it to the next network layer; a backward propagation module adapted to: quantize the activation value gradient output by the next network layer into a third block of floating-point numbers, wherein the bit widths of the third block of floating-point numbers are a first predetermined value; calculate the activation value of the current network layer based on the first block of floating-point numbers and the second block of floating-point numbers, and output it to the next network layer; The width of the fourth block of floating-point numbers is the second predetermined value; the activation value gradient of the previous network layer is calculated based on the third block of floating-point numbers and the first block of floating-point numbers, and the gradient is output to the previous network layer; the parameter gradient of the current network layer is calculated based on the third block of floating-point numbers and the second block of floating-point numbers; the parameter update module is adapted to: calculate the parameter update value of the current network layer based on the parameter gradient, generate a fourth block of floating-point numbers corresponding to the parameter update value, wherein the bit width of the fourth block of floating-point numbers is the third predetermined value; update the parameters of the current network layer based on the first block of floating-point numbers and the fourth block of floating-point numbers, and generate a first block of floating-point numbers corresponding to the updated parameters.

[0025] According to another aspect of the present invention, a computing device is provided, comprising: at least one processor; and a memory storing program instructions, wherein the program instructions are configured to be suitable for execution by the at least one processor, and the program instructions include instructions for executing the above method.

[0026] According to another aspect of the present invention, a readable storage medium storing program instructions is provided. When the program instructions are read and executed by a computing device, the computing device executes the above method.

[0027] According to the training scheme of the neural network model of the embodiment of the present invention, based on the combination of block floating-point quantization and delayed update, the parameter value, the delayed update value and the parameter update value have the same decimal point position through the shared exponent control of the block floating-point number, and the part of the parameter update value that is valid relative to the delayed update value (the overlapping part of the mantissa of the parameter update value and the delayed update value) is added to the delayed update value. When the delayed update value is large enough, the parameter value is updated according to the delayed update value. In this way, low-precision parameter values can be used in the forward propagation, backpropagation and parameter update processes, without the need to additionally save the high-precision floating-point values corresponding to the low-precision parameter values, and without the need to frequently perform conversion operations between high-precision and low-precision type data, thereby saving storage resources while improving the training speed of the neural network model.

[0028] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0030] Figure 1 shows a structural diagram of a data center according to an embodiment of the present invention;

[0031] Figure 2 FIG2 shows an internal structure diagram of a server in a data center according to an embodiment of the present invention;

[0032] Figure 3 is a diagram showing the connection relationship between a scheduling unit and an acceleration unit in a server according to an embodiment of the present invention;

[0033] Figure 4 is a diagram of the internal structure of an acceleration unit core according to one embodiment of the present invention;

[0034] Figure 5 A schematic diagram of a method for representing block floating-point numbers in an embodiment of the present invention is shown;

[0035] Figure 6 A flowchart of a method 600 for training a neural network model according to one embodiment of the present invention is shown;

[0036] Figure 7 Schematic diagram of the training process of the convolutional layer in an embodiment of the present invention is shown;

[0037] Figure 8 Schematic diagram of parameter updating process in an embodiment of the present invention is shown;

[0038] Figure 9 A schematic diagram of a training device 900 for a neural network model according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0039] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0040] First, an implementation environment of the training method of the neural network model according to an embodiment of the present invention is introduced.

[0041] Data Center

[0042] Data centers are globally coordinated networks of specialized equipment used to transmit, accelerate, display, compute, and store data on the internet's infrastructure. In the future, data centers will become a competitive asset for businesses. With the widespread use of data centers, artificial intelligence and other technologies are increasingly being applied to data centers. Neural networks, a key AI technology, are already widely used in data center big data analytics.

[0043] In traditional large data centers, the network structure is usually as follows Figure 1 As shown in Figure 1, this is the hierarchical inter-networking model. This model consists of the following parts:

[0044] Server 140 : Each server 140 is a processing and storage entity of the data center. The processing and storage of a large amount of data in the data center are completed by these servers 140 .

[0045] Access switch 130: Access switches 130 connect servers 140 to the data center. One access switch 130 connects to multiple servers 140. Access switches 130 are typically located at the top of a rack, and are therefore also called top-of-rack switches. They physically connect to the servers.

[0046] Aggregation switch 120: Each aggregation switch 120 connects to multiple access switches 130 and provides other services, such as firewall, intrusion detection, network analysis, etc.

[0047] Core switch 110: Core switch 110 provides high-speed forwarding for packets entering and leaving the data center and provides connectivity for aggregation switch 120. The entire data center network is divided into an L3 routing network and an L2 routing network. Core switch 110 generally provides a resilient L3 routing network for the entire data center network.

[0048] Typically, aggregation switch 120 is the demarcation point between the L2 and L3 routing networks. The network below aggregation switch 120 is the L2 network, and the network above it is the L3 network. Each aggregation switch group manages a point of delivery (POD), each of which is an independent VLAN network. Servers can migrate within a POD without changing their IP addresses or default gateways, as each POD corresponds to a single L2 broadcast domain.

[0049] The Spanning Tree Protocol (STP) is typically used between aggregation switches 120 and access switches 130. STP ensures that only one aggregation switch 120 is available for a given VLAN, with other aggregation switches 120 being used only in the event of a failure. This means that horizontal scalability is impossible at the aggregation switch 120 level because even if multiple aggregation switches 120 are added, only one is still operational.

[0050] server

[0051] Since the server 140 is the actual processing device in the data center, Figure 2 The figure shows a block diagram of the internal structure of a server 140. The server 140 includes a memory 210, a scheduling unit cluster 270 and an acceleration unit cluster 280 connected by a bus. The scheduling unit cluster 270 includes a plurality of scheduling units 220. The acceleration unit cluster 280 includes a plurality of acceleration units 230. In the embodiment of the present disclosure, the acceleration unit is a special processing unit designed mainly to accelerate the operation processing speed of the neural network model, which can be embodied as a processing unit (NPU) specially designed for neural network operation processing, a graphics processing unit (GPU), an application-specific integrated circuit (ASIC) and a field-programmable gate array (FPGA). The scheduling unit is a processing unit that schedules the acceleration unit and allocates a sequence of instructions to be executed to each acceleration unit. It can take the form of a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) and the like.

[0052] The traditional central processing unit (CPU) architecture design results in the control unit and storage unit occupying a large portion of the space, while the computing unit occupies insufficient space. Therefore, while it is very effective in logic control, it is inefficient in large-scale parallel computing. Therefore, various specialized acceleration units have been developed to more effectively improve the computing speed of calculations for different functions and different fields. The acceleration unit proposed in the present invention is a processing unit specifically designed to accelerate the computing speed of neural network models. It adopts a data-driven parallel computing architecture and is used to process a large number of operations (such as convolution and pooling) of each neural network node. Because the data and intermediate results of the large number of operations (such as convolution and pooling) of each neural network node are closely related and are frequently used throughout the entire computing process, the existing CPU architecture requires frequent access to external memory due to the small memory capacity within the CPU core, resulting in inefficient processing. Using this acceleration unit specifically designed to accelerate the computing speed of neural network models, each core has on-chip memory with a storage capacity suitable for neural network calculations, avoiding frequent access to external memory, greatly improving processing efficiency and computing performance.

[0053] The acceleration unit 230 must accept the scheduling of the scheduling unit 220. Figure 2 As shown, the memory 210 stores various neural network models, including the nodes of these models and the weights and bias data of the nodes. These neural network models are used when needed. Figure 2 A scheduling unit 220 is deployed to an acceleration unit 230. That is, the scheduling unit 220 can send the address of the parameters in the model (such as the weights and biases of each node) in the memory 210 to the acceleration unit 230 in the form of instructions. When the acceleration unit 230 actually uses the neural network model for calculation, it will directly address these parameters in the memory 210 according to the addresses of these parameters in the memory 210, and temporarily store them in its on-chip memory. When the acceleration unit 230 actually uses the neural network model for calculation, the scheduling unit 220 will also send the input parameters of the model to the acceleration unit 230 in the form of instructions, and temporarily store them in the on-chip memory of the acceleration unit 230. In this way, the acceleration unit 230 can perform inference calculations based on these input parameters and the parameters in the model (such as weights and biases).

[0054] Internal structure of the scheduling unit and acceleration unit

[0055] The following combination Figure 3 The internal structure diagram of the scheduling unit 220 and the acceleration unit 230 specifically illustrates how the scheduling unit 220 schedules the acceleration unit 230 to work.

[0056] like Figure 3As shown, the scheduling unit 220 includes multiple processor cores 222 and a cache 221 shared by the multiple processor cores 222. Each processor core 222 includes an instruction fetch unit 203, an instruction decoding unit 224, an instruction issuing unit 225, and an instruction executing unit 226.

[0057] The instruction fetch unit 223 is used to move the instruction to be executed from the memory 210 to the instruction register (which may be Figure 3 The register file 229 shown is used to store a register for instructions) and receives the next instruction fetch address or calculates the next instruction fetch address according to an instruction fetch algorithm. The instruction fetch algorithm includes, for example, increasing or decreasing the address according to the instruction length.

[0058] After fetching the instruction, the dispatch unit 220 enters the instruction decoding stage. The instruction decoding unit 224 decodes the fetched instruction according to the predetermined instruction format to obtain the operand fetch information required by the fetched instruction, thereby preparing for the operation of the instruction execution unit 226. The operand fetch information may point to an immediate value, a register, or other software / hardware that can provide a source operand.

[0059] The instruction issuing unit 225 is located between the instruction decoding unit 224 and the instruction executing unit 226 and is used for scheduling and controlling instructions to efficiently distribute each instruction to different instruction executing units 226 , thereby enabling parallel operation of multiple instructions.

[0060] After the instruction issuing unit 225 sends the instruction to the instruction executing unit 226, the instruction executing unit 226 begins executing the instruction. However, if the instruction executing unit 226 determines that the instruction should be executed by an acceleration unit, it forwards the instruction to the corresponding acceleration unit for execution. For example, if the instruction is for neural network inference, the instruction executing unit 226 will no longer execute the instruction and instead send it via the bus to the acceleration unit 230 for execution.

[0061] The acceleration unit 30 includes multiple cores 236 ( Figure 3 Four cores are shown in the figure, but those skilled in the art should understand that the acceleration unit 230 may also include other numbers of cores 236), a command processor 237, a direct storage access mechanism 235, and a bus channel 231.

[0062] The bus channel 231 is a channel for instructions to enter and exit the acceleration unit 230 from the bus.

[0063] Direct Memory Access (DMA) mechanism 235 is a feature provided by some computer bus architectures that enables data to be written directly from an attached device to the computer's mainboard memory. This greatly improves data access efficiency compared to a method where all data transfers between devices must pass through the dispatch unit. Because of this mechanism, the core of the acceleration unit 230 can directly access memory 210 and read parameters in the neural network model (such as the weights and biases of each node), greatly improving data access efficiency.

[0064] The command processor 237 assigns instructions sent from the scheduling unit 220 to the acceleration unit 230 to a core 236 for execution. The instruction execution unit 226 sends a sequence of pending instructions to be executed by the acceleration unit 230 to the acceleration unit 230. After entering the sequence of pending instructions from the bus channel 231, it is cached in the command processor 237, which then selects a core 236 and assigns the instruction sequence to it for execution. The command processor 237 also manages synchronization between cores 236.

[0065] Acceleration unit core

[0066] Figure 4 FIG. 2 is a diagram showing the internal structure of the acceleration unit core 236 according to an embodiment of the present invention.

[0067] In one embodiment, Figure 4 As shown, the acceleration unit core 236 includes a tensor engine 310 , a pooling engine 320 , a memory copy engine 330 , a sequencer 350 , an instruction cache 340 , an on-chip memory 360 , and a constant buffer 370 .

[0068] The instruction sequence assigned by the command processor 237 to the acceleration unit core 236 is first cached in the instruction cache 340. The sequencer 350 then fetches instructions from the instruction cache 340 in first-in, first-out order and, depending on the nature of the instruction, assigns them to the tensor engine 310, pooling engine 320, or memory copy engine 330 for execution. The tensor engine 310 is responsible for processing operations such as convolution and matrix multiplication in the neural network model. The pooling engine 320 is responsible for processing pooling operations in the neural network model. The memory copy engine 330 is responsible for copying operands stored in the on-chip memory 360 within the core 236 to the memory shared between the cores 236, or to the on-chip memory 360 within other cores 236. The sequencer 350 determines whether to assign the instruction to the tensor engine 310, pooling engine 320, or memory copy engine 330 based on the nature of the operation, such as convolution, matrix multiplication, pooling, or operand copy.

[0069] On-chip memory 360 is an in-core memory that stores model parameters in the neural network model, as well as input parameters and various intermediate results during actual use of the neural network model. Constant buffer 370 is a buffer that stores constant parameters in the neural network model (e.g., hyperparameters) in addition to weight parameters. As described above, when the scheduling unit 220 pre-configures the neural network model in the acceleration unit 230, the scheduling unit 220 sends the addresses of the model parameters in memory 210 to the acceleration unit 230 in the form of instructions. These parameters include node weights and other parameters (e.g., hyperparameters). For weights, the acceleration unit 230 retrieves them from the corresponding locations in memory 210 during actual neural network model operations and stores them in on-chip memory 360. For other parameters, the acceleration unit 230 retrieves them from the corresponding locations in memory 210 during actual neural network model operations and stores them in constant buffer 370. Furthermore, when the instruction to actually start inference is assigned by the command processor 237 to the core 236 for execution, the input parameters (inputs to the neural network model) in the instruction are also stored in on-chip memory 360. In addition, after the tensor engine 310 and the pooling engine 320 perform convolution or pooling operations, the various intermediate results obtained are also stored in the on-chip memory 360.

[0070] The training method for the neural network model of the embodiment of the present invention can be executed in the server 140 of the above-mentioned data center. Depending on the size of the neural network model and the size of the training data set, the model training method can be executed by an acceleration unit 230 in one server 140, or distributed training of the neural network model can be performed by multiple acceleration units 230 in one or more servers 120. The acceleration unit 230 can obtain the neural network model and training data from the memory 210 and train the neural network model based on the obtained training data.

[0071] Neural network model training involves three steps: forward propagation, backpropagation, and parameter updates. Current research on low-precision quantization and compression techniques focuses primarily on the forward reasoning phase of neural network models, with less research on low-precision backpropagation and parameter updates. Because backpropagation requires twice as much computation as forward propagation, quantizing the backward gradient propagation is crucial.

[0072] However, during the training of low-precision neural network models, the parameter update values are too small compared to the low-precision parameter values, making it impossible to directly accumulate the parameter update values to the low-precision parameters. To address this problem, low-precision parameter values are usually used in forward propagation and directional propagation, while the corresponding high-precision floating-point values (usually single-precision floating-point numbers) are additionally stored for the low-precision parameter values for parameter updates. This not only increases additional storage resources, but also requires frequent conversion operations between low-precision and high-precision data types, resulting in low training efficiency.

[0073] To this end, an embodiment of the present invention proposes a neural network model training scheme based on a combination of block floating point (BFP) and lazy update. In this scheme, the parameter value, the delayed update value and the parameter update value have the same decimal point position through the shared exponent control of the block floating point number, and the part of the parameter update value that is valid relative to the delayed update value (for example, the overlapping part of the mantissa of the parameter update value and the delayed update value) is added to the delayed update value. When the delayed update value is large enough, the parameter value is updated according to the delayed update value. Among them, the parameter update value refers to the value to be updated of the parameter value calculated based on the parameter gradient and the optimization algorithm (such as the stochastic gradient descent method, the momentum gradient descent method, etc.) in one iteration; the parameter delayed update value refers to the part of the parameter update value that has not been updated to the parameter value from the first iteration to the previous iteration. For example, a total of n iterations have been carried out so far, and the parameter update value at the i-th iteration is x i , x i The part updated to the parameter value is y i , then the delayed update value is In this way, low-precision parameter values can be used in the forward propagation, backpropagation and parameter update processes, without the need to additionally save the high-precision floating-point values corresponding to the low-precision parameter values, and without the need to frequently perform conversion operations between high-precision and low-precision data types, thereby saving storage resources and improving the training speed of the neural network model.

[0074] The following first introduces the principle of quantizing multiple floating-point data into a block of floating-point data. In the model training method embodiment later, the process of delayed update is introduced in detail.

[0075] A block floating-point number is a data format that uses a shared exponent for an entire block of integer data. This format quantizes multiple floating-point data, using the largest data element as a basis to form an exponent (integer data) that serves as the shared exponent for the multiple floating-point data elements. The shared exponent is then used to generate the mantissa (integer data) for each floating-point data element. In other words, multiple floating-point data elements form a data block, which can be represented as a block floating-point number.

[0076] Figure 5 A schematic diagram of a method for representing block floating-point numbers in an embodiment of the present invention is shown. Figure 5 The block floating-point number shown is a block floating-point number with a bit width of 8 bits, a mantissa part of which is 8 bits (including the sign bit), and an exponent part of which is also 8 bits (including the sign bit). The block floating-point number corresponds to 3 floating-point data.

[0077] It should be noted that in the embodiments of the present invention, the bit width of a block floating-point number refers to the bit width of the mantissa (i.e., the number of bits in the mantissa), and that the bit width of the exponent is the same for block floating-point numbers of different bit widths, for example, 8 bits. Thus, an 8-bit block floating-point number has both an 8-bit exponent and a 8-bit mantissa; a 16-bit block floating-point number has a 16-bit exponent and an 8-bit mantissa; a 32-bit block floating-point number has a 32-bit exponent and an 8-bit mantissa; and an n-bit block floating-point number has an n-bit exponent and an 8-bit mantissa.

[0078] To quantize floating-point numbers into integers, affine quantization is usually used. The formula is as follows:

[0079] r=s(qz)

[0080] Here, s (expansion factor) and z (zero point) represent the quantization parameters, r is a floating-point value, and q is the quantized value. Typically, s is represented using 32-bit single-precision floating-point or fixed-point, and z is mapped to the true zero point of the quantized value. Both z and q are integer data.

[0081] To more effectively improve implementation efficiency in customized hardware, the expansion coefficient is constrained to a format with an exponent of 2 during symmetric quantization, and z is set to 0. This quantization method is used in embodiments of the present invention to replace the high-complexity multiplication calculations between fixed-point and floating-point numbers through bit shifting.

[0082] In this way, the quantization formula can be expressed as:

[0083]

[0084] in, is the expansion coefficient, E s is the exponent, q is the mantissa. For multiple floating point numbers r, the range of their values is r∈[r min ,r max ], then the shared index of the block floating point numbers corresponding to multiple floating point numbers can be expressed as follows:

[0085] E s =ceil(log2{max(|r min |,|r max |)})-(b-1)

[0086] Among them, r min is the minimum value of multiple floating-point numbers, r max The ceil() function rounds up to an integer. b represents the bit width of the floating-point number. For example, for a 32-bit single-precision floating-point number, b=32.

[0087] The multiplication of two floating point numbers is:

[0088] i a·b =i a ·i b ,E a·b =E a +E b

[0089] Among them, i a 、i b are the mantissa parts of two blocks of floating-point numbers, E a 、E b The exponent part of the two blocks of floating point numbers, i a·b is the mantissa of the product, E a·b In the embodiment of the present invention, intermediate calculation results are usually temporarily stored as 32-bit block floating-point numbers. The main purpose is to prevent overflow in subsequent multiplication and accumulation operations in convolution. In this way, a new shared exponent is obtained based on the above formula.

[0090] The above provides an implementation method for quantizing multiple floating-point numbers into block floating-point numbers. In the training of the neural network model of the embodiment of the present invention, this method can be used to quantize multiple floating-point numbers into block floating-point numbers, or other implementation methods of quantizing multiple floating-point numbers into block floating-point numbers in the prior art can be used. The present invention does not limit the specific quantization method used, and those skilled in the art can make reasonable choices based on specific needs.

[0091] Figure 6 A flow chart of a training method 600 for a neural network model according to an embodiment of the present invention is shown. The method 600 can be executed in the aforementioned server 140, for example, by an acceleration unit 230 in one server 140, or by multiple acceleration units 230 in one or more servers 120 to perform distributed training on the neural network model. The acceleration unit 230 can obtain the neural network model and training data from the memory 210, and train the neural network model based on the obtained training data. The method 600 can be applied to various scenarios such as images, voice, video, and machine translation. For example, in an image scenario, the corresponding neural network model can be an image classification model, a target detection model, etc.; in a machine translation scenario, the corresponding neural network model can be a neural network machine translation model. As Figure 6 As shown, the method 600 includes a forward propagation step S602, a backpropagation step S604 and a parameter updating step S606.

[0092] In step S602, the sample features of the training samples are input into the neural network model, and the model output of the neural network model is obtained after being processed by each network layer of the neural network model. The loss function is calculated based on the model output and the sample labels of the training samples.

[0093] Training data can include image samples, speech samples, and natural language processing samples. For example, if the neural network model to be trained is a neural machine translation model, each piece of training data is a text pair, representing the correspondence between a text in a first language and a text in a second language. The first language text is the sample feature, serving as the model input, and the second language text is the sample label.

[0094] The first network layer of the neural network model takes sample features as input and calculates activation values based on the input and the parameters (weights, biases) of the first network layer. Then, the second network layer takes the activation values output by the first network layer as input and calculates activation values based on the input and the parameters of the second network layer, and so on. Finally, the activation values output by the output layer of the neural network model are used as the output of the model, and the loss function (loss) is calculated based on the model output and the sample labels of the training samples.

[0095] In an embodiment of the present invention, when executing the forward propagation process, the model input, the activation values of each network layer, and the parameters of each network layer are quantized into block floating-point numbers. The specific quantization method can adopt the method described above, or other block floating-point quantization methods in the prior art. For ease of description, the input of the first layer of the network is also referred to as the activation value, and the block floating-point number corresponding to the parameter of each network layer is referred to as the first block floating-point number, and the block floating-point number corresponding to the activation value is referred to as the second block floating-point number. The first block floating-point number and the second block floating-point number use the same bit width, for example, both are 8-bit block floating-point numbers.

[0096] In this way, in the forward propagation, the processing performed by the current network layer is: obtaining the first block of floating-point numbers corresponding to the parameters of the current network layer; quantizing the activation value output by the previous network layer into the second block of floating-point numbers; calculating the activation value of the current network layer based on the first block of floating-point numbers and the second block of floating-point numbers, and outputting it to the next network layer.

[0097] It should be noted that the network layer typically calculates activation values in tensors. This means that activation values are calculated based on the input activation value tensor and the parameter tensor of the network layer. The activation value tensor is a tensor composed of all activation values of a network layer (or network units within a network layer), and the parameter tensor is a tensor composed of all parameters of a network layer (network units within a network layer). Accordingly, block floating-point numbers also use tensors as their granularity. The first block floating-point number is the block floating-point number obtained by quantizing the data block corresponding to the parameter tensor, and the second block floating-point number is the block floating-point number obtained by quantizing the data block corresponding to the activation value tensor.

[0098] A neural network model typically consists of multiple convolutional layers and one or more fully connected layers. A convolutional layer includes a convolution unit, a pooling unit, and an activation function unit, while a fully connected layer includes a linear processing unit and an activation function unit. The pooling unit does not have parameters or parameter gradients.

[0099] Figure 7 Schematic diagram of the training process of the convolutional layer in an embodiment of the present invention is shown. Figure 7 As shown in Figure 2, the forward propagation process performed in the convolutional layer includes the following steps:

[0100] 1) Use direct memory access (DMA) to obtain the first block of floating-point numbers corresponding to the parameters of this network layer and the activation value output by the previous network layer from the memory (DDR), quantize the activation value into a second block of floating-point numbers, input the first block of floating-point numbers and the second block of floating-point numbers into the convolution unit for processing, and output the processed linear value. In addition, the first block of floating-point numbers and the second block of floating-point numbers are temporarily stored in the memory for access during backpropagation. In this embodiment, the first block of floating-point numbers and the second block of floating-point numbers are both 8-bit block floating-point numbers (8-bit BFP), and the output linear value is a 32-bit block floating-point number (32-bit BFP);

[0101] Assuming that the current convolution layer is the lth layer, the processing performed in the convolution unit is:

[0102] I l =I l-1 *W l

[0103] Among them, * represents convolution operation, I l-1 The second floating point number corresponding to the activation value output by the previous network layer, I l is the linear value obtained by convolution processing, W l The first block of floating-point numbers corresponding to the parameters of the convolution unit.

[0104] 2) Quantize the output linear value into a quantized linear value with a bit width of 8 (8-bit BFP);

[0105] 3) Input the quantized linear value to the pooling unit for processing and output the pooled value (8-bit BFP);

[0106] 4) The pooled value is input to the activation function unit for processing, and the activation value (32-bit BFP) is output. The forward propagation of the current convolutional layer is completed.

[0107] Similarly, the forward propagation process performed in the fully connected layer includes:

[0108] 1) Input the first block of floating-point numbers and the second block of floating-point numbers into the linear processing unit for processing, and output the processed linear values (32-bit BFP);

[0109] 2) Quantize the output linear value into a quantized linear value with a bit width of 8 (8-bit BFP);

[0110] 3) The quantized linear value is input to the activation function unit for processing, and the activation value (32-bit BFP) is output. The forward propagation of the current fully connected layer is completed.

[0111] After the forward propagation is completed, the method 600 proceeds to step S604. In step S604, the activation value gradient of the output layer is calculated based on the loss function, and starting from the output layer, the parameter gradient of the current network layer and the activation value gradient of the next network layer are calculated layer by layer.

[0112] In this embodiment of the present invention, when performing the backpropagation process, the activation value gradient is quantized into a block floating-point number. The specific quantization method can adopt the method described above, or other block floating-point quantization methods known in the art. For ease of description, the block floating-point number corresponding to the activation value gradient is referred to as the third block floating-point number. The bit width of the third block floating-point number is larger than that of the first and second block floating-point numbers, for example, a 16-bit block floating-point number (16-bit BFP).

[0113] In this way, in backpropagation, the processing performed by the current network layer is: quantizing the activation value gradient output by the next network layer into a third block of floating-point numbers; calculating the activation value gradient of the previous network layer based on the third block of floating-point numbers and the first block of floating-point numbers, and outputting it to the previous network layer; calculating the parameter gradient of the current network layer based on the third block of floating-point numbers and the second block of floating-point numbers.

[0114] Continue to refer to Figure 7 , the backpropagation process performed in the convolutional layer of the neural network model includes the following steps:

[0115] 1) Quantize the activation value gradient output by the next network layer into a third block of floating point numbers (16-bit BFP);

[0116] The applicant has found that in order to ensure that the training accuracy of the neural network model is not lost, the gradient of back propagation requires a larger bit width than the parameters and activation values. Therefore, in an embodiment of the present invention, the bit width of various gradients in back propagation is set to 16 bits.

[0117] 2) Based on the activation function used by the activation function unit, reverse derivative (activation derivative) of the third block of floating-point numbers is performed to obtain the pooling value gradient (32-bit BFP);

[0118] Backward derivation is performed according to the chain rule. If the activation value gradient is known, the activation function can be differentiated. The derivative result is multiplied by the activation value gradient to obtain the derivative of the input of the activation function unit (that is, the output of the pooling unit), which is called the pooling value gradient in this invention.

[0119] In this step and subsequent steps, regarding the principle and process of using the chain rule for derivation in back propagation, reference can be made to relevant existing technologies and will not be repeated here.

[0120] 3) Based on the pooling template used by the pooling unit, the pooling value gradient is reversely derived (pooling derivative) to obtain the linear value gradient (32-bit BFP);

[0121] 4) Quantize the linear value gradient to quantized linear value gradient (16-bit BFP)

[0122] 5) Based on the first block of floating-point numbers (i.e., the parameters of the convolution unit), the quantized linear value gradient is reversely derived (convolution derivative 1) to obtain the activation value gradient of the previous network layer (32-bit BFP);

[0123] 6) Based on the second block of floating-point numbers (i.e., the activation values input to the convolution unit), the linear value gradient is reversely derived (convolution derivative 2) to obtain the parameter gradient of the current network layer (32-bit BFP).

[0124] Assuming that the current convolution layer is the lth layer, the back propagation process corresponding to the convolution unit is:

[0125] g l-1 =g l *rot180(W l )

[0126] g w =I l-1 *g l

[0127] Among them, * represents convolution operation, W l The first block of floating-point numbers corresponding to the parameters of the convolution unit, g l is the pooling derivative result, g l-1is the activation value gradient of the previous network layer, Rot180() means flipping the matrix 180 degrees, I l-1 is the activation value gradient input to the current network layer, g w is the parameter gradient of the current network layer.

[0128] Due to g l-1 The calculation of g runs through the entire back propagation data flow, and g w The calculation of g l-1 The calculation of is parallel and will not block the data flow to the lower layer. Therefore, in the back propagation stage, the activation value gradient can be expressed as two different data precisions. l-1 The calculation is quantized to 16-bit BFP format. For g w , the 32-bit BFP format is maintained.

[0129] Similarly, the backpropagation process performed in the fully connected layer includes:

[0130] 1) Quantize the activation value gradient output by the next network layer into a third block of floating point numbers (16-bit BFP);

[0131] 2) Perform reverse derivation of the third block of floating-point numbers based on the activation function used by the activation function unit to obtain the linear value gradient (32-bit BFP);

[0132] 3) Quantize the linear value gradient to quantized linear value gradient (16-bit BFP);

[0133] 4) Based on the first block of floating-point numbers (i.e., the parameters of the linear processing unit), the quantized linear value gradient is reversely derived to obtain the activation value gradient of the previous network layer (32-bit BFP);

[0134] 5) Based on the second block of floating-point numbers (i.e., the activation values input to the linear processing unit), the linear value gradient is reversely derived to obtain the parameter gradient of the current network layer (32-bit BFP).

[0135] After backpropagation is completed, method 600 proceeds to step S606. In step S606, the parameters of the neural network model are updated. The parameters of each network layer of the neural network model can be updated sequentially or in parallel. First, the parameter update value of the current network layer is calculated based on the parameter gradient of the current network layer, and a fourth block of floating-point numbers (32-bit BFP) corresponding to the parameter update value is generated. Then, the parameters of the current network layer are updated based on the first block of floating-point numbers and the fourth block of floating-point numbers, and a first block of floating-point numbers corresponding to the updated parameters is generated.

[0136] Since the parameter update value is too small compared to the low-precision parameter value, it is impossible to directly add the parameter update value to the low-precision parameter. Therefore, in step 1, the lazy update method is used to update the parameter. The parameter value, the delayed update value, and the parameter update value have the same decimal point position through the shared exponent control of the block floating-point number, and the part of the parameter update value that is valid relative to the delayed update value (for example, the overlapping part of the mantissa of the parameter update value and the delayed update value) is added to the delayed update value. When the delayed update value is large enough, the parameter value is updated according to the delayed update value. Among them, the parameter update value refers to the value to be updated of the parameter value calculated based on the parameter gradient and the optimization algorithm (such as stochastic gradient descent method, momentum gradient descent method, etc.) in one iteration; the parameter delayed update value refers to the part of the parameter update value that has not been updated to the parameter value from the first iteration to the last iteration.

[0137] Figure 8 FIG. 1 shows a schematic diagram of the parameter updating process in an embodiment of the present invention. Figure 8 As shown in FIG, the process of delaying the update of parameters includes:

[0138] 1) Get the fifth block of floating-point numbers (16-bit BFP) corresponding to the delay update value after the previous iteration;

[0139] 2) right-shifting the mantissas of the fourth block of floating-point numbers by a first predetermined value, and adding the shifted mantissas to the fifth block of floating-point numbers, where the first predetermined value is the difference between the exponents of the fifth block of floating-point numbers and the exponents of the fourth block of floating-point numbers. When performing the mantissa shift operation, the truncated portion is added to the untruncated portion using rounding.

[0140] 3) Calculate the difference between the first block of floating-point numbers and the fifth block of floating-point numbers after the mantissa is shifted right by a second predetermined value to obtain a first difference. In one implementation, the second predetermined value is the bit width of the fifth block of floating-point numbers - 1 (15 in the figure). Similarly, when performing the mantissa shift operation, the truncated portion is added to the untruncated portion in a rounded manner, which is equivalent to updating the highest bit of the delayed update value to the parameter in a rounded manner. In another implementation, the second predetermined value can also be another value so that the shifted delayed update value overlaps with the mantissa of the parameter value, and the overlapping portion of the delayed update value is updated to the parameter.

[0141] 4) Calculate the sum of the first block of floating-point numbers and the fifth block of floating-point numbers, and then subtract the first difference to obtain the fifth block of floating-point numbers corresponding to the delayed update value after this iteration;

[0142] 5) Update the first block of floating-point numbers to the first difference value.

[0143] The above delayed update process is expressed as follows:

[0144] w acc =w acc +(w ch >>(E acc -E ch )).round()

[0145] w wu =w-(w acc >>(b-1)).round()

[0146] w acc ←(w+w acc )-w wu

[0147] w←w wu

[0148] Among them, w is the mantissa of the parameter value, w ch Update the value of the parameter to the end of the value, E ch The exponential part of the parameter update value, w acc The mantissa of the delayed update value, E acc is the exponential part of the delayed update value, w wu is an intermediate variable, b is the bit width of the delayed update value (16 in this embodiment), round() is rounding, and ← means assigning the value on the right to the left.

[0149] After executing steps S602 to S606, one iteration is completed. The above steps S602 to S606 can be repeatedly executed. When the neural network model converges, or the number of iterations of the parameters reaches a preset number, the training is stopped to obtain a trained neural network model.

[0150] In one embodiment, the neural network model to be trained is a neural network machine translation model. Then, the final parameters are applied to the model to obtain a trained neural network machine translation model. Then, the information to be translated is translated based on the neural network machine translation model to obtain a translation result, and the translation result is output.

[0151] According to the training scheme of the neural network model of the embodiment of the present invention, based on the combination of block floating-point quantization and delayed update, the parameter value, the delayed update value, and the parameter update value are controlled to have the same decimal point position through the shared exponent of the block floating-point number. The part of the parameter update value that is valid relative to the delayed update value is added to the delayed update value. When the delayed update value is large enough, the parameter value is updated according to the delayed update value. In this way, low-precision parameter values can be used in the forward propagation, backpropagation, and parameter update processes, without the need to additionally save the high-precision floating-point values corresponding to the low-precision parameter values, nor the need to frequently perform conversion operations between high-precision and low-precision types of data, thereby saving storage resources while improving the training speed of the neural network model.

[0152] Thus, in some application scenarios, the above-mentioned training method can also be executed by a terminal device, in which the above-mentioned acceleration unit, such as a neural network processing unit (NPU) or a graphics processing unit (GPU), is deployed. The terminal device can be a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a speaker computing device, a vehicle computing device (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a wearable device including a computing device (e.g., a watch including a computing device, glasses including a computing device), or a home device including a computing device (e.g., a speaker including a computing device, a television including a computing device, a washing machine including a computing device).

[0153] Figure 9 FIG. 1 is a schematic diagram showing a training apparatus 900 for a neural network model according to an embodiment of the present invention. Figure 9 , the apparatus 900 includes a forward propagation module 910, a backward propagation module 920, and a parameter updating module 930, wherein:

[0154] The forward propagation module 910 is adapted to:

[0155] Get the first block of floating-point numbers corresponding to the parameters of the current network layer;

[0156] quantizing the activation values output by the previous network layer into a second block of floating-point numbers, wherein the bit widths of the first block of floating-point numbers and the second block of floating-point numbers are a first predetermined value;

[0157] Calculate the activation value of the current network layer based on the first block of floating-point numbers and the second block of floating-point numbers, and output it to the next network layer;

[0158] The back-propagation module 920 is adapted to:

[0159] quantizing the activation value gradient output by the next network layer into a third block of floating-point numbers, wherein the bit width of the third block of floating-point numbers is a second predetermined value;

[0160] Calculate the activation value gradient of the previous network layer based on the third block of floating-point numbers and the first block of floating-point numbers, and output it to the previous network layer;

[0161] Calculate the parameter gradient of the current network layer based on the third block of floating-point numbers and the second block of floating-point numbers;

[0162] The parameter update module 930 is adapted to:

[0163] Calculating parameter update values of the current network layer based on the parameter gradient, and generating a fourth block of floating-point numbers corresponding to the parameter update values, wherein a bit width of the fourth block of floating-point numbers is a third predetermined value;

[0164] The parameters of the current network layer are updated based on the first block of floating-point numbers and the fourth block of floating-point numbers, and the first block of floating-point numbers corresponding to the updated parameters are generated.

[0165] The specific processing performed by the forward propagation module 910, the back propagation module 920 and the parameter updating module 930 can refer to the above method 600 and will not be described in detail here.

[0166] The various techniques described herein may be implemented in conjunction with hardware or software, or a combination thereof. Thus, the methods and apparatus of the present invention, or certain aspects or portions of the methods and apparatus of the present invention, may be implemented in the form of program codes (i.e., instructions) embedded in a tangible medium, such as a removable hard disk, a USB flash drive, a floppy disk, a CD-ROM, or any other machine-readable storage medium, wherein when the program is loaded into a machine such as a computer and executed by the machine, the machine becomes an apparatus for practicing the present invention.

[0167] When the program code is executed on a programmable computer, the computing device generally includes a processor, a storage medium readable by the processor (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. The memory is configured to store the program code; the processor is configured to execute the method of the present invention according to the instructions in the program code stored in the memory.

[0168] By way of example and not limitation, readable media include readable storage media and communication media. Readable storage media store information such as computer-readable instructions, data structures, program modules, or other data. Communication media typically embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and include any information delivery medium. Combinations of any of the above are also included within the scope of readable media.

[0169] In the description provided herein, the algorithms and displays are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems may also be used in conjunction with the examples of the present invention. Based on the above description, it is apparent that the structure required for constructing such systems is well understood. In addition, the present invention is not directed to any specific programming language. It should be understood that various programming languages may be utilized to implement the present invention described herein, and the description of specific languages above is provided for the purpose of disclosing preferred embodiments of the present invention.

[0170] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0171] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Accordingly, the claims that follow the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of the invention.

[0172] Those skilled in the art will appreciate that the modules, units, or components of the devices in the examples disclosed herein may be arranged in the device described in the embodiment, or alternatively may be located in one or more devices different from the devices in the examples. The modules in the foregoing examples may be combined into one module or further divided into multiple submodules.

[0173] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition may be divided into multiple submodules or subunits or subcomponents. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed herein may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0174] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of the present invention and to form different embodiments. For example, in the claims below, any of the claimed embodiments may be used in any combination.

[0175] In addition, some of the embodiments are described herein as methods or combinations of method elements that can be implemented by a processor of a computer system or by other devices that perform the functions described. Thus, a processor having the necessary instructions for implementing the method or method element forms a device for implementing the method or method element. Furthermore, the elements described herein of the device embodiments are examples of devices for implementing the functions performed by the elements for the purpose of implementing the invention.

[0176] As used herein, unless otherwise specified, the use of ordinal numbers "first," "second," "third," etc. to describe common objects merely indicates that different instances of similar objects are involved and are not intended to imply that the objects so described must have a given order in time, space, ranking, or in any other manner.

[0177] Although the present invention has been described with respect to a limited number of embodiments, it will be apparent to those skilled in the art, having benefit of the foregoing description, that other embodiments are contemplated within the scope of the invention thus described. Furthermore, it should be noted that the language used in this specification has been selected primarily for readability and didactic purposes, rather than for the purpose of explaining or limiting the subject matter of the present invention. Consequently, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is intended to be illustrative and not restrictive of the scope of the invention, which is defined by the appended claims.

Claims

1. A method for training a neural network model, executed in a server, wherein the neural network model includes multiple network layers, the method comprising: Forward propagation step: Get the first block of floating-point numbers corresponding to the parameters of the current network layer; quantizing the activation values output by the previous network layer into a second block of floating-point numbers, wherein the bit widths of the first block of floating-point numbers and the second block of floating-point numbers are a first predetermined value; Calculate the activation value of the current network layer based on the first block of floating-point numbers and the second block of floating-point numbers, and output it to the next network layer; Backward propagation step: quantizing the activation value gradient output by the next network layer into a third block of floating-point numbers, wherein the bit width of the third block of floating-point numbers is a second predetermined value; Calculate the activation value gradient of the previous network layer based on the third block of floating-point numbers and the first block of floating-point numbers, and output it to the previous network layer; Calculate the parameter gradient of the current network layer based on the third block of floating-point numbers and the second block of floating-point numbers; Parameter update steps: Calculating a parameter update value of the current network layer based on the parameter gradient, and generating a fourth block of floating-point numbers corresponding to the parameter update value, wherein a bit width of the fourth block of floating-point numbers is a third predetermined value; The parameters of the current network layer are updated based on the first block of floating-point numbers and the fourth block of floating-point numbers to generate a first block of floating-point numbers corresponding to the updated parameters. The updating of the parameters of the current network layer based on the first block of floating-point numbers and the fourth block of floating-point numbers to generate a first block of floating-point numbers corresponding to the updated parameters includes: obtaining a delay update value after the last iteration, and adding a portion of the fourth block of floating-point numbers that is valid relative to the delay update value to the delay update value; in response to the delay update value being greater than a preset threshold, updating the first block of floating-point numbers based on the delay update value to obtain a first block of floating-point numbers corresponding to the updated parameters.

2. The method according to claim 1, wherein The current network layer is a fully connected layer, and the fully connected layer includes a linear processing unit and an activation function unit; Calculating the activation value of the current network layer based on the first block of floating-point numbers and the second block of floating-point numbers includes: Inputting the first block of floating-point numbers and the second block of floating-point numbers into a linear processing unit for processing, and outputting a linear value having a bit width of a third predetermined value; quantizing the linear value into a quantized linear value having a bit width of a first predetermined value; The quantized linear value is input into the activation function unit for processing, and an activation value with a bit width of a third predetermined value is output.

3. The method according to claim 2, wherein: The step of calculating the activation value gradient of the previous network layer based on the third block of floating-point numbers and the first block of floating-point numbers includes: Reversely deriving the third block of floating-point numbers based on the activation function used by the activation function unit to obtain a linear value gradient with a bit width of a third predetermined value; quantizing the linear value gradient into a quantized linear value gradient with a bit width of a second predetermined value; Reversely derive the quantized linear value gradient based on the first block of floating-point numbers to obtain the activation value gradient of the previous network layer; Calculating the parameter gradient of the current network layer based on the third block of floating-point numbers and the second block of floating-point numbers includes: The linear value gradient is reversely derived based on the second block of floating-point numbers to obtain the parameter gradient of the current network layer.

4. The method according to claim 1, wherein The current network layer is a convolutional layer, which includes a convolution unit, a pooling unit and an activation function unit; Calculating the activation value of the current network layer based on the first block of floating-point numbers and the second block of floating-point numbers includes: Inputting the first block of floating-point numbers and the second block of floating-point numbers into a convolution unit for processing, and outputting a linear value with a bit width of a third predetermined value; quantizing the linear value into a quantized linear value having a bit width of a first predetermined value; Inputting the quantized linear value into a pooling unit for processing, and outputting a pooling value; The pooled value is input into the activation function unit for processing, and an activation value with a bit width of a third predetermined value is output.

5. The method according to claim 4, wherein: The step of calculating the activation value gradient of the previous network layer based on the third block of floating-point numbers and the first block of floating-point numbers includes: Performing reverse derivation of the third block of floating-point numbers based on the activation function used by the activation function unit to obtain a pooling value gradient with a bit width of a third predetermined value; Reversely deriving the pooling value gradient based on the pooling template used by the pooling unit to obtain a linear value gradient; quantizing the linear value gradient into a quantized linear value gradient with a bit width of a second predetermined value; Reversely derive the quantized linear value gradient based on the first block of floating-point numbers to obtain the activation value gradient of the previous network layer; Calculating the parameter gradient of the current network layer based on the third block of floating-point numbers and the second block of floating-point numbers includes: The linear value gradient is reversely derived based on the second block of floating-point numbers to obtain the parameter gradient of the current network layer.

6. The method according to any one of claims 1 to 5, wherein: The first predetermined value is smaller than the second predetermined value, and the second predetermined value is smaller than the third predetermined value.

7. The method of claim 6, wherein: The first predetermined value is 8, the second predetermined value is 16, and the third predetermined value is 32.

8. The method of claim 1, wherein: The updating of the parameters of the current network layer based on the first block of floating-point numbers and the fourth block of floating-point numbers to generate the first block of floating-point numbers corresponding to the updated parameters includes: Obtaining a fifth block of floating-point numbers corresponding to the delay update value after the previous iteration, wherein the delay update value is a portion of the parameter update value that has not been updated to the parameter, and the bit width of the fifth block of floating-point numbers is a second predetermined value; Shifting the mantissa of the fourth block of floating-point numbers rightward by a first predetermined value, and adding the mantissa to the fifth block of floating-point numbers; Calculating the difference between the first block of floating-point numbers and the fifth block of floating-point numbers after the mantissas are shifted right by a second predetermined value to obtain a first difference value; Calculate the sum of the first block of floating-point numbers and the fifth block of floating-point numbers, and then subtract the first difference to obtain the fifth block of floating-point numbers corresponding to the delayed update value after this iteration; The first block of floating-point numbers is updated to the first difference value.

9. The method of claim 8, wherein the first predetermined value is a difference between an exponent of the fifth block of floating-point numbers and an exponent of the fourth block of floating-point numbers, and the second predetermined value is a bit width of the fifth block of floating-point numbers - 1.

10. The method of claim 8, wherein: When performing a shift operation on the mantissa, the truncated portion is added to the untruncated portion by rounding.

11. The method of claim 1, wherein: The server includes an acceleration unit, and the training method is suitable for being executed by the acceleration unit.

12. The method of claim 11, wherein: The acceleration unit is a neural network processing unit NPU or a graphics processing unit GPU.

13. The method of claim 11, wherein: The server is deployed in a data center.

14. A method for training a neural network model, executed in a terminal device, wherein the neural network model includes multiple network layers, the method comprising: Forward propagation step: Get the first block of floating-point numbers corresponding to the parameters of the current network layer; quantizing the activation values output by the previous network layer into a second block of floating-point numbers, wherein the bit widths of the first block of floating-point numbers and the second block of floating-point numbers are a first predetermined value; Calculate the activation value of the current network layer based on the first block of floating-point numbers and the second block of floating-point numbers, and output it to the next network layer; Backward propagation step: quantizing the activation value gradient output by the next network layer into a third block of floating-point numbers, wherein the bit width of the third block of floating-point numbers is a second predetermined value; Calculate the activation value gradient of the previous network layer based on the third block of floating-point numbers and the first block of floating-point numbers, and output it to the previous network layer; Calculate the parameter gradient of the current network layer based on the third block of floating-point numbers and the second block of floating-point numbers; Parameter update steps: Calculating a parameter update value of the current network layer based on the parameter gradient, and generating a fourth block of floating-point numbers corresponding to the parameter update value, wherein a bit width of the fourth block of floating-point numbers is a third predetermined value; The parameters of the current network layer are updated based on the first block of floating-point numbers and the fourth block of floating-point numbers to generate a first block of floating-point numbers corresponding to the updated parameters. The updating of the parameters of the current network layer based on the first block of floating-point numbers and the fourth block of floating-point numbers to generate a first block of floating-point numbers corresponding to the updated parameters includes: obtaining a delay update value after the last iteration, and adding a portion of the fourth block of floating-point numbers that is valid relative to the delay update value to the delay update value; in response to the delay update value being greater than a preset threshold, updating the first block of floating-point numbers based on the delay update value to obtain a first block of floating-point numbers corresponding to the updated parameters.

15. The training method according to claim 14, wherein: The terminal device is a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a speaker computing device, a vehicle computing device, a wearable device including a computing device, or a home device including a computing device.

16. The training method according to claim 14 or 15, wherein: The terminal device includes an acceleration unit, and the training method is suitable for being executed by the acceleration unit.

17. The training method according to claim 16, wherein: The acceleration unit is a neural network processing unit NPU or a graphics processing unit GPU.

18. A training device for a neural network model, comprising: Forward propagation module, suitable for: Get the first block of floating-point numbers corresponding to the parameters of the current network layer; quantizing the activation values output by the previous network layer into a second block of floating-point numbers, wherein the bit widths of the first block of floating-point numbers and the second block of floating-point numbers are a first predetermined value; Calculate the activation value of the current network layer based on the first block of floating-point numbers and the second block of floating-point numbers, and output it to the next network layer; Backpropagation module, suitable for: quantizing the activation value gradient output by the next network layer into a third block of floating-point numbers, wherein the bit width of the third block of floating-point numbers is a second predetermined value; Calculate the activation value gradient of the previous network layer based on the third block of floating-point numbers and the first block of floating-point numbers, and output it to the previous network layer; Calculate the parameter gradient of the current network layer based on the third block of floating-point numbers and the second block of floating-point numbers; Parameter update module, suitable for: Calculating a parameter update value of the current network layer based on the parameter gradient, and generating a fourth block of floating-point numbers corresponding to the parameter update value, wherein a bit width of the fourth block of floating-point numbers is a third predetermined value; The parameters of the current network layer are updated based on the first block of floating-point numbers and the fourth block of floating-point numbers to generate a first block of floating-point numbers corresponding to the updated parameters. The updating of the parameters of the current network layer based on the first block of floating-point numbers and the fourth block of floating-point numbers to generate a first block of floating-point numbers corresponding to the updated parameters includes: obtaining a delay update value after the last iteration, and adding a portion of the fourth block of floating-point numbers that is valid relative to the delay update value to the delay update value; in response to the delay update value being greater than a preset threshold, updating the first block of floating-point numbers based on the delay update value to obtain a first block of floating-point numbers corresponding to the updated parameters.

19. A computing device comprising: at least one processor; and A memory storing program instructions, wherein the program instructions are configured to be executed by the at least one processor, and the program instructions include instructions for executing the method according to any one of claims 1 to 17.

20. A readable storage medium storing program instructions, wherein when the program instructions are read and executed by a computing device, the computing device is caused to execute the method according to any one of claims 1 to 17.

Images