A distributed training method, gradient communication method, device and computing device

Abstract

The application discloses a distributed training method and gradient communication method and device of a deep neural network and a computing device. The method comprises the following steps: in each iteration step, a training data subset is obtained from a training data set, and a gradient of network parameters of the deep neural network on the training data subset is calculated as a local gradient; and the local gradient of each computing node is exchanged with other computing nodes in an alternating manner of sparse communication and dense communication, wherein the sparse communication refers to a communication mode in which the local gradient is sparsified and then exchanged with other computing nodes, the dense communication refers to a communication mode in which the local gradient is directly exchanged with other computing nodes without sparsification, in the sparse communication mode, important gradients of each computing node are exchanged with other computing nodes, the important gradient is a local gradient with a gradient value greater than a preset threshold, and in the dense communication mode, all local gradients of each computing node are exchanged with other computing nodes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a distributed training method of deep neural network, a gradient communication method, device and computing equipment. BACKGROUND

[0002] With the increasing size of deep neural networks (DNNs) and training data sets, the training of DNNs requires huge computing power, storage space, communication bandwidth and power consumption. The current mainstream DNNs training platform is based on a distributed multi-node system, which uses parallel computing characteristics to greatly accelerate the training process. However, frequent gradient exchange communication increases the transmission bandwidth pressure, resulting in low platform expansion capability and long communication time. Therefore, communication time has become a bottleneck for the development of distributed training.

[0003] Gradient compression, as a very promising method, accelerates distributed training by reducing communication volume. There are two main strategies for gradient quantization and gradient sparsity. Gradient quantization reduces the bit width of gradient data to achieve gradient data compression. Gradient sparsity selects important gradients for parameter update, and the remaining unimportant gradients, also known as delayed gradients, are accumulated locally with the delayed gradients left over from the last iteration and do not perform parameter update.

[0004] However, in the case of high sparsity, on the one hand, due to the lack of gradient information and the influence of delayed gradients, the final model convergence performance is poor, and on the other hand, since only a few important gradients are selected for sparse communication each time, this situation will cause excessive accumulation of delayed gradients, resulting in the problem of gradient explosion.

[0005] Therefore, how to provide a distributed gradient sparse communication method without affecting the model convergence performance has become a problem that needs to be solved by those skilled in the art. SUMMARY

[0006] In view of the above problems, the present application is proposed to provide a distributed training method of deep neural network, a gradient communication method, device and computing equipment to overcome the above problems or at least partially solve the above problems.

[0007] According to one aspect of the present application, a gradient communication method of deep neural network is provided, which is applied to a distributed training framework of deep neural network, the distributed computing framework comprising a plurality of computing nodes, and performing the following steps on each computing node:

[0008] In each iteration step, a training data subset is obtained from a training data set, and the gradient of the network parameters of the deep neural network on the training data subset is calculated as a local gradient;

[0009] exchange the local gradients with other computing nodes in an alternating manner of sparse communication and dense communication, wherein the sparse communication refers to a communication manner of exchanging the local gradients with other computing nodes after the local gradients are sparsified, the dense communication refers to a communication manner of directly exchanging the local gradients with other computing nodes without sparsifying the local gradients, in the sparse communication manner, important gradients of the local gradients are exchanged with other computing nodes, the important gradients are the local gradients whose gradient values are greater than a preset threshold, in the dense communication manner, all the local gradients are exchanged with other computing nodes.

[0010] Optionally, in the gradient communication method, the alternating manner of sparse communication and dense communication comprises: performing sparse communication for a predetermined number of times, and then performing dense communication once.

[0011] Optionally, in the gradient communication method, before the step of exchanging the important gradients with other computing nodes, the method further comprises: for each network layer of the deep neural network, selecting a predetermined number of local gradients from the local gradients corresponding to the network layer in descending order of gradient values as the important gradients.

[0012] Optionally, in the gradient communication method, the exchanging the important gradients with other computing nodes comprises: for each network layer of the deep neural network, based on the important gradients of the network layer, sparsifying a gradient tensor of the network layer to form a sparse gradient tensor, wherein the gradient tensor is a tensor composed of all the local gradients of the network layer; and sending the sparse gradient tensor to other computing nodes after encoding.

[0013] Optionally, in the gradient communication method, the sparsifying the gradient tensor of the network layer comprises: setting the local gradients other than the important gradients in the gradient tensor to zero.

[0014] Optionally, in the gradient communication method, the encoding is run-length encoding.

[0015] Optionally, in the gradient communication method, before the step of exchanging the local gradients with other computing nodes, the method further comprises: updating the local gradients to the sum of the local gradients and delayed gradients of the last iteration.

[0016] Optionally, in the gradient communication method, before the step of exchanging all the local gradients with other computing nodes, the method further comprises: setting the delayed gradients of the current iteration to zero.

[0017] Optionally, in the gradient communication method of the present application, before the step of exchanging local gradients with other computing nodes, there is further included: storing local gradients other than important gradients as delayed gradients of the current iteration.

[0018] Optionally, in the gradient communication method of the present application, the step of exchanging local gradients with other computing nodes includes: communicating with other computing nodes in an all-reduce manner to exchange local gradients with each other.

[0019] According to another aspect of the present application, there is provided a distributed training method applied to a distributed training framework of a deep neural network, the distributed computing framework including a plurality of computing nodes, and performing the following steps on each computing node:

[0020] In each iteration step, a batch of training data subsets are obtained from a training data set, and gradients of network parameters of the deep neural network on the training data subsets are calculated as local gradients;

[0021] In an alternating manner of sparse communication and dense communication, local gradients are exchanged with other computing nodes, wherein sparse communication refers to a communication manner of exchanging gradients with other computing nodes after sparse processing of local gradients, and dense communication refers to a communication manner of directly exchanging gradients with other computing nodes without sparse processing of local gradients, in the sparse communication manner, important gradients are exchanged with other computing nodes, and important gradients are local gradients with gradient values greater than a preset threshold, in the dense communication manner, all local gradients are exchanged with other computing nodes;

[0022] A gradient average value of local gradients exchanged by all computing nodes in the current iteration step is calculated, and network parameters of the deep neural network are updated based on the gradient average value.

[0023] Optionally, in the distributed training method of the present application, before the step of exchanging local gradients with other computing nodes, there is further included: updating local gradients as a sum of local gradients and delayed gradients of the last iteration.

[0024] Optionally, in the distributed training method of the present application, before the step of exchanging all local gradients with other computing nodes, there is further included: setting delayed gradients of the current iteration to zero.

[0025] Optionally, in the distributed training method of the present application, before the step of exchanging important gradients with other computing nodes, there is further included: storing local gradients other than important gradients as delayed gradients of the current iteration.

[0026] Optionally, in the distributed training method of the present application, the updating of the network parameters of the deep neural network based on the gradient average value comprises: updating the network parameters of the deep neural network by using a gradient descent method without momentum based on the gradient average value; or updating the network parameters of the deep neural network by using a gradient descent method with momentum based on the gradient average value.

[0027] Optionally, in the distributed training method of the present application, the method further comprises: stopping the training when the deep neural network converges, or the number of iterations of the network parameters reaches a preset number.

[0028] According to another aspect of the present application, a distributed training method is provided, which is applied to a distributed training framework of a deep neural network, the distributed computing framework comprising a plurality of terminal devices, and each terminal device performing the following steps:

[0029] In each iteration step, a batch of training data subsets are obtained from a training data set, and the gradients of the network parameters of the deep neural network on the training data subsets are calculated as local gradients;

[0030] The local gradients are exchanged with other terminal devices in an alternating manner of sparse communication and dense communication, wherein the sparse communication refers to a communication manner of exchanging gradients with other computing nodes after the local gradients are sparsified, the dense communication refers to a communication manner of directly exchanging gradients with other computing nodes without sparsifying the local gradients, in the sparse communication manner, important gradients are exchanged with other terminal devices, the important gradients being the local gradients whose gradient values are greater than a preset threshold, in the dense communication manner, all local gradients are exchanged with other terminal devices;

[0031] A gradient average value of the exchanged local gradients of all terminal devices in the current iteration step is calculated, and the network parameters of the deep neural network are updated based on the gradient average value.

[0032] Optionally, in the distributed training method of the present application, the terminal device is a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a sound box computing device, a computing device of a vehicle, a wearable device comprising a computing device, or a home device comprising a computing device.

[0033] According to another aspect of the present application, a distributed training apparatus is provided, comprising:

[0034] A gradient calculation unit is adapted to obtain a training data subset from a training data set in each iteration step, and calculate the gradients of the network parameters of the deep neural network on the training data subset as local gradients;

[0035] a gradient exchange unit adapted to exchange local gradients with other computing nodes in an alternating manner of sparse communication and dense communication, wherein the sparse communication refers to a communication manner of exchanging gradients with other computing nodes after the local gradients are sparsified, and the dense communication refers to a communication manner of exchanging gradients with other computing nodes without sparsifying the local gradients, in the sparse communication manner, important gradients are exchanged with other computing nodes, and the important gradients are local gradients whose gradient values are greater than a preset threshold, in the dense communication manner, all local gradients are exchanged with other computing nodes;

[0036] a parameter updating unit adapted to calculate a gradient average value of the local gradients exchanged by all computing nodes in the current iteration step, and update network parameters of the deep neural network based on the gradient average value.

[0037] According to still another aspect of the present application, there is provided a distributed training system comprising a plurality of computing nodes, each of which comprises the distributed training apparatus as described above.

[0038] According to still another aspect of the present application, there is provided a computing device comprising at least one processor and a memory having stored program instructions configured to be executed by the at least one processor, the program instructions comprising instructions for performing the method as described above.

[0039] According to still another aspect of the present application, there is provided a readable storage medium having stored program instructions, which, when read and executed by a computing device, cause the computing device to perform the method as described above.

[0040] According to the gradient communication scheme of the present application, the gradient communication is performed in a periodic alternating manner of multiple sparse gradients combined with one dense gradient, which effectively reduces the gradient transmission amount, speeds up the communication process, and each dense gradient communication effectively compensates for the lost gradient information, improving the convergence performance of the model. In addition, each dense gradient communication can clear all previous local delayed gradients, effectively solving the problem of gradient explosion caused by excessive accumulation of local gradients. Further, the global momentum stochastic gradient descent algorithm is used, which takes into account the long-term gradient information, further improving the convergence performance of the model.

[0041] The above description is only a summary of the technical scheme of the present application. In order to more clearly understand the technical means of the present application, the contents of the specification can be implemented, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described below. BRIEF DESCRIPTION OF DRAWINGS

[0042] Various other advantages and benefits will become apparent to those of ordinary skill in the art, upon reading the following detailed description of the preferred embodiment. The accompanying drawings are included to provide a better understanding of the preferred embodiment, and are not intended to limit the scope of the application, and, wherein:

[0043] Figure 1 A structural diagram of a data center to which an embodiment of the present application is applied is shown;

[0044] Figure 2 An internal structural diagram of a server in a data center according to an embodiment of the present application is shown;

[0045] Figure 3 A connection relationship diagram of a scheduling unit and an acceleration unit inside a server according to an embodiment of the present application is shown;

[0046] Figure 4 An internal structural diagram of an acceleration unit core according to an embodiment of the present application is shown;

[0047] Figure 5 A flowchart of a gradient communication method 500 according to an embodiment of the present application is shown;

[0048] Figure 6 A comparison diagram of bandwidth occupation of dense, sparse and sparse-dense alternating communication in distributed training is shown;

[0049] Figure 7 A flowchart of a distributed training method 700 according to an embodiment of the present application is shown;

[0050] Figure 8 A flowchart of a distributed training method 800 according to an embodiment of the present application is shown;

[0051] Figure 9 A schematic diagram of a distributed training device 900 according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0052] Exemplary embodiments of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms without being limited by the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0053] First, an implementation environment of a distributed training method of an embodiment of the present application is introduced.

[0054] Data center

[0055] Data centers are globally collaborative networks of specific devices used to deliver, accelerate, present, compute, and store data information over the Internet network infrastructure. In the future, data centers will also become assets for enterprise competition. With the wide application of data centers, artificial intelligence and other technologies are increasingly applied to data centers. As an important technology of artificial intelligence, neural networks have been widely applied to data center big data analysis and operation.

[0056] In a traditional large data center, the network structure is usually as shown in Figure 1 , that is, a hierarchical inter-networking model. This model includes the following parts:

[0057] Servers 140: Each server 140 is a processing and storage entity of the data center, and the processing and storage of a large amount of data in the data center are completed by these servers 140.

[0058] Access switches 130: Access switches 130 are switches used to allow servers 140 to access the data center. One access switch 130 accesses multiple servers 140. Access switches 130 are usually located at the top of the rack, so they are also called Top of Rack switches, which are physically connected to servers.

[0059] Aggregation switches 120: Each aggregation switch 120 connects multiple access switches 130 and provides other services such as firewall, intrusion detection, network analysis, etc.

[0060] Core switches 110: Core switches 110 provide high-speed forwarding for packets entering and exiting the data center and provide connectivity for aggregation switches 120. The network of the entire data center is divided into L3 layer routing network and L2 layer routing network, and core switches 110 usually provide a resilient L3 layer routing network for the entire network of the data center.

[0061] Usually, aggregation switches 120 are the demarcation points of L2 and L3 layer routing networks, and the aggregation switches 120 below are L2 networks and the aggregation switches 120 above are L3 networks. Each group of aggregation switches manages a delivery point (POD, Point Of Delivery), and each POD is an independent VLAN network. Migration of servers within a POD does not need to modify IP addresses and default gateways, because a POD corresponds to an L2 broadcast domain.

[0062] The Spanning Tree Protocol (STP) is usually used between the aggregation switches 120 and the access switches 130. The STP makes only one aggregation layer switch 120 available for a VLAN network, and other aggregation switches 120 are used only in case of failure (dashed lines in the figure above). That is, at the level of the aggregation switches 120, horizontal expansion is not possible, because even if multiple aggregation switches 120 are added, only one is in operation.

[0063] Server

[0064] Since the server 140 is a real processing device of the data center, Figure 2 A structural block diagram of a server 140 is shown. The server 140 includes a bus-connected memory 210, a scheduling unit cluster 270, and an acceleration unit cluster 280. The scheduling unit cluster 270 includes multiple scheduling units 220. The acceleration unit cluster 280 includes multiple acceleration units 230. The acceleration unit is a special processing unit designed mainly to accelerate the operation processing speed of a neural network model in the embodiments of the present disclosure, which can be embodied as a processing unit (NPU) specially designed for neural network operation processing, a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), and the like. The scheduling unit is a processing unit that schedules the acceleration units and allocates to each acceleration unit a to-be-executed instruction sequence to be executed, which can be in the form of a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), and the like.

[0065] The traditional central processing unit architecture design makes the control unit and the storage unit occupy a large part of the space in the architecture, while the computing unit occupies less space, so it is very effective in logical control, but not efficient in large-scale parallel computing. Therefore, various special acceleration units have been developed to improve the operation speed of computing in different functions and different fields. The acceleration unit proposed in the present application is a processing unit specially used to accelerate the operation speed of neural network model. It is a data-driven parallel computing architecture used to process a large number of operations (such as convolution, pooling, etc.) of each neural network node. Since the data and intermediate results in the large number of operations (such as convolution, pooling, etc.) of each neural network node are closely related throughout the computing process and are frequently used, the existing central processing unit architecture has a small memory capacity in the core, so it needs to access the external memory frequently, resulting in low processing efficiency. The acceleration unit specially used to accelerate the operation speed of neural network model can greatly improve the processing efficiency and computing performance because each core has on-chip memory with a storage capacity suitable for neural network computing, avoiding frequent access to external memory.

[0066] The acceleration unit 230 is scheduled by the scheduling unit 220. As shown in Figure 2 , the memory 210 stores various neural network models, including the nodes of these models and the weight data of the nodes, etc. These neural network models are deployed to an acceleration unit 230 by one scheduling unit 220 in the Figure 2 . That is, the scheduling unit 220 can send the address of the parameters (such as the weights of each node) in the memory 210 to the acceleration unit 230 in the form of an instruction. When the acceleration unit 230 actually uses the neural network model for calculation, it will directly address these parameters in the memory 210 according to the address of these parameters (such as weights) in the memory 210, and temporarily store them in its on-chip memory. When the acceleration unit 230 actually uses the neural network model for calculation, the scheduling unit 220 will also send the input parameters of the model to the acceleration unit 230 in the form of an instruction, and temporarily store them in the on-chip memory of the acceleration unit 230. In this way, the acceleration unit 230 can perform inference calculation according to these input parameters and the parameters (such as weights) in the model.

[0067] Internal structure of the scheduling unit and the acceleration unit

[0068] The following will be combined Figure 3 The internal structure diagram of the scheduling unit 220 and the acceleration unit 230 to explain how the scheduling unit 220 schedules the acceleration unit 230 to work.

[0069] As shown in Figure 3As shown, the scheduling unit 220 includes multiple processor cores 222 and a cache 221 shared by the multiple processor cores 222. Each processor core 222 includes an instruction fetch unit 203, an instruction decode unit 224, an instruction issue unit 225, and an instruction execution unit 226.

[0070] Instruction fetch unit 223 is used to move the instruction to be executed from memory 210 to instruction register (which may be...). Figure 3 The instruction is stored in one of the registers in the register file 229 shown, and the next fetch address is received or calculated according to the fetch algorithm, which may include, for example, incrementing or decrementing the address based on the instruction length.

[0071] After the instruction is fetched, the scheduling unit 220 enters the instruction decoding stage. The instruction decoding unit 224 decodes the fetched instruction according to a predetermined instruction format to obtain the operand fetch information required by the fetched instruction, thereby preparing for the operation of the instruction execution unit 225. Operand fetch information includes, for example, pointers to immediate values, registers, or other software / hardware that can provide source operands.

[0072] The instruction issuing unit 225 is located between the instruction decoding unit 224 and the instruction execution unit 226. It is used for instruction scheduling and control to efficiently allocate each instruction to different instruction execution units 226, making parallel operation of multiple instructions possible.

[0073] After instruction issuing unit 225 sends an instruction to instruction execution unit 226, instruction execution unit 226 begins executing the instruction. However, if instruction execution unit 226 determines that the instruction should be executed by an acceleration unit, it forwards it to the corresponding acceleration unit for execution. For example, if the instruction is a neural network inference instruction, instruction execution unit 226 will not execute the instruction but will instead send it to acceleration unit 230 via the bus for execution.

[0074] The acceleration unit 30 contains multiple 236 cores. Figure 3 Four cores are shown, but those skilled in the art will understand that the acceleration unit 230 may also contain other numbers of cores 236, command processor 237, direct memory access mechanism 235, and bus channel 231.

[0075] Bus channel 231 is the channel through which instructions enter and exit the acceleration unit 230 via the bus.

[0076] Direct Memory Access (DMA) mechanism 235 is a feature provided by some computer bus architectures that enables data to be written directly from an attached device to the memory of the computer motherboard. This approach greatly improves the efficiency of data access compared to the approach where all data transfers between devices go through a scheduling unit. Because of this mechanism, the cores of the acceleration unit 230 can directly access the memory 210 to read parameters (e.g., weights of nodes) in the neural network model, greatly improving the efficiency of data access.

[0077] The command processor 237 assigns the instructions sent by the scheduling unit 220 to the cores 236 for execution. The instruction execution unit 226 sends a sequence of instructions to be executed by the acceleration unit 230 to the acceleration unit 230. After the sequence of instructions enters the bus channel 231, it is buffered in the command processor 237, from which the cores 236 are selected by the command processor 237 to execute the sequence of instructions. In addition, the command processor 237 is also responsible for synchronization operations between the cores 236.

[0078] Acceleration unit core

[0079] Figure 4 is an internal structure diagram of an acceleration unit core 236 according to an embodiment of the present disclosure.

[0080] In one embodiment, as shown in Figure 4 the acceleration unit core 236 includes a tensor engine 310, a pooling engine 320, a memory copy engine 330, a sequencer 350, an instruction cache 340, an on-chip memory 360, and a constant buffer 370.

[0081] The sequence of instructions assigned to the acceleration unit core 236 by the command processor 237 first enters the instruction cache 340 for buffering. Then, the sequencer 350 fetches instructions from the instruction cache 340 in a first-in-first-out order and assigns them to the tensor engine 310, the pooling engine 320, or the memory copy engine 330 for execution according to the nature of the instructions. The tensor engine 310 is responsible for processing convolution and matrix multiplication operations in the neural network model. The pooling engine 320 is responsible for processing pooling operations in the neural network model. The memory copy engine 330 is responsible for copying operands stored in the on-chip memory 360 in the core 236 to a shared memory between the cores 236 or to the on-chip memory 360 in other cores 236. The sequencer 350 determines whether to assign an instruction to the tensor engine 310, the pooling engine 320, or the memory copy engine 330 according to the nature of the fetched instruction, such as convolution, matrix multiplication, pooling, or operand copying.

[0082] On-chip memory 360 is an on-chip memory that stores weight parameters in the neural network model, and input parameters and various intermediate results when the neural network model is actually used. Constant buffer 370 is a buffer that stores other constant parameters (for example, hyperparameters in the neural network model) in the neural network model, except for the weight parameters. As described above, in the process of scheduling unit 220 pre-configuring the neural network model in acceleration unit 230, scheduling unit 220 sends the addresses of the parameters in the model in memory 210 to acceleration unit 230 in the form of instructions. These parameters include the weights and other parameters (such as hyperparameters) of the nodes. For weights, acceleration unit 230 takes it from the corresponding location in memory 210 when the neural network model is actually operated, and places it in on-chip memory 360. For other parameters, acceleration unit 230 takes it from the corresponding location in memory 210 when the neural network model is actually operated, and places it in constant buffer 370. In addition, when the instruction to actually start inference is assigned by command processor 237 to core 236 for execution, the input parameters (inputs to the neural network model) in the instruction are also stored in on-chip memory 360. In addition, when tensor engine 310 and pooling engine 320 perform convolution or pooling operations, the various intermediate results obtained are also stored in on-chip memory 360.

[0083] The distributed training method of the embodiment of the present application can be executed in the above-mentioned data center. Specifically, each acceleration unit in the plurality of servers included in the data center can be regarded as a computing node, and the plurality of computing nodes can form a distributed training framework for distributed training. These computing nodes can be CPU devices, GPU devices, NPU (neural network processor) devices, or other types of computing devices, and the computing nodes are connected in a ring, a mesh, or other manners.

[0084] The computing node can obtain the deep neural network and the training data subset from the memory 210, and train the deep neural network based on the obtained training data subset. During the model training process, the computing node needs to exchange gradient data with other computing nodes, and update the network parameters (model parameters) of the deep neural network based on the average value of the gradient data of all computing nodes, for example, using the gradient descent method to update the network parameters of the deep neural network.

[0085] Data parallelism (Data Parallel) is a form of distributed training of a deep neural network, which divides the training data into multiple parts and trains on different computing nodes. If the computing nodes do not share a common memory, but only have a capacity-limited local memory, and the size of the training data set is too large to be stored in the local memory, the training data set needs to be divided and distributed to each computing node, and then the computing nodes train the deep neural network according to the local data allocated to each computing node.

[0086] The distributed training process of a typical deep neural network is as follows:

[0087] (1) At each iteration step (training step), a computing node obtains a training data subset from the training data set. Usually, the number of training data in the training data subset is much smaller than the number of training data in the training data set, so it can also be called mini-batch training data.

[0088] (2) The gradient of the network parameters (weights, biases, etc.) of the deep neural network on the mini-batch training data is calculated. Usually, the gradient calculated on one computing node is also called the local gradient of the computing node.

[0089] (3) The local gradient is sparsified, that is, important gradients are selected from the local gradient for communication. Important gradients are local gradients whose values are greater than a preset threshold.

[0090] (4) Communicate with other computing nodes to exchange important gradients with each other.

[0091] (5) Calculate the gradient average of the important gradients of all computing nodes, and update the network parameters of the deep neural network based on the gradient average.

[0092] Repeat steps (1) to (5) until the deep neural network model converges or the number of iterations of the network parameters reaches a preset number, and stop training.

[0093] In the above process, sparse gradient communication is used at each iteration step. Although it can reduce the bandwidth occupation of gradient communication and reduce the communication cost, in the case of high sparsity, on the one hand, due to the lack of gradient information and the influence of delayed gradient, the final model convergence performance becomes poor, on the other hand, since only a few important gradients are selected for sparse communication each time, this situation will cause excessive accumulation of delayed gradient, leading to the problem of gradient explosion.

[0094] Therefore, in the embodiments of the present application, a simple and effective distributed gradient compression sparse communication method is proposed, which adopts multiple sparse gradients combined with one dense gradient for periodic alternating gradient communication, which can achieve a compromise between gradient sparse communication acceleration and model convergence performance. Among them, sparse communication refers to the communication mode of exchanging gradients with other computing nodes after sparsifying the local gradient, and dense communication refers to the communication mode of directly exchanging gradients with other computing nodes without sparsifying the local gradient.

[0095] Figure 5A flow chart of a gradient communication method 500 according to an embodiment of the present application is shown. The method 500 is applied in a distributed computing framework of a deep neural network, and is executed in each computing node of the distributed computing framework, which can be implemented as the aforementioned acceleration unit 230. The method 500 can be applied in various scenarios such as image, speech, video, machine translation, etc. For example, in an image scenario, the corresponding deep neural network can be an image classification model or an object detection model; in a machine translation scenario, the corresponding deep neural network can be a neural network machine translation model. The neural network machine translation model is a sequence-to-sequence model with an encoder made of gated recurrent units, a decoder made of gated recurrent units, and an attention mechanism. The method 500 performs the following steps on each computing node:

[0096] In step S502, in each iteration step, a subset of training data, i.e., a mini-batch of training data, is obtained from the training data set, and the gradient of the network parameters of the deep neural network on the subset of training data is calculated as a local gradient.

[0097] The type of training data can be image samples, speech samples, or natural language processing samples. For example, when the deep neural network to be trained is a neural network machine translation model, each piece of training data is a text pair, which is a corresponding relationship between a first language text and a second language text.

[0098] In the embodiment of the present application, the distributed training adopts a data parallel (Data Parallel) mode. In the data parallel mode, the training data set stored in the memory is used as a sampling data source, and the computing nodes use random sampling to obtain multiple training data from the training data set as mini-batch training data. Specifically, random sampling can be performed in a with-replacement manner, which can ensure that the local training data on each computing node is independent and identically distributed with the original training data. In addition, the number of pieces of training data obtained in each iteration step, i.e., the size of the mini-batch data, can be determined according to the memory size of the computing node.

[0099] The mini-batch training data is input to the deep neural network to perform forward propagation, the output of the deep neural network is compared with the expected label data to calculate the loss error, and then the loss error is back-propagated to calculate the gradient of each parameter of the deep neural network on the mini-batch training data layer by layer as a local gradient. Generally, all local gradients of each network layer of the deep neural network form a gradient tensor, and each network parameter in the network layer corresponds to an element in the gradient tensor.

[0100] Due to the alternation of sparse communication and dense communication between the computing nodes, the sparse communication refers to the communication mode of exchanging the gradient with other computing nodes after the local gradient is sparsified, and the dense communication refers to the communication mode of directly exchanging the gradient with other computing nodes without sparsifying the local gradient. In the sparse communication mode, the important gradient, i.e., the local gradient whose value is greater than a preset threshold, is exchanged with other computing nodes each time, and then, the update of the network parameter is also based on the important gradient. In addition to the important gradient, other gradients do not perform the update of the parameter in this iteration, and are stored as the delayed gradient for the update of the parameter in the subsequent iteration. Therefore, the local gradient calculated in the current iteration step is also updated as the sum of the local gradient and the delayed gradient of the last iteration.

[0101] In step S504, it is judged which gradient communication mode is adopted in the current iteration step. If the sparse communication mode is adopted, step S506 is entered, and if the dense communication mode is adopted, step S508 is entered.

[0102] In the embodiment of the present application, the gradient communication is performed in the alternation of sparse communication and dense communication, i.e., the sparse communication is performed for a predetermined number of times, and then the dense communication is performed once. An iteration counter t (indicating which iteration step the current iteration step is) and a sparse communication step length S can be set, and the remainder operation is performed by using the iteration counter t and the sparse communication step length S. If t%S≠0, it is judged that the current communication is the sparse communication, otherwise, it is judged that the current communication is the dense communication.

[0103] It should be noted that the sparse communication step length S is a hyperparameter setting. If the setting is too small, the gradient communication amount is too large, which affects the training time, and if the setting is too large, the gradient information is lost seriously. The better S value can be obtained through experiments, for example, S=6.

[0104] In step S506, the sparse communication is performed with other computing nodes to exchange the important gradient with each other, i.e., the synchronization of the important gradient is performed by all the computing nodes. Specifically, the All-reduce mode can be used to perform the sparse communication with other computing nodes to exchange the important gradient with each other.

[0105] Before the sparse communication is performed, the important gradient needs to be selected. Specifically, for each network layer of the deep neural network, a predetermined number of local gradients are selected as the important gradient from the local gradient corresponding to the network layer in the order of gradient value from large to small.

[0106] Then, in each computing node, for each network layer of the deep neural network, the gradient tensor of the network layer needs to be sparsified based on the important gradient of the network layer, that is, the local gradient other than the important gradient in the gradient tensor is set to zero to form a sparse gradient tensor, and the sparse gradient tensor is encoded, for example, using run-length encoding, and is sent to other computing nodes to complete the exchange of important gradients.

[0107] Run-length encoding is a relatively simple compression algorithm, and the basic idea is to use (the number of consecutive occurrences, a certain character) to describe the character that appears repeatedly and continuously multiple times. For example, the existing character data AAAAABBBBCCC is encoded as 5A4B3C after run-length encoding. In the embodiment of the present application, since a large number of elements in the sparse gradient tensor are 0, the communication volume can be greatly reduced by run-length encoding.

[0108] As described above, in the sparse communication mode, the important gradient is exchanged with other computing nodes each time, and then the update of the network parameters is also based on the important gradient, and the other gradients other than the important gradient do not perform the update of the parameters in this iteration. Therefore, after the important gradient is selected, and before the important gradient of each computing node is exchanged with other computing nodes, the local gradient other than the important gradient is further stored as the delayed gradient of this iteration, and the delayed gradient corresponding to the important gradient is set to zero.

[0109] In step S508, dense communication is performed with other computing nodes to exchange all local gradients of each other, that is, all gradients are synchronized among all computing nodes. Specifically, the all-reduce mode can be used to perform sparse communication with other computing nodes to exchange all local gradients of each other.

[0110] In the embodiment of the present application, the local computing node also sets the delayed gradient of this iteration to zero before exchanging all local gradients of each other with other computing nodes. In this way, by clearing all the delayed gradients of the local computing node before each dense gradient communication, the problem of gradient explosion caused by excessive accumulation of local gradients is effectively solved.

[0111] Figure 6 A comparison diagram of bandwidth occupation of distributed training dense, sparse, and sparse-dense alternating communication is shown. As shown in Figure 6 The present application uses multiple sparse gradients combined with one dense gradient for periodic alternating gradient communication, which effectively reduces the gradient transmission volume and speeds up the communication process compared with using dense communication for each iteration, and each dense gradient communication effectively compensates for the lost gradient information and improves the convergence performance of the model.

[0112] The following describes a distributed training method based on the above gradient communication method.

[0113] Figure 7 A flow chart of a distributed training method 700 according to an embodiment of the present application is shown. The method 700 is applied in a distributed computing framework of a deep neural network, and is executed in each computing node of the distributed computing framework, which can be implemented as the aforementioned acceleration unit 230. The method 700 can be applied in various scenarios such as image, speech, video, machine translation, etc. For example, in an image scenario, the corresponding deep neural network can be an image classification model, an object detection model, etc.; in a machine translation scenario, the corresponding deep neural network can be a neural network machine translation model. As shown in the figure, the method 700 performs the following steps on each computing node: Figure 7

[0114] In step S702, in each iteration step, a subset of training data, i.e. a mini-batch of training data, is obtained from the training data set, and the gradient of the network parameters of the deep neural network on the subset of training data is calculated as a local gradient.

[0115] The type of training data can be image samples, speech samples, natural language processing samples. For example, when the deep neural network to be trained is a neural network machine translation model, each piece of training data is a text pair, which is the corresponding relationship between a first language text and a second language text.

[0116] In the embodiment of the present application, the distributed training adopts a data parallel (Data Parallel) mode. In the data parallel mode, the training data set stored in the memory is taken as a sampling data source, and the computing nodes obtain multiple training data from the training data set by random sampling as a mini-batch of training data. Specifically, random sampling can be performed in a with-replacement manner, which can ensure that the local training data on each computing node is independent and identically distributed with the original training data. In addition, the number of training data obtained in each iteration step, i.e. the size of the mini-batch data, can be determined according to the memory size of the computing node.

[0117] The mini-batch of training data is input to the deep neural network to perform forward propagation, the output of the deep neural network is compared with the expected label data to calculate the loss error, and then the loss error is back-propagated to calculate the gradient of each parameter of the deep neural network on the mini-batch of training data layer by layer as a local gradient. Generally, all local gradients of each network layer of the deep neural network constitute a gradient tensor, and each network parameter in the network layer corresponds to an element in the gradient tensor.

[0118] ​In the sparse communication mode, each time, the important gradient is exchanged with other computing nodes, and then, the update of the network parameter is also based on the important gradient, and other gradients except the important gradient do not perform the update of the parameter in this iteration, and are stored as the delayed gradient for the update of the parameter in the subsequent iteration process. Therefore, in the current iteration step, the calculated local gradient is also updated as the sum of the local gradient and the delayed gradient in the last iteration.

[0119] In step S704, it is judged whether the gradient communication mode adopted in the current iteration step is the sparse communication mode or the dense communication mode. If the sparse communication mode is adopted, step S706 is entered, and if the dense communication mode is adopted, step S708 is entered.

[0120] In the embodiment of the present application, the gradient communication is performed in the mode of alternation of the sparse communication and the dense communication, that is, after a predetermined number of sparse communications are performed, one dense communication is performed. An iteration counter t (indicating the current iteration step is the iteration step) and a sparse communication step S can be set, and the iteration counter t and the sparse communication step S are used to perform the modulo operation. If t%S≠0, it is judged that this communication is the sparse communication, otherwise, it is judged that this communication is the dense communication.

[0121] In step S706, the sparse communication is performed with other computing nodes to exchange the important gradient of each other, that is, the synchronization of the important gradient is performed among all computing nodes. Specifically, the all-reduce mode can be used to perform the sparse communication with other computing nodes to exchange the important gradient of each other.

[0122] Before the sparse communication is performed, the important gradient needs to be selected. Specifically, for each network layer of the deep neural network, a predetermined number of local gradients are selected as the important gradient from the local gradient corresponding to the network layer in the order of the gradient value from large to small.

[0123] Then, in each computing node, for each network layer of the deep neural network, the gradient tensor of the network layer needs to be sparsified based on the important gradient of the network layer, that is, the local gradient except the important gradient in the gradient tensor is set to zero to form the sparse gradient tensor, and the sparse gradient tensor is encoded, for example, after the run-length encoding, is sent to other computing nodes to complete the exchange of the target gradient block.

[0124] As described above, in the sparse communication mode, the important gradient is exchanged with other computing nodes each time, and then the update of the network parameters is also based on the important gradient, and other gradients except the important gradient do not perform the update of the parameters in this iteration. Therefore, after the important gradient is selected, and before the important gradient of each computing node is exchanged with other computing nodes, the local gradient except the important gradient is further stored as the delayed gradient of this iteration, and the delayed gradient corresponding to the important gradient is set to zero.

[0125] In step S708, the dense communication is performed with other computing nodes, and all the gradients are exchanged with each other, that is, the synchronization of all the gradients is performed among all the computing nodes. Specifically, the all-reduce mode can be used to perform the sparse communication with other computing nodes, and all the gradients are exchanged with each other.

[0126] In the embodiment of the present application, the local computing node also sets the delayed gradient of this iteration to zero before the all the gradients are exchanged with other computing nodes. In this way, by clearing all the delayed gradients of the local computing node before each dense gradient communication, the gradient explosion problem caused by the excessive accumulation of local gradients is effectively solved.

[0127] In step S710, the gradient average of the local gradient exchanged by all the computing nodes in the current iteration step is calculated, and the network parameters of the deep neural network are updated based on the gradient average. Specifically, the gradient descent method without momentum can be used to update the network parameters of the deep neural network, or the gradient descent method with momentum can be used to update the network parameters of the deep neural network.

[0128] After steps S702-S710 are performed, one iteration is completed. The above steps S502-S510 can be repeatedly performed, and when the deep neural network converges or the iteration number of the network parameters reaches a preset number, the training is stopped, and the deep neural network on each computing device is the trained deep neural network.

[0129] In one embodiment, the deep neural network to be trained is a neural network machine translation model, the final network parameters are applied to the model to obtain a trained neural network machine translation model, then the neural network machine translation model is used to translate the information to be translated to obtain a translation result, and the translation result is output.

[0130] Hereinafter, a specific implementation of the distributed training method is introduced by taking a global momentum distributed stochastic gradient descent (SGD) algorithm as an example.

[0131] Figure 8A flowchart of a distributed training method 800 according to an embodiment of the present invention is shown. Method 800 is applied in a distributed computing framework for a deep neural network and is executed in each computing node of the distributed computing framework. Specifically, the computing node can be implemented as the aforementioned computing device acceleration unit 230. Figure 8 As shown, method 800 performs the following steps on each compute node (worker node):

[0132] 1) Set the hyperparameters for distributed training, including the training dataset θ and the mini-batch size N. b Number of nodes N, sparse communication step size S;

[0133] 2) The local worker node reads a small batch of data D into memory;

[0134] 3) Determine if this is the first iteration; if so, initialize the model parameters w. t-1 (Parameters at step t-1), local delay gradient (The delayed gradient of the k-th working node at step t-1, also known as the delayed gradient), global momentum accumulation V t (Global momentum accumulation at step t), where w t-1 Initialize using random numbers in the range [-1, 1] generated by a normal distribution. and V t Initialize to 0;

[0135] 4) The local worker node inputs the training data into the network for forward propagation, compares the output with the expected label data to calculate the loss error (Loss), and performs backpropagation based on the Loss, calculating the gradient of each parameter layer by layer. in Let N be the local gradient of the k-th working node at step t. b For batch data size, For model parameters w t-1 The local gradient is calculated based on a single training data point in the mini-batch training dataset D.

[0136] 5) Apply the local delay gradient generated in the previous iteration. With the newly generated Accumulate to generate new

[0137] 6) Perform a modulo operation using the iteration counter t and the preset sparse communication step size S. If t%S! = 0, then determine that this communication is sparse communication, and proceed with step 7) on the new... Perform sparse compression of the gradient; otherwise, proceed directly to dense communication as described in step 9).

[0138] 7) Gradient sparsity compression processing is performed, specifically as follows:

[0139] A hierarchical Top-k fine-grained sparsity strategy is adopted, that is, the sparsity threshold of each layer is selected layer by layer (L is the number of network layers included in the deep neural network); then, a mask is generated using the threshold val of each layer According to the mask, important sparse gradients are selected And the local accumulated gradient is updated After the first θ% sparse gradients are selected, 8) is performed.

[0140] It should be understood that the top-k fine-grained sparsity strategy is a strategy for selecting important gradients based on absolute value thresholds. First, each layer of the deep neural network is unfolded into a one-dimensional tensor and sorted in ascending order according to the gradient absolute value. Then, a threshold corresponding to the preset sparsity percentage θ is selected. Finally, the mask Mask of the important gradient is derived according to the threshold.

[0141] 8) The selected sparse gradients are encoded using run-length encoding, and then communicated to other nodes based on the All-reduce distributed framework to other nodes and aggregated and averaged to obtain

[0142] 9) If this communication is dense communication, a dense communication is performed, that is, the of 5) is sent to other nodes for aggregated and averaged to obtain At the same time, the local accumulated gradient is cleared.

[0143] 10) The global momentum distributed stochastic gradient descent algorithm is adopted to perform long-term gradient compensation operation, so that the model update considers the global convergence information, thereby accelerating the convergence speed of the training. Specifically, it includes the global momentum accumulation process V t+1 = m·V t + UG t , and then the new global momentum is used for each parameter update process: w t+1 = w t - η·V t+1 .

[0144] 11) Determine whether there is still untrained data in the local node. If yes, the next iteration is performed, and the iteration counter t is incremented. Otherwise, it indicates that the training is completed, and then the trained model parameters are output. Wherein, a small batch of data is used for each training, and the total training data amount and the small batch size are fixed as N b , the total number of iterations T is obtained, so as to determine whether there is still untrained data.​

[0145] According to the gradient communication scheme of the application, the periodic alternating gradient communication is performed by combining multiple sparse gradients with one dense gradient, which effectively reduces the gradient transmission amount, speeds up the communication process, and each dense gradient communication effectively compensates for the lost gradient information, improving the convergence performance of the model. In addition, each dense gradient communication can clear all previous local delayed gradients, effectively solving the gradient explosion problem caused by excessive accumulation of local gradients.

[0146] In this way, in some application scenarios, the above-mentioned distributed training method can also be executed by a terminal device, and the terminal device is deployed with an acceleration unit such as a neural network processing unit (NPU), a graphics processing unit (GPU), etc. The terminal device can be a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a sound box computing device, a computing device of a vehicle (for example, a vehicle-mounted communication system, a vehicle-mounted entertainment system, a vehicle-mounted navigation system), a wearable device including a computing device (for example, a watch with a computing device, glasses with a computing device), or a home device including a computing device (for example, a sound box with a computing device, a television with a computing device, a washing machine with a computing device).

[0147] Figure 9 A schematic diagram of a distributed training apparatus 900 according to an embodiment of the application is shown. Referring to Figure 9 , the apparatus 900 comprises:

[0148] The gradient calculation unit 910 is adapted to obtain a training data subset from the training data set in each iteration step, and calculate the local gradient of the network parameters of the deep neural network on the training data subset;

[0149] The gradient exchange unit 920 is adapted to exchange the local gradients with other computing nodes in an alternating manner of sparse communication and dense communication, wherein in the sparse communication manner, the important gradients are exchanged with other computing nodes, and in the dense communication manner, all the gradients are exchanged with other computing nodes;

[0150] The parameter updating unit 930 is adapted to calculate the gradient average value of the local gradients exchanged by all computing nodes in the current iteration step, and update the network parameters of the deep neural network based on the gradient average value.

[0151] The specific processes performed by the gradient calculation unit 910, the gradient exchange unit 920 and the parameter updating unit 930 can refer to the above-mentioned method 700 or method 800, which will not be repeated here.

[0152] The various techniques described herein can be implemented in combination with hardware or software, or a combination thereof. Thus, the methods and apparatus of the present invention, or certain aspects or portions thereof, can take the form of program code (i.e., instructions) embedded in a tangible medium, such as a removable hard disk, USB flash drive, floppy disk, CD-ROM, or any other machine-readable storage medium, wherein when the program is loaded into and executed by a machine such as a computer, the machine becomes an apparatus for practicing the present invention.

[0153] When the program code is executed on a programmable computer, the computing device generally includes a processor, a processor-readable storage medium (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. The memory is configured to store program code; the processor is configured to execute the method of the present invention according to instructions in the program code stored in the memory.

[0154] By way of example, and not limitation, readable media include readable storage media and communication media. Readable storage media stores information such as computer-readable instructions, data structures, program modules, or other data. Communication media generally embodies computer-readable instructions, data structures, program modules, or other data in the form of modulated data signals such as carrier waves or other transmission mechanisms, and includes any information delivery medium. Any combination of the above is also included within the scope of readable media.

[0155] In the specification provided herein, the algorithms and displays are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used with the examples of this invention. The required structure for constructing such systems is apparent from the above description. Furthermore, this invention is not directed to any particular programming language. It should be understood that the contents of the invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing preferred embodiments of the invention.

[0156] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0157] Similarly, it is to be understood that the embodiments of the present application can be placed into practice notwithstanding modifications to form yet further embodiments of the present application. As such, the terms and expressions have been used on the true basis of their understood meanings in general usage, without being limited to a narrow patent- or legal- dictionary meaning that can have been invoked by the juristic or judicial authorities in any particular manner at any particular time.

[0158] Those skilled in the art will understand that the modules, or units, or components of the devices in the examples disclosed herein can be arranged in the devices as described in the examples, or alternatively can be located in one or more devices different from the devices in the examples. The modules in the foregoing examples can be combined into one module or further divided into multiple sub-modules.

[0159] Those skilled in the art will understand that the modules in the devices in the examples can be adaptively changed and disposed in one or more devices different from the examples. The modules or units or components in the examples can be combined into one module or unit or component, and further divided into multiple sub-modules or sub-units or sub-components. Except that at least some of such features and / or processes or units are mutually exclusive, any combination of all the features disclosed in the specification (including the accompanying claims, abstract and drawings), and all the processes or units of any method or device disclosed thus can be adopted. Unless explicitly stated otherwise, each feature disclosed in the specification (including the accompanying claims, abstract and drawings) can be replaced by an alternative feature providing the same, equivalent or similar function.

[0160] Further, those skilled in the art will understand that the combination of features of different embodiments means within the scope of the present application and forms different embodiments, although some of the examples described herein include certain features rather than others included in other examples. For example, in the following claims, any one of the claimed embodiments can be used in any combination.

[0161] Furthermore, some of the embodiments described herein are of a "method" or a "process" that can be embodied in software, firmware or hardware, and when embodied in software, can be implemented with computer- executable instructions. The rules of equivalence, as well as the "means-plus-function" or "step-plus-function" clauses below, are meant to cover both "articles of manufacture" and "combinations of articles of manufacture" that consist of one or more computer- readable media storing computer-executable instructions.

[0162] As used herein, unless otherwise indicated, the use of the ordinal adjectives "first", "second", "third" and so forth, merely designate different instances of an object, and do not imply a specific order or sequence unless explicitly stated.

[0163] While the application has been described in terms of several embodiments, those skilled in the art will recognize that the application can be practiced with modifications and alterations limited only by the spirit and scope of the inventiveness. Furthermore, the purpose of the description is to enable any person skilled in the art to practice the application as described in the specification and claimed claims. The contexts of the specification make it apparent to those skilled in the art that unnecessary obfuscation is to be avoided. Therefore, the use of any and all examples, or exemplary language (e.g., "for example", "for instance", "as an example", or the like), is intended merely to better illuminate the application and does not pose a limitation to the scope of the application unless claimed. Numerous modifications and adaptations will be apparent to those skilled in the art without departing from the scope of the present application as defined in the appended claims. Accordingly, the disclosure is both an enabling disclosure and a best mode of the present application, and each combination and permutation of features described and / or pictured herein is specifically contemplated. Therefore, the scope of the application is limited only by the following claims and the equivalents thereof.

Claims

1. A gradient communication method applied in a distributed training framework for a deep neural network, the distributed training framework comprising multiple computing nodes, and performing the following steps on each computing node: In each iteration step, a subset of training data is obtained from the training dataset by random sampling, and the gradient of the network parameters of the deep neural network on the subset of training data is calculated as the local gradient. Update the local gradient to the sum of the local gradient and the delayed gradient from the previous iteration; The algorithm employs alternating sparse and dense communication methods to exchange local gradients with other computing nodes. Sparse communication refers to exchanging gradients with other computing nodes after sparsifying the local gradients, while dense communication refers to exchanging gradients with other computing nodes directly without sparsifying the local gradients. In sparse communication, local gradients other than important gradients are stored as delayed gradients for the current iteration, and important gradients (gradients with values ​​greater than a preset threshold) are exchanged with other computing nodes. In dense communication, delayed gradients for the current iteration are set to zero, and all local gradients are exchanged with other computing nodes.

2. The method as described in claim 1, wherein, The alternation of sparse and dense communication includes performing a predetermined number of sparse communication operations followed by a dense communication operation.

3. The method as described in claim 1 or 2, wherein, Before the step of exchanging important gradients with other computing nodes, the method further includes: for each network layer of the deep neural network, selecting a predetermined number of local gradients as important gradients from the local gradients corresponding to that network layer in descending order of gradient value.

4. The method of claim 1, wherein, The exchange of important gradients with other computing nodes includes: For each network layer of the deep neural network, based on the important gradients of that network layer, the gradient tensor of that network layer is sparsified to form a sparse gradient tensor, wherein the gradient tensor is a tensor composed of all the local gradients of that network layer. After encoding the sparse gradient tensor, it is sent to other computing nodes.

5. The method of claim 4, wherein the sparsification of the gradient tensor of the network layer comprises: Set the local gradients in the gradient tensor, excluding the important gradients, to zero.

6. The method as described in claim 4 or 5, wherein, The encoding is run-length encoding.

7. The method as described in claim 1 or 2, wherein, The exchange of local gradients with other computing nodes includes: It uses an All-reduce approach to communicate with other computing nodes and exchange their local gradients.

8. A distributed training method, applied in a distributed training framework for deep neural networks, the distributed training framework comprising multiple computing nodes, and performing the following steps on each computing node: In each iteration step, a batch of training data subsets is obtained from the training dataset by random sampling, and the gradient of the network parameters of the deep neural network on the training data subset is calculated as the local gradient; Update the local gradient to the sum of the local gradient and the delayed gradient from the previous iteration; The algorithm employs alternating sparse and dense communication methods to exchange local gradients with other computing nodes. Sparse communication refers to exchanging gradients with other computing nodes after sparsifying the local gradients, while dense communication refers to exchanging gradients with other computing nodes directly without sparsifying the local gradients. In sparse communication, local gradients other than important gradients are stored as delayed gradients for the current iteration, and important gradients (gradients greater than a preset threshold) are exchanged with other computing nodes. In dense communication, delayed gradients for the current iteration are set to zero, and all local gradients are exchanged with other computing nodes. Calculate the average gradient of the local gradients exchanged by all computing nodes in the current iteration step, and update the network parameters of the deep neural network based on the average gradient.

9. The method of claim 8, wherein, The step of updating the network parameters of the deep neural network based on the average gradient includes: Based on the average gradient, the network parameters of the deep neural network are updated using gradient descent without momentum; or Based on the average gradient, the network parameters of the deep neural network are updated using the gradient descent method with momentum.

10. The method of claim 8 or 9, further comprising: Training stops when the deep neural network converges, or when the number of iterations of the network parameters reaches a preset number.

11. A distributed training method, applied in a distributed training framework for deep neural networks, the distributed training framework comprising multiple terminal devices, and performing the following steps on each terminal device: In each iteration step, a batch of training data subsets is obtained from the training dataset by random sampling, and the gradient of the network parameters of the deep neural network on the training data subset is calculated as the local gradient; Update the local gradient to the sum of the local gradient and the delayed gradient from the previous iteration; The algorithm employs alternating sparse and dense communication methods to exchange local gradients with other terminal devices. Sparse communication refers to exchanging gradients with other computing nodes after sparsifying the local gradients, while dense communication refers to exchanging gradients with other computing nodes directly without sparsifying the local gradients. In sparse communication, local gradients other than important gradients are stored as delayed gradients for the current iteration, and important gradients (gradients greater than a preset threshold) are exchanged with other terminal devices. In dense communication, delayed gradients for the current iteration are set to zero, and all local gradients are exchanged with other terminal devices. Calculate the average gradient of the local gradients exchanged by all terminal devices in the current iteration step, and update the network parameters of the deep neural network based on the average gradient.

12. The distributed training method as described in claim 11, wherein, The terminal device is a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a speaker computing device, a vehicle computing device, a wearable device that includes a computing device, or a home appliance that includes a computing device.

13. A distributed training device, comprising: The gradient calculation unit is adapted to obtain a subset of training data from the training dataset by random sampling in each iteration step, calculate the gradient of the network parameters of the deep neural network on the subset of training data as the local gradient, and update the local gradient to the sum of the local gradient and the delayed gradient of the previous iteration. The gradient exchange unit is suitable for exchanging local gradients with other computing nodes using alternating sparse and dense communication methods. Sparse communication refers to the communication method of exchanging gradients with other computing nodes after sparsifying the local gradients, while dense communication refers to the communication method of exchanging gradients with other computing nodes directly without sparsifying the local gradients. In the sparse communication method, the local gradients except for the important gradients are stored as the delayed gradients of the current iteration, and the important gradients are exchanged with other computing nodes. The important gradients are the local gradients whose gradient values ​​are greater than a preset threshold. In the dense communication method, the delayed gradients of the current iteration are set to zero, and all local gradients are exchanged with other computing nodes. The parameter update unit is adapted to calculate the average gradient of the local gradients exchanged by all computing nodes in the current iteration step, and update the network parameters of the deep neural network based on the average gradient.

14. A distributed training system comprising multiple computing nodes, each computing node comprising the distributed training apparatus as described in claim 13.

15. A computing device, comprising: At least one processor; and A memory storing program instructions, wherein the program instructions are configured to be executed by the at least one processor, the program instructions including instructions for performing the method as described in any one of claims 1-12.

16. A readable storage medium storing program instructions that, when read and executed by a computing device, cause the computing device to perform the method as described in any one of claims 1-12.

Images