A distributed training method, gradient communication method, device and computing equipment

Abstract

The present invention discloses a distributed training method, gradient communication method, apparatus, and computing device for a deep neural network. The method comprises: obtaining a training data subset from a training data set, calculating the gradient of the network parameters of the deep neural network on the training data subset as a local gradient; for each network layer of the deep neural network, calculating the gradient metric value of each gradient block in the network layer, and selecting a predetermined number of target gradient blocks in descending order of the gradient metric value, wherein the gradient block includes multiple local gradients with consecutive storage addresses; and communicating with other computing nodes to exchange each other's target gradient blocks.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a distributed training method, gradient communication method, apparatus, and computing device for deep neural networks. Background Art

[0002] As deep neural networks (DNNs) and training datasets continue to grow, their training requires enormous computing power, storage space, communication bandwidth, and power consumption. Current mainstream DNN training platforms are based on distributed multi-node systems, leveraging parallel computing to significantly accelerate the training process. However, frequent gradient exchange communications increase transmission bandwidth pressure, leading to problems such as limited platform scalability and long communication times. Therefore, communication time has become a bottleneck in the development of distributed training.

[0003] Gradient compression is a promising method for accelerating distributed training by reducing communication traffic. It primarily includes two strategies: gradient quantization and gradient sparsification. Gradient quantization compresses gradient data by reducing the bit width of gradient data. Gradient sparsification uses a correlation evaluation method to select important gradients for parameter updates first. The remaining unimportant gradients, also known as delayed gradients, are locally accumulated with the delayed gradients from the previous iteration without any parameter updates.

[0004] Among gradient sparsification methods, the current mainstream strategy is the top-k sparsification strategy, which selects the top k gradients of each neural network layer as important gradients based on their importance. However, the important gradients selected by the top-k sparsification strategy suffer from discontinuous memory addresses and irregular indexing. Furthermore, the sorting and random sampling required to select gradient importance are extremely hardware-unfriendly. Furthermore, irregular sparse data introduces complex access control logic, resulting in a lengthy sparsification process and making hardware implementation difficult.

[0005] Therefore, how to provide a hardware-friendly gradient sparse method with continuous memory addresses, simple control logic, and accelerated sparse processing is an urgent problem that needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of the above problems, the present invention is proposed to provide a distributed training method, gradient communication method, apparatus and computing device for deep neural networks that overcome the above problems or at least partially solve the above problems.

[0007] According to one aspect of the present invention, a gradient communication method for a deep neural network is provided, which is applied to a distributed training framework for a deep neural network. The distributed computing framework includes multiple computing nodes, and the following steps are performed on each computing node:

[0008] Obtaining a training data subset from a training data set, and calculating the gradient of a network parameter of the deep neural network on the training data subset as a local gradient;

[0009] For each network layer of the deep neural network, calculate the gradient metric value of each gradient block in the network layer, and select a predetermined number of target gradient blocks in descending order of the gradient metric value, wherein the gradient block includes a plurality of local gradients with consecutive storage addresses;

[0010] Communicate with other computing nodes and exchange each other's target gradient blocks.

[0011] Optionally, in the gradient communication method of the present invention, the network layer includes a convolutional layer, and the gradient block is a plurality of local gradients corresponding to a filter in the convolutional layer.

[0012] Optionally, in the gradient communication method of the present invention, the network layer includes a fully connected layer, and the gradient block is a plurality of local gradients corresponding to network parameters between a network node in the fully connected layer and all network nodes in the previous network layer.

[0013] Optionally, in the gradient communication method of the present invention, the network layer includes a bias layer, and the gradient block is a local gradient corresponding to a network node in the bias layer.

[0014] Optionally, in the gradient communication method of the present invention, the gradient metric value is the L1 norm or L2 norm of the gradient block.

[0015] Optionally, in the gradient communication method of the present invention, the predetermined number is 1.

[0016] Optionally, in the gradient communication method of the present invention, the communicating with other computing nodes and exchanging each other's target gradient blocks includes: for each network layer of the deep neural network, based on the target gradient block of the network layer, performing sparsification processing on the gradient tensor of the network layer to form a sparse gradient tensor, wherein the gradient tensor is a tensor composed of all local gradients of the network layer; and encoding the sparse gradient tensor and sending it to other computing nodes.

[0017] Optionally, in the gradient communication method of the present invention, the performing sparse processing on the gradient tensor of the network layer includes: setting local gradients in the gradient tensor except for the target gradient block to zero.

[0018] Optionally, in the gradient communication method of the present invention, the encoding is run-length encoding.

[0019] Optionally, in the gradient communication method of the present invention, before the step of calculating the gradient metric value of each gradient block in the network layer, the method further includes: updating the local gradient to the sum of the local gradient and the delayed gradient of the previous iteration; and after the step of selecting a predetermined number of target gradient blocks in descending order of gradient metric value, the method further includes: storing the local gradients other than the target gradient blocks as the delayed gradients of the current iteration.

[0020] Optionally, in the gradient communication method of the present invention, communicating with other computing nodes and exchanging each other's target gradient blocks includes: communicating with other computing nodes in an All-reduce manner and exchanging each other's target gradient blocks.

[0021] According to another aspect of the present invention, a distributed training method is provided for use in a distributed training framework for a deep neural network. The distributed computing framework includes multiple computing nodes, and the following steps are performed on each computing node:

[0022] Obtaining a training data subset from a training data set, and calculating the gradient of a network parameter of the deep neural network on the training data subset as a local gradient;

[0023] For each network layer of the deep neural network, calculate the gradient metric value of each gradient block in the network layer, and select a predetermined number of target gradient blocks in descending order of the gradient metric value, wherein the gradient block includes a plurality of local gradients with consecutive storage addresses;

[0024] Communicate with other computing nodes and exchange each other's target gradient blocks;

[0025] Calculate the average gradient of the target gradient blocks of all computing nodes, and update the network parameters of the deep neural network based on the average gradient.

[0026] Optionally, in the distributed training method of the present invention, before the step of calculating the gradient metric value of each gradient block in the network layer, the method further includes: updating the local gradient to the sum of the local gradient and the delayed gradient of the previous iteration;

[0027] After selecting a predetermined number of target gradient blocks in descending order of gradient metric values, the method further includes storing local gradients other than the target gradient blocks as delayed gradients of this iteration.

[0028] Optionally, in the distributed training method of the present invention, updating the network parameters of the deep neural network based on the gradient average includes: updating the network parameters of the deep neural network based on the gradient average by using a gradient descent method without momentum; or updating the network parameters of the deep neural network based on the gradient average by using a gradient descent method with momentum.

[0029] Optionally, the distributed training method of the present invention further includes: stopping training when the deep neural network converges, or when the number of iterations of the network parameters reaches a preset number.

[0030] According to another aspect of the present invention, a distributed training method is provided for use in a distributed training framework for a deep neural network. The distributed computing framework includes multiple terminal devices, and the following steps are performed on each terminal device:

[0031] Obtaining a training data subset from a training data set, and calculating the gradient of a network parameter of the deep neural network on the training data subset as a local gradient;

[0032] For each network layer of the deep neural network, calculate the gradient metric value of each gradient block in the network layer, and select a predetermined number of target gradient blocks in descending order of the gradient metric value, wherein the gradient block includes a plurality of local gradients with consecutive storage addresses;

[0033] Communicate with other terminal devices and exchange target gradient blocks with each other;

[0034] Calculate the gradient average of the target gradient blocks of all terminal devices, and update the network parameters of the deep neural network based on the gradient average.

[0035] Optionally, in the distributed training method of the present invention, the terminal device is a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a speaker computing device, a vehicle computing device, a wearable device including a computing device, or a home device including a computing device.

[0036] According to another aspect of the present invention, there is provided a distributed training device, comprising:

[0037] a gradient calculation unit, adapted to obtain a training data subset from a training data set, and calculate the gradient of the network parameters of the deep neural network on the training data subset as a local gradient;

[0038] a gradient selection unit adapted to calculate, for each network layer of the deep neural network, a gradient metric value of each gradient block in the network layer, and select a predetermined number of target gradient blocks in descending order of the gradient metric values, wherein the gradient block includes a plurality of local gradients having consecutive storage addresses;

[0039] A gradient exchange unit, adapted to communicate with other computing nodes and exchange target gradient blocks with each other;

[0040] A parameter updating unit is adapted to calculate the gradient average of the target gradient blocks of all computing nodes, and update the network parameters of the deep neural network based on the gradient average.

[0041] According to another invention of the present invention, a distributed training system is provided, comprising a plurality of computing nodes, each computing node comprising the above-mentioned distributed training device.

[0042] According to another aspect of the present invention, a computing device is provided, comprising: at least one processor; and a memory storing program instructions, wherein the program instructions are configured to be suitable for execution by the at least one processor, and the program instructions include instructions for executing the above method.

[0043] According to another aspect of the present invention, a readable storage medium storing program instructions is provided. When the program instructions are read and executed by a computing device, the computing device executes the above method.

[0044] In distributed sparse communication, the present invention proposes a coarse-grained important gradient selection method, that is, important gradient selection is performed in units of gradient blocks. On the one hand, it avoids time-consuming problems such as sorting a large amount of gradient data, greatly accelerating the sparse process. On the other hand, the storage addresses of the obtained important gradients are continuous, and no additional resources are required to process sparse format data, which facilitates hardware memory access control and achieves the goals of high performance and low power consumption, thereby improving the overall training efficiency of distributed training of deep neural networks.

[0045] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0047] Figure 1 shows a structural diagram of a data center according to an embodiment of the present invention;

[0048] Figure 2 FIG2 shows an internal structure diagram of a server in a data center according to an embodiment of the present invention;

[0049] Figure 3 is a diagram showing the connection relationship between a scheduling unit and an acceleration unit in a server according to an embodiment of the present invention;

[0050] Figure 4 is a diagram of the internal structure of an acceleration unit core according to one embodiment of the present disclosure;

[0051] Figure 5 1 shows a flow chart of a gradient communication method 500 according to one embodiment of the present invention;

[0052] Figure 6 shows a flow chart of a distributed training method 600 according to one embodiment of the present invention;

[0053] Figure 7 1. A flow chart of a distributed training method 700 according to an embodiment of the present invention is shown;

[0054] Figure 8 A schematic diagram of a distributed training device 800 according to an embodiment of the present invention is shown;

[0055] Figure 9 Shows a schematic diagram of deep neural network filter calculation;

[0056] Figure 10 A schematic diagram comparing fine-grained and coarse-grained sparse strategies is shown. DETAILED DESCRIPTION

[0057] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0058] First, the implementation environment of the distributed training method according to the embodiment of the present invention is introduced.

[0059] Data Center

[0060] Data centers are globally coordinated networks of specialized equipment used to transmit, accelerate, display, compute, and store data on the internet's infrastructure. In the future, data centers will become a competitive asset for businesses. With the widespread use of data centers, artificial intelligence and other technologies are increasingly being applied to data centers. Neural networks, a key AI technology, are already widely used in data center big data analytics.

[0061] In traditional large data centers, the network structure is usually as follows Figure 1 As shown in Figure 1, this is the hierarchical inter-networking model. This model consists of the following parts:

[0062] Server 140 : Each server 140 is a processing and storage entity of the data center. The processing and storage of a large amount of data in the data center are completed by these servers 140 .

[0063] Access switch 130: Access switches 130 connect servers 140 to the data center. One access switch 130 connects to multiple servers 140. Access switches 130 are typically located at the top of a rack, and are therefore also called top-of-rack switches. They physically connect to the servers.

[0064] Aggregation switch 120: Each aggregation switch 120 connects to multiple access switches 130 and provides other services, such as firewall, intrusion detection, network analysis, etc.

[0065] Core switch 110: Core switch 110 provides high-speed forwarding for packets entering and leaving the data center and provides connectivity for aggregation switch 120. The entire data center network is divided into an L3 routing network and an L2 routing network. Core switch 110 generally provides a resilient L3 routing network for the entire data center network.

[0066] Typically, aggregation switch 120 is the demarcation point between the L2 and L3 routing networks. The network below aggregation switch 120 is the L2 network, and the network above it is the L3 network. Each aggregation switch group manages a point of delivery (POD), each of which is an independent VLAN network. Servers can migrate within a POD without changing their IP addresses or default gateways, as each POD corresponds to a single L2 broadcast domain.

[0067] The Spanning Tree Protocol (STP) is typically used between aggregation switches 120 and access switches 130. STP ensures that only one aggregation switch 120 is available for a given VLAN network, with other aggregation switches 120 being used only in the event of a failure (dashed lines in the figure above). This means that horizontal scalability is impossible at the aggregation switch 120 level because even if multiple aggregation switches 120 are added, only one is still operational.

[0068] server

[0069] Since the server 140 is the actual processing device in the data center, Figure 2 The figure shows a block diagram of the internal structure of a server 140. The server 140 includes a memory 210, a scheduling unit cluster 270 and an acceleration unit cluster 280 connected by a bus. The scheduling unit cluster 270 includes a plurality of scheduling units 220. The acceleration unit cluster 280 includes a plurality of acceleration units 230. In the embodiment of the present disclosure, the acceleration unit is a special processing unit designed mainly to accelerate the operation processing speed of the neural network model, which can be embodied as a processing unit (NPU) specially designed for neural network operation processing, a graphics processing unit (GPU), an application-specific integrated circuit (ASIC) and a field-programmable gate array (FPGA). The scheduling unit is a processing unit that schedules the acceleration unit and allocates a sequence of instructions to be executed to each acceleration unit. It can take the form of a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) and the like.

[0070] The traditional central processing unit (CPU) architecture design results in the control unit and storage unit occupying a large portion of the space, while the computing unit occupies insufficient space. Therefore, while it is very effective in logic control, it is inefficient in large-scale parallel computing. Therefore, various specialized acceleration units have been developed to more effectively improve the computing speed of calculations for different functions and different fields. The acceleration unit proposed in the present invention is a processing unit specifically designed to accelerate the computing speed of neural network models. It adopts a data-driven parallel computing architecture and is used to process a large number of operations (such as convolution and pooling) of each neural network node. Because the data and intermediate results of the large number of operations (such as convolution and pooling) of each neural network node are closely related and are frequently used throughout the entire computing process, the existing CPU architecture requires frequent access to external memory due to the small memory capacity within the CPU core, resulting in inefficient processing. Using this acceleration unit specifically designed to accelerate the computing speed of neural network models, each core has on-chip memory with a storage capacity suitable for neural network calculations, avoiding frequent access to external memory, greatly improving processing efficiency and computing performance.

[0071] The acceleration unit 230 must accept the scheduling of the scheduling unit 220. Figure 2 As shown, the memory 210 stores various neural network models, including the nodes of these models and the weight data of the nodes. These neural network models are used when needed. Figure 2 A scheduling unit 220 is deployed to an acceleration unit 230. That is, the scheduling unit 220 can send the addresses of the parameters in the model (such as the weights of each node) in the memory 210 to the acceleration unit 230 in the form of instructions. When the acceleration unit 230 actually uses the neural network model for calculation, it will directly address these parameters in the memory 210 according to the addresses of these parameters (such as weights) in the memory 210, and temporarily store them in its on-chip memory. When the acceleration unit 230 actually uses the neural network model for calculation, the scheduling unit 220 will also send the input parameters of the model to the acceleration unit 230 in the form of instructions, and temporarily store them in the on-chip memory of the acceleration unit 230. In this way, the acceleration unit 230 can perform inference calculations based on these input parameters and the parameters in the model (such as weights).

[0072] Internal structure of the scheduling unit and acceleration unit

[0073] The following combination Figure 3 The internal structure diagram of the scheduling unit 220 and the acceleration unit 230 specifically illustrates how the scheduling unit 220 schedules the acceleration unit 230 to work.

[0074] like Figure 3As shown, the scheduling unit 220 includes multiple processor cores 222 and a cache 221 shared by the multiple processor cores 222. Each processor core 222 includes an instruction fetch unit 203, an instruction decoding unit 224, an instruction issuing unit 225, and an instruction executing unit 226.

[0075] The instruction fetch unit 223 is used to move the instruction to be executed from the memory 210 to the instruction register (which may be Figure 3 The register file 229 shown is used to store a register for instructions) and receives the next instruction fetch address or calculates the next instruction fetch address according to an instruction fetch algorithm. The instruction fetch algorithm includes, for example, increasing or decreasing the address according to the instruction length.

[0076] After fetching the instruction, the dispatch unit 220 enters the instruction decoding stage. The instruction decoding unit 224 decodes the fetched instruction according to the predetermined instruction format to obtain the operand fetch information required by the fetched instruction, thereby preparing for the operation of the instruction execution unit 225. The operand fetch information may point to an immediate value, a register, or other software / hardware that can provide a source operand.

[0077] The instruction issuing unit 225 is located between the instruction decoding unit 224 and the instruction executing unit 226 and is used for scheduling and controlling instructions to efficiently distribute each instruction to different instruction executing units 226 , thereby enabling parallel operation of multiple instructions.

[0078] After the instruction issuing unit 225 sends the instruction to the instruction executing unit 226, the instruction executing unit 226 begins executing the instruction. However, if the instruction executing unit 226 determines that the instruction should be executed by an acceleration unit, it forwards the instruction to the corresponding acceleration unit for execution. For example, if the instruction is for neural network inference, the instruction executing unit 226 will no longer execute the instruction and instead send it via the bus to the acceleration unit 230 for execution.

[0079] The acceleration unit 30 includes multiple cores 236 ( Figure 3 Four cores are shown in the figure, but those skilled in the art should understand that the acceleration unit 230 may also include other numbers of cores 236), a command processor 237, a direct storage access mechanism 235, and a bus channel 231.

[0080] The bus channel 231 is a channel for instructions to enter and exit the acceleration unit 230 from the bus.

[0081] Direct Memory Access (DMA) mechanism 235 is a feature provided by some computer bus architectures that enables data to be written directly from an attached device to the computer's mainboard memory. This greatly improves data access efficiency compared to a method where all data transfers between devices must pass through the dispatch unit. Because of this mechanism, the core of the acceleration unit 230 can directly access memory 210 and read parameters (e.g., the weights of each node) in the neural network model, greatly improving data access efficiency.

[0082] The command processor 237 assigns instructions sent from the scheduling unit 220 to the acceleration unit 230 to a core 236 for execution. The instruction execution unit 226 sends a sequence of pending instructions to be executed by the acceleration unit 230 to the acceleration unit 230. After entering the sequence of pending instructions from the bus channel 231, it is cached in the command processor 237, which then selects a core 236 and assigns the instruction sequence to it for execution. The command processor 237 also manages synchronization between cores 236.

[0083] Acceleration unit core

[0084] Figure 4 FIG. 2 is a diagram of the internal structure of the acceleration unit core 236 according to one embodiment of the present disclosure.

[0085] In one embodiment, Figure 4 As shown, the acceleration unit core 236 includes a tensor engine 310 , a pooling engine 320 , a memory copy engine 330 , a sequencer 350 , an instruction cache 340 , an on-chip memory 360 , and a constant buffer 370 .

[0086] The instruction sequence assigned by the command processor 237 to the acceleration unit core 236 is first cached in the instruction cache 340. The sequencer 350 then fetches instructions from the instruction cache 340 in first-in, first-out order and, depending on the nature of the instruction, assigns them to the tensor engine 310, pooling engine 320, or memory copy engine 330 for execution. The tensor engine 310 is responsible for processing operations such as convolution and matrix multiplication in the neural network model. The pooling engine 320 is responsible for processing pooling operations in the neural network model. The memory copy engine 330 is responsible for copying operands stored in the on-chip memory 360 within the core 236 to the memory shared between the cores 236, or to the on-chip memory 360 within other cores 236. The sequencer 350 determines whether to assign the instruction to the tensor engine 310, pooling engine 320, or memory copy engine 330 based on the nature of the operation, such as convolution, matrix multiplication, pooling, or operand copy.

[0087] On-chip memory 360 is an in-core memory that stores weight parameters in the neural network model, as well as input parameters and various intermediate results during actual use of the neural network model. Constant buffer 370 is a buffer that stores other constant parameters in the neural network model (e.g., hyperparameters) in addition to weight parameters. As described above, when the scheduling unit 220 pre-configures the neural network model in the acceleration unit 230, the scheduling unit 220 sends the addresses of the model parameters in memory 210 to the acceleration unit 230 in the form of instructions. These parameters include node weights and other parameters (e.g., hyperparameters). For weights, the acceleration unit 230 retrieves them from the corresponding locations in memory 210 during actual neural network model operations and stores them in on-chip memory 360. For other parameters, the acceleration unit 230 retrieves them from the corresponding locations in memory 210 during actual neural network model operations and stores them in constant buffer 370. Furthermore, when the instruction to actually start inference is assigned by the command processor 237 to the core 236 for execution, the input parameters (inputs to the neural network model) in the instruction are also stored in on-chip memory 360. In addition, after the tensor engine 310 and the pooling engine 320 perform convolution or pooling operations, the various intermediate results obtained are also stored in the on-chip memory 360.

[0088] The distributed training method of an embodiment of the present invention can be executed in the above-mentioned data center. Specifically, each acceleration unit in the multiple servers included in the data center can be regarded as a computing node, and multiple computing nodes can be composed of a distributed training framework for distributed training. These computing nodes can be CPU devices, GPU devices, NPU (neural network processor) devices or other types of computing devices. The computing nodes are connected in a ring shape, a mesh shape or other ways.

[0089] The computing node can obtain the deep neural network and the training data subset from the memory 210, and train the deep neural network based on the obtained training data subset. During the model training process, the computing node needs to exchange gradient data with other computing nodes and update the network parameters (model parameters) of the deep neural network based on the average value of the gradient data of all computing nodes. For example, the gradient descent method is used to update the network parameters of the deep neural network.

[0090] Data parallelism is a form of distributed training for deep neural networks. It divides the training data into multiple parts and trains them on different compute nodes. If the compute nodes do not have shared public memory, but only have limited local memory, and the training dataset is too large to store in local memory, the training dataset needs to be divided and distributed to each compute node. The compute nodes then train the deep neural network based on their allocated local data.

[0091] The typical distributed training process of a deep neural network is:

[0092] (1) In each iteration (training step), the computing node obtains a subset of training data from the training data set. Usually, the number of training data in the training data subset is much smaller than the number of training data included in the training data set. Therefore, it can also be called a mini-batch of training data.

[0093] (2) Calculate the gradient of the network parameters (weights, biases, etc.) of the deep neural network on the small batch training data. Usually, the gradient calculated on a computing node is also called the local gradient of the computing node;

[0094] (3) Performing sparse processing on the local gradients, that is, selecting important gradients from the local gradients for communication. Important gradients are local gradients whose gradient values ​​are greater than a preset threshold;

[0095] (4) Communicate with other computing nodes and exchange important gradients with each other;

[0096] (5) Calculate the gradient average of the important gradients of all computing nodes and update the network parameters of the deep neural network based on the gradient average.

[0097] Repeat steps (1) to (5) until the deep neural network model converges or the number of iterations of the network parameters reaches a preset number, and then stop training.

[0098] The traditional top-k sparsification strategy is a fine-grained sparsification strategy. For each layer of a deep neural network, important gradients are selected by applying a predetermined threshold, with gradients greater than the threshold being considered important. This selection process is based on individual gradients, hence the term "fine-grained." On the one hand, selecting important gradients requires sorting a large amount of gradient data, which is very time-consuming. On the other hand, the selected important gradients are isolated, have discontinuous memory addresses, and are sparse data with irregular indexes. Irregular sparse data introduces complex access control logic, resulting in a lengthy sparsification process and difficulties in hardware implementation.

[0099] Therefore, in an embodiment of the present invention, a coarse-grained sparse strategy is proposed. The coarse-grained selection object is the data block level, and important gradient selection is performed in units of data blocks (gradient blocks). For example, for the convolution layer, the selection object is the filter (convolution kernel) level, that is, the important filter is selected (corresponding to the gradients of C×H×W parameters, where C, H, and W are the channels, height, and width of the filter, respectively). The storage addresses of the C×H×W gradients inside the filter are continuous, and only one base address is required. Then, sparse communication is performed based on the selected target gradient block. On the one hand, time-consuming problems such as sorting a large amount of gradient data are avoided, greatly accelerating the sparse process. On the other hand, the storage addresses of the important gradients obtained are continuous, and no additional resources are required to process sparse format data, which facilitates hardware memory access control implementation.

[0100] Figure 5 A flow chart of a gradient communication method 500 according to an embodiment of the present invention is shown. Method 300 is applied to a distributed computing framework of a deep neural network and is executed in each computing node of the distributed computing framework. The computing node can be specifically implemented as the aforementioned acceleration unit 230. Method 500 can be applied to various scenarios such as images, voice, video, and machine translation. For example, in an image scenario, the corresponding deep neural network can be an image classification model, a target detection model, etc.; in a machine translation scenario, the corresponding deep neural network can be a neural network machine translation model. Among them, the neural network machine translation model is a sequence-to-sequence model having an encoder made by a gated recurrent unit, an encoder made by a gated recurrent unit, and an attention mechanism. Method 500 performs the following steps on each computing node:

[0101] In step S502, in each iteration step, a training data subset, i.e., a mini-batch of training data, is obtained from the training data set, and the gradient of the network parameters of the deep neural network on the training data subset is calculated as the local gradient.

[0102] Training data can include image samples, speech samples, and natural language processing samples. For example, if the deep neural network to be trained is a neural network machine translation model, each piece of training data is a text pair, representing the correspondence between a text in the first language and a text in the second language.

[0103] In an embodiment of the present invention, distributed training uses a data parallel approach. In the data parallel approach, a training data set stored in a memory is used as a sampling data source, and a computing node uses random sampling to obtain multiple training data from the training data set as small batches of training data. Specifically, random sampling can be performed with replacement, which ensures that the local training data on each computing node is independent and identically distributed with the original training data. In addition, the number of training data items obtained in each iteration, i.e., the batch size of the small batch of data, can be determined based on the memory size of the computing node.

[0104] Mini-batch training data is fed into a deep neural network for forward propagation. The output of the deep neural network is compared with the expected labeled data to calculate the loss error. Backward propagation is then performed based on the loss error, calculating the gradients of each parameter of the deep neural network layer by layer on the mini-batch training data as local gradients. Typically, all local gradients of each network layer of the deep neural network form a gradient tensor, with each network parameter in the network layer corresponding to an element in the gradient tensor.

[0105] Because compute nodes use sparse gradient communication, they exchange important gradients with other compute nodes each time. Network parameter updates are also based on these important gradients. Other gradients are not used to update parameters in this iteration, but are instead stored as delayed gradients for subsequent parameter updates. Therefore, in the current iteration, the calculated local gradient is updated to the sum of the local gradient and the delayed gradient from the previous iteration.

[0106] In step S504, for each network layer of the deep neural network, the gradient metric value of each gradient block in the network layer is calculated, and a predetermined number of target gradient blocks are selected in descending order of the gradient metric values. For example, the gradient block with the largest gradient metric value is selected as the target gradient block. In this embodiment of the present invention, the gradient metric value can be the L1 norm or L2 norm of the gradient block. Of course, the gradient metric value can also be other metrics that represent the overall gradient value of the gradient block. This embodiment of the present invention does not limit the specific metric method.

[0107] In this embodiment of the present invention, a gradient block refers to a data block consisting of multiple local gradients with consecutive storage addresses. A deep neural network includes convolutional layers, bias layers, pooling layers, and fully connected layers. The only network layers with parameters and gradients are the convolutional, bias, and fully connected layers; the pooling layer does not have parameters or gradients.

[0108] For a convolutional layer, a gradient block is a plurality of local gradients corresponding to all network parameters included in a filter in the convolutional layer. Each filter corresponds to a gradient block, and a convolutional layer corresponds to multiple gradient blocks. Figure 9 Figure 2 shows a schematic diagram of deep neural network filter calculation. Figure 9 As shown, each convolution layer includes multiple filter kernels (i.e., convolution kernels, 4 of which are shown in the figure, namely filter K1, filter K2, filter K3 and filter K4). The size of the filter is C×K×K, where C, K, and K are the number of channels, height, and width of the filter respectively. Each filter includes C×K×K network parameters, each network parameter corresponds to a gradient, and all gradients corresponding to a filter constitute a gradient block. The scale of the gradient block is the same as the scale of the filter. Given the input feature map Ifmap, filter Kernel, and bias Bias, the output feature map Ofmap l =f((∑Ifmap l-1 *Kernel l )+Bias l ).

[0109] For the bias layer, the gradient block is the local gradient corresponding to a network node in the bias layer. Each network node corresponds to a gradient block (a gradient block with only one element), and a bias layer corresponds to multiple gradient blocks. Since each network parameter in the bias layer corresponds to a different filter, the parameter gradient of the bias layer is a one-dimensional vector with the same number of filters. The bias layer is treated similarly to other layers, selecting the gradient with the largest L1 norm or L2 norm, that is, the gradient with the largest absolute value.

[0110] For the fully connected layer, the gradient block is a plurality of local gradients corresponding to the network parameters between a network node (i.e., a neuron) in the fully connected layer and all the network nodes of the previous network layer. Each network node corresponds to a gradient block, and a fully connected layer corresponds to multiple gradient blocks. Specifically, assuming that the parameter size of the fully connected layer is F×L, where F is the output dimension and L is the input dimension (one-dimensional vector), when processing the fully connected layer, the embodiment of the present invention selects L with the largest L1 norm or L2 norm of the gradient in F. F can be equivalent to the number of filters f in the convolutional layer, and L is equivalent to a single filter, except that the dimensions of the two are different, the former is 1-dimensional and the latter is 3-dimensional. Therefore, the gradient block in the fully connected layer corresponds to the gradient of the parameter corresponding to a certain L, and its size is determined by the input dimension. The storage of each L-sized gradient parameter corresponding to the output dimension F is continuous. Therefore, the memory address of the gradient block in the fully connected layer selected by coarse granularity is also continuous.

[0111] As mentioned above, due to the sparse gradient communication between compute nodes, only important gradients are exchanged with other compute nodes each time. Network parameter updates are then based on these important gradients. Other gradients are not used to update parameters in this iteration, but are instead stored as delayed gradients for subsequent parameter updates. Specifically, local gradients other than the target gradient block are stored as delayed gradients for this iteration, and the delayed gradients corresponding to the target gradient block are set to zero.

[0112] In step S506, the target gradient blocks are exchanged with other computing nodes, that is, all computing nodes synchronize their target gradient blocks. Specifically, an all-reduce approach can be used to communicate with other computing nodes and exchange their target gradient blocks.

[0113] In this embodiment of the present invention, the communication between computing nodes to exchange target gradient blocks is sparse communication. In each computing node, for each layer of the deep neural network, the gradient tensor of that layer is sparsified based on the target gradient block of that layer to form a sparse gradient tensor. This sparse gradient tensor is then encoded, for example using run-length encoding, and sent to other computing nodes to complete the exchange of target gradient blocks.

[0114] Run-length encoding is a relatively simple compression algorithm. Its basic concept is to describe repeated and consecutive characters using (number of consecutive occurrences, a certain character). For example, the existing character data AAAAABBBBCCC becomes 5A4B3C after run-length encoding. In the embodiment of the present invention, because a large number of elements in the sparse gradient tensor are zero, run-length encoding can greatly reduce communication traffic.

[0115] Figure 10 This is a comparison of the implementation principles of the coarse-grained gradient sparse strategy provided by the present invention and the existing fine-grained strategy top-k, as shown in Figure 10 As shown, the important gradients (filter gradient blocks) selected by the present invention have continuous addresses in the memory pool, so burst transmission can be supported without requiring additional resources to store indexes and values ​​separately, which is convenient for hardware implementation.

[0116] The following introduces a distributed training method based on the above gradient communication method.

[0117] Figure 6The flowchart of the distributed training method 600 according to one embodiment of the present invention is shown. The method 600 is applied to the distributed computing framework of the deep neural network and is executed in each computing node of the distributed computing framework. The computing node can be specifically implemented as the aforementioned acceleration unit 230. The method 600 can be applied to various scenarios such as images, voice, video, and machine translation. For example, in the image scenario, the corresponding deep neural network can be an image classification model, a target detection model, etc.; in the machine translation scenario, the corresponding deep neural network can be a neural network machine translation model. Figure 6 As shown, method 600 performs the following steps on each computing node:

[0118] In step S602, in each iteration step, a training data subset, i.e., a small batch of training data, is obtained from the training data set, and the local gradient of the network parameters of the deep neural network on the small batch of training data is calculated.

[0119] Training data can include image samples, speech samples, and natural language processing samples. For example, if the deep neural network to be trained is a neural network machine translation model, each piece of training data is a text pair, representing the correspondence between a text in the first language and a text in the second language.

[0120] In an embodiment of the present invention, distributed training utilizes a data-parallel approach. In this data-parallel approach, a training dataset stored in a data storage device is used as the sampling data source, and computing nodes utilize random sampling to obtain multiple training data from the training dataset as mini-batch training data. Specifically, random sampling can be performed with replacement, ensuring that the local training data on each computing node is independent and identically distributed with the original training data. Furthermore, the number of training data items obtained in each iteration, i.e., the batch size of the mini-batch data, can be determined based on the memory size of the computing node.

[0121] Mini-batch training data is fed into a deep neural network for forward propagation. The output of the deep neural network is compared with the expected labeled data to calculate the loss error. Backward propagation is then performed based on the loss error, calculating the gradients of each parameter of the deep neural network layer by layer on the mini-batch training data as local gradients. Typically, all local gradients of each network layer of the deep neural network form a gradient tensor, with each network parameter in the network layer corresponding to an element in the gradient tensor.

[0122] Because compute nodes use sparse gradient communication, they exchange important gradients with other compute nodes each time. Network parameter updates are also based on these important gradients. Other gradients are not used to update parameters in this iteration, but are instead stored as delayed gradients for subsequent parameter updates. Therefore, in the current iteration, the calculated local gradient is updated to the sum of the local gradient and the delayed gradient from the previous iteration.

[0123] In step S604, for each network layer of the deep neural network, the gradient metric value of each gradient block in the network layer is calculated, and a predetermined number of target gradient blocks are selected in descending order of the gradient metric values. For example, the gradient block with the largest gradient metric value is selected as the target gradient block. In this embodiment of the present invention, the gradient metric value can be the L1 norm or L2 norm of the gradient block. Of course, the gradient metric value can also be other metrics that represent the overall gradient value of the gradient block. This embodiment of the present invention does not limit the specific metric method.

[0124] In this embodiment of the present invention, a gradient block refers to a data block consisting of multiple local gradients with consecutive storage addresses. A deep neural network includes convolutional layers, bias layers, pooling layers, and fully connected layers. The only network layers with parameters and gradients are the convolutional, bias, and fully connected layers; the pooling layer does not have parameters or gradients.

[0125] For a convolutional layer, a gradient block is a plurality of local gradients corresponding to all network parameters included in a filter in the convolutional layer. Each filter corresponds to a gradient block, and a convolutional layer corresponds to multiple gradient blocks.

[0126] For the bias layer, the gradient block is the local gradient corresponding to a network node in the bias layer. Each network node corresponds to a gradient block (a gradient block with only one element), and a bias layer corresponds to multiple gradient blocks. Since each network parameter in the bias layer corresponds to a different filter, the parameter gradient of the bias layer is a one-dimensional vector with the same number of filters. The bias layer is treated similarly to other layers, selecting the gradient with the largest L1 norm or L2 norm, that is, the gradient with the largest absolute value.

[0127] For a fully connected layer, a gradient block is a number of local gradients corresponding to the network parameters between a network node (i.e., a neuron) in the fully connected layer and all network nodes in the previous network layer. Each network node has one gradient block, and a fully connected layer corresponds to multiple gradient blocks.

[0128] As mentioned above, due to the sparse gradient communication between compute nodes, only important gradients are exchanged with other compute nodes each time. Network parameter updates are then based on these important gradients. Other gradients are not used to update parameters in this iteration, but are instead stored as delayed gradients for subsequent parameter updates. Specifically, local gradients other than the target gradient block are stored as delayed gradients for this iteration, and the delayed gradients corresponding to the target gradient block are set to zero.

[0129] In step S606, the target gradient blocks are exchanged with other computing nodes, that is, all computing nodes synchronize their target gradient blocks. Specifically, an all-reduce approach can be used to communicate with other computing nodes and exchange their target gradient blocks.

[0130] In this embodiment of the present invention, the communication between computing nodes to exchange target gradient blocks is sparse communication. In each computing node, for each layer of the deep neural network, the gradient tensor of that layer is sparsified based on the target gradient block of that layer to form a sparse gradient tensor. This sparse gradient tensor is then encoded, for example using run-length encoding, and sent to other computing nodes to complete the exchange of target gradient blocks.

[0131] It should be noted that the processing speeds of different compute nodes may vary. Because of synchronous training, after the fastest compute node completes local gradient calculation and target gradient block selection, it needs to wait for the slowest compute node to complete this action. Only after the target gradient blocks on all compute nodes are ready can gradient communication between all compute nodes begin.

[0132] In step S608, the average gradient of the target gradient blocks of all computing nodes is calculated, and the network parameters of the deep neural network are updated based on the average gradient. Specifically, the network parameters of the deep neural network can be updated using the gradient descent method without momentum, or using the gradient descent method with momentum.

[0133] After executing steps S602 to S608, one iteration is completed. Steps S602 to S608 can be executed repeatedly. When the deep neural network converges, or the number of iterations of the network parameters reaches a preset number, training is stopped, and the deep neural network on each computing device is now a trained deep neural network.

[0134] In one embodiment, the deep neural network to be trained is a neural network machine translation model. Then, the final network parameters are applied to the model to obtain a trained neural network machine translation model. Then, the information to be translated is translated based on the neural network machine translation model to obtain a translation result, and the translation result is output.

[0135] The following uses the global momentum distributed stochastic gradient descent (SGD) algorithm as an example to introduce a specific implementation of the distributed training method.

[0136] Figure 7 1 shows a flow chart of a distributed training method 700 according to an embodiment of the present invention. The method 700 is applied to a distributed computing framework of a deep neural network and is executed in each computing node of the distributed computing framework 100. The computing node can be specifically implemented as the aforementioned acceleration unit 230. Figure 7 As shown, method 700 performs the following steps on each computing node (worker node):

[0137] 1) Set the hyperparameters for distributed training, including the training dataset θ and the mini-batch size N b , the number of nodes N;

[0138] 2) The local worker node reads a small batch of data D into memory;

[0139] 3) Determine whether it is the first iteration, if so, initialize the model parameter w t-1 (parameters of step t-1), local delayed gradient (delayed gradient of the kth worker node at step t-1, also known as delayed gradient), global momentum accumulation V t (global momentum accumulation at step t), where w t-1 Use normal distribution to generate random numbers in [-1,1] for initialization. and V t Initialized to 0;

[0140] 4) The local worker node inputs the training data into the network for forward transmission, compares the output with the expected label data to obtain the loss error Loss, performs backpropagation based on the Loss, and calculates the gradient of each parameter layer by layer in is the local gradient of the kth worker node at step t, N b is the batch data size, The model parameter is w t-1 is the local gradient calculated based on a training data in the mini-batch training dataset D;

[0141] 5) The local delayed gradient generated by the last iteration With this new Accumulate and generate new Then the new Perform coarse-grained sparse compression of gradients;

[0142] 6) Perform coarse-grained sparse compression processing operations. The specific steps are as follows:

[0143] 61) Loop through each layer. For example, in the hardware implementation process, the data stream is processed one by one according to the input. The L1 norm of the lth layer (l = 0, 1 ... L-1, the deep neural network includes L network layers) and the fth filter (f = 0, 1 ... F-1, the network layer includes F filters) (i.e., each gradient block) is calculated respectively: (where C, H, and W are the number of channels, height, and width of the filter respectively);

[0144] 62) Each layer selects a filter with the largest L1 norm And get its position index indx in the layer to which it belongs max ;

[0145] 63) The L1 norm of the maximum L1 norm filter obtained in 62) is used as the reference value of the important filter of this layer, and the mask is obtained according to the reference value Mask is a 0-1 tensor of the same size and shape as the gradient parameters of each layer;

[0146] It is necessary to further understand the principle of obtaining the mask based on the above reference value, by obtaining the maximum L1 norm filter corresponding to the layer Set the masks corresponding to all filters in the layer that are not equal to the reference value to zero, provided that the mask is a tensor with all initial values ​​of 1 and the same dimensions and shape (C, H, W) as the gradient parameters of each layer. For example, if there are F C×H×W filters in a layer and F corresponding C×H×W initial masks, after processing, only one C×H×W mask is all 1, and the remaining F-1 masks are all 0.

[0147] 64) Select important sparse gradients based on the mask And update the local accumulated gradient

[0148] 7) Use run-length encoding to encode the selected important sparse gradients;

[0149] 8) Based on the All-reduce distributed framework, it transmits and exchanges data to other nodes Worker.

[0150] Sparse communication operations will Send it to other worker nodes and perform aggregation and average to get Multiple nodes perform the same operations above, reflecting the advantages of the distributed synchronous update strategy.

[0151] 9) Using the global momentum distributed stochastic gradient descent algorithm, long-term gradient compensation is performed, so that the model update takes into account the global convergence information, thereby accelerating the convergence speed of training. Specifically, it includes the global momentum accumulation process V t+1 =m·V t +UG t , and then use the new global momentum to update each parameter: w t+1 =w t -η·V t+1 .

[0152] 10) Determine whether the local node still has untrained data. If so, proceed to the next iteration and increment the iteration counter t. Otherwise, the training is completed and the trained model parameters are output. Each training uses a small batch of data. The total training data volume and the small batch size are fixed to N. b , we can get the total number of iterations T, and thus determine whether there is any untrained data.

[0153] In method 700, a coarse-grained sparse strategy is used to determine the importance of gradients based on the L1 norm of the hierarchical filters. The filter with the largest L1 norm in each layer of the deep neural network is selected and its parameters are updated first. Other gradients that are not considered important are accumulated and updated in the next iteration. Compared with the fine-grained top-k sparse method, this strategy avoids time-consuming issues such as additional random sampling and large number sorting, greatly accelerating the sparse process. At the same time, the addresses of important gradients obtained are continuous, eliminating the need for additional resources to process sparse format data, facilitating hardware memory access control and achieving high performance and low power consumption.

[0154] In distributed sparse communication, the present invention proposes a coarse-grained important gradient selection method, that is, important gradient selection is performed in units of gradient blocks. On the one hand, it avoids time-consuming problems such as sorting a large amount of gradient data, greatly accelerating the sparse process. On the other hand, the storage addresses of the obtained important gradients are continuous, and no additional resources are required to process sparse format data, which facilitates hardware memory access control and achieves the goals of high performance and low power consumption, thereby improving the overall training efficiency of distributed training of deep neural networks.

[0155] Thus, in some application scenarios, the above-mentioned distributed training method can also be executed by a terminal device, in which the above-mentioned acceleration unit, such as a neural network processing unit (NPU) or a graphics processing unit (GPU), is deployed. The terminal device can be a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a speaker computing device, a vehicle computing device (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a wearable device including a computing device (e.g., a watch including a computing device, glasses including a computing device), or a home device including a computing device (e.g., a speaker including a computing device, a television including a computing device, a washing machine including a computing device).

[0156] Figure 8 FIG. 8 is a schematic diagram showing a distributed training device 800 according to an embodiment of the present invention. Figure 8 , the apparatus 800 includes:

[0157] a gradient calculation unit 810 adapted to obtain a training data subset from a training data set and calculate a local gradient of a network parameter of the deep neural network on the training data subset;

[0158] a gradient selection unit 820 adapted to calculate, for each network layer of the deep neural network, a gradient metric value of each gradient block in the network layer, and select a predetermined number of target gradient blocks in descending order of gradient metric values, wherein the gradient block includes a plurality of local gradients having consecutive storage addresses;

[0159] A gradient exchange unit 830 is adapted to communicate with other computing nodes and exchange target gradient blocks with each other;

[0160] The parameter updating unit 840 is adapted to calculate the gradient average of the target gradient blocks of all computing nodes and update the network parameters of the deep neural network based on the gradient average.

[0161] The specific processing performed by the gradient calculation unit 810, the gradient selection unit 820, the gradient exchange unit 830, and the parameter updating unit 840 may refer to the above-mentioned method 600 or method 700, and will not be described in detail here.

[0162] The various techniques described herein may be implemented in conjunction with hardware or software, or a combination thereof. Thus, the methods and apparatus of the present invention, or certain aspects or portions of the methods and apparatus of the present invention, may be implemented in the form of program codes (i.e., instructions) embedded in a tangible medium, such as a removable hard disk, a USB flash drive, a floppy disk, a CD-ROM, or any other machine-readable storage medium, wherein when the program is loaded into a machine such as a computer and executed by the machine, the machine becomes an apparatus for practicing the present invention.

[0163] When the program code is executed on a programmable computer, the computing device generally includes a processor, a storage medium readable by the processor (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. The memory is configured to store the program code; the processor is configured to execute the method of the present invention according to the instructions in the program code stored in the memory.

[0164] By way of example and not limitation, readable media include readable storage media and communication media. Readable storage media store information such as computer-readable instructions, data structures, program modules, or other data. Communication media typically embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and include any information delivery medium. Combinations of any of the above are also included within the scope of readable media.

[0165] In the description provided herein, the algorithms and displays are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems may also be used in conjunction with the examples of the present invention. Based on the above description, it is apparent that the structure required for constructing such systems is well understood. In addition, the present invention is not directed to any specific programming language. It should be understood that various programming languages ​​may be utilized to implement the present invention described herein, and the description of specific languages ​​above is provided for the purpose of disclosing preferred embodiments of the present invention.

[0166] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0167] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Accordingly, the claims that follow the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of the invention.

[0168] Those skilled in the art will appreciate that the modules, units, or components of the devices in the examples disclosed herein may be arranged in the device described in the embodiment, or alternatively may be located in one or more devices different from the devices in the examples. The modules in the foregoing examples may be combined into one module or further divided into multiple submodules.

[0169] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition may be divided into multiple submodules or subunits or subcomponents. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed herein may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0170] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of the present invention and to form different embodiments. For example, in the claims below, any of the claimed embodiments may be used in any combination.

[0171] In addition, some of the embodiments are described herein as methods or combinations of method elements that can be implemented by a processor of a computer system or by other devices that perform the functions described. Thus, a processor having the necessary instructions for implementing the method or method element forms a device for implementing the method or method element. Furthermore, the elements described herein of the device embodiments are examples of devices for implementing the functions performed by the elements for the purpose of implementing the invention.

[0172] As used herein, unless otherwise specified, the use of ordinal numbers "first," "second," "third," etc. to describe common objects merely indicates that different instances of similar objects are involved and are not intended to imply that the objects so described must have a given order in time, space, ranking, or in any other manner.

[0173] Although the present invention has been described with respect to a limited number of embodiments, it will be apparent to those skilled in the art, having benefit of the foregoing description, that other embodiments are contemplated within the scope of the invention thus described. Furthermore, it should be noted that the language used in this specification has been selected primarily for readability and didactic purposes, rather than for the purpose of explaining or limiting the subject matter of the present invention. Consequently, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is intended to be illustrative and not restrictive of the scope of the invention, which is defined by the appended claims.

Claims

1. A gradient communication method is applied to a distributed training framework for deep neural networks, wherein the distributed training framework includes multiple computing nodes and performs the following steps on each computing node: Obtaining a training data subset from a training data set, and calculating the gradient of a network parameter of the deep neural network on the training data subset as a local gradient; For each network layer of the deep neural network, the gradient metric value of each gradient block in the network layer is calculated, and a predetermined number of target gradient blocks are selected in descending order of the gradient metric value, wherein: The gradient block includes a plurality of local gradients having consecutive storage addresses; Communicate with other computing nodes and exchange each other's target gradient blocks.

2. The method according to claim 1, wherein The network layer includes a convolutional layer, and the gradient block is a plurality of local gradients corresponding to a filter in the convolutional layer.

3. The method according to claim 1 or 2, wherein The network layer includes a fully connected layer, and the gradient block is a plurality of local gradients corresponding to network parameters between a network node in the fully connected layer and all network nodes in the previous network layer.

4. The method according to claim 1, wherein The network layer includes a bias layer, and the gradient block is a local gradient corresponding to a network node in the bias layer.

5. The method according to claim 1, wherein The gradient metric value is the L1 norm or L2 norm of the gradient block.

6. The method of claim 1, wherein: The predetermined number is 1.

7. The method of claim 1, wherein: The communication with other computing nodes and the exchange of target gradient blocks with each other include: For each network layer of the deep neural network, based on the target gradient block of the network layer, performing a sparsification process on the gradient tensor of the network layer to form a sparse gradient tensor, wherein the gradient tensor is a tensor composed of all local gradients of the network layer; After encoding the sparse gradient tensor, it is sent to other computing nodes.

8. The method of claim 7, wherein the step of performing sparsification processing on the gradient tensor of the network layer comprises: The local gradients in the gradient tensor except for the target gradient block are set to zero.

9. The method of claim 7, wherein: The encoding is run-length encoding.

10. The method of claim 1, wherein: Before the step of calculating the gradient metric value of each gradient block in the network layer, the method further includes: updating the local gradient to the sum of the local gradient and the delayed gradient of the previous iteration; After selecting a predetermined number of target gradient blocks in descending order of gradient metric values, the method further includes storing local gradients other than the target gradient blocks as delayed gradients of this iteration.

11. The method of claim 1, wherein: The communication with other computing nodes and the exchange of target gradient blocks with each other include: Use the All-reduce method to communicate with other computing nodes and exchange each other's target gradient blocks.

12. A distributed training method is applied to a distributed training framework for a deep neural network, wherein the distributed training framework includes multiple computing nodes and performs the following steps on each computing node: Obtaining a training data subset from a training data set, and calculating the gradient of a network parameter of the deep neural network on the training data subset as a local gradient; For each network layer of the deep neural network, the gradient metric value of each gradient block in the network layer is calculated, and a predetermined number of target gradient blocks are selected in descending order of the gradient metric value, wherein: The gradient block includes a plurality of local gradients having consecutive storage addresses; Communicate with other computing nodes and exchange each other's target gradient blocks; Calculate the average gradient of the target gradient blocks of all computing nodes, and update the network parameters of the deep neural network based on the average gradient.

13. The method of claim 12, wherein: Before the step of calculating the gradient metric value of each gradient block in the network layer, the method further includes: updating the local gradient to the sum of the local gradient and the delayed gradient of the previous iteration; After selecting a predetermined number of target gradient blocks in descending order of gradient metric values, the method further includes storing local gradients other than the target gradient blocks as delayed gradients of this iteration.

14. The method according to claim 12 or 13, wherein: The updating of the network parameters of the deep neural network based on the gradient average comprises: Based on the gradient average, updating the network parameters of the deep neural network using a gradient descent method without momentum; or Based on the gradient average, the network parameters of the deep neural network are updated using the gradient descent method with momentum.

15. The method of claim 12, further comprising: When the deep neural network converges, or the number of iterations of the network parameters reaches a preset number, training is stopped.

16. A distributed training method, applied to a distributed training framework for a deep neural network, wherein the distributed training framework includes multiple terminal devices and performs the following steps on each terminal device: Obtaining a training data subset from a training data set, and calculating the gradient of a network parameter of the deep neural network on the training data subset as a local gradient; For each network layer of the deep neural network, the gradient metric value of each gradient block in the network layer is calculated, and a predetermined number of target gradient blocks are selected in descending order of the gradient metric value, wherein: The gradient block includes a plurality of local gradients having consecutive storage addresses; Communicate with other terminal devices and exchange target gradient blocks with each other; Calculate the gradient average of the target gradient blocks of all terminal devices, and update the network parameters of the deep neural network based on the gradient average.

17. The distributed training method according to claim 16, wherein: The terminal device is a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a speaker computing device, a vehicle computing device, a wearable device including a computing device, or a home device including a computing device.

18. A distributed training device comprising: a gradient calculation unit, adapted to obtain a training data subset from a training data set and calculate the gradient of a network parameter of the deep neural network on the training data subset as a local gradient; a gradient selection unit adapted to calculate, for each network layer of the deep neural network, a gradient metric value of each gradient block in the network layer, and select a predetermined number of target gradient blocks in descending order of the gradient metric values, wherein the gradient block includes a plurality of local gradients having consecutive storage addresses; A gradient exchange unit, adapted to communicate with other computing nodes and exchange target gradient blocks with each other; A parameter updating unit is adapted to calculate the gradient average of the target gradient blocks of all computing nodes, and update the network parameters of the deep neural network based on the gradient average.

19. A distributed training system comprising a plurality of computing nodes, each computing node comprising the distributed training apparatus according to claim 18.

20. A computing device comprising: at least one processor; and A memory storing program instructions, wherein the program instructions are configured to be executed by the at least one processor, and the program instructions include instructions for executing the method according to any one of claims 1 to 17.

21. A readable storage medium storing program instructions, wherein when the program instructions are read and executed by a computing device, the computing device is caused to execute the method according to any one of claims 1 to 17.

Images